| dc.description.abstract | Just as the perception of simple events such as clapping hands requires a linkage of sound with the movements that produce the sound, the integration of more complex events such as describing how to give an injection often requires a linkage between the instructor’s utterances and their actions. However, the mechanism for integrating these more complex multimodal events is unclear. For example, it is possible that predictive temporal relationships are important for multimodal event understanding, but it is also possible that this form of understanding arises much more from meaningful causal between-event links that are temporally unspecified. This latter approach might be supported by a cognitive temporal integration window, within which multimodal event information integrates flexibly with few default commitments about specific temporal relationships. To test this hypothesis, we assessed the consequences of disrupting temporal relationships between instructors’ actions and their speech in narrated screen-capture instructional videos (Experiment 1) and live-action instructional videos (Experiment 2) by displacing the audio channel forward or backward relative to the video by 0, 1, 3, or 7 seconds. We assessed learning, event segmentation, disruption awareness, segmentation uncertainty, and perceived workload. Across two experiments, 7-second temporal disruptions consistently increased uncertainty and workload, and decreased learning in Experiment 2. None of these effects appeared for 3-second disruptions, which were barely detectable. 1-second disruptions produced no effects and were undetectable, even though much intra-event information falls within this range. Our results suggest an event-integration window that supports integration of events independent of constraining temporal relationships between subevents. | |
