Speech-Gesture Integration in Adults with Moderate-Severe Traumatic Brain Injury

Abstract

Language is multimodal, containing speech and gesture. Listeners must bind co-occurring information from verbal and visual modalities to form an integrated representation of the speaker’s message. Across two experiments, I tested whether this process is disrupted by moderate-severe traumatic brain injury (TBI). In experiment 1, participants with TBI and non-injured participants watched videos of a narrator telling short stories. The narrator provided some information only in gesture. Participants retold the stories immediately after, 20-minutes later, and one-week later. We tested whether participants reported information from gesture in their retellings. Despite recalling fewer story details, participants with TBI were as likely to report information from gesture as their non-injured peers, and both groups were more likely to report information from gesture one week later compared to immediately after hearing the stories. In experiment 2, we tested participants’ ability to integrate speech and gesture in-the-moment during multimodal language processing. Participants watched videos of a speaker producing a sentence (e.g., “The girl will eat the very good sandwich”) paired with either a meaningful gesture or meaningless grooming movement at the verb. We measured participants’ eye fixations in a visual scene to the target picture (e.g., sandwich), a verb-related competitor (e.g., apple) and distractor images (e.g., guitar, piano). Although participants in both groups were more likely to fixate the target item when the speaker produced a gesture compared to a grooming movement, this effect was significantly attenuated in the TBI group. Although experiment 1 suggests that gesture may support comprehension and memory for stories in adults with TBI, experiment 2 provides evidence that the time course of speech-gesture integration can be disrupted after TBI. This disruption is likely to scale up more complex communication contexts that require the processing and integration of multimodal signals. I discuss clinical implications for improving ecological validity of communication assessment and continued need to identify communicative and cognitive functions of gesture to support rehabilitation after brain injury.