Hand Gestures During Talking Can Influence How Others Hear You

Making hand movements while talking is a common gesture that everyone does. However, this often-ignored gesture can influence on how someone perceives the words spoken, according to a new study published in ‘Proceedings of the Royal Society’ on January 27.

Known as beat gestures, the up and down movements of the hand while speaking can affect some fundamental aspects of how the words are perceived by others. “Politicians use these gestures to get their message across,” said Hans Rutger Bosker, the lead author of the study from the Max Planck Institute of Psycholinguistics in Nijmegen, Netherlands, while giving an example.

Bosker and his colleagues were interested in finding out how these normal gestures can be important in sound recognition. For this, the researchers recruited participants and presented them videos where Bosker was saying Dutch words and recorded how the listeners perceived them. The words uttered by Bosker had two meanings based on how they are said with the hand gestures. For example, if we take an English word, say OBJECT which has two syllables -- OB-JECT -- which syllable is accompanied by the hand gestures can have remarkable impact on the perception of the listeners.

The team found that the participants, on an average, were 20% more likely to hear stress on a syllable if it is accompanied with a beat gesture. Commenting on the findings, Bosker said, “The timing of even the simplest hand movement is vital to face-to-face communication. We’ve shown how multimodal speech perception really is.”

This could simply be a learned association. But, at the same time there could be evolutionary implications as well. Wim Pouw of Radboud University, who was not involved in the study, has some explanations regarding this. According to him, manual gestures coupled with vocal system activity can increase lung pressure. The gestures use muscles and interact with the vocal system. The lung pressure can affect vocal qualities associated with stressed speech.

The experiment was conducted only in Dutch language, but Bosker believes that this could be extended to other languages as well and it can be a generalised phenomenon. However, he says that it is speculative and researches are necessary for a generalisation.

This research is also important during the COVID pandemic. “With people wearing face masks, we can’t lip read. Our data explains how much communication can be improved if we gesture alone,” Bosker says.

This research is an evidence of the McGurk effect. in simple words, the McGurk effect is a phenomenon which shows the interaction between hearing and vision as far speech recognition is concerned. Visual inputs can heavily influence the perception of words and in some cases, visual inputs can change the meaning of speech sounds.

The study is also an input to the open question whether synchrony between hand gestures and utterance of a word can impact the sounds a listener hears. Researches done over a decade now show that hand movements are temporally synchronized with the words we speak.