From Sounds to Actions: how Gestures Depict Auditory Information

Abstract

The Gesture as Simulated Action (GSA) Model holds that gestures result from simulated actions, which are in turn led by simulations of motor representations and visuospatial representations (Hostetter & Alibali, 2008). The GSA also predicts that, all things being equal, the action component of the mental simulation predicts variations in the gesture rate. Chapter 2 addressed the question of whether audiospatial representations can also lead to gesture production. To test this hypothesis, musical pitch has been chosen because its mental representation is constituted by a spatial and an auditory component, as shown by Connell, Cai, and Holler (2013). Chapter 2 presents two experiments where participants described sequences of sounds that varied in pitch, from higher to lower pitch and vice versa, and an analysis of how this information was conveyed through the gesture modality was conducted. In Study 1, participants verbally described short sequences of musical notes to an addressee. Information about sequences of notes featuring shifts from low to high pitches was conveyed through upward hand movements, and shifts from high to low pitch led to downward hand movements. As a result, gestures conveyed spatial information about pitch such that higher pitches were allocated to higher locations compared to lower pitches. The second experiment aimed at ruling out the possibility that speaking (e.g., labels such as 'high', 'low', 'up', or 'down') shaped this gestural behaviour. Participants were given an articulatory suppression task which required them to hum the sound sequences and therefore prevented them from employing verbal descriptions. The gestural behaviour was consistent with that observed in the previous experiment. In summary, the gestures produced while communicating about sound pitch emerged from the spatial representation of pitch in the auditory modality, and this was independent of the speech production process. Chapter 2 will conclude discussing the implications of these findings on gesture production models, with a focus on the GSA framework. Chapter 3 tested whether the action component of the mental simulation underlying gestures can alone predict variations in the gesture rate. The GSA claims that "as one moves from visual images through spatial images to motor images, the amount of action simulation increases and so does the likelihood of gesture" (Hostetter & Alibali, 2008, p. 510). Results suggest that the strength of action simulation and visual simulation together might predict variations in the gesture rate. However, the experiment did not have enough power to provide definite evidence for it, or to disentangle the effect of the action component from the effect of the visual component on gesture rate. These results are discussed in relation to future research. Finally, Chapter 4 provides a brief discussion of all the findings of earlier chapters and their theoretical meaning.

Date of Award	1 Aug 2015
Original language	English
Awarding Institution	The University of Manchester
Supervisor	Louise Connell (Supervisor) & Judith Holler (Supervisor)