From Single Words to Compositional Language via Gestures: Applications in Robot Language Learning

Abstract

Infant development is remarkable during the first years of life. Cognitive growth is accelerated, with one of the key areas being language and communication. Language development in children is multimodal and cumulative. In terms of language growth, infants go from using gestures, to early words and combining these with gestures, until they form multi-word phrases. In fact, gestures and speech are integrated. Our research is fuelled by the fact that language is the most natural interface for humans to interact and yet is still in its infancy when it comes to human-robot interaction through natural language. This provides a rich source of inspiration and motivation for modelling early language acquisition on robotic platforms. Our work presents developmental robotics models leading to a novel framework focusing on the transition from single words to multi-words. This thesis first introduces a cognitive model that leverages the mask regional convolutional network for gesture-word comprehension on the iCub humanoid robot. The model is evaluated on a gesture-object dataset collected as part of our work. One of our key scientific contributions is the integration of a two-stage cascaded mask regional convolutional neural network with a matrix completion framework to model the single word to multi-word transition aided by a deictic gesture. The structured cognitive architecture is deployed on the iCub to illustrate its applicability for modelling language acquisition on an embodied agent. Our results further strengthen the capabilities of these frameworks. This thesis concludes by outlining how our contribution formulates potential guidelines for future efforts in robotic models of early language acquisition. Our work gives a computational model of moving from single words to compositional language, by exploiting current state-of-the-art machine learning techniques to realise this.

Date of Award	1 Aug 2020
Original language	English
Awarding Institution	The University of Manchester
Supervisor	Gavin Brown (Supervisor) & Angelo Cangelosi (Supervisor)