Mapping language to vision in a real-world robotic scenario

Karla Stepanova, Frederico B. Klein, Angelo Cangelosi, Michal Vavrecka

Research output: Contribution to journalArticlepeer-review

269 Downloads (Pure)


Language has evolved over centuries and was gradually enriched and improved. The question, how people find assignment between meanings and referents, remains unanswered. There are many of computational models based on the statistical co-occurrence of meaning-reference pairs. Unfortunately, these mapping strategies show poor performance in an environment with a higher number of objects or noise. Therefore, we propose a more robust noise-resistant algorithm. We tested the performance of this novel algorithm with simulated and physical iCub robots. We developed a testing scenario consisting of objects with varying visual properties presented to the robot accompanied by utterances describing the given object. The results suggest that the proposed mapping procedure is robust, resistant against noise and shows better performance than one-step mapping for all levels of noise in the linguistic input, as well as slower performance degradation with increasing noise. Furthermore, the proposed procedure increases the clustering accuracy of both modalities.

Original languageEnglish
Article number8325300
Pages (from-to)784-794
Number of pages11
JournalIEEE Transactions on Cognitive and Developmental Systems
Issue number3
Early online date26 Mar 2018
Publication statusPublished - 1 Sept 2018


  • cognitive modeling
  • Computational modeling
  • cross-situational learning
  • Grounding
  • iCub robot
  • language acquisition.
  • Linguistics
  • Robots
  • Robustness
  • symbol grounding
  • Task analysis
  • Visualization


Dive into the research topics of 'Mapping language to vision in a real-world robotic scenario'. Together they form a unique fingerprint.

Cite this