A multimodal corpus for the evaluation of computational models for (grounded) language acquisition

Abstract

This paper describes the design and ac-quisition of a German multimodal cor-pus for the development and evaluation of computational models for (grounded) lan-guage acquisition and algorithms enabling corresponding capabilities in robots. The corpus contains parallel data from multi-ple speakers/actors, including speech, vi-sual data from different perspectives and body posture data. The corpus is designed to support the development and evalua-tion of models learning rather complex grounded linguistic structures, e.g. syn-tactic patterns, from sub-symbolic input. It provides moreover a valuable resource for evaluating algorithms addressing sev-eral other learning processes, e.g. concept formation or acquisition of manipulation skills. The corpus will be made available to the public. 1