Grounding Language in the World: Schema Theory Meets Semiotics

Abstract

A theoretical framework for grounding language is introduced that provides a computational path from sensing and motor action to words and speech acts. The approach combines concepts from semiotics and schema theory to develop a holis-tic approach to linguistic meaning. Schemas serve as structured beliefs that are grounded in an agent’s physical environment through a causal-predictive cycle of ac-tion and perception. Speech acts are interpreted in terms of grounded schemas. The theory reflects lessons learned from implementations of several language processing robots and provides a framework for the design of situated, physically-embedded natural language processing systems, and for various other multimedia and cross-modal processing systems that straddle symbolic and non-symbolic realms.