SemEval-2014 Task 6: Supervised Semantic Parsing of Robotic Spatial Commands

Abstract

SemEval-2014 Task 6 aims to advance semantic parsing research by providing a high-quality annotated dataset to com- pare and evaluate approaches. The task focuses on contextual parsing of robotic commands, in which the additional con- text of spatial scenes can be used to guide a parser to control a robot arm. Six teams submitted systems using both rule-based and statistical methods. The best per- forming (hybrid) system scored 92.5% and 90.5% for parsing with and without spatial context. However, the best per- forming statistical system scored 87.35% and 60.84% respectively, indicating that generalized understanding of commands given to a robot remains challenging, de- spite the fixed domain used for the task.