Using Depth for Improving Referring Expression Comprehension in\n Real-World Environments

Abstract

In a human-robot collaborative task where a robot helps its partner by\nfinding described objects, the depth dimension plays a critical role in\nsuccessful task completion. Existing studies have mostly focused on\ncomprehending the object descriptions using RGB images. However, 3-dimensional\nspace perception that includes depth information is fundamental in real-world\nenvironments. In this work, we propose a method to identify the described\nobjects considering depth dimension data. Using depth features significantly\nimproves performance in scenes where depth data is critical to disambiguate the\nobjects and across our whole evaluation dataset that contains objects that can\nbe specified with and without the depth dimension.\n