Leveraging Explainability for Comprehending Referring Expressions in the\n Real World

Abstract

For effective human-robot collaboration, it is crucial for robots to\nunderstand requests from users and ask reasonable follow-up questions when\nthere are ambiguities. While comprehending the users' object descriptions in\nthe requests, existing studies have focused on this challenge for limited\nobject categories that can be detected or localized with existing object\ndetection and localization modules. On the other hand, in the wild, it is\nimpossible to limit the object categories that can be encountered during the\ninteraction. To understand described objects and resolve ambiguities in the\nwild, for the first time, we suggest a method by leveraging explainability. Our\nmethod focuses on the active regions of a scene to find the described objects\nwithout putting the previous constraints on object categories and natural\nlanguage instructions. We evaluate our method in varied real-world images and\nobserve that the regions suggested by our method can help resolve ambiguities.\nWhen we compare our method with a state-of-the-art baseline, we show that our\nmethod performs better in scenes with ambiguous objects which cannot be\nrecognized by existing object detectors.\n