AgentMario: A Multitask Agent for Robotic Interaction With Locker Systems

Abstract

A robotic locker system is needed where automated storage and retrieval of items are required without the need for staff presence. For example, a robot can provide 7/24 available services of medical items pick-up and return, during the COVID-19 pandemic (or under other emergencies). A robotic locker system is usually equipped with a user-friendly intuitive interface (e.g., a touchscreen); meanwhile, the robot desires a multitask agent that can observe, understand, and operate the locker’s interface to complete many tasks of storing/accessing/shipping items. In this article, we study building a multitask agent for interacting with robotic locker systems, called AgentMario. Without human intervention for a specific task, AgentMario decomposes solving a task into learning basic skills (states or user interfaces) and planning over the skills (finding the next state/interface). When the agent is solving a task, our search algorithm walks on the finite state machine graph and generates the proper plans (operation sequence) for the agent. In experiments, our method accomplishes four diverse tasks of picking-up/storing/dropping-off/shipping items. By employing image recognition and mechanical automation technologies, we implement AgentMario with a robot arm to enable contactless operation over the locker’s interface. Experimental results show that our method outperforms baselines in most tasks by a large margin.