Multi-Level Robotic Pouring Using Offline Reinforcement Learning

Abstract

This paper presents a novel learning-based method for a robotic manipulator to achieve liquid pouring across different liquid levels using only visual sensors. Previous works have relied on either online reinforcement learning or imitation learning, which are limited by sim-to-real gaps or data efficiency. In this paper, we propose to combine supervised learning and offline reinforcement learning, utilizing a human demonstration dataset containing only successful pouring at the highest level. Specifically, our approach employs supervised learning for a visual classifier, transforming the visual input into a categorical distribution over liquid levels. The offline reinforcement learning method is applied to a binary-conditioned control policy, which takes a binary signal and the robot's proprioception as inputs to determine the action. The binary signal indicates whether the desired liquid level has been reached based on the target liquid level. Through experiments, our method demonstrates superior effectiveness compared to a state-of-the-art imitation learning algorithm. Moreover, tests in unseen scenarios and multiple liquid level commands verify the generalization and transferability of our approach.