Integrating Reinforcement Learning and Virtual Fixtures for Safer Automatic Robotic Surgery

Abstract

A primary concern in robotic automation is safety, especially in surgical scenarios. In this paper, we propose a virtual fixture (VF) based safe reinforcement learning framework to ensure safety constraints. The framework ensures that the agent, particularly multi-joint robotic manipulator agents, acts within the hard constraints. In the training phase, VF confines the exploration of the agent within a safe operational space. The core idea is that once the agent violates the VF, it will be pushed back to the safe region. Then, the safe action corrected by the VF is collected and forms a safe experience used for subsequent policy optimization, which we refer to as safety experience reshaping (SER). Subsequently, we design a visual module to detect safety constraints to construct the VF and transfer the trained policy to the real robot. We compare our framework to 5 state-of-the-art RL methods and a nonlearning-based method. Results show that our framework gets a lower rate of constraint violations and better performance in task success. Furthermore, in addition to the static constraint tasks, we also designed two tasks involving dynamic constraints, highlighting the superiority of our method in handling dynamic constraints. The videos of our physical experiment can be found in the following links (Lymph node removal, Human-robot collaboration 1, Human-robot collaboration 2).