An Approach to Design a Biomechanically-Inspired Reward Function to Solve a Patience Cube Under Reinforcement Learning Framework

摘要

This paper presents an approach to design a reward function by adopting both control theoretic and biomechanical perspectives. In reinforcement learning (RL), a reward function plays a crucial role for an RL agent training; especially, a task learning time and a task performance. Accordingly, designing a reward function becomes a key issue to train an RL agent generating human-like policy/strategy to perform dexterous manipulation. Since human beings are good at producing heuristic approaches to complete a given task, determining a set of basis functions as well as corresponding weights used not to be so straightforward. In this study, we consider solving a patience cube as an example of a dexterous manipulation task. In our approach, we first employed a quadratic regulator form as a backbone of a desired reward function. Next, the kinematic data of a controlled object and the sEMG data of a human expert were measured while performing a demonstration to solve a patience cube. Then, from the measured data, the weights of the basis functions were determined by utilizing muscle synergy extraction and inverse optimal control as two key tools. Finally, an RL agent was trained by the designed reward function and comparative analysis versus the other RL agents trained by prototypical weight settings was followed. The result showed that the RL agent trained by our approach yielded human-like learning curve as well as policy successfully and outperformed the others in terms of a task success rate and a task completion time. These findings substantiated the feasibility of extending our approach to an assistive robotic manipulator or prosthesis design to perform the activities of daily living.