Machine Learning-based Human-Following System: Following the Predicted Position of a Walking Human

Abstract

Human–robot interaction (HRI) has been widely researched in diverse applications. A robot following a person is one such scenario investigated in the HRI field. However, human movements and actions are complex and can change dramatically. We herein demonstrate a machine learning-based system that allows a person-following robot to track in real-time the predicted future motion of a walking human, from a first-person perspective. We assume that a depth sensor that can detect the human skeleton is loaded on a mobile robot to provide data on the user’s motion from a first-person perspective. The system calculates the coordinates of the center of gravity (COG) and 25 body joints of the user. These coordinates of COG and 25 body joints are relative to the robot based on the position of the person tracked, and these are used for the input dataset of a neural network (NN) that predicts human motion. A five-layered NN estimates the relative vectors in real-time between the current person’s COG and the future position of the 25 body joints. Using a proportional–integral–derivative (PID) controller, the person-following robot can track the predicted position of a walking human 0.5 s in advance to increase the robustness of following and to avoid delays.