MIntNet: Rapid Motion Intention Forecasting of Coupled Human-Robot Systems With Simulation-to-Real Autoregressive Neural Networks

Abstract

This letter describes the use of a simulation-to-real training pipeline using autoregressive neural networks (MIntNet) for coupled-human robot motion intention prediction. Using only general prior knowledge about the interaction task, a large simulation dataset was generated and used to train a multi-output variation of the classic autoregressive model. The network used an encoding-decoding method to construct condensed representations of the coupled system kinematics over a sequence of time windows and generated their condensed latent representations to predict multiple sequences of the future system states. This method was then tested on 10 real human subjects' data for the interaction task and the simulation-to-real generalization performance was evaluated for the proposed network along with alternative implementations of standard multilayered perceptron, convolutional, and long-short term memory based networks. Results show the proposed network has better generalization performance compared to the alternatives, capable of closely predicting positions during fast motion along non-constant curvatures subject to low-frequency disturbances. The MIntNet was able to accurately predict future positions in a 200 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\text{ms}$</tex-math></inline-formula> window with errors of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$3.1 \pm 4.8$</tex-math></inline-formula> <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\text{mm}$</tex-math></inline-formula> averaged over the prediction window with inference times of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$0.26 \pm 0.44$</tex-math></inline-formula> <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\text{ms}$</tex-math></inline-formula>. Performance was higher for short range predictions with errors over the time window growing as <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$2.3 \pm 3.4$</tex-math></inline-formula> <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\text{mm}$</tex-math></inline-formula> at 50 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\text{ms}$</tex-math></inline-formula>, <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$2.4 \pm 4.4$</tex-math></inline-formula> <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\text{mm}$</tex-math></inline-formula> at 100 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\text{ms}$</tex-math></inline-formula>, and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$5.5 \pm 6.7$</tex-math></inline-formula> <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\text{mm}$</tex-math></inline-formula> at 200 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\text{ms}$</tex-math></inline-formula>. Together these properties allow for agile predictions of motion intention that can be used to inform assistive control policies for e