Human–Robot Cooperative Piano Playing With Learning-Based Real-Time Music Accompaniment

Abstract

Recent advances in machine learning have paved the way for the development of musical and entertainment robots. However, human–robot cooperative instrument playing remains a challenge, particularly due to the intricate motor coordination and temporal synchronization. In this article, we propose a theoretical framework for human–robot cooperative piano playing based on nonverbal cues. First, we present a music improvisation model that employs a recurrent neural network (RNN) to predict appropriate chord progressions based on the human's melodic input. Second, we propose a behavior-adaptive controller to facilitate seamless temporal synchronization, allowing the cobot to generate harmonious acoustics. The collaboration takes into account the bidirectional information flow between the human and robot. We have developed an entropy-based system to assess the quality of cooperation by analyzing the impact of different communication modalities during human–robot collaboration. Experiments demonstrate that our RNN-based improvisation can achieve a 93% accuracy rate. Meanwhile, with the MPC adaptive controller, the robot could respond to the human teammate in homophony performances with real-time accompaniment. Our designed framework has been validated to be effective in allowing humans and robots to work collaboratively in the artistic piano-playing task.