Robotic Arm Generative Painting Through Real-time Analysis of Music Performance

Abstract

This paper describes a prototype audio-visual performance of a Ufactory Uarm Swift and a live musician. In this setting, the robotic arm was used as an AI agent to create a visual representation of a musical work in real-time. An A4 white canvas was gradually filled with a mixture of black, blue, red and yellow paints across the span of approximately eight minutes. The musician, performing on an acoustic violin, fitted with a custom built audio interface, performed multiple versions of an improvisatory work developed specifically for the prototype performance. The following sections discuss our technical approach to programming and implementing the Ufactory Uarm Swift as a painting arm, reflections of the musical process and propose future directions for this project.