Generative AI for automated task modelling and task allocation in human robot collaborative applications

Abstract

Task modelling and assignments generation is a complex and time-consuming activity despite the availability of modern CAx and planning tools. This paper proposes an AI based framework using Large Multi-Modal Models and a Digital Twin to automatically create task models, sequences and assignment plans through the processing of video streams involving visual and audio cues on the recorded resources, tools, and tasks. The same LMMs perform the task-to-resource allocation considering metrics such as human factors and resource workload. A case study on the assembly of white goods showcases reduction in manual planning, enhanced resources utilization and improved human-robot collaborative applications.