Integrating skill knowledge and multimodal information to enhance 3C assembly skills

摘要

Purpose To solve the 3C assembly task, the traditional robot needs a lot of manual coding and the skills learned are faced with the problems of adapting to the scene and task diversity, lack of generalization ability and so on. This paper aims to propose a skill knowledge and multimodal information fusion algorithm (SKMIF). Design/methodology/approach This method combines skill knowledge and multimodal information in large language model to enhance 3C assembly task skills. The SKMIF algorithm is used to conduct experiments in simulated and real 3C assembly tasks, which verifies the effectiveness of the method in single-task and multi-task scenarios, and solves the problem of insufficient generalization ability of automated programming. Findings Through the transfer from simulation to reality, the assembly strategy learned in the virtual environment is applied to the real scene, which significantly improves the assembly accuracy and success rate in the real environment. The verification of the enhanced 3C soft-row line assembly task shows that the success rate is 96%. Originality/value This paper proposes a new algorithm to enhance 3C assembly skills, to improve generalization ability and adaptability to multitasking environments.