Integrating Computer Vision and language model for interactive AI - Robot

Abstract

This research presents an innovative approach to integrating robotic systems with artificial intelligence, addressing long-standing challenges in their functionality. Utilizing YOLOv9 object detection model, Large Language Model LLM, Retrieval Augmented Generation RAG, alongside a 6 degree of freedom robotic arm 6DOF, the design achieves seamless interaction and multifaceted functionality. Key design choices, including hardware selection (Raspberry Pi 58GB) and utilization of various techniques, optimize performance. Experimental results demonstrate a confidence level of 1.0 at a confidence threshold of 0.970 for all classes combined. In the F1-Confidence curve, the all-classes curve achieves an F1 score of 0.84 at a confidence threshold of 0.524, showcasing overall strong performance and a balanced trade-off between precision and recall. The synthesis of diverse methodological approaches underscores the developmental trajectory of this sophisticated robotic assistant, showcasing its transformative potential in promoting effective operations beyond conventional paradigms.