Hybrid Uncalibrated Visual Servoing Control of Harvesting Robots With RGB-D Cameras

Abstract

Visual servoing (VS) control has seen wide adoption in harvesting robots. However, parameter calibration is cumbersome, which makes the use of VS robotic systems inconvenient. Besides, dynamic fruits usually lead to a degeneration of control while tracking. To overcome the drawbacks, we present a new image-based uncalibrated visual servoing (IBUVS) control approach, consisting of a hybrid visual configuration and an adaptive tracking controller, referred to as hybrid-IBUVS. Specifically, our hybrid-IBUVS employs an eye-in-hand camera and a fixed red–green–blue-depth camera to construct a hybrid VS system, basing on multiobject detection and edge-computing technologies. Meanwhile, we also propose adaptive laws to online estimate the uncalibrated parameters of the cameras and robot dynamics. Furthermore, our hybrid-IBUVS uses an adaptive tracking controller to guarantee the harvesting robot to track a predefined trajectory to approach a fruit target. By Lyapunov stability theory, asymptotic convergence of the proposed control scheme is rigorously proven. Experimental results demonstrate the effectiveness of the proposed scheme. All shown results supported the research claims.