Fruit Distribution Acquisition With Multi-Vision for Multi-Arm Harvesting Robots

Abstract

This study focuses on the precise localization of fruits using multiple stereo visual perception units in a multi-arm collaborative harvesting robot. Fruit localization in overlapping camera fields is challenging in complex and unstructured orchard environments. This is due to the inaccuracies in estimating the fruit centroid of the same target fruit under different camera perspectives, making it difficult to determine whether different centroid positions belong to the same fruit. To address this issue, a novel fruit localization method based on multiple visual units is proposed in this work to reduce mislocalization of fruits in overlapping camera fields. Firstly, a target-based multi-camera extrinsic calibration method is employed to obtain accurate external parameter matrices between multiple cameras and the robot base coordinate system. Secondly, to enhance the recognition and localization accuracy of fruits under a single camera, a multitask fruit detection and segmentation model is used, along with a frustum-based fruit localization method. Finally, to reduce the problem of repeated detections of the same fruit in overlapping camera fields, a filtering method based on fruit surface point cloud matching is proposed. Experimental results demonstrate that the proposed method improves fruit localization accuracy and reduces the probability of repeated false detections. Visualized point cloud information generated by the stereo cameras is provided to showcase the effectiveness of the proposed method.