Picking point localization method based on semantic reasoning for complex picking scenarios in vineyards

Abstract

In the complex orchard environment, precise picking point localization is crucial for the automation of fruit picking robots. However, existing methods are prone to positioning errors when dealing with complex scenarios such as short peduncles, partial occlusion, or complete misidentification, which can affect the actual work efficiency of the fruit picking robot. This study proposes an enhanced picking point localization method based on semantic reasoning for complex picking scenarios in vineyard. It innovatively designs three modules: the semantic reasoning module (SRM), the ROI threshold adjustment strategy (RTAS), and the picking point location optimization module (PPOM). The SRM is applied to handle the scenarios of grape peduncles being obstructed by obstacles, partial misidentification of peduncles, and complete misidentification of peduncles. The RTAS addresses the issue of low and short peduncles during the picking process. Finally, the PPOM optimizes the final position of the picking point, allowing the robotic arm to perform the picking operation with greater flexibility. Experimental results show that SegFormer achieves an mIoU (mean Intersection over Union) of 84.54 %, with B_IoU and P_IoU reaching 73.90 % and 75.63 %, respectively. Additionally, the success rate of the improved fruit picking point localization algorithm reached 94.96 %, surpassing the baseline algorithm by 8.12 %. The algorithm's average processing time is 0.5428 ± 0.0063 s, meeting the practical requirements for real-time picking. • Propose a high-precision picking point localization method, meeting real-time needs. • Semantic reasoning achieves advanced inference from existing semantic information. • Three algorithms—SRM, RTAS, PPOM—improve picking point accuracy in complex scenarios.