Peduncle Detection of Ripe Strawberry to Localize Picking Point Using DF-Mask R-CNN and Monocular Depth

Abstract

Accurate localization of picking points and depth estimation is critical for implementing a robotic strawberry harvesting system. Due to the delicate nature of strawberries, harvesting must be performed without bruising or damage, typically by grasping and cutting the peduncle of the ripe strawberry. However, accurately detecting and localizing the thin peduncle in a cluttered environment is a significant challenge. This study proposed depth fused Mask R-CNN (DF-Mask R-CNN), which integrates depth information of the scene with the RGB image to enhance the detection, localization, and segmentation of strawberries and their peduncles in a greenhouse environment. To generate a dense depth map, a cutting-edge monocular depth estimator, ZoeDepth was used. The proposed DF-Mask R-CNN with ResNet101-FPN exhibited superior instance segmentation performance, with an overall mAP of 81.9%, with mAP<sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">small</sub> at 33.3%, mAP<sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">medium</sub> at 78.79%, mAP<sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">large</sub> at 88.8 and AP<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">IOU</sup>=0.5 at 98.1%. In tests with 300 ripe strawberry samples, the method demonstrated a robust picking point detection, with a mean absolute error and root mean square error of 1.98 cm and 2.12 cm, respectively. These results highlight the effectiveness of the DF-Mask R-CNN model combined with the ZoeDepth estimator in enhancing the detection, localization, and segmentation of strawberries and their peduncles. This approach enables precise picking point localization and depth estimation for efficient vision systems for robotic strawberry harvesting.