A pose-versatile imaging system for comprehensive 3D modeling of planar-canopy fruit trees for automated orchard operations

Abstract

• Imaging systems can resolve occlusions by integrating captures from multiple poses. • Pose-versatile imaging systems address captures with low-confidence depth values. • Developed pose-versatile system enhanced 3D model accuracy over fixed-pose system. • The low-cost, fast-computing system showed potential for robotic orchard operations. High-density tree fruit production systems employ SNAP (Simple, Narrow, Accessible and Productive) canopy architectures, such as the UFO (Upright Fruiting Offshoots) system, that require intensive management practices. The growing adoption of these production systems in the USA, along with the decline of farm labor in the country, has sparked interest in automating manual orchard operations. Machine vision plays a key role in the development of robotic solutions because the success of these robots largely depends on the ability of the imaging systems (ISs) to quickly and accurately generate three-dimensional (3D) models of the surroundings. Tree models, for example, are essential to determine cutting points and guide cutting tools to the correct positions when pruning selectively. However, the ISs proposed in recent studies do not produce sufficiently comprehensive models, are expensive, and/or are impractical for commercial applications. In this study, a novel, time-efficient, and pose-versatile imaging system (Mobile IS) was developed and tested to overcome these issues. The Mobile IS utilized off-the-shelf cameras to capture an initial 3D point cloud model of a scene and then dynamically refined the model in real time by integrating additional point clouds from close range and different poses using simple photogrammetric techniques. To evaluate the performance of the Mobile IS, a wide range of UFO-trained tree offshoot diameters (OSDs), and side-branch lengths (SBLs) and spacings (SBSs)—parameters on which the pruning rules for UFOs are based—were measured on the models reconstructed by the Mobile IS and a fixed-pose imaging system (Fixed IS) and compared to ground truth. Mobile IS models exhibited higher accuracy compared to the Fixed IS models, as evidenced by the root mean square (RMS) errors of the Mobile IS measurements ( RMS EOSD of 4.9 mm, RMS ESBL of 8.0 mm, and RMS ESBS of 3.6 mm) and the Fixed IS measurements ( RMS EOSD of 5.9 mm, RMS ESBL of 18.1 mm, and RMS ESBS of 3.8 mm). The versatility of pose enabled the Mobile IS to overcome occlusions and areas with low-confidence depth values. The results suggest that the Mobile IS holds promise as an IS for various robotic applications, including automated pruning, thinning and harvesting, across different tree fruit crops and canopy architectures.