Three-View Cotton Flower Counting through Multi-Object Tracking and Multi-Modal Imaging

摘要

<b><sc>Abstract.</sc></b> Monitoring the number of cotton flowers can provide important information for breeders to assess the flowering time and the productivity of genotypes because flowering marks the transition from vegetative growth to reproductive growth and impacts the number of bolls developed. Typically, manual counting is difficult, time-consuming, and tedious. To count cotton flowers efficiently and accurately, we propose a multi-view and multi-modal imaging approach based on object tracking using both RGB and depth images collected by three cameras mounted on a ground robotic platform. Tracking by detection algorithm was employed to track flowers from three views at the same time. Furthermore, a YOLOv8 object detector was trained to detect flowers in images and a deep learning-based optical flow model RAFT (Recurrent All-pairs Field Transforms) was used to estimate motion between two adjacent frames. IoU and distance costs were used to associate flowers in the tracking algorithm, and tracked flowers were segmented to obtain pixel positions for depth extraction. These positions from two side views were then projected onto the middle image coordinate using camera calibration parameters. Finally, all flowers in the middle image coordinate were clustered by a constrained hierarchy clustering algorithm to remove duplicated counts. The results showed that the mAP of trained YOLOv8x was 96.4%. The counting results of the developed method were highly correlated with those of manual counts with an R² of 0.92. Besides, the Mean Relative Error of all testing videos was 6.22%. Overall, the proposed method provides an efficient and effective approach to solving the cotton flower counting from multiple views, which is beneficial for breeders to dissect genetic mechanisms of flowering time with an unprecedented spatial and temporal resolution.