Multi-Object Tracking for Cotton Boll Counting in Ground Videos Based on Transformer

Abstract

<b><sc>Abstract.</sc></b> The number of cotton bolls is a crucial phenotyping trait, which is essential for both breeders and growers. This trait offers insights into the plant‘s genetic and physiological growth mechanisms and supports decisions in crop management. Counting every cotton boll in the field is difficult for humans, as it is labor-intensive and tedious. With the development of computer vision and robotics, ground robots equipped with deep learning algorithms provide an efficient solution for plant phenotyping. In this paper, we developed a transformer and multi-object tracking-based approach for counting cotton bolls from videos collected by a ground phenotyping robot. RT-DETR, a transformer-based detector, was employed to detect bounding boxes of cotton bolls in each frame. To avoid repeated counting across frames, the FlowFormer model was introduced to predict the dense optical flow between adjacent frames, where the movement of each pixel between two frames can be obtained. The positions of each bounding box in the previous frame can then be estimated in the current frame using the average movement of all pixels within the bounding box. A two-stage association strategy was utilized to associate the detected bounding boxes with the estimated bounding boxes in the current frame, thus avoiding repeated counting. Moreover, a counting line was set to further improve counting accuracy, alleviating the ID switch problem where a single cotton boll could be assigned multiple IDs, leading to duplicated counting if it was not tracked correctly. The cotton boll was counted only when its trajectory intersected with the counting line, which ameliorated the ID switch issue. The experimental results showed that the mAP0.5 of the RT-DETR was 0.929 for cotton boll detection. Furthermore, the developed approach achieved a moderately strong correlation between the predicted number and the ground truth with an R² of 0.61 and a MAPE of 11.86%. Overall, the developed approach provides an efficient and effective tool that significantly benefits breeders and growers in cotton breeding and yield estimation.