FruitQuery: A lightweight query-based instance segmentation model for in-field fruit ripeness determination

Abstract

Accurate fruit instance segmentation at different ripeness stages is critical for developing autonomous harvesting robots, particularly given the unstructured in-field conditions. In this paper, we combine two in-field fruit datasets of peaches and strawberries for multiple ripeness stages determination, and propose a lightweight query-based instance segmentation model named FruitQuery. The combined dataset contains 3 peach ripeness stages and 4 strawberry ripeness stages, covering various unstructured conditions of two popular fruits. The model FruitQuery consists of three parts: a backbone, a pixel decoder and Transformer decoders. Efficient multi-head self-attention modules are introduced to the backbone to reduce computational overhead, and a pyramid pooling module is added to the pixel decoder to enhance multi-scale feature fusion. Transformer decoders are then applied to learn a fixed number of queries from features and generate instance masks, avoiding postprocessing like non-maximum suppression. FruitQuery runs in an end-to-end way and incorporates the convolution and Transformer to capture fine-grained features related to different fruits at different ripeness stages. Extensive experiments on the combined fruit dataset demonstrate that our FruitQuery achieves the highest average precision of 67.02 with only 14.08M parameters, outperforming 13 state-of-the-art models with 33 variants. It is noted that FruitQuery surpasses three series of YOLO (v8, v9 and v10) by a large margin. Ablation studies and visualizations also show its robust feature extraction with fewer parameter usage, indicating that the query-based design is effective in localizing fruit. These results highlight FruitQuery's compelling balance between segmentation performance and model size, offering the potential for in-field application. • 1 : We labelled the public strawberry dataset (StrawDI_Db1) with four ripeness stages and made the ripeness labels publicly available to support the research community. • 2 : We merged our published peach dataset (NinePeach) and StrawDI_Db1 into a single dataset, enabling models to handle the instance segmentation of two fruits simultaneously. The combination includes diverse similar variations in appearance, lighting conditions, occlusions, and background textures. • 3 : We present FruitQuery, a lightweight query-based instance segmentation model for fruit ripeness determination. It integrates both convolution and Transformer features and runs in an end-to-end way. • 4 : Our FruitQuery achieves the highest 67.02 average precision with only 14.08M parameters, outperforming 13 other state-of-the-art models with 33 variants. Notably, it surpasses three series of YOLO (v8, v9 and v10) and larger Transformer-based models.