Method for the real-time detection of tomato ripeness using a phenotype robot and RP-YolactEdge

Abstract

In order to address the challenge of non-destructive detection of tomato fruit ripeness in controlled environments,this study proposed a real-time instance segmentation method based on the edge device. This method combined the principlesof phenotype robots and machine vision based on deep learning. A compact and remotely controllable phenotype detectionrobot was employed to acquire precise data on tomato ripeness. The video data were then processed by using an efficientbackbone and the FeatFlowNet structure for feature extraction and analysis of key-frame to non-key-frame mapping from videodata. To enhance the diversity of training datasets and the generalization of the model, an innovative approach was chosen byusing random enhancement techniques. Besides, the PolyLoss optimization technique was applied to further improve theaccuracy of the ripeness multi-class detection tasks. Through validation, the method of this study achieved real-time processingspeeds of 90.1 fps (RTX 3070Ti) and 65.5 fps (RTX 2060 S), with an average detection accuracy of 97% compared tomanually measured results. This is more accurate and efficient than other instance segmentation models according to actualtesting in a greenhouse. Therefore, the results of this research can be deployed in edge devices and provide technical support forunmanned greenhouse monitoring devices or fruit-picking robots in facility environments. Keywords: instance segmentation, phenotype robot, tomato, greenhouse-based plant phenotyping, ripeness detection DOI: 10.25165/j.ijabe.20241702.8403. Citation: Wang Y Q, Gou W B, Wang C Y, Fan J C, Wen W L, Lu X J, et al. Method for the real-time detection of tomato ripeness using a phenotype robot and RP-YolactEdge. Int J Agric & Biol Eng, 2024; 17(2): 200–210.