TU-IR Apple Image Dataset: Benchmarking, Challenges, and Asymmetric Characterization for Bruise Detection in Application of Automatic Harvesting

Abstract

With the blooming interest in computer vision-based technologies for future automation of food producers, there is a need for incorporating an automatic bruise detection module in robotic apple harvesting because of decreasing accessibility and growing labor costs. Although numerous studies have been published for automatic quality inspection of fruit and other agricultural products, there is a lack of publicly available image-based datasets for quality inspection/automatic detection of bruises. Toward the aim of developing a bruise detection system for apple harvesting, especially at night time, this article describes the designing issues (i.e., protocol) and creation of a new infrared imaging-based dataset titled “TU-IR Apple Image Dataset,” which contains 1375 infrared images of apples defining four major categories of bruises (i.e., fresh, slight, moderate, and severe). Along with the infrared images, ground truths (in the form of binary masks) and measurements of suspicious bruised regions are defined. This study also investigates the efficiency of infrared imaging technology for automatic bruise detection in apples by performing an analysis of temperature-based, intensity-based, texture-based, shape-based, and deep convolutional neural network-based features. The classification performance was evaluated using eight different feature sets. Based on the experimental results, considering the most outer-performed classifier, deep convolutional neural networks as a fixed feature extraction method were found to provide the highest prediction performance for discriminating between fresh and three categories of bruises in apples with an average accuracy, specificity, and sensitivity of 93.87%, 80.57%, and 92.02%, respectively.