A CRF that combines touch and vision for haptic mapping

摘要

Robots could benefit from maps that represent haptic properties of their surroundings. By touching locations with tactile sensors, robots can infer haptic properties of their surroundings, but touching all locations would be prohibitive. We present an algorithm that uses touch and vision to efficiently produce a dense haptic map. Our approach assumes that surfaces near a robot that are visually similar are more likely to have similar haptic properties. Given an image and sparse haptic labels, our algorithm uses a dense conditional random field (CRF) to produce a haptic map with labels for all image pixels. In an evaluation using images with idealized haptic labels, our algorithm substantially outperformed a previous algorithm. It also enabled a real robot to label leaves and trunks after reaching into artificial foliage. In addition, we show that our algorithm can use a convolutional neural network (CNN) for material recognition from Bell et al. that we modified and fine-tuned. This CNN provides estimated probabilities for haptic labels using vision alone, which enables the algorithm to infer haptic labels before the robot makes contact with anything. In our evaluation, using this CNN further improved performance.