Uncalibrated 6-DoF Robotic Grasping With RGB-D Sensor: A Keypoint-Driven Servoing Method

Abstract

Existing 6-degree-of-freedom (6-DoF) robotic grasping methods based on 3-D pose estimation often suffer from long-standing issues of quantization errors, inference delays, and susceptibility to interference. We explore a novel approach that reformulates 6-DoF grasping as a minimizing problem of the projection error of 2-D keypoints. We first input the RGB stream from the red–green–blue and depth (RGB-D) sensor into an efficient single-stage keypoint detector to extract sparse keypoints of the target. Subsequently, we augment the training data using domain randomization to reduce real-world annotation costs and improve robustness to occlusions and extreme lighting. We then use an image-based visual servoing (IBVS) controller, integrating keypoint features with the depth data of the RGB-D sensor, to transform keypoint deviations into the corresponding joint velocity commands for accurate robot tracking. Real-world grasping experiments demonstrate that our method achieves over 70% grasping success in scenarios with unknown hand–eye calibration. Moreover, it maintains pixel-level accuracy under cluttered static and dynamic conditions.