Toward long-term and accurate Augmented-Reality display for minimally-invasive surgery

Abstract

Augmented-Reality (AR) displays increase surgeon's visual awareness of high-risk surgical targets (e.g., the location of a tumor) by accurately overlaying pre-operative radiological 3-D model onto the intra-operative laparoscopic video. Existing AR systems lack in accuracy and robustness against frequent illumination changes, camera motions, and organ occlusions, which rapidly cause the loss of image (anchor) points, and thus the loss of the AR display after a few seconds. In this paper, we present a new AR system, which represents the first step toward long term and accurate augmented surgical display. Our system leverages feature matching to automatically recover the overlay by predicting the image locations of a high number of anchor points that were lost after a sudden image change. Additionally, a weighted sliding-window least-squares approach is also used to increase the accuracy of the AR display over time. The effectiveness of the proposed system in maintaining a long term, stable, and accurate augmentation has been tested over a set of real partial-nephrectomy laparascopic monocular videos from a DaVinci surgical robot.