Surgical instruments tracking based on deep learning with lines detection and spatio-temporal context

Abstract

Visual tracking has a number of applications on robotic minimally invasive surgery (RMIS). In this paper, we propose a visual tracking method for two-dimensional tool detection using convolutional neural network (CNN) with line segment detector (LSD) and tracking using spatio-temporal context (STC) learning. We train a CNN with the datasets labeled by LSD to detect the tool's tip on the basis of the lines' positions quickly and accurately and utilize the STC to track the tool in real time. We experimentally compared our method to two other visual tracking methods on three surgical datasets. The results show that our method has a very good performance in both speed and accuracy.