Task-Oriented Interventionists’ Operating Skills Recognition Based on Multimodal Data Hierarchical Fusion

Abstract

The operating skills of interventionists have an important impact on the surgical treatment effect, and combining operating skills with surgical robots is a promising method to improve the intelligence level of robots. However, the unstructured operating skills recognition is a complex and lengthy process, and related research is limited. Moreover, compared with the method of skills recognition for specific vascular segment operation data, the skills recognition throughout the entire surgical process is more meaningful and more challenging. In this study, a data acquisition solution integrating four modal sensors was used to collect operation data from expert and novice interventionists from model and in vivo animal experimental platforms. A hierarchical fusion strategy based on the Shannon entropy weight method is proposed for feature fusion and behavior recognition, with average recognition accuracies of 96.84% (model) and 93.26% (in vivo). Further qualitative and quantitative analyses effectively evaluated skill level differences between expert and novice interventionists. These promising results indicate the possibility of applying this fusion strategy to clinical operating skills recognition. Simultaneously, the digitization of operating skills could be used as control indicator to improve the intelligence level of surgical robots.