Multidimensional light field endoscope for robotic and AI guided surgery

Abstract

The multidimensional light field information pertaining to the surgical site has the potential to enhance the precision and elevate the degree of intelligence of robotic surgical procedures as well as AI-driven diagnosis and treatment. Endoscopes are evolving towards advancements in 3D technology, near-infrared (NIR) fluorescence imaging, and artificial intelligence. Here, we report a multidimensional light field endoscope that mimics the stereoscopic vision of the human eye and the polarized spectral vision of the mantis shrimp's compound eye. This endoscope is capable of not only capturing three primary color images featuring distinct polarization states from two varying viewpoints, enabling scene color 3D reconstruction and polarization degree computation, but also acquiring ICG NIR fluorescence information without electronic staining. Color 3D images serve as effective aids for robots in surgical navigation, while polarization state information and ICG fluorescence data offer a clear distinction between lesions and vital organs, significantly elevating the intelligence level of robotic and AI-guided surgeries. We designed a series of experiments to verify the imaging performance of different channels of the endoscope and proposed algorithms to achieve color image restoration, 3D reconstruction, polarization extraction, and color fluorescence without electronic staining. The multidimensional light field endoscope will become an indispensable wisdom-eyes for robotic and AI-guided surgery.