H-RSSG: High-Fidelity Robotic Surgical Scene Generation With Implicit Deformable Neural Radiance Field

Abstract

Artificial intelligence-generated content (AIGC) aims to represent new content synthesized by learning rules from the existing data and has expanded its applications to the health domain, particularly generating medical robotic surgical scenes. Most pioneering methods rely primarily on 2D representations and thus will inevitably suffer from scene distortion when large surgical tool movements and intense soft tissue deformation are encountered. Recent works instead employ explicit 3D structural representations or implicit neural rendering to improve performance under large pose changes. Nevertheless, the fidelity of structure and texture is not so desirable, especially for novel-view synthesis. In this paper, we propose H-RSSG(high-fidelity robotic surgical scene generation with an implicit deformable neural radiance field), which achieves high-fidelity and free-view robotic-surgical-scenes synthesis. The H-RSSG explores the potential of artificial intelligence in generating high-fidelity medical robotic surgical scenes. In addition to generating robotic surgical scenes, the H-RSSG provides standardized evaluation metrics by establishing a benchmark dataset. Moreover, H-RSSG enhances the robustness of AIGC-generated scenes by combining adjacent view feature fusion and multi-view correspondence loss to improve spatial context consistency. A notable advantage of the H-RSSG is its integration of an extended transform-based depth perception module with a lightweight neural network used to estimate masking depths even in the presence of surgical instruments. This results in spatiotemporally consistent high-fidelity robotic surgical scenes. The results of the experiments conducted on the self-established and EndoNeRF datasets have a PRMSE of about 0.0341, an average improvement of 12.1% in PSNR, and an average improvement of 6.69% in SSIM, which demonstrates that the proposed H-RSSG can outperform the benchmark models’ performance, achieving state-of-the-art results. This study highlights the AIGC’s potential to create highly realistic surgical training environments for medical professionals and applications in the medical industry.Video is available at:H-RSSG.