Deep Learning-Based 3D Surgical Scene Understanding for Cochlear Implant Surgery

Abstract

Computer vision has the potential to revolutionize precision surgery, preoperative planning, and surgical education by delivering crucial insights that improve procedural accuracy, reduce risks, and enhance patient outcomes. However, the surgical setting is inherently complex, featuring specialized lighting, occlusions from blood and tissue, and constrained imaging capabilities, which pose significant challenges for real-time image analysis and interpretation. To overcome these complexities, we propose a suite of deep learning-based techniques specifically tailored to the unique demands of surgical environments. Our focus includes segmentation of surgical tools to accurately identify and track instrument boundaries; 3D reconstruction of tools to enhance spatial understanding; depth map estimation of the surgical area to capture relative object distances; and dynamic scene reconstruction to provide continuous updates on tool positions and orientations. These endeavors aim to provide surgeons with real-time, precise 3D information on the pose, position, and depth changes of surgical instruments. Our framework leverages foundation models and self-supervised learning techniques, tailored specifically for cochlear implant insertion surgery, where visualization is restricted to a single monocular microscope camera. By developing methods that operate effectively within these constraints—and integrating optimizations for inference speed and accuracy enhancement—our approach achieves real-time inference speed and improved accuracy, offering actionable insights for intraoperative localization of surgical instruments in 3D space, ultimately strengthening surgeons' ability to interpret and navigate the surgical scene. Through these advancements, our methodologies aim to significantly improve the precision and safety of cochlear implantation, contributing broadly to the field of image-guided surgery and fostering new standards in surgical care.