Generative AI-empowered digital twin for immersive training in cardiac surgery

Cardiovascular diseases remain the leading cause of mortality worldwide (WHO,2019). In particular, coronary artery disease often requires coronary artery bypass grafting, a highly technical procedure requiring more than 15 years of training and relying on wet-labs, simulators and direct observation in the operating room. These traditional training methods suffer from limited availability, reproducibility and interactivity. This project proposes a technological and methodological rupture to overcome these limitations based on the cross fertilization between immersive (enabling safe, repeatable and personalized training experiences [1]) and generative (creation of artificial and personalized visual and artificial content [2], [3], [4]) technologies for cardiac surgery. The objective is to design a generative digital twin of cardiac surgery, capable of reconstructing [4], simulating [5], [6], and enriching real surgical interventions for immersive and adaptive training [7], [8]. To that end, the core ambition is to build a controllable virtual environment generated from real multimodal data — videos, depth, motion, and sound — combined with text-to-3D generative AI [2], [9], [10]. The system will enable trainees to visualize, interact with, and rehearse surgical gestures in extended reality, while dynamically adapting to user expertise or scenario variations. 

Introducing generative modeling to transcend the limitations of current handcrafted simulators and scripted experiences involves several challenges. Currently, the use of Extended Reality (XR) in cardiac surgery remains restricted to serious game or teleportation [1] due to the lack of publicly available open-surgery dataset [8], [11]. Next, computer vision and generative AI methods to reconstruct and analyze dynamic 3D scenes can be limited by constrains such as deformable tissues, occlusions, and lighting variability [6], [12], [13]. To that end, it will build upon recent advances in neural scene representations [9] and differentiable point-based rendering [13], combining implicit and explicit modeling approaches [3] with text-to-3D generative diffusion, to produce semantically controllable and anatomically faithful environments [2], [10], [14], [15]. In the medium term, we will be able to automatically generate realistic and customizable scenarios from text and image prompts while considering the previous constraints[6]. 

The project involves several actors specialized in (i) computer vision and VR/AR (ARMEDIA, SAMOVAR, Télécom SudParis); (ii) models for generative AI (SOP, SAMOVAR, Télécom SudParis) and (iii) cardiac surgery with expertise in XR (Pr. Patrick Nataf, INSERM). Consequently, a database from surgeon-view has already been partially acquired. The PhD student will extend and optimize the capture setup, develop pre-processing and annotation pipelines (segmentation, summarization, description, medical ontology), fine-tune and implement generative models for reconstruction and scene synthesis and develop a multimodal-prompt-based 3D generation engine for surgical scenario. The expected outcome is a proof-of-concept AI-driven cardiac training simulator integrated into existing XR infrastructures at INSERM. The different teams have been approached by P. Spinga, a former student of Télécom SudParis, ranked 16/195 (1st year) and 8/241 (2nd year) and a former research engineer at Dassault Systèmes - Healthcare department. 

In conclusion, this research aims at opening the way toward a complete digital twin of cardiac surgery — a cognitive and perceptual companion that could one day assist surgeons before, during, and after interventions, transforming both medical education and the future of surgery itself [7].