Physics-based Deep Neural Network for Augmented Reality during Liver Surgery

Jean-Nicolas Brunet (1) , Andrea Mendizabal (1) , Antoine Petit (1) , Nicolas Golse (2) , Eric Vibert (2) , Stéphane Cotin (1)

1 MIMESIS - Computational Anatomy and Simulation for Medicine
2 CHB - Centre hépato-biliaire

In this paper we present an approach combining a finite element method and a deep neural network to learn complex elastic deformations with the objective of providing augmented reality during hep-atic surgery. Derived from the U-Net architecture, our network is built entirely from physically-based simulations of a preoperative segmenta-tion of the organ. These simulations are performed using an immersed-boundary method, which offers several numerical and practical benefits, such as not requiring boundary-conforming volume elements. We perform a quantitative assessment of the method using synthetic and ex vivo patient data. Results show that the network is capable of solving the deformed state of the organ using only a sparse partial surface displacement data and achieve similar accuracy as a FEM solution, while being about 100x faster. When applied to an ex vivo liver example, we achieve the registration in only 3 ms with a mean target registration error (TRE) of 2.9 mm.

Mots clés: augmented reality; real-time simulation; Deep learning

Domaines: Intelligence artificielle [cs.AI]; Modélisation et simulation

Jean-Nicolas Brunet, Andrea Mendizabal, Antoine Petit, Nicolas Golse, Eric Vibert, et al.. Physics-based Deep Neural Network for Augmented Reality during Liver Surgery. MICCAI 2019 - 22nd International Conference on Medical Image Computing and Computer Assisted Intervention, Oct 2019, Shenzhen, China. pp.8, ⟨10.1007/978-3-030-32254-0_16⟩. ⟨hal-02158862v2⟩