Publications HAL de nazim haouchine

2023

Book sections

titre: Augmented Reality for Computer-Guided Interventions
auteur: Stéphane Cotin, Nazim Haouchine
article: Andrew Yeh Ching Nee; Soh Kim Ong. Springer Handbook of Augmented Reality, Springer, pp.689, 2023, 978-3-030-67822-7. ⟨10.1007/978-3-030-67822-7_28⟩
Accès au bibtex

2022

Journal articles

titre: Pose Estimation and Non-rigid Registration for Augmented Reality during Neurosurgery
auteur: Nazim Haouchine, Parikshit Juvekar, Michael Nercessian, William Wells, Alexandra Golby, Sarah Frisken
article: IEEE Transactions on Biomedical Engineering, 2022, 69 (4), pp.1310 - 1317. ⟨10.1109/TBME.2021.3113841⟩
resume: Objective: A craniotomy is the removal of a part of the skull to allow surgeons to have access to the brain and treat tumors. When accessing the brain, a tissue deformation occurs and can negatively influence the surgical procedure outcome. In this work, we present a novel Augmented Reality neurosurgical system to superimpose pre-operative 3D meshes derived from MRI onto a view of the brain surface acquired during surgery. Methods: Our method uses cortical vessels as main features to drive a rigid then non-rigid 3D/2D registration. We first use a feature extractor network to produce probability maps that are fed to a pose estimator network to infer the 6-DoF rigid pose. Then, to account for brain deformation, we add a nonrigid refinement step formulated as a Shape-from-Template problem using physics-based constraints that helps propagate the deformation to sub-cortical level and update tumor location. Results: We tested our method retrospectively on 6 clinical datasets and obtained low pose error, and showed using synthetic dataset that considerable brain shift compensation and low TRE can be achieved at cortical and sub-cortical levels. Conclusion: The results show that our solution achieved accuracy below the actual clinical errors demonstrating the feasibility of practical use of our system. Significance: This work shows that we can provide coherent Augmented Reality visualization of 3D cortical vessels observed through the craniotomy using a single camera view and that cortical vessels provide strong features for performing both rigid and non-rigid registration.
Accès au texte intégral et bibtex

titre: Cortical Vessel Segmentation for Neuronavigation using Vesselness-enforced Deep Neural Networks
auteur: Nazim Haouchine, Michael Nercessian, Parikshit Juvekar, Alexandra Golby, Sarah Frisken
article: IEEE Transactions on Medical Robotics and Bionics, 2022
resume: We propose in this paper an efficient method to segment cortical vessels in craniotomy images acquired by the surgical microscope. Our method uses a vesselness-enforced convolutional neural network to classify each pixel of a craniotomy image as a vessel or surrounding tissue. This permits training the network not only on appearance-based features but also on geometrical-based constraints that will ensure the continuity of the vascular trees. Our solution uses neural style transfer to generate new instances of images from manually labeled data leading to augment the training dataset in an anatomically semantic manner. The generated images improve the generalization of our model to various types of cortical surface appearances and vascular geometries. We conducted experiments on real images from human patients that demonstrate that accurate intraoperative cortical vessel segmentation can be achieved.
Accès au texte intégral et bibtex

2021

Conference papers

titre: Estimation of High Framerate Digital Subtraction Angiography Sequences at Low Radiation Dose
auteur: Nazim Haouchine, Parikshit Juvekar, Xin Xiong, Jie Luo, Tina Kapur, Rose Du, Alexandra Golby, Sarah Frisken
article: MICCAI 2021 - Medical Image Computing and Computer Assisted Interventions, Sep 2021, Strasbourg, France
resume: Digital Subtraction Angiography (DSA) provides high resolution image sequences of blood flow through arteries and veins and is considered the gold standard for visualizing cerebrovascular anatomy for neurovascular interventions. However, acquisition frame rates are typically limited to 1-3 fps to reduce radiation exposure, and thus DSA sequences often suffer from stroboscopic effects. We present the first approach that permits generating high frame rate DSA sequences from low frame rate acquisitions eliminating these artifacts without increasing the patient's exposure to radiation. Our approach synthesizes new intermediate frames using a phase-aware Convolutional Neural Network. This network accounts for the non-linear blood flow progression due to vessel geometry and initial velocity of the contrast agent. Our approach outperforms existing methods and was tested on several low frame rate DSA sequences of the human brain resulting in sequences of up to 17 fps with smooth and continuous contrast flow, free of flickering artifacts.
Accès au texte intégral et bibtex

titre: Deep Cortical Vessel Segmentation Driven By Data Augmentation With Neural Image Analogy
auteur: Michael Nercessian, Nazim Haouchine, Parikshit Juvekar, Sarah Frisken, Alexandra Golby
article: ISBI 2021 - IEEE International Symposium on Biomedical Imaging, Apr 2021, Nice, France
resume: During a craniotomy, a bone flap is temporarily removed from the skull to reveal the brain for surgery. The cortical vessels located at the surface of the brain are considered strong features to guide surgeons during the procedure, since they appear in both preoperative and intraoperative images and are an indication of how the brain may have shifted. We propose a method utilizing a deep neural network to perform cortical vessel segmentation in craniotomy images captured through the surgical microscope. Following a U-Net architecture, our solution classifies each pixel of a craniotomy image as vessel, parenchyma, or surrounding tissue and background. We use neural image analogy to build a diverse training set of unique images mirroring cortical anatomy generated from a limited amount of manually labeled data. The synthesized images enhance generalization of our model to various types of cortical surface appearances and geometries. Experiments on real data from human patients show that intraoperative cortical vessel segmentation can be performed accurately.
Accès au texte intégral et bibtex

2020

Journal articles

titre: Predicted Microscopic Cortical Brain Images for Optimal Craniotomy Positioning and Visualization
auteur: Nazim Haouchine, Pariskhit Juvekar, Alexandra Golby, Sarah Frisken
article: Computer Methods in Biomechanics and Biomedical Engineering: Imaging & Visualization, 2020, ⟨10.1080/21681163.2020.1834874⟩
resume: During a craniotomy, the skull is opened to allow surgeons to have access to the brain and perform the procedure. The position and size of this opening are chosen in a way to avoid critical structures, such as vessels, and facilitate the access to tumors. Planning the operation is done based on pre-operative images and does not account for intra-operative surgical events. We present a novel image-guided neurosurgical system to optimize the craniotomy opening. Using physics-based modeling we define a cortical deformation map that estimates the displacement field at candidate craniotomy locations. This deformation map is coupled with an image analogy algorithm that produces realistic synthetic images that can be used to predict both the geometry and the appearance of the brain surface before opening the skull. These images account for cortical vessel deformations that may occur after opening the skull and is rendered in a way that increases the surgeon's understanding and assimilation. Our method was tested retrospectively on patients data showing good results and demonstrating the feasibility of practical use of our system.
Accès au texte intégral et bibtex

Conference papers

titre: Deformation Aware Augmented Reality for Craniotomy using 3D/2D Non-rigid Registration of Cortical Vessels
auteur: Nazim Haouchine, Parikshit Juvekar, William M Wells Iii, Stéphane Cotin, Alexandra Golby, Sarah Frisken
article: Medical Image Computing and Computer Assisted Intervention, Oct 2020, Lima, Peru. pp.735--744
resume: Intra-operative brain shift is a well-known phenomenon that describes non-rigid deformation of brain tissues due to gravity and loss of cerebrospinal fluid among other phenomena. This has a negative influence on surgical outcome that is often based on pre-operative planning where the brain shift is not considered. We present a novel brain-shift aware Augmented Reality method to align pre-operative 3D data onto the deformed brain surface viewed through a surgical microscope. We formulate our non-rigid registration as a Shape-from-Template problem. A pre-operative 3D wire-like deformable model is registered onto a single 2D image of the cortical vessels, which is automatically segmented. This 3D/2D registration drives the underlying brain structures, such as tumors, and compensates for the brain shift in sub-cortical regions. We evaluated our approach on simulated and real data composed of 6 patients. It achieved good quantitative and qualitative results making it suitable for neurosurgical guidance.
Accès au texte intégral et bibtex

titre: Alignment of Cortical Vessels viewed through the Surgical Microscope with Preoperative Imaging to Compensate for Brain Shift
auteur: Nazim Haouchine, Parikshit Juvekar, Alexandra Golby, William M Wells, Stéphane Cotin, Sarah Frisken
article: Medical Imaging 2020: Image-Guided Procedures, Robotic Interventions, and Modeling, Feb 2020, Houston, United States
resume: Brain shift is a non-rigid deformation of brain tissue that is affected by loss of cerebrospinal fluid, tissue manipulation and gravity among other phenomena. This deformation can negatively influence the outcome of a surgical procedure since surgical planning based on pre-operative image becomes less valid. We present a novel method to compensate for brain shift that maps preoperative image data to the deformed brain during intra-operative neurosurgical procedures and thus increases the likelihood of achieving a gross total resection while decreasing the risk to healthy tissue surrounding the tumor. Through a 3D/2D non-rigid registration process, a 3D articulated model derived from pre-operative imaging is aligned onto 2D images of the vessels viewed through the surgical miscroscopic intra-operatively. The articulated 3D vessels constrain a volumetric biomechanical model of the brain to propagate cortical vessel deformation to the parenchyma and in turn to the tumor. The 3D/2D non-rigid registration is performed using an energy minimization approach that satisfies both projective and physical constraints. Our method is evaluated on real and synthetic data of human brain showing both quantitative and qualitative results and exhibiting its particular suitability for real-time surgical guidance.
Accès au texte intégral et bibtex

2019

Conference papers

titre: Deformed Reality
auteur: Antoine Petit, Nazim Haouchine, Frédérick Roy, Dan B Goldman, Stéphane Cotin
article: Computer Graphics & Visual Computing (Eurographics), Sep 2019, Bangor, United Kingdom
resume: We present Deformed Reality, a new way of interacting with an augmented reality environment by manipulating 3D objects in an intuitive and physically-consistent manner. Using the core principle of augmented reality to estimate rigid pose over time, our method makes it possible for the user to deform the targeted object while it is being rendered with its natural texture, giving the sense of a interactive scene editing. Our framework follows a computationally efficient pipeline that uses a proxy CAD model for both pose computation, physically-based manipulations and scene appearance estimation. The final composition is built upon a continuous image completion and re-texturing process to preserve visual consistency. The presented results show that our method can open new ways of using augmented reality by not only augmenting the environment but also interacting with objects intuitively.
Accès au texte intégral et bibtex

2018

Journal articles

titre: Calipso: Physics-based Image and Video Editing through CAD Model Proxies
auteur: Nazim Haouchine, Frédérick Roy, Hadrien Courtecuisse, Matthias Niessner, Stéphane Cotin
article: The Visual Computer, 2018, 36, pp.211-226. ⟨10.1007/s00371-018-1600-0⟩
resume: We present Calipso, an interactive method for editing images and videos in a physically-coherent manner. Our main idea is to realize physics-based manipulations by running a full physics simulation on proxy geometries given by non-rigidly aligned CAD models. Running these simulations allows us to apply new, unseen forces to move or deform selected objects, change physical parameters such as mass or elasticity, or even add entire new objects that interact with the rest of the underlying scene. In our method the user makes edits directly in 3D; these edits are processed by the simulation and then transfered to the target 2D content using shape-to-image correspondences in a photo-realistic rendering process. To align the CAD models, we introduce an efficient CAD-to-image alignment procedure that jointly minimizes for rigid and non-rigid alignment while preserving the high-level structure of the input shape. Moreover, the user can choose to exploit image flow to estimate scene motion, producing coherent physical behavior with ambient dynamics. We demonstrate physics-based editing on a wide range of examples producing myriad physical behavior while preserving geometric and visual consistency.
Accès au texte intégral et bibtex

titre: Vision-based Force Feedback Estimation for Robot-assisted Surgery using Instrument-constrained Biomechanical 3D Maps
auteur: Nazim Haouchine, Winnie Kuang, Stéphane Cotin, Michael Yip
article: IEEE Robotics and Automation Letters, 2018, ⟨10.1109/LRA.2018.2810948⟩
resume: We present a method for estimating visual and haptic force feedback on robotic surgical systems that currently do not include significant force feedback for the operator. Our approach permits to compute contact forces between instruments and tissues without additional sensors, relying only on endoscopic images acquired by a stereoscopic camera. Using an underlying biomechanical model built on-the-fly from the organ shape and by considering the surgical tool as boundary conditions acting on the surface of the model, contact force can be estimated at the tip of the tool. At the same time these constraints generate stresses that permit to compose a new endoscopic image as visual feedback for the surgeon. The results are demonstrated on in- vivo sequences of a human liver during robotic surgery, while quantitative validation is performed on an DejaVu and ex-vivo experimentation with ground truth to show the advantage of our approach.
Accès au texte intégral et bibtex

2017

Conference papers

titre: DejaVu: Intra-operative Simulation for Surgical Gesture Rehearsal
auteur: Nazim Haouchine, Danail Stoyanov, Frederick Roy, Stéphane Cotin
article: Medical Image Computing and Computer Assisted Interventions Conference MICCAI 2017, Oct 2017, Quebec City, Canada
resume: Advances in surgical simulation and surgical augmented reality have changed the way surgeons prepare for practice and conduct medical procedures. Despite considerable interest from surgeons, the use of simulation is still predominantly confined to pre-operative training of surgical tasks and the lack of robustness of surgical augmented reality means that it is seldom used for surgical guidance. In this paper, we present DejaVu, a novel surgical simulation approach for intra-operative surgical gesture rehearsal. With DejaVu we aim at bridging the gap between pre-operative surgical simulation and crucial but not yet robust intra-operative surgical augmented reality. By exploiting intra-operative images we produce a simulation that faithfully matches the actual procedure without visual discrepancies and with an underlying physical modelling that performs real-time deformation of organs and surrounding tissues, surgeons can interact with the targeted organs through grasping, pulling or cutting to immediately rehearse their next gesture. We present results on different in vivo surgical procedures and demonstrate the feasibility of practical use of our system.
Accès au texte intégral et bibtex

titre: Silhouette-based Pose Estimation for Deformable Organs Application to Surgical Augmented Reality
auteur: Yinoussa Adagolodjo, Raffaella Trivisonne, Nazim Haouchine, Stéphane Cotin, Hadrien Courtecuisse
article: IROS 2017 - IEEE/RSJ International Conference on Intelligent Robots and Systems, Sep 2017, Vancouver, Canada
resume: — In this paper we introduce a method for semi-automatic registration of 3D deformable models using 2D shape outlines (silhouettes) extracted from a monocular camera view. Our framework is based on the combination of a biomechanical model of the organ with a set of projective constraints influencing the deformation of the model. To enforce convergence towards a global minimum for this ill-posed problem we interactively provide a rough (rigid) estimation of the pose. We show that our approach allows for the estimation of the non-rigid 3D pose while relying only on 2D information. The method is evaluated experimentally on a soft silicone gel model of a liver, as well as on real surgical data, providing augmented reality of the liver and the kidney using a monocular laparoscopic camera. Results show that the final elastic registration can be obtained in just a few seconds, thus remaining compatible with clinical constraints. We also evaluate the sensitivity of our approach according to both the initial alignment of the model and the silhouette length and shape.
Accès au texte intégral et bibtex

titre: Image-driven Stochastic Identification of Boundary Conditions for Predictive Simulation
auteur: Igor Peterlik, Nazim Haouchine, Lukáš Ručka, Stéphane Cotin
article: 20th International Conference on Medical Image Computing and Computer Assisted Intervention, Sep 2017, Québec, Canada
resume: In computer-aided interventions, biomechanical models reconstructed from the pre-operative data are used via augmented reality to facilitate the intra-operative navigation. The predictive power of such models highly depends on the knowledge of boundary conditions. However , in the context of patient-specific modeling, neither the pre-operative nor the intra-operative modalities provide a reliable information about the location and mechanical properties of the organ attachments. We present a novel image-driven method for fast identification of boundary conditions which are modelled as stochastic parameters. The method employs the reduced-order unscented Kalman filter to transform in real-time the probability distributions of the parameters, given observations extracted from intra-operative images. The method is evaluated using synthetic, phantom and real data acquired in vivo on a porcine liver. A quantitative assessment is presented and it is shown that the method significantly increases the predictive power of the biomechanical model.
Accès au texte intégral et bibtex

titre: Template-based Monocular 3D Recovery of Elastic Shapes using Lagrangian Multipliers
auteur: Nazim Haouchine, Stéphane Cotin
article: Computer Vision and Pattern Recognition (CVPR), Jul 2017, Honolulu, Hawai, United States
resume: We present in this paper an efficient template-based method for 3D recovery of elastic shapes from a fixed monocular camera. By exploiting the object's elasticity, in contrast to isometric methods that use inextensibility constraints , a large range of deformations can be handled. Our method is expressed as a saddle point problem using La-grangian multipliers resulting in a linear system which unifies both mechanical and optical constraints and integrates Dirichlet boundary conditions, whether they are fixed or free. We experimentally show that no prior knowledge on material properties is needed, which exhibit the generic usability of our method with elastic and inelastic objects with different kinds of materials. Comparisons with existing techniques are conducted on synthetic and real elastic objects with strains ranging from 25% to 130% resulting to low errors.
Accès au texte intégral et bibtex

Poster communications

titre: Deformed Reality: Proof of concept and preliminary results
auteur: Nazim Haouchine, Antoine Petit, Frederick Roy, Stéphane Cotin
article: ISMAR 2017 - 16th IEEE International Symposium on Mixed and Augmented Reality, Oct 2017, Nantes, France. 2017
resume: We introduce " Deformed Reality " , a new paradigm to interactively manipulate objects in a scene in a deformable manner. Using the core principle of augmented reality to estimate rigid pose over time, our method enables the user to deform the targeted object while it is being rendered with its natural texture, giving the sense of a real-time object editing in user environment. The presented results show that our method can open new ways of using augmented reality by not only augmenting the scene but also interacting with it in a non-rigid manner.
Accès au texte intégral et bibtex

2016

Journal articles

titre: Handling Topological Changes during Elastic Registration: Application to Augmented Reality in Laparoscopic Surgery
auteur: Christoph Joachim Paulus, Nazim Haouchine, Seong-Ho Kong, Renato Vianna Soares, David Cazier, Stéphane Cotin
article: International Journal of Computer Assisted Radiology and Surgery, 2016, 12, pp.461-470. ⟨10.1007/s11548-016-1502-4⟩
resume: Purpose: Locating the internal structures of an organ is a critical aspect of many surgical procedures. Minimally invasive surgery, associated with augmented reality techniques, offers the potential to visualize inner structures, allowing for improved analysis, depth perception or for supporting planning and decision systems. Methods: Most of the current methods dealing with rigid or non-rigid augmented reality make the assumption that the topology of the organ is not modified. As surgery relies essentially on cutting and dissection of anatomical structures, such methods are limited to the early stages of the surgery. We solve this shortcoming with the introduction of a method for physics-based elastic registration using a single view from a monocular camera. Singularities caused by topological changes are detected and propagated to the pre-operative model. This significantly improves the coherence between the actual laparoscopic view and the model, and provides added value in terms of navigation and decision-making, e.g. by overlaying the internal structures of an organ on the laparoscopic view. Results: Our real time augmentation method is assessed on several scenarios, using synthetic objects and real organs. In all cases, the impact of our approach is demonstrated, both qualitatively and quantitatively. Conclusion: The presented approach tackles the challenge of localizing internal structures throughout a complete surgical procedure, even after surgical cuts. This information is crucial for surgeons to improve the outcome for their surgical procedure and avoid complications.
Accès au texte intégral et bibtex

titre: Robust Augmented Reality registration method for Localization of Solid Organs’ Tumors Using CT-derived Virtual Biomechanical Model and Fluorescent Fiducials
auteur: Seong-Ho Kong, Nazim Haouchine, Renato Soares, Andrey S Klymchenko, Bohdan Andreiuk, Bruno Marques, Galyna Shabat, Thierry Piéchaud, Michele Diana, Stéphane Cotin, Jacques Marescaux
article: Surgical Endoscopy, 2016, ⟨10.1007/s00464-016-5297-8⟩
resume: Accurate localization of solid organs tumors is crucial to ensure both radicality and organ function preservation. Augmented Reality (AR) is the fusion of computer-generated and real-time images. AR can be used in surgery as a navigation tool, by creating a patient-specific virtual model through 3D software manipulation of DICOM imaging (e.g. CT-scan). The virtual model can be superimposed to the real-time images to obtain the enhanced real-time localization. However, the 3D virtual model is rigid, and does not take into account inner structures’ deformations. We present a concept of automated navigation system, enabling transparency visualization of internal anatomy and tumor’s margins, while the organs undergo deformation during breathing or surgical manipulation.
Accès au texte intégral et bibtex

Conference papers

titre: Using Contours as Boundary Conditions for Elastic Registration during Minimally Invasive Hepatic Surgery
auteur: Nazim Haouchine, Frederick Roy, Lionel Untereiner, Stéphane Cotin
article: International Conference on Intelligent Robots and Systems, Oct 2016, Daejeon, South Korea
resume: We address in this paper the ill-posed problem of initial alignment of pre-operative to intra-operative data for augmented reality during minimally invasive hepatic surgery. This problem consists of finding the rigid transformation that relates the scanning reference and the endoscopic camera pose, and the non-rigid transformation undergone by the liver w.r.t its scanned state. Most of the state-of-the-art methods assume a known initial registration. Here, we propose a method that permits to recover the deformation undergone by the liver while simultaneously finding the rotational and translational parts of the transformation. Our formulation considers the boundaries of the liver with its surrounding tissues as hard constraints directly encoded in an energy minimization process. We performed experiments on real in-vivo data of human hepatic surgery and synthetic data, and compared our method with related works.
Accès au texte intégral et bibtex

titre: Simultaneous Pose Estimation and Augmentation of Elastic Surfaces from a Moving Monocular Camera
auteur: Nazim Haouchine, Marie-Odile Berger, Stephane Cotin
article: International Symposium on Mixed and Augmented Reality, Sep 2016, Merida, Mexico
resume: We present in this paper an original method to estimate the pose of a monocular camera while simultaneously modeling and capturing the elastic deformation of the object to be augmented. Our method tackles a challenging problem where ambiguities between rigid motion and non-rigid deformation are present. This issue represents a major lock for the establishment of an efficient surgical augmented reality where endoscopic camera moves and organs deform. Using an underlying physical model to estimate the low stressed regions our algorithm separates the rigid body motion from the elastic deformations using polar decomposition of the strain tensor. Following this decomposition, a constrained minimization, that encodes both the optical and the physical constraints, is resolved at each frame. Results on real and simulated data are exposed to show the effectiveness of our approach.
Accès au texte intégral et bibtex

titre: Segmentation and Labelling of Intra-operative Laparoscopic Images using Structure from Point Cloud
auteur: Nazim Haouchine, Stephane Cotin
article: International Symposium on Biomedical Imaging : "From Nano to Macro" (ISBI 2016), Apr 2016, Prague, Czech Republic
resume: We present in this paper an automatic method for segmenting and labelling of liver its surrounding tissues in intra-operative laparoscopic images. The goal is to be able to distinguished between the different structure that compose a common intra-operative hepatic surgery scene. This will permits to improve the registration between pre-operative data and intra-operative images for task such as Augmented Reality. Our segmentation method consider the scene as a 3D structured point cloud instead of a laparoscopic images in order to exploit powerful informations such as curvature and normals, in addition to visual cues that permits to efficiently classify the scene. Our approach works well on sparse and noisy point clouds, thanks to a surface approximation stage, and unlike existing approaches, is independent of organs texture in the image. Experiements performed on challenging human hepatic surgery confirm that accurate segmentation and labelling are possible using 3D structure information and appropriate visual cues.
Accès au texte intégral et bibtex

2015

Journal articles

titre: Patient-specific Biomechanical Modeling for Guidance during Minimally-invasive Hepatic Surgery
auteur: Rosalie Plantefève, Igor Peterlik, Nazim Haouchine, Stéphane Cotin
article: Annals of Biomedical Engineering, 2015
resume: During the minimally-invasive liver surgery, only the partial surface view of the liver is usually provided to the surgeon via the laparoscopic camera. Therefore, it is necessary to estimate the actual position of the internal structures such as tumors and vessels from the pre-operative images. Nevertheless, such task can be highly challenging since during the intervention, the abdominal organs undergo important deformations due to the pneumoperitoneum, respiratory and cardiac motion and the interaction with the surgical tools. Therefore, a reliable automatic system for intra-operative guidance requires fast and reliable registration of the pre- and intra-operative data. In this paper we present a complete pipeline for the registration of pre-operative patient-specific image data to the sparse and incomplete intra-operative data. While the intra-operative data is represented by a point cloud extracted from the stereo-endoscopic images, the pre-operative data is used to reconstruct a biomechanical model which is necessary for accurate estimation of the position of the internal structures, considering the actual deformations. This model takes into account the patient-specific liver anatomy composed of parenchyma, vascularization and capsule, and is enriched with anatomical boundary conditions transferred from an atlas. The registration process employs the iterative closest point technique together with a penalty-based method. We perform a quantitative assessment based on the evaluation of the target registration error on synthetic data as well as a qualitative assessment on real patient data. We demonstrate that the proposed registration method provides good results in terms of both accuracy and robustness w. r. t. the quality of the intra-operative data.
Accès au texte intégral et bibtex

titre: Impact of Soft Tissue Heterogeneity on Augmented Reality for Liver Surgery
auteur: Nazim Haouchine, Stephane Cotin, Igor Peterlik, Jeremie Dequidt, Mario Sanz Lopez, Erwan Kerrien, Marie-Odile Berger
article: IEEE Transactions on Visualization and Computer Graphics, 2015, 21 (5), pp.584 - 597. ⟨10.1109/TVCG.2014.2377772⟩
resume: This paper presents a method for real-time augmented reality of internal liver structures during minimally invasive hepatic surgery. Vessels and tumors computed from pre-operative CT scans can be overlaid onto the laparoscopic view for surgery guidance. Compared to current methods, our method is able to locate the in-depth positions of the tumors based on partial three-dimensional liver tissue motion using a real-time biomechanical model. This model permits to properly handle the motion of internal structures even in the case of anisotropic or heterogeneous tissues, as it is the case for the liver and many anatomical structures. Experimentations conducted on phantom liver permits to measure the accuracy of the augmentation while real-time augmentation on in vivo human liver during real surgery shows the benefits of such an approach for minimally invasive surgery.
Accès au texte intégral et bibtex

titre: Monocular 3D Reconstruction and Augmentation of Elastic Surfaces with Self-occlusion Handling
auteur: Nazim Haouchine, Jeremie Dequidt, Marie-Odile Berger, Stephane Cotin
article: IEEE Transactions on Visualization and Computer Graphics, 2015, pp.14. ⟨10.1109/TVCG.2015.2452905⟩
resume: This paper focuses on the 3D shape recovery and augmented reality on elastic objects with self-occlusions handling, using only single view images. Shape recovery from a monocular video sequence is an underconstrained problem and many approaches have been proposed to enforce constraints and resolve the ambiguities. State-of-the art solutions enforce smoothness or geometric constraints, consider specific deformation properties such as inextensibility or resort to shading constraints. However, few of them can handle properly large elastic deformations. We propose in this paper a real-time method that uses a mechanical model and able to handle highly elastic objects. The problem is formulated as an energy minimization problem accounting for a non-linear elastic model constrained by external image points acquired from a monocular camera. This method prevents us from formulating restrictive assumptions and specific constraint terms in the minimization. In addition, we propose to handle self-occluded regions thanks to the ability of mechanical models to provide appropriate predictions of the shape. Our method is compared to existing techniques with experiments conducted on computer-generated and real data that show the effectiveness of recovering and augmenting 3D elastic objects. Additionally, experiments in the context of minimally invasive liver surgery are also provided and results on deformations with the presence of self-occlusions are exposed.
Accès au texte intégral et bibtex

Conference papers

titre: Framework for augmented reality in Minimally Invasive laparoscopic surgery
auteur: Bruno Marques, Rosalie Plantefeve, Frédérick Roy, Nazim Haouchine, Emmanuel Jeanvoine, Igor Peterlik, Stéphane Cotin
article: HealthCom 2015, Oct 2015, Boston, United States. ⟨10.1109/HealthCom.2015.7454467⟩
resume: This article presents a framework for fusing pre-operative data and intra-operative data for surgery guidance. This framework is employed in the context of Minimally Invasive Surgery (MIS) of the liver. From stereoscopic images a three dimensional point cloud is reconstructed in real-time. This point cloud is then used to register a patient-specific biomechanical model derived from Computed Tomography images onto the laparoscopic view. In this way internal structures such as vessels and tumors can be visualized to help the surgeon during the procedure. This is particularly relevant since abdominal organs undergo large deformations in the course of the surgery, making it difficult for surgeons to correlate the laparoscopic view with the pre-operative images. Our method has the potential to reduce the duration of the operation as the biomechanical model makes it possible to estimate the in-depth position of tumors and vessels at any time of the surgery, which is essential to the surgical decision process. Results show that our method can be successfully applied during laparoscopic procedure without interfering with the surgical work flow.
Accès au texte intégral et bibtex

titre: Surgical Augmented Reality with Topological Changes
auteur: Christoph J. Paulus, Nazim Haouchine, David Cazier, Stéphane Cotin
article: Medical Image Computing and Computer Assisted Interventions, Oct 2015, München, Germany
resume: The visualization of internal structures of organs in minimally invasive surgery is an important avenue for improving the perception of the surgeon, or for supporting planning and decision systems. However, current methods dealing with non-rigid augmented reality only provide augmentation when the topology of the organ is not modified. In this paper we solve this shortcoming by introducing a method for physics-based non-rigid augmented reality. Singularities caused by topo-logical changes are detected and propagated to the pre-operative model. This significantly improves the coherence between the actual laparascopic view and the model, and provides added value in terms of navigation and decision making. Our real time augmentation algorithm is assessed on a video showing the cut of a porcine liver's lobe in minimal invasive surgery.
Accès au texte intégral et bibtex

titre: Surgical Augmented Reality with Topological Changes
auteur: Christoph Paulus, Nazim Haouchine, David Cazier, Stéphane Cotin
article: MICCAI 2015: Medical Image Computing and Computer-Assisted Intervention, Oct 2015, Munich, Germany. ⟨10.1007/978-3-319-24553-9_51⟩
Accès au texte intégral et bibtex

titre: Augmented Reality during Cutting and Tearing of Deformable Objects
auteur: Christoph J. Paulus, Nazim Haouchine, David Cazier, Stephane Cotin
article: The 14th IEEE International Symposium on Mixed and Augmented Reality, Sep 2015, Fukuoka, Japan. pp.6
resume: Current methods dealing with non-rigid augmented reality only provide an augmented view when the topology of the tracked object is not modified, which is an important limitation. In this paper we solve this shortcoming by introducing a method for physics-based non-rigid augmented reality. Singularities caused by topological changes are detected by analyzing the displacement field of the underlying deformable model. These topological changes are then applied to the physics-based model to approximate the real cut. All these steps, from deformation to cutting simulation, are performed in real-time. This significantly improves the coherence between the actual view and the model, and provides added value.
Accès au texte intégral et bibtex

titre: Improving depth perception during surgical augmented reality
auteur: Bruno Marques, Nazim Haouchine, Rosalie Plantefeve, Stephane Cotin
article: SIGGRAPH [Poster], Aug 2015, Los Angeles, United States. pp.Article No. 24, ⟨10.1145/2787626.2792654⟩
resume: This study suggests a method to compensate the loss of depth perception while enhancing organ vessels and tumors to surgeons. This method relies on a combination of contour rendering technique and adaptive alpha blending to effectively perceive the vessels and tumors depth. In addition, this technique is designed to achieve real-time to satisfy the requirements of clinical routines, and has been tested on real human surgery.
Accès au texte intégral et bibtex

titre: Fracture in Augmented Reality
auteur: Nazim Haouchine, Alexandre Bilger, Jeremie Dequidt, Stephane Cotin
article: SIGGRAPH [Poster], Aug 2015, Los Angeles, United States
resume: We propose in this study an image-guided mesh cutting method to handle real-time augmentation of paper tearing. This method relies on the combination of visually-based fracture tracking algorithm and a physics-based model that is dynamically superimposed on the image.
Accès au texte intégral et bibtex

titre: Surgery Training, Planning and Guidance Using the SOFA Framework
auteur: Hugo Talbot, Nazim Haouchine, Igor Peterlik, Jeremie Dequidt, Christian Duriez, Hervé Delingette, Stephane Cotin
article: Eurographics, May 2015, Zurich, Switzerland
resume: In recent years, an active development of novel technologies dealing with medical training, planning and guidance has become an increasingly important area of interest in both research and health-care manufacturing. A combination of advanced physical models, realistic human-computer interaction and growing computational power is bringing new solutions in order to help both medical students and experts to achieve a higher degree of accuracy and reliability in surgical interventions. In this paper, we present three different examples of medical physically-based simulations implemented in a common software platform called SOFA. Each example represents a different application: training for cardiac electrophysiology, pre-operative planning of cryosurgery and per-operative guidance for laparoscopy. The goal of this presentation is to evaluate the realism, accuracy and efficiency of the simulations, as well as to demonstrate the potential and flexibility of the SOFA platform.
Accès au texte intégral et bibtex

Theses

titre: Image-guided Simulation for Augmented Reality during Hepatic Surgery
auteur: Nazim Haouchine
article: Computer Science [cs]. Université de Lille1, 2015. English. ⟨NNT : ⟩
resume: The main objective of this thesis is to provide surgeons with tools for pre and intra-operative decision support during minimally invasive hepatic surgery. These interventions are usually based on laparoscopic techniques or, more recently, flexible endoscopy. During such operations, the surgeon tries to remove a significant number of liver tumors while preserving the functional role of the liver. This involves defining an optimal hepatectomy, i.e. ensuring that the volume of post-operative liver is at least at 55% of the original liver and the preserving at hepatic vasculature. Although intervention planning can now be considered on the basis of preoperative patient-specific, significant movements of the liver and its deformations during surgery data make this very difficult to use planning in practice. The work proposed in this thesis aims to provide augmented reality tools to be used in intra-operative conditions in order to visualize the position of tumors and hepatic vascular networks at any time.
Accès au texte intégral et bibtex

2014

Conference papers

titre: Automatic Alignment of pre and intraoperative Data using Anatomical Landmarks for Augmented Laparoscopic Liver Surgery
auteur: Rosalie Plantefeve, Nazim Haouchine, Jean Pierre Radoux, Stéphane Cotin
article: International Symposium on Biomedical Simulation ISBMS, Oct 2014, Strasbourg, France
resume: Each year in Europe 50,000 new liver cancer cases are diagnosed for which hepatic surgery combined to chemotherapy is the most common treatment. In particular the number of laparoscopic liver surgeries has increased significantly over the past years. This type of minimally invasive procedure which presents many benefits for the patient is challenging for the surgeons due to the limited field of view. Recently new augmented reality techniques which merge preoperative data and intraoperative images and permit to visualize internal structures have been proposed to help surgeons during this type of surgery. One of the difficulties is to align preoperative data with the intraoperative images. We propose in this paper a semi-automatic approach for solving the ill-posed problem of initial alignment for Augmented Reality systems during liver surgery. Our registration method relies on anatomical landmarks extracted from both the laparoscopic images and three-dimensional model, using an image-based soft-tissue reconstruction technique and an atlas-based approach, respectively. The registration evolves automatically from a quasi-rigid to a non-rigid registration. Furthermore, the surface-driven deformation is induced in the volume via a patient specific biomechanical model. The experiments conducted on both synthetic and in vivo data show promising results with a registration error of 2 mm when dealing with a visible surface of 30% of the whole liver.
Accès au texte intégral et bibtex

titre: The Role of Ligaments: Patient-Specific or Scenario-Specific ?
auteur: Julien Bosman, Nazim Haouchine, Jérémie Dequidt, Igor Peterlik, Stéphane Cotin, Christian Duriez
article: International Symposium on Biomedical Simulation ISBMS, Oct 2014, Strasbourg, France
resume: In this paper, we present a preliminary study dealing with the importance of correct modeling of connective tissues such as ligaments in laparoscopic liver surgery simulation. We show that the model of these tissues has a significant impact on the overall results of the simulation. This is demonstrated numerically using two different scenarios from the laparoscopic liver surgery, both resulting in important deformation of the liver: insufflation of the abdominal cavity with gas (pneumoperitoneum) and manipulation with the liver lobe using a surgical instrument (grasping pincers). For each scenario, a series of simulations is performed with or without modeling the deformation of the ligaments (fixed constraints or biomechanical model with the parameter of the literature). The numerical comparison shows that modeling the ligament deformations can be at least as important as the correct selection of the patient-specific parameters, nevertheless this observation depends on the simulated scenario.
Accès au texte intégral et bibtex

titre: Single View Augmentation of 3D Elastic Objects
auteur: Nazim Haouchine, Jérémie Dequidt, Marie-Odile Berger, Stéphane Cotin
article: International Symposium on Mixed and Augmented Reality - ISMAR, Sep 2014, Munich, Germany
resume: This paper proposes an efficient method to capture and augment highly elastic objects from a single view. 3D shape recovery from a monocular video sequence is an underconstrained problem and many approaches have been proposed to enforce constraints and resolve the ambiguities. State-of-the art solutions enforce smoothness or geometric constraints, consider specific deformation properties such as inextensibility or ressort to shading constraints. However, few of them can handle properly large elastic deformations. We propose in this paper a real-time method which makes use of a me chanical model and is able to handle highly elastic objects. Our method is formulated as a energy minimization problem accounting for a non-linear elastic model constrained by external image points acquired from a monocular camera. This method prevents us from formulating restrictive assumptions and specific constraint terms in the minimization. The only parameter involved in the method is the Young's modulus where we show in experiments that a rough estimate of its value is sufficient to obtain a good reconstruction. Our method is compared to existing techniques with experiments conducted on computer-generated and real data that show the effectiveness of our approach. Experiments in the context of minimally invasive liver surgery are also provided.
Accès au texte intégral et bibtex

titre: Réalité augmentée pour la chirurgie minimalement invasive du foie utilisant un modèle biomécanique guidé par l'image
auteur: Nazim Haouchine, Stéphane Cotin, Jérémie Dequidt, Erwan Kerrien, Marie-Odile Berger
article: Reconnaissance de Formes et Intelligence Artificielle (RFIA) 2014, Jun 2014, Rouen, France
resume: Cet article présente une méthode de réalité augmentée pour la chirurgie minimalement invasive du foie. Le réseau vasculaire et les tumeurs internes reconstruites à partir des données pré-opératoires (IRM ou CT) peuvent ainsi être visualisées dans l'image laparoscopique afin de guider les gestes du chirurgien pendant l'opération. Cette méthode est capable de propager les déformations 3D de la surface du foie à ses structures internes grâce à un modèle biomécanique sous-jacent qui prend en compte l'anisotropie et l'hétérogénéité du tissu hépatique. Des résultats sont montrés sur une vidéo in-vivo d'un foie humain acquise pendant une opération et sur un foie en silicone.
Accès au texte intégral et bibtex

titre: Towards an Accurate Tracking of Liver Tumors for Augmented Reality in Robotic Assisted Surgery
auteur: Nazim Haouchine, Jérémie Dequidt, Igor Peterlik, Erwan Kerrien, Marie-Odile Berger, Stéphane Cotin
article: International Conference on Robotics and Automation (ICRA), Jun 2014, Hong Kong, China
resume: This article introduces a method for tracking the internal structures of the liver during robot-assisted procedures. Vascular network, tumors and cut planes, computed from pre-operative data, can be overlaid onto the laparoscopic view for image-guidance, even in the case of large motion or deformation of the organ. Compared to current methods, our method is able to precisely propagate surface motion to the internal structures. This is made possible by relying on a fast yet accurate biomechanical model of the liver combined with a robust visual tracking approach designed to properly constrain the model. Augmentation results are demonstrated on in-vivo sequences of a human liver during robotic surgery, while quantitative validation is performed on an ex-vivo porcine liver experimentation. Validation results show that our approach gives an accurate surface registration with an error of less than 6mm on the position of the tumor.
Accès au texte intégral et bibtex

2013

Journal articles

titre: Deformation-based Augmented Reality for Hepatic Surgery
auteur: Nazim Haouchine, Jérémie Dequidt, Marie-Odile Berger, Stéphane Cotin
article: Studies in Health Technology and Informatics, 2013, 184
resume: In this paper we introduce a method for augmenting the laparoscopic view during hepatic tumor resection. Using augmented reality techniques, vessels, tumors and cutting planes computed from pre-operative data can be overlaid onto the laparoscopic video. Compared to current techniques, which are limited to a rigid registration of the pre-operative liver anatomy with the intra-operative image, we propose a real-time, physics-based, non-rigid registration. The main strength of our approach is that the deformable model can also be used to regularize the data extracted from the computer vision algorithms. We show preliminary results on a video sequence which clearly highlights the interest of using physics-based model for elastic registration.
Accès au texte intégral et bibtex

Conference papers

titre: Image-guided Simulation of Heterogeneous Tissue Deformation For Augmented Reality during Hepatic Surgery
auteur: Nazim Haouchine, Jérémie Dequidt, Igor Peterlik, Erwan Kerrien, Marie-Odile Berger, Stéphane Cotin
article: ISMAR - IEEE International Symposium on Mixed and Augmented Reality 2013, Oct 2013, Adelaide, Australia
resume: This paper presents a method for real-time augmentation of vas- cular network and tumors during minimally invasive liver surgery. Internal structures computed from pre-operative CT scans can be overlaid onto the laparoscopic view for surgery guidance. Com- pared to state-of-the-art methods, our method uses a real-time biomechanical model to compute a volumetric displacement field from partial three-dimensional liver surface motion. This permits to properly handle the motion of internal structures even in the case of anisotropic or heterogeneous tissues, as it is the case for the liver and many anatomical structures. Real-time augmentation results are presented on in vivo and ex vivo data and illustrate the benefits of such an approach for minimally invasive surgery.
Accès au texte intégral et bibtex

2012

Conference papers

titre: Physics-based Augmented Reality for 3D Deformable Object
auteur: Nazim Haouchine, Jérémie Dequidt, Erwan Kerrien, Marie-Odile Berger, Stéphane Cotin
article: Eurographics Workshop on Virtual Reality Interaction and Physical Simulation, Dec 2012, Darmstadt, Germany
resume: This paper introduces an original method to perform augmented or mixed reality on deformable objects. Compared to state-of-the-art techniques, our method is able to track deformations of volumetric objects and not only surfacic objects. A flexible framework that relies on the combination of a 3D motion estimation and a physics-based deformable model used as a regularization and interpolation step allows to perform a non-rigid and robust registration. Results are exposed, based on computer-generated datasets and video sequences of real environments in order to assess the relevance of our approach.
Accès au texte intégral et bibtex