This PDF file contains the front matter associated with SPIE Proceedings Volume 9415, including the Title Page, Copyright information, Table of Contents, Invited Panel Discussion, and Conference Committee listing.

A mitral valve reconstruction (MVR) is a complex operation in which the functionality of incompetent mitral valves is re-established by applying surgical techniques. This work deals with predictive biomechanical simulations of operation scenarios for an MVR, and the simulation's integration into a knowledge-based surgery assistance system. We present a framework for the definition of the corresponding surgical workflow, which combines semantically enriched surgical expert knowledge with a biomechanical simulation. Using an ontology, 'surgical rules' which describe decision and assessment criteria for surgical decision-making are represented in a knowledge base. Through reasoning these 'rules' can then be applied on patient-specific data in order to be converted into boundary conditions for the biomechanical soft tissue simulation, which is based on the Finite Elements Method (FEM). The simulation, which is implemented in the open-source C++ FEM software HiFlow³, is controlled via the Medical Simulation Markup Language (MSML), and makes use of High Performance Computing (HPC) methods to cope with real-time requirements in surgery. The simulation results are presented to surgeons to assess the quality of the virtual reconstruction and the consequential remedial effects on the mitral valve and its functionality. The whole setup has the potential to support the intraoperative decision-making process during MVR where the surgeon usually has to make fundamental decision under time pressure.

Cardiac valvular stenosis, prolapse and regurgitation are increasingly common conditions, particularly in an elderly population with limited potential for on-pump cardiac surgery. NeoChord©, MitraClip© and numerous stent-based transcatheter aortic valve implantation (TAVI) devices provide an alternative to intrusive cardiac operations; performed while the heart is beating, these procedures require surgeons and cardiologists to learn new image-guidance based techniques. Developing these visual aids and protocols is a challenging task that benefits from sophisticated simulators. Existing models lack features needed to simulate off-pump valvular procedures: functional, dynamic valves, apical and vascular access, and user flexibility for different activation patterns such as variable heart rates and rapid pacing. We present a left ventricle phantom with these characteristics. The phantom can be used to simulate valvular repair and replacement procedures with magnetic tracking, augmented reality, fluoroscopy and ultrasound guidance. This tool serves as a platform to develop image-guidance and image processing techniques required for a range of minimally invasive cardiac interventions. The phantom mimics in vivo mitral and aortic valve motion, permitting realistic ultrasound images of these components to be acquired. It also has a physiological realistic left ventricular ejection fraction of 50%. Given its realistic imaging properties and non-biodegradable composition—silicone for tissue, water for blood—the system promises to reduce the number of animal trials required to develop image guidance applications for valvular repair and replacement. The phantom has been used in validation studies for both TAVI image-guidance techniques¹, and image-based mitral valve tracking algorithms².

Beating heart valve therapies rely extensively on image guidance to treat patients who would be considered inoperable with conventional surgery. Mitral valve repair techniques including the MitrClip, NeoChord, and emerging transcatheter mitral valve replacement techniques rely on transesophageal echocardiography for guidance. These images are often difficult to interpret as the tool will cause shadowing artifacts that occlude tissue near the target site. Here, we integrate ultrasound imaging directly into the NeoChord device. This provides an unobstructed imaging plane that can visualize the valve lea ets as they are engaged by the device and can aid in achieving both a proper bite and spacing between the neochordae implants. A proof of concept user study in a phantom environment is performed to provide a proof of concept for this device.

Two-dimensional (2D) ultrasound (US) has been the clinical standard for over two decades for monitoring and assessing cardiac function and providing support via intra-operative visualization and guidance for minimally invasive cardiac interventions. Developments in three-dimensional (3D) image acquisition and transducer design and technology have revolutionized echocardiography imaging enabling both real-time 3D trans-esophageal and intra-cardiac image acquisition. However, in most cases the clinicians do not access the entire 3D image volume when analyzing the data, rather they focus on several key views that render the cardiac anatomy of interest during the US imaging exam. This approach enables image acquisition at a much higher spatial and temporal resolution. Two such common approaches are the bi-plane and tri-plane data acquisition protocols; as their name states, the former comprises two orthogonal image views, while the latter depicts the cardiac anatomy based on three co-axially intersecting views spaced at 60⁰ to one another. Since cardiac anatomy is continuously changing, the intra-operative anatomy depicted using real-time US imaging also needs to be updated by tracking the key features of interest and endocardial left ventricle (LV) boundaries. Therefore, rapid automatic feature tracking in US images is critical for three reasons: 1) to perform cardiac function assessment; 2) to identify location of surgical targets for accurate tool to target navigation and on-target instrument positioning; and 3) to enable pre- to intra-op image registration as a means to fuse pre-op CT or MR images used during planning with intra-operative images for enhanced guidance. In this paper we utilize monogenic filtering, graph-cut based segmentation and robust spline smoothing in a combined work flow to process the acquired tri-plane TEE time series US images and demonstrate robust and accurate tracking of the LV endocardial features. We reconstruct the endocardial LV geometry using the tri-plane contours and spline interpolation, and assess the accuracy of the proposed work flow against gold-standard results from the GE Echopac PC clinical software according to quantitative clinical LV characterization parameters, such as the length, circumference, area and volume. Our proposed combined work flow leads to consistent, rapid and automated identification of the LV endocardium, suitable for intra-operative applications and "on-the-fly" computer-assisted assessment of ejection fraction for cardiac function monitoring.Two-dimensional (2D) ultrasound (US) has been the clinical standard for over two decades for monitoring and assessing cardiac function and providing support via intra-operative visualization and guidance for minimally invasive cardiac interventions. Developments in three-dimensional (3D) image acquisition and transducer design and technology have revolutionized echocardiography imaging enabling both real-time 3D trans-esophageal and intra-cardiac image acquisition. However, in most cases the clinicians do not access the entire 3D image volume when analyzing the data, rather they focus on several key views that render the cardiac anatomy of interest during the US imaging exam. This approach enables image acquisition at a much higher spatial and temporal resolution. Two such common approaches are the bi-plane and tri-plane data acquisition protocols; as their name states, the former comprises two orthogonal image views, while the latter depicts the cardiac anatomy based on three co-axially intersecting views spaced at 600 to one another. Since cardiac anatomy is continuously changing, the intra-operative anatomy depicted using real-time US imaging also needs to be updated by tracking the key features of interest and endocardial left ventricle (LV) boundaries. Therefore, rapid automatic feature tracking in US images is critical for three reasons: 1) to perform cardiac function assessment; 2) to identify location of surgical targets for accurate tool to target navigation and on-target instrument positioning; and 3) to enable pre- to intra-op image registration as a means to fuse pre-op CT or MR images used during planning with intra-operative images for enhanced guidance. In this paper we utilize monogenic filtering, graph-cut based segmentation and robust spline smoothing in a combined work flow to process the acquired tri-plane TEE time series US images and demonstrate robust and accurate tracking of the LV endocardial features. We reconstruct the endocardial LV geometry using the tri-plane contours and spline interpolation, and assess the accuracy of the proposed work flow against gold-standard results from the GE Echopac PC clinical software according to quantitative clinical LV characterization parameters, such as the length, circumference, area and volume. Our proposed combined work flow leads to consistent, rapid and automated identification of the LV endocardium, suitable for intra-operative applications and on-the- y" computer-assisted assessment of ejection fraction for cardiac function monitoring.

Cardiac ablation consists of navigating a catheter into the heart and delivering RF energy to electrically isolate tissue regions that generate or propagate arrhythmia. Besides the challenges of accurate and precise targeting of the arrhythmic sites within the beating heart, limited information is currently available to the cardiologist regarding intricate electrodetissue contact, which directly impacts the quality of produced lesions. Recent advances in ablation catheter design provide intra-procedural estimates of tissue-catheter contact force, but the most direct indicator of lesion quality for any particular energy level and duration is the tissue-catheter contact area, and that is a function of not only force, but catheter pose and material elasticity as well.

In this experiment, we have employed real-time ultrasound (US) imaging to determine the complete interaction between the ablation electrode and tissue to accurately estimate contact, which will help to better understand the effect of catheter pose and position relative to the tissue. By simultaneously recording tracked position, force reading and US image of the ablation catheter, the differing material properties of polyvinyl alcohol cryogel^[1] phantoms are shown to produce varying amounts of tissue depression and contact area (implying varying lesion quality) for equivalent force readings. We have shown that the elastic modulus significantly affects the surface-contact area between the catheter and tissue at any level of contact force. Thus we provide evidence that a prescribed level of catheter force may not always provide sufficient contact area to produce an effective ablation lesion in the prescribed ablation time.

Many technical innovations in multimodal radiologic imaging and bronchoscopy have emerged recently in the effort against lung cancer. Modern X-ray computed-tomography (CT) scanners provide three-dimensional (3D) high-resolution chest images, positron emission tomography (PET) scanners give complementary molecular imaging data, and new integrated PET/CT scanners combine the strengths of both modalities. State-of-the-art bronchoscopes permit minimally invasive tissue sampling, with vivid endobronchial video enabling navigation deep into the airway-tree periphery, while complementary endobronchial ultrasound (EBUS) reveals local views of anatomical structures outside the airways. In addition, image-guided intervention (IGI) systems have proven their utility for CT-based planning and guidance of bronchoscopy. Unfortunately, no IGI system exists that integrates all sources effectively through the complete lung-cancer staging work flow. This paper presents a prototype of a computer-based multimodal IGI system that strives to fill this need. The system combines a wide range of automatic and semi-automatic image-processing tools for multimodal data fusion and procedure planning. It also provides a flexible graphical user interface for follow-on guidance of bronchoscopy/EBUS. Human-study results demonstrate the system’s potential.

We present an analysis of the registration component of a proposed image guidance system for image guided liver surgery, using contrast enhanced CT. The analysis is performed on a visually realistic liver phantom and in-vivo porcine data. A robust registration process that can be deployed clinically is a key component of any image guided surgery system. It is also essential that the accuracy of the registration can be quantified and communicated to the surgeon. We summarise the proposed guidance system and discuss its clinical feasibility. The registration combines an intuitive manual alignment stage, surface reconstruction from a tracked stereo laparoscope and a rigid iterative closest point registration to register the intra-operative liver surface to the liver surface derived from CT. Testing of the system on a liver phantom shows that subsurface landmarks can be localised to an accuracy of 2.9 mm RMS. Testing during five porcine liver surgeries demonstrated that registration can be performed during surgery, with an error of less than 10 mm RMS for multiple surface landmarks.

We present a system for registering the coordinate frame of an endoscope to pre- or intra- operatively acquired CT data based on optimizing the similarity metric between an endoscopic image and an image predicted via rendering of CT. Our method is robust and semi-automatic because it takes account of physical constraints, specifically, collisions between the endoscope and the anatomy, to initialize and constrain the search. The proposed optimization method is based on a stochastic optimization algorithm that evaluates a large number of similarity metric functions in parallel on a graphics processing unit. Images from a cadaver and a patient were used for evaluation. The registration error was 0.83 mm and 1.97 mm for cadaver and patient images respectively. The average registration time for 60 trials was 4.4 seconds. The patient study demonstrated robustness of the proposed algorithm against a moderate anatomical deformation.

One of the most complex and difficult tasks for surgeons during minimally invasive interventions is suturing. A prerequisite to assist the suturing process is the tracking of the needle. The endoscopic images provide a rich source of information which can be used for needle tracking. In this paper, we present an image-based method for markerless needle tracking. The method uses a color-based and geometry-based segmentation to detect the needle. Once an initial needle detection is obtained, a region of interest enclosing the extracted needle contour is passed on to a reduced segmentation. It is evaluated with in vivo images from da Vinci interventions.

The challenge is to accurately guide the surgical tool within the three-dimensional (3D) surgical field for roboticallyassisted operations such as tumor margin removal from a debulked brain tumor cavity. The proposed technique is 3D image-guided surgical navigation based on matching intraoperative video frames to a 3D virtual model of the surgical field. A small laser-scanning endoscopic camera was attached to a mock minimally-invasive surgical tool that was manipulated toward a region of interest (residual tumor) within a phantom of a debulked brain tumor. Video frames from the endoscope provided features that were matched to the 3D virtual model, which were reconstructed earlier by raster scanning over the surgical field. Camera pose (position and orientation) is recovered by implementing a constrained bundle adjustment algorithm. Navigational error during the approach to fluorescence target (residual tumor) is determined by comparing the calculated camera pose to the measured camera pose using a micro-positioning stage. From these preliminary results, computation efficiency of the algorithm in MATLAB code is near real-time (2.5 sec for each estimation of pose), which can be improved by implementation in C++. Error analysis produced 3-mm distance error and 2.5 degree of orientation error on average. The sources of these errors come from 1) inaccuracy of the 3D virtual model, generated on a calibrated RAVEN robotic platform with stereo tracking; 2) inaccuracy of endoscope intrinsic parameters, such as focal length; and 3) any endoscopic image distortion from scanning irregularities. This work demonstrates feasibility of micro-camera 3D guidance of a robotic surgical tool.

Positive surgical margins (PSMs) found following prostate cancer surgery are a significant risk factor for post-operative disease recurrence. Noxious adjuvant radiation and chemical-based therapies are typically offered to men with PSMs. Unfortunately, no real-time intraoperative technology is currently available to guide surgeons to regions of suspicion during the initial prostatectomy where immediate surgical excisions could be used to reduce the chance of PSMs. A microendoscopic electrical impedance sensing probe was developed with the intention of providing real-time feedback regarding margin status to surgeons during robot-assisted laparoscopic prostatectomy (RALP) procedures. A radially configured 17-electrode microendoscopic probe was designed, constructed, and initially evaluated through use of gelatin-based phantoms and an ex vivo human prostate specimen. Impedance measurements are recorded at 10 frequencies (10 kHz - 100 kHz) using a high-speed FPGA-based electrical impedance tomography (EIT) system. Tetrapolar impedances are recorded from a number of different electrode configurations strategically chosen to sense tissue in a pre-defined sector underlying the probe face. A circular electrical impedance map (EIM) with several color-coded pie-shaped sectors is created to represent the impedance values of the probed tissue. Gelatin phantom experiments show an obvious distinction in the impedance maps between high and low impedance regions. Similarly, the EIM generated from the ex vivo prostate case shows distinguishing features between cancerous and benign regions. Based on successful development of this probe and these promising initial results, EIMs of additional prostate specimens are being collected to further evaluate this approach for intraoperative surgical margin assessment during RALP procedures.

Accurate and reliable identification of thalamic nuclei is important for surgical interventions and neuroanatomical studies. This is a challenging task due to their small sizes and low intra-thalamic contrast in standard T1-weighted or T2- weighted images. Previously proposed techniques rely on diffusion imaging or functional imaging. These require additional scanning and suffer from the low resolution and signal-to-noise ratio in these images. In this paper, we aim to directly segment the thalamic nuclei in standard 3T T1-weighted images using shape models. We manually delineate the structures in high-field MR images and build high resolution shape models from a group of subjects. We then investigate if the nuclei locations can be inferred from the whole thalamus. To do this, we hierarchically fit joint models. We start from the entire thalamus and fit a model that captures the relation between the thalamus and large nuclei groups. This allows us to infer the boundaries of these nuclei groups and we repeat the process until all nuclei are segmented. We validate our method in a leave-one-out fashion with seven subjects by comparing the shape-based segmentations on 3T images to the manual contours. Results we have obtained for major nuclei (dice coefficients ranging from 0.57 to 0.88 and mean surface errors from 0.29mm to 0.72mm) suggest the feasibility of using such joint shape models for localization. This may have a direct impact on surgeries such as Deep Brain Stimulation procedures that require the implantation of stimulating electrodes in specific thalamic nuclei.

In interventional radiology, navigating devices under the sole guidance of fluoroscopic images inside a complex architecture of tortuous and narrow vessels like the cerebral vascular tree is a difficult task. Visualizing the device in 3D could facilitate this navigation. For curvilinear devices such as guide-wires and catheters, a 3D reconstruction may be achieved using two simultaneous fluoroscopic views, as available on a biplane acquisition system. The purpose of this paper is to present a new automatic three-dimensional curve reconstruction method that has the potential to reconstruct complex 3D curves and does not require a perfect segmentation of the endovascular device. Using epipolar geometry, our algorithm translates the point correspondence problem into a segment correspondence problem. Candidate 3D curves can be formed and evaluated independently after identifying all possible combinations of compatible 3D segments. Correspondence is then inherently solved by looking in 3D space for the most coherent curve in terms of continuity and curvature. This problem can be cast into a graph problem where the most coherent curve corresponds to the shortest path of a weighted graph. We present quantitative results of curve reconstructions performed from numerically simulated projections of tortuous 3D curves extracted from cerebral vascular trees affected with brain arteriovenous malformations as well as fluoroscopic image pairs of a guide-wire from both phantom and clinical sets. Our method was able to select the correct 3D segments in 97.5% of simulated cases thus demonstrating its ability to handle complex 3D curves and can deal with imperfect 2D segmentation.

In epilepsy cases, subdural electrodes are often implanted to acquire intracranial EEG (iEEG) for seizure localization and resection planning. However, the electrodes may shift significantly between implantation and resection, during the time that the patient is monitored for iEEG recording. As a result, the accuracy of surgical planning based on electrode locations at the time of resection can be compromised. Previous studies have only quantified the electrode shift with respect to the skull, but not with respect to the cortical surface, because tracking cortical shift between surgeries is challenging. In this study, we use an intraoperative stereovision (iSV) system to visualize and localize the cortical surface as well as electrodes, record three-dimensional (3D) locations of the electrodes in MR space at the time of implantation and resection, respectively, and quantify the raw displacements, i.e., with respect to the skull. Furthermore, we track the cortical surface and quantify the shift between surgeries using an optical flow (OF) based motion-tracking algorithm. Finally, we compute the electrode shift with respect to the cortical surface by subtracting the cortical shift from raw measured displacements. We illustrate the method using one patient example. In this particular patient case, the results show that the electrodes not only shifted significantly with respect to the skull (8.79 ± 3.00 mm in the lateral direction, ranging from 2.88 mm to 12.87 mm), but also with respect to the cortical surface (7.20 ± 3.58 mm), whereas the cortical surface did not shift significantly in the lateral direction between surgeries (2.23 ± 0.76 mm).

Providing suitable training for aspiring neurosurgeons is becoming more and more problematic. The increasing popularity of the endovascular treatment of intracranial aneurysms leads to a lack of simple surgical situations for clipping operations, leaving mainly the complex cases, which present even experienced surgeons with a challenge. To alleviate this situation, we have developed a training simulator with haptic interaction allowing trainees to practice virtual clipping surgeries on real patient-specific vessel geometries. By using specialized finite element (FEM) algorithms (fast finite element method, matrix condensation) combined with GPU acceleration, we can achieve the necessary frame rate for smooth real-time interaction with the detailed models needed for a realistic simulation of the vessel wall deformation caused by the clamping with surgical clips. Vessel wall geometries for typical training scenarios were obtained from 3D-reconstructed medical image data, while for the instruments (clipping forceps, various types of clips, suction tubes) we use models provided by manufacturer Aesculap AG. Collisions between vessel and instruments have to be continuously detected and transformed into corresponding boundary conditions and feedback forces, calculated using a contact plane method. After a training, the achieved result can be assessed based on various criteria, including a simulation of the residual blood flow into the aneurysm. Rigid models of the surgical access and surrounding brain tissue, plus coupling a real forceps to the haptic input device further increase the realism of the simulation.

This paper presents a methodology for the digital formatting of a printed atlas of the brainstem and the delineation of cranial nerves from this digital atlas. It also describes on-going work on the 3D resampling and refinement of the 2D functional regions and nerve contours. In MRI-based anatomical modeling for neurosurgery planning and simulation, the complexity of the functional anatomy entails a digital atlas approach, rather than less descriptive voxel or surface-based approaches. However, there is an insufficiency of descriptive digital atlases, in particular of the brainstem. Our approach proceeds from a series of numbered, contour-based sketches coinciding with slices of the brainstem featuring both closed and open contours. The closed contours coincide with functionally relevant regions, whereby our objective is to fill in each corresponding label, which is analogous to painting numbered regions in a paint-by-numbers kit. Any open contour typically coincides with a cranial nerve. This 2D phase is needed in order to produce densely labeled regions that can be stacked to produce 3D regions, as well as identifying the embedded paths and outer attachment points of cranial nerves. Cranial nerves are modeled using an explicit contour based technique called 1-Simplex. The relevance of cranial nerves modeling of this project is two-fold: i) this atlas will fill a void left by the brain segmentation communities, as no suitable digital atlas of the brainstem exists, and ii) this atlas is necessary to make explicit the attachment points of major nerves (except I and II) having a cranial origin. Keywords: digital atlas, contour models, surface models

Recent research has revealed that incision planning in laser surgery deploying stylus and tablet outperforms state-of-the-art micro-manipulator-based laser control. Providing more detailed quantitation regarding that approach, a comparative study of six tablet-based strategies for laser path planning is presented. Reference strategy is defined by monoscopic visualization and continuous path drawing on a graphics tablet. Further concepts deploying stereoscopic or a synthesized laser view, point-based path definition, real-time teleoperation or a pen display are compared with the reference scenario. Volunteers were asked to redraw and ablate stamped lines on a sample. Performance is assessed by measuring planning accuracy, completion time and ease of use. Results demonstrate that significant differences exist between proposed concepts. The reference strategy provides more accurate incision planning than the stereo or laser view scenario. Real-time teleoperation performs best with respect to completion time without indicating any significant deviation in accuracy and usability. Point-based planning as well as the pen display provide most accurate planning and increased ease of use compared to the reference strategy. As a result, combining the pen display approach with point-based planning has potential to become a powerful strategy because of benefiting from improved hand-eye-coordination on the one hand and from a simple but accurate technique for path definition on the other hand. These findings as well as the overall usability scale indicating high acceptance and consistence of proposed strategies motivate further advanced tablet-based planning in laser microsurgery.

Cochlear implants (CIs) are neural prosthetics that stimulate the auditory nerve pathways within the cochlea using an implanted electrode array to restore hearing. After implantation, the CI is programmed by an audiologist who determines which electrodes are active, i.e., the electrode configuration, and selects other stimulation settings. Recent clinical studies by our group have shown that hearing outcomes can be significantly improved by using an image-guided electrode configuration selection technique we have designed. Our goal in this work is to automate the electrode configuration selection step with the long term goal of developing a fully automatic system that can be translated to the clinic. Until now, the electrode configuration selection step has been performed by an expert with the assistance of image analysis-based estimates of the electrode-neural interface. To automatically determine the electrode configuration, we have designed an optimization approach and propose the use of a cost function with feature terms designed to interpret the image analysis data in a similar fashion as the expert. Further, we have designed an approach to select parameters in the cost function using our database of existing electrode configuration plans as training data. The results we present show that our automatic approach results in electrode configurations that are better or equally as good as manually selected configurations in over 80% of the cases tested. This method represents a crucial step towards clinical translation of our image-guided cochlear implant programming system.

Catheter path reconstruction is a necessary step in many clinical procedures, such as cardiovascular interventions and high-dose-rate brachytherapy. To overcome limitations of standard imaging modalities, electromagnetic tracking has been employed to reconstruct catheter paths. However, tracking errors pose a challenge in accurate path reconstructions. We address this challenge by means of a filtering technique incorporating the electromagnetic measurements with the nonholonomic motion constraints of the sensor inside a catheter. The nonholonomic motion model of the sensor within the catheter and the electromagnetic measurement data were integrated using an extended Kalman filter. The performance of our proposed approach was experimentally evaluated using the Ascension’s 3D Guidance trakStar electromagnetic tracker. Sensor measurements were recorded during insertions of an electromagnetic sensor (model 55) along ten predefined ground truth paths. Our method was implemented in MATLAB and applied to the measurement data. Our reconstruction results were compared to raw measurements as well as filtered measurements provided by the manufacturer. The mean of the root-mean-square (RMS) errors along the ten paths was 3.7 mm for the raw measurements, and 3.3 mm with manufacturer’s filters. Our approach effectively reduced the mean RMS error to 2.7 mm. Compared to other filtering methods, our approach successfully improved the path reconstruction accuracy by exploiting the sensor’s nonholonomic motion constraints in its formulation. Our approach seems promising for a variety of clinical procedures involving reconstruction of a catheter path.

This work proposes new methods for a microstereotactic frame based on bone cement fixation. Microstereotactic frames are under investigation for minimal invasive temporal bone surgery, e.g. cochlear implantation, or for deep brain stimulation, where products are already on the market. The correct pose of the microstereotactic frame is either adjusted outside or inside the operating room and the frame is used for e.g. drill or electrode guidance. We present a patientspecific, disposable frame that allows intraoperative, sterile pose-setting. Key idea of our approach is bone cement between two plates that cures while the plates are positioned with a mechatronics system in the desired pose. This paper includes new designs of microstereotactic frames, a system for alignment and first measurements to analyze accuracy and applicable load.

Brachytherapy and thermal ablation are relatively new approaches in robot-assisted minimally invasive interventions for treating malignant tumors. Ultrasound remains the most favored choice for imaging feedback, the benefits being cost effectiveness, radiation free, and easy access in an OR. However it does not generally provide high contrast, noise free images. Distortion occurs when the sound waves pass through a medium that contains air and/or when the target organ is deep within the body. The distorted images make it quite difficult to recognize and localize tumors and surgical tools. Often tools, such as a bevel-tipped needle, deflect from its path during insertion, making it difficult to detect the needle tip using a single perspective view. The shifting of the target due to cardiac and/or respiratory motion can add further errors in reaching the target. This paper describes a comprehensive system that uses robot dexterity to capture 2D ultrasound images in various pre-determined modes for generating 3D ultrasound images and assists in maneuvering a surgical tool. An interactive 3D virtual reality environment is developed that visualizes various artifacts present in the surgical site in real-time. The system helps to avoid image distortion by grabbing images from multiple positions and orientation to provide a 3D view. Using the methods developed for this application, an accuracy of 1.3 mm was achieved in target attainment in an in-vivo experiment subjected to tissue motion. An accuracy of 1.36 mm and 0.93 mm respectively was achieved for the ex-vivo experiments with and without external induced motion. An ablation monitor widget that visualizes the changes during the complete ablation process and enables evaluation of the process in its entirety is integrated.

Each year in the U.S., 7.4 million surgical procedures involving the major vessels are performed. Many of our patients require multiple surgeries, and many of the procedures include “surgical exploration”. Procedures of this kind come with a significant amount of risk, carrying up to a 17.4% predicted mortality rate. This is especially concerning for our target population of pediatric patients with congenital abnormalities of the heart and major pulmonary vessels. This paper offers a novel approach to surgical planning which includes studying virtual and physical models of pulmonary vasculature of an individual patient before operation obtained from conventional 3D X-ray computed tomography (CT) scans of the chest. These models would provide clinicians with a non-invasive, intricately detailed representation of patient anatomy, and could reduce the need for invasive planning procedures such as exploratory surgery. Researchers involved in the AirPROM project have already demonstrated the utility of virtual and physical models in treatment planning of the airways of the chest. Clinicians have acknowledged the potential benefit from such a technology. A method for creating patient-derived physical models is demonstrated on pulmonary vasculature extracted from a CT scan with contrast of an adult human. Using a modified version of the NIH ImageJ program, a series of image processing functions are used to extract and mathematically reconstruct the vasculature tree structures of interest. An auto-generated STL file is sent to a 3D printer to create a physical model of the major pulmonary vasculature generated from 3D CT scans of patients.

Percutaneous treatment of scaphoid fractures has found increasing interest in recent years as it promises to minimize soft-tissue damage, and minimizes the risk of infections and the loss of the joint stability. However, as this procedure is mostly performed on 2D fluoroscopic images, the accurate localization of the scaphoid bone for fracture fixation renders extremely challenging. In this work, we thus propose the integration of a statistical wrist model with 3D intraoperative ultrasound for accurate localization of the scaphoid bone. We utilize a previously developed statistical wrist model and register it to bone surfaces in ultrasound images using a probabilistic approach that involves expectation-maximization. We utilize local phase symmetry to detect features in noisy ultrasound images; in addition, we use shadow information in ultrasound images to enhance and set apart bone from other features. Feasibility experiments are performed by registering the wrist model to 3D ultrasound volumes of two different wrists at two different wrist positions. And the result indicates a potential of the proposed technique for localization of the scaphoid bone in ultrasound images.

A robust atlas-to-subject registration using a statistical deformation model (SDM) is presented. The SDM uses statistics of voxel-wise displacement learned from pre-computed deformation vectors of a training dataset. This allows an atlas instance to be directly translated into an intensity volume and compared with a patient’s intensity volume. Rigid and nonrigid transformation parameters were simultaneously optimized via the Covariance Matrix Adaptation – Evolutionary Strategy (CMA-ES), with image similarity used as the objective function. The algorithm was tested on CT volumes of the pelvis from 55 female subjects. A performance comparison of the CMA-ES and Nelder-Mead downhill simplex optimization algorithms with the mutual information and normalized cross correlation similarity metrics was conducted. Simulation studies using synthetic subjects were performed, as well as leave-one-out cross validation studies. Both studies suggested that mutual information and CMA-ES achieved the best performance. The leave-one-out test demonstrated 4.13 mm error with respect to the true displacement field, and 26,102 function evaluations in 180 seconds, on average.

Robots have been shown to be useful in assisting surgeons in a variety of bone drilling and milling procedures. Examples include commercial systems for joint repair or replacement surgeries, with in vitro feasibility recently shown for mastoidectomy. Typically, the robot is guided along a path planned on a CT image that has been registered to the physical anatomy in the operating room, which is in turn registered to the robot. The registrations often take advantage of the high accuracy of fiducial registration, but, because no real-world registration is perfect, the drill guided by the robot will inevitably deviate from its planned path. The extent of the deviation can vary from point to point along the path because of the spatial variation of target registration error. The allowable deviation can also vary spatially based on the necessary safety margin between the drill tip and various nearby anatomical structures along the path. Knowledge of the expected spatial distribution of registration error can be obtained from theoretical models or experimental measurements and used to modify the planned path. The objective of such modifications is to achieve desired probabilities for sparing specified structures. This approach has previously been studied for drilling straight holes but has not yet been generalized to milling procedures, such as mastoidectomy, in which cavities of more general shapes must be created. In this work, we present a general method for altering any path to achieve specified probabilities for any spatial arrangement of structures to be protected. We validate the method via numerical simulations in the context of mastoidectomy.

Physics based deformable registration can provide physically consistent image match of deformable soft tissues. In order to help radiologist/surgeons to determine the status of malicious tumors, we often need to accurately align the regions with embedded tumors. This is a very challenging task since the tumor and the surrounding tissues have very different tissue properties such as stiffness and elasticity. In order to address this problem, based on minimum strain energy principle in elasticity theory, we propose to partition the whole region of interest into smaller sub-regions and dynamically adjust weights of vessel segments and bifurcation points in each sub-region in the registration objective function. Our previously proposed fast vessel registration is used as a component in the inner loop. We have validated the proposed method using liver MR images from human subjects. The results show that our method can detect the large registration errors and improve the registration accuracy in the neighborhood of the tumors and guarantee the registration errors to be within acceptable accuracy. The proposed technique has the potential to significantly improve the registration capability and the quality of clinical diagnosis and treatment planning.

Soft tissue deformation represents a significant error source in current surgical navigation systems used for open hepatic procedures. While numerous algorithms have been proposed to rectify the tissue deformation that is encountered during open liver surgery, clinical validation of the proposed methods has been limited to surface based metrics and sub-surface validation has largely been performed via phantom experiments. Tracked intraoperative ultrasound (iUS) provides a means to digitize sub-surface anatomical landmarks during clinical procedures. The proposed method involves the validation of a deformation correction algorithm for open hepatic image-guided surgery systems via sub-surface targets digitized with tracked iUS. Intraoperative surface digitizations were acquired via a laser range scanner and an optically tracked stylus for the purposes of computing the physical-to-image space registration within the guidance system and for use in retrospective deformation correction. Upon completion of surface digitization, the organ was interrogated with a tracked iUS transducer where the iUS images and corresponding tracked locations were recorded. After the procedure, the clinician reviewed the iUS images to delineate contours of anatomical target features for use in the validation procedure. Mean closest point distances between the feature contours delineated in the iUS images and corresponding 3-D anatomical model generated from the preoperative tomograms were computed to quantify the extent to which the deformation correction algorithm improved registration accuracy. The preliminary results for two patients indicate that the deformation correction method resulted in a reduction in target error of approximately 50%.

This paper presents a new needle detection technique for ultrasound guided interventions based on the spectral properties of small displacements arising from hand tremour or intentional motion. In a block-based approach, the displacement map is computed for each block of interest versus a reference frame, using an optical flow technique. To compute the flow parameters, the Lucas-Kanade approach is used in a multiresolution and regularized form. A least-squares fit is used to estimate the flow parameters from the overdetermined system of spatial and temporal gradients. Lateral and axial components of the displacement are obtained for each block of interest at consecutive frames. Magnitude-squared spectral coherency is derived between the median displacements of the reference block and each block of interest, to determine the spectral correlation. In vivo images were obtained from the tissue near the abdominal aorta to capture the extreme intrinsic body motion and insertion images were captured from a tissue-mimicking agar phantom. According to the analysis, both the involuntary and intentional movement of the needle produces coherent displacement with respect to a reference window near the insertion site. Intrinsic body motion also produces coherent displacement with respect to a reference window in the tissue; however, the coherency spectra of intrinsic and needle motion are distinguishable spectrally. Blocks with high spectral coherency at high frequencies are selected, estimating a channel for needle trajectory. The needle trajectory is detected from locally thresholded absolute displacement map within the initial estimate. Experimental results show the RMS localization accuracy of 1:0 mm, 0:7 mm, and 0:5 mm for hand tremour, vibrational and rotational needle movements, respectively.

Image-guided surgery systems are often used to provide surgeons with informational support. Due to several unique advantages such as ease of use, real-time image acquisition, and no ionizing radiation, ultrasound is a common intraoperative medical imaging modality used in image-guided surgery systems. To perform advanced forms of guidance with ultrasound, such as virtual image overlays or automated robotic actuation, an ultrasound calibration process must be performed. This process recovers the rigid body transformation between a tracked marker attached to the transducer and the ultrasound image. Point-based phantoms are considered to be accurate, but their calibration framework assumes that the point is in the image plane. In this work, we present the use of an active point phantom and a calibration framework that accounts for the elevational uncertainty of the point. Given the lateral and axial position of the point in the ultrasound image, we approximate a circle in the axial-elevational plane with a radius equal to the axial position. The standard approach transforms all of the imaged points to be a single physical point. In our approach, we minimize the distances between the circular subsets of each image, with them ideally intersecting at a single point. We simulated in noiseless and noisy cases, presenting results on out-of-plane estimation errors, calibration estimation errors, and point reconstruction precision. We also performed an experiment using a robot arm as the tracker, resulting in a point reconstruction precision of 0.64mm.

Accurate needle placement is essential in percutaneous procedures such as radiofrequency ablation (RFA) of liver tumors. Use of real-time navigation of an interventional needle can improve targeting accuracy and yield precise measurements of the needle tip inside the body. An emerging technology based on Fiber Bragg Grating (FBG) sensors has demonstrated the potential of estimating shapes at high frequencies (up to 20 kHz), fast enough for real-time applications. In this paper, we present a calibration procedure for this novel needle tracking technology using strain measurements obtained from fiber Bragg gratings (FBGs). Three glass fibers equipped with two FBGs each were incorporated into a 19G needle. The 3D needle shape is reconstructed based on a polynomial fitting of strain measurements obtained from the fibers. The real-time information provided by the needle tip position and shape allows tracking of the needle deflections during tissue insertion. An experimental setup was designed to yield a calibration that is insensitive to ambient temperature fluctuations and robust to slight external disturbances. We compare the shape of the 3D reconstructed needle to measurements obtained from camera images, as well as assess needle tip tracking accuracy on a ground-truth phantom. Initial results show that the tracking errors for the needle tip are under 1mm, while 3D shape deflections are minimal near the needle tip. The accuracy is appropriate for applications such as RFA of liver tumors.

Tracking technology is often necessary for image-guided surgical interventions. Optical tracking is one the options, but it suffers from line of sight and workspace limitations. Optical tracking is accomplished by attaching a rigid body marker, having a pattern for pose detection, onto a tool or device. A larger rigid body results in more accurate tracking, but at the same time large size limits its usage in a crowded surgical workspace. This work presents a prototype of a novel optical tracking method using a virtual rigid body (VRB). We define the VRB as a 3D rigid body marker in the form of pattern on a surface generated from a light source. Its pose can be recovered by observing the projected pattern with a stereo-camera system. The rigid body's size is no longer physically limited as we can manufacture small size light sources. Conventional optical tracking also requires line of sight to the rigid body. VRB overcomes these limitations by detecting a pattern projected onto the surface. We can project the pattern onto a region of interest, allowing the pattern to always be in the view of the optical tracker. This helps to decrease the occurrence of occlusions. This manuscript describes the method and results compared with conventional optical tracking in an experiment setup using known motions. The experiments are done using an optical tracker and a linear-stage, resulting in targeting errors of 0.38mm±0.28mm with our method compared to 0.23mm±0.22mm with conventional optical markers. Another experiment that replaced the linear stage with a robot arm resulted in rotational errors of 0.50±0.31° and 2.68±2.20° and the translation errors of 0.18±0.10 mm and 0.03±0.02 mm respectively.

We present a fiber optical shape sensing system that allows to track the shape of a standard telecom fiber with fiber Bragg grating. The shape sensing information is combined with a medical visualization platform to visualize the shape sensing information together with medical images and post-processing results like 3D models, vessel graphs, or segmentation results. The framework has a modular nature to use it for various medical applications like catheter or needle based interventions. The technology has potential in the medical area as it is MR-compatible and can easily be integrated in catheters and needles due to its small size.

A method is developed to build patient-specific motion models based on 4DCBCT images taken at treatment time and use them to generate 3D time-varying images (referred to as 3D fluoroscopic images). Motion models are built by applying Principal Component Analysis (PCA) on the displacement vector fields (DVFs) estimated by performing deformable image registration on each phase of 4DCBCT relative to a reference phase. The resulting PCA coefficients are optimized iteratively by comparing 2D projections captured at treatment time with projections estimated using the motion model. The optimized coefficients are used to generate 3D fluoroscopic images. The method is evaluated using anthropomorphic physical and digital phantoms reproducing real patient trajectories. For physical phantom datasets, the average tumor localization error (TLE) and (95th percentile) in two datasets were 0.95 (2.2) mm. For digital phantoms assuming superior image quality of 4DCT and no anatomic or positioning disparities between 4DCT and treatment time, the average TLE and the image intensity error (IIE) in six datasets were smaller using 4DCT-based motion models. When simulating positioning disparities and tumor baseline shifts at treatment time compared to planning 4DCT, the average TLE (95th percentile) and IIE were 4.2 (5.4) mm and 0.15 using 4DCT-based models, while they were 1.2 (2.2) mm and 0.10 using 4DCBCT-based ones, respectively. 4DCBCT-based models were shown to perform better when there are positioning and tumor baseline shift uncertainties at treatment time. Thus, generating 3D fluoroscopic images based on 4DCBCT-based motion models can capture both inter- and intra- fraction anatomical changes during treatment.

Visual tracking of surgical instruments is an essential part of eye surgery, and plays an important role for the surgeons as well as it is a key component of robotics assistance during the operation time. The difficulty of detecting and tracking medical instruments in-vivo images comes from its deformable shape, changes in brightness, and the presence of the instrument shadow. This paper introduces a new approach to detect the tip of surgical tool and its width regardless of its head shape and the presence of the shadows or vessels. The approach relies on integrating structural information about the strong edges from the RGB color model, and the tool location-based information from L*a*b color model. The probabilistic Hough transform was applied to get the strongest straight lines in the RGB-images, and based on information from the L* and a*, one of these candidates lines is selected as the edge of the tool shaft. Based on that line, the tool slope, the tool centerline and the tool tip could be detected. The tracking is performed by keeping track of the last detected tool tip and the tool slope, and filtering the Hough lines within a box around the last detected tool tip based on the slope differences. Experimental results demonstrate the high accuracy achieved in term of detecting the tool tip position, the tool joint point position, and the tool centerline. The approach also meets the real time requirements.

Lung Cancer is the leading cause of cancer death in both men and women. Among various treatment methods currently being used in the clinic, External Beam Radiation Therapy (EBRT) is used widely not only as the primary treatment method, but also in combination with chemotherapy and surgery. However, this method may lack desirable dosimetric accuracy because of respiration induced tumor motion. Recently, biomechanical modeling of the respiratory system has become a popular approach for tumor motion prediction and compensation. This approach requires reasonably accurate data pertaining to thoracic pressure variation, diaphragm position and biomechanical properties of the lung tissue in order to predict the lung tissue deformation and tumor motion. In this paper, we present preliminary results of an in vivo study obtained from a Finite Element Model (FEM) of the lung developed to predict tumor motion during respiration.

Anecdotally, surgeons sometimes observe large errors when using image guidance in endonasal surgery. We hypothesize that one contributing factor is the possibility that operating room personnel might accidentally bump the optically tracked rigid body attached to the patient after registration has been performed. In this paper we explore the registration error at the skull base that can be induced by simulated bumping of the rigid body, and find that large errors can occur when simulated bumps are applied to the rigid body. To address this, we propose a new fixation method for the rigid body based on granular jamming (i.e. using particles like ground coffee). Our results show that our granular jamming fixation prototype reduces registration error by 28%-68% (depending on bump direction) in comparison to a standard Brainlab reference headband.

This paper proposes an automatic method for vertebra localization, labeling, and segmentation in multi-slice Magnetic Resonance (MR) images. Prior work in this area on MR images mostly requires user interaction while our method is fully automatic. Cubic intensity-based features are extracted from image voxels. A deep learning approach is used for simultaneous localization and identification of vertebrae. The localized points are refined by local thresholding in the region of the detected vertebral column. Thereafter, a statistical multi-vertebrae model is initialized on the localized vertebrae. An iterative Expectation Maximization technique is used to register the vertebral body of the model to the image edges and obtain a segmentation of the lumbar vertebral bodies. The method is evaluated by applying to nine volumetric MR images of the spine. The results demonstrate 100% vertebra identification and a mean surface error of below 2.8 mm for 3D segmentation. Computation time is less than three minutes per high-resolution volumetric image.

Defining therapeutic protocols in asthma and monitoring patient response require a more in-depth knowledge on the disease severity and treatment outcome based on quantitative indicators. This paper aims at grading severity in asthma based on objective morphological measurements obtained in automated fashion from 3-D multi-slice computed tomography (MSCT) image datasets. These measures attempt to capture and quantify the airway remodeling process involved in asthma, both at the level of the airway wall thickness and airway lumen. Two morphological changes are thus targeted here, (1) the airway wall thickening measured as a global index characterizing the increase of wall thickness above a normal value of wall-to-lumen-radius ratio, and (2) the bronchoarterial ratio index assessed globally from numerous locations in the lungs. The combination of these indices provides a grading of the severity of the remodeling process in asthma which correlates with the known phenotype of the patients investigated. Preliminary application to assess the patient response in thermoplasty trials is also considered from the point of view of the defined indices.

Tracked ultrasound is a safe alternative to X-ray for imaging bones. The interpretation of bony structures is challenging as ultrasound has no specific intensity characteristic of bones. Several image segmentation algorithms have been devised to identify bony structures. We propose an open-source framework that would aid in the development and comparison of such algorithms by quantitatively measuring segmentation performance in the ultrasound images. True-positive and false-negative metrics used in the framework quantify algorithm performance based on correctly segmented bone and correctly segmented boneless regions. Ground-truth for these metrics are defined manually and along with the corresponding automatically segmented image are used for the performance analysis. Manually created ground truth tests were generated to verify the accuracy of the analysis. Further evaluation metrics for determining average performance per slide and standard deviation are considered. The metrics provide a means of evaluating accuracy of frames along the length of a volume. This would aid in assessing the accuracy of the volume itself and the approach to image acquisition (positioning and frequency of frame). The framework was implemented as an open-source module of the 3D Slicer platform. The ground truth tests verified that the framework correctly calculates the implemented metrics. The developed framework provides a convenient way to evaluate bone segmentation algorithms. The implementation fits in a widely used application for segmentation algorithm prototyping. Future algorithm development will benefit by monitoring the effects of adjustments to an algorithm in a standard evaluation framework.

The cortical surface of the human brain consists of a large number of folds forming valleys and ridges, the gyri and sulci. Often, it is desirable to perform a segmentation of a brain image into these underlying structures in order to assess parameters relative to these functional components. Typical examples for this include measurements of cortical thickness for individual functional areas, or the correlation of functional areas derived from fMRI data to corresponding anatomical areas seen in structural imaging. In this paper, we present a novel interactive technique, that allows for fast and intuitive segmentation of these functional areas from T1-weighted MR images of the brain. Our segmentation approach is based exclusively on morphological image processing operations, eliminating the requirement for explicit reconstruction of the brains surface.

With the rapid growth of positron emission tomography/computed tomography (PET/CT)-based medical applications, body-wide anatomy recognition on whole-body PET/CT images becomes crucial for quantifying body-wide disease burden. This, however, is a challenging problem and seldom studied due to unclear anatomy reference frame and low spatial resolution of PET images as well as low contrast and spatial resolution of the associated low-dose CT images. We previously developed an automatic anatomy recognition (AAR) system [15] whose applicability was demonstrated on diagnostic computed tomography (CT) and magnetic resonance (MR) images in different body regions on 35 objects. The aim of the present work is to investigate strategies for adapting the previous AAR system to low-dose CT and PET images toward automated body-wide disease quantification. Our adaptation of the previous AAR methodology to PET/CT images in this paper focuses on 16 objects in three body regions – thorax, abdomen, and pelvis – and consists of the following steps: collecting whole-body PET/CT images from existing patient image databases, delineating all objects in these images, modifying the previous hierarchical models built from diagnostic CT images to account for differences in appearance in low-dose CT and PET images, automatically locating objects in these images following object hierarchy, and evaluating performance. Our preliminary evaluations indicate that the performance of the AAR approach on low-dose CT images achieves object localization accuracy within about 2 voxels, which is comparable to the accuracies achieved on diagnostic contrast-enhanced CT images. Object recognition on low-dose CT images from PET/CT examinations without requiring diagnostic contrast-enhanced CT seems feasible.

Minimally invasive neuroendoscopic surgery provides an alternative to open craniotomy for many skull base lesions. These techniques provides a great benefit to the patient through shorter ICU stays, decreased post-operative pain and quicker return to baseline function. However, density of critical neurovascular structures at the skull base makes planning for these procedures highly complex. Furthermore, additional surgical portals are often used to improve visualization and instrument access, which adds to the complexity of pre-operative planning. Surgical approach planning is currently limited and typically involves review of 2D axial, coronal, and sagittal CT and MRI images. In addition, skull base surgeons manually change the visualization effect to review all possible approaches to the target lesion and achieve an optimal surgical plan. This cumbersome process relies heavily on surgeon experience and it does not allow for 3D visualization. In this paper, we describe a rapid pre-operative planning system for skull base surgery using the following two novel concepts: importance-based highlight and mobile portal. With this innovation, critical areas in the 3D CT model are highlighted based on segmentation results. Mobile portals allow surgeons to review multiple potential entry portals in real-time with improved visualization of critical structures located inside the pathway. To achieve this we used the following methods: (1) novel bone-only atlases were manually generated, (2) orbits and the center of the skull serve as features to quickly pre-align the patient’s scan with the atlas, (3) deformable registration technique was used for fine alignment, (4) surgical importance was assigned to each voxel according to a surgical dictionary, and (5) pre-defined transfer function was applied to the processed data to highlight important structures. The proposed idea was fully implemented as independent planning software and additional data are used for verification and validation. The experimental results show: (1) the proposed methods provided greatly improved planning efficiency while optimal surgical plans were successfully achieved, (2) the proposed methods successfully highlighted important structures and facilitated planning, (3) the proposed methods require shorter processing time than classical segmentation algorithms, and (4) these methods can be used to improve surgical safety for surgical robots.

An operating room is a stressful work environment. Nevertheless, all involved persons have to work safely as there is no space for mistakes. To ensure a high level of concentration and seamless interaction, all involved persons have to know their own tasks and the tasks of their colleagues. The entire team must work synchronously at all times. To optimize the overall workflow, a task manager supporting the team was developed. In parallel, a common conceptual design of a business process visualization was developed, which makes all relevant information accessible in real-time during a surgery. In this context an overview of all processes in the operating room was created and different concepts for the graphical representation of these user-dependent processes were developed. This paper describes the concept of the task manager as well as the general concept in the field of surgery.

Complete surgical removal of tumor tissue is essential for postoperative prognosis after surgery. Intraoperative tumor imaging and visualization are an important step in aiding surgeons to evaluate and resect tumor tissue in real time, thus enabling more complete resection of diseased tissue and better conservation of healthy tissue. As an emerging modality, hyperspectral imaging (HSI) holds great potential for comprehensive and objective intraoperative cancer assessment. In this paper, we explored the possibility of intraoperative tumor detection and visualization during surgery using HSI in the wavelength range of 450 nm - 900 nm in an animal experiment. We proposed a new algorithm for glare removal and cancer detection on surgical hyperspectral images, and detected the tumor margins in five mice with an average sensitivity and specificity of 94.4% and 98.3%, respectively. The hyperspectral imaging and quantification method have the potential to provide an innovative tool for image-guided surgery.

This work investigates combination of Optical Coherence Tomography and two cameras, observing a microscopic scene. Stereo vision provides realistic images, but is limited in terms of penetration depth. Optical Coherence Tomography (OCT) enables access to subcutaneous structures, but 3D-OCT volume data do not give the surgeon a familiar view. The extension of the stereo camera setup with OCT imaging combines the benefits of both modalities. In order to provide the surgeon with a convenient integration of OCT into the vision interface, we present an automated image processing analysis of OCT and stereo camera data as well as combined imaging as augmented reality visualization. Therefore, we care about OCT image noise, perform segmentation as well as develop proper registration objects and methods. The registration between stereo camera and OCT results in a Root Mean Square error of 284 μm as average of five measurements. The presented methods are fundamental for fusion of both imaging modalities. Augmented reality is shown as application of the results. Further developments lead to fused visualization of subcutaneous structures, as information of OCT images, into stereo vision.

Purpose: Robotic C-arm systems are capable of general noncircular orbits whose trajectories can be driven by the particular imaging task. However obtaining accurate calibrations for reconstruction in such geometries can be a challenging problem. This work proposes a method to perform a unique geometric calibration of an arbitrary C-arm orbit by registering 2D projections to a previously acquired 3D image to determine the transformation parameters representing the system geometry.

Methods: Experiments involved a cone-beam CT (CBCT) bench system, a robotic C-arm, and three phantoms. A robust 3D-2D registration process was used to compute the 9 degree of freedom (DOF) transformation between each projection and an existing 3D image by maximizing normalized gradient information with a digitally reconstructed radiograph (DRR) of the 3D volume. The quality of the resulting “self-calibration” was evaluated in terms of the agreement with an established calibration method using a BB phantom as well as image quality in the resulting CBCT reconstruction.

Results: The self-calibration yielded CBCT images without significant difference in spatial resolution from the standard (“true”) calibration methods (p-value >0.05 for all three phantoms), and the differences between CBCT images reconstructed using the “self” and “true” calibration methods were on the order of 10^-3 mm^-1. Maximum error in magnification was 3.2%, and back-projection ray placement was within 0.5 mm.

Conclusion: The proposed geometric “self” calibration provides a means for 3D imaging on general noncircular orbits in CBCT systems for which a geometric calibration is either not available or not reproducible. The method forms the basis of advanced “task-based” 3D imaging methods now in development for robotic C-arms.

Robot-assisted laparoscopic partial nephrectomies (RALPN) are performed to treat patients with locally confined renal carcinoma. There are well-documented benefits to performing partial (opposed to radical) kidney resections and to using robot-assisted laparoscopic (opposed to open) approaches. However, there are challenges in identifying tumor margins and critical benign structures including blood vessels and collecting systems during current RALPN procedures. The primary objective of this effort is to couple multiple image and data streams together to augment visual information currently provided to surgeons performing RALPN and ultimately ensure complete tumor resection and minimal damage to functional structures (i.e. renal vasculature and collecting systems). To meet this challenge we have developed a framework and performed initial feasibility experiments to couple pre-operative high-resolution anatomic images with intraoperative MRI, ultrasound (US) and optical-based surface mapping and kidney tracking. With these registered images and data streams, we aim to overlay the high-resolution contrast-enhanced anatomic (CT or MR) images onto the surgeon’s view screen for enhanced guidance. To date we have integrated the following components of our framework: 1) a method for tracking an intraoperative US probe to extract the kidney surface and a set of embedded kidney markers, 2) a method for co-registering intraoperative US scans with pre-operative MR scans, and 3) a method for deforming pre-op scans to match intraoperative scans. These components have been evaluated through phantom studies to demonstrate protocol feasibility.

Purpose. To extend the functionality of radiographic / fluoroscopic imaging systems already within standard spine surgery workflow to: 1) provide guidance of surgical device analogous to an external tracking system; and 2) provide intraoperative quality assurance (QA) of the surgical product.

Methods. Using fast, robust 3D-2D registration in combination with 3D models of known components (surgical devices), the 3D pose determination was solved to relate known components to 2D projection images and 3D preoperative CT in near-real-time. Exact and parametric models of the components were used as input to the algorithm to evaluate the effects of model fidelity. The proposed algorithm employs the covariance matrix adaptation evolution strategy (CMA-ES) to maximize gradient correlation (GC) between measured projections and simulated forward projections of components. Geometric accuracy was evaluated in a spine phantom in terms of target registration error at the tool tip (TRE_x), and angular deviation (TRE_Φ) from planned trajectory.

Results. Transpedicle surgical devices (probe tool and spine screws) were successfully guided with TRE_x<2 mm and TRE_Φ <0.5° given projection views separated by at least >30° (easily accommodated on a mobile C-arm). QA of the surgical product based on 3D-2D registration demonstrated the detection of pedicle screw breach with TRE_x<1 mm, demonstrating a trend of improved accuracy correlated to the fidelity of the component model employed.

Conclusions. 3D-2D registration combined with 3D models of known surgical components provides a novel method for near-real-time guidance and quality assurance using a mobile C-arm without external trackers or fiducial markers. Ongoing work includes determination of optimal views based on component shape and trajectory, improved robustness to anatomical deformation, and expanded preclinical testing in spine and intracranial surgeries.

Translation of any novel and existing 3D-2D image registration methods into clinical image-guidance systems is limited due to lack of their objective validation on clinical image datasets. The main reason is that, besides the calibration of the 2D imaging system, a reference or ”gold standard” registration is very difficult to obtain on clinical image datasets. In the context of cerebral endovascular image-guided interventions (EIGIs), we present a calibration device in the form of a headband with integrated fiducial markers and, secondly, propose an automated pipeline comprising 3D and 2D image processing, analysis and annotation steps, the result of which is a retrospective calibration of the 2D imaging system and an optimal, i.e., “gold standard” registration of 3D and 2D images. The device and methods were used to create the ”gold standard” on 15 datasets of 3D and 2D cerebral angiograms, whereas each dataset was acquired on a patient undergoing EIGI for either aneurysm coiling or embolization of arteriovenous malformation. The use of the device integrated seamlessly in the clinical workflow of EIGI. While the automated pipeline eliminated all manual input or interactive image processing, analysis or annotation. In this way, the time to obtain the ”gold standard” was reduced from 30 to less than one minute and the “gold standard” of 3D-2D registration on all 15 datasets of cerebral angiograms was obtained with a sub-0.1 mm accuracy.

In low-dose prostate brachytherapy treatment, a large number of radioactive seeds is implanted in and adjacent to the prostate gland. Planning of this treatment involves the determination of a Planning Target Volume (PTV), followed by defining the optimal number of seeds, needles and their coordinates for implantation. The two major planning tasks, i.e. PTV determination and seed definition, are associated with inter- and intra-expert variability. Moreover, since these two steps are performed in sequence, the variability is accumulated in the overall treatment plan. In this paper, we introduce a model based on a data fusion technique that enables joint determination of PTV and the minimum Prescribed Isodose (mPD) map. The model captures the correlation between different information modalities consisting of transrectal ultrasound (TRUS) volumes, PTV and isodose contours. We take advantage of joint Independent Component Analysis (jICA) as a linear decomposition technique to obtain a set of joint components that optimally describe such correlation. We perform a component stability analysis to generate a model with stable parameters that predicts the PTV and isodose contours solely based on a new patient TRUS volume. We propose a framework for both modeling and prediction processes and evaluate it on a dataset of 60 brachytherapy treatment records. We show PTV prediction error of 10:02±4:5% and the V100 isodose overlap of 97±3:55% with respect to the clinical gold standard.

Magnetic resonance imaging (MRI)-targeted, 3D transrectal ultrasound (TRUS)-guided “fusion” prostate biopsy aims to reduce the 21–47% false negative rate of clinical 2D TRUS-guided sextant biopsy. Although it has been reported to double the positive yield, MRI-targeted biopsy still has a substantial false negative rate. Therefore, we propose optimization of biopsy targeting to meet the clinician’s desired tumor sampling probability, optimizing needle targets within each tumor and accounting for uncertainties due to guidance system errors, image registration errors, and irregular tumor shapes. As a step toward this optimization, we obtained multiparametric MRI (mpMRI) and 3D TRUS images from 49 patients. A radiologist and radiology resident contoured 81 suspicious regions, yielding 3D surfaces that were registered to 3D TRUS. We estimated the probability, P, of obtaining a tumor sample with a single biopsy, and investigated the effects of systematic errors and anisotropy on P. Our experiments indicated that a biopsy system’s lateral and elevational errors have a much greater effect on sampling probabilities, relative to its axial error. We have also determined that for a system with RMS error of 3.5 mm, tumors of volume 1.9 cm³ and smaller may require more than one biopsy core to ensure 95% probability of a sample with 50% core involvement, and tumors 1.0 cm³ and smaller may require more than two cores.

Radiotherapy is frequently used to treat unoperated or partially resected tumors. Tumor movement, e.g. caused by respiration, is a major challenge in this context. Markers can be implanted around the tumor prior to radiation therapy for accurate tracking of tumor movement. However, accurate placement of these markers while keeping a secure margin around the target and while taking into account critical structures is a difficult task. Computer-assisted needle insertion has been an active field of research in the past decades. However, the challenge of navigated marker placement for motion compensated radiotherapy has not yet been addressed. This work presents a system to support marker implantation for radiotherapy under consideration of safety margins and optimal marker configuration. It is designed to allow placement of markers both percutaneously and during an open liver surgery. To this end, we adapted the previously proposed EchoTrack system which integrates ultrasound (US) imaging and electromagnetic (EM) tracking in a single mobile modality. The potential of our new marker insertion concept was evaluated in a phantom study by inserting sets of three markers around dedicated targets (n=22) simultaneously spacing the markers evenly around the target as well as placing the markers in a defined distance to the target. In all cases the markers were successfully placed in a configuration fulfilling the predefined criteria. This includes a minimum distance of 18.9 ± 2.4 mm between marker and tumor as well as a divergence of 2.1 ± 1.5 mm from the planned marker positions. We conclude that our system has high potential to facilitate the placement of markers in suitable configurations for surgeons without extensive experience in needle punctions as high quality configurations were obtained even by medical non-experts.

Assessing the therapeutic benefit of surgical navigation systems is a challenging problem in image-guided surgery. The exact clinical indications for patients that may benefit from these systems is not always clear, particularly for abdominal surgery where image-guidance systems have failed to take hold in the same way as orthopedic and neurosurgical applications. We report interim analysis of a prospective clinical trial for localizing small colorectal liver metastases using the Explorer system (Path Finder Technologies, Nashville, TN). Colorectal liver metastases are small lesions that can be difficult to identify with conventional intraoperative ultrasound due to echogeneity changes in the liver as a result of chemotherapy and other preoperative treatments. Interim analysis with eighteen patients shows that 9 of 15 (60%) of these occult lesions could be detected with image guidance. Image guidance changed intraoperative management in 3 (17%) cases. These results suggest that image guidance is a promising tool for localization of small occult liver metastases and that the indications for image-guided surgery are expanding.

Medical image segmentation is crucial for quantitative organ analysis and surgical planning. Since interactive segmentation is not practical in a production-mode clinical setting, automatic methods based on 3D object appearance models have been proposed. Among them, approaches based on object atlas are the most actively investigated. A key drawback of these approaches is that they require a time-costly image registration process to build and deploy the atlas. Object cloud models (OCM) have been introduced to avoid registration, considerably speeding up the whole process, but they have not been compared to object atlas models (OAM). The present paper fills this gap by presenting a comparative analysis of the two approaches in the task of individually segmenting nine anatomical structures of the human body. Our results indicate that OCM achieve a statistically significant better accuracy for seven anatomical structures, in terms of Dice Similarity Coefficient and Average Symmetric Surface Distance.

Image non-uniformity and intensity non-standardness are two major hurdles encountered in human and computer interpretation and analysis of magnetic resonance (MR) images. Automated methods for image non-uniformity correction (NC) and intensity standardization (IS) may fail because solutions for them require identifying regions representing the same tissue type for several different tissues, and the automatic strategies, irrespective of the approach, may fail in this task. This paper presents interactive strategies to overcome this problem: interactive NC and interactive IS. The methods require sample tissue regions to be specified for several different types of tissues. Interactive NC estimates the degree of non-uniformity at each voxel in a given image, builds a global function for non-uniformity correction, and then corrects the image to improve quality. Interactive IS includes two steps: a calibration step and a transformation step. In the first step, tissue intensity signatures of each tissue from a few subjects are utilized to set up key landmarks in a standardized intensity space. In the second step, a piecewise linear intensity mapping function is built between the same tissue signatures derived from the given image and those in the standardized intensity space to transform the intensity of the given image into standardized intensity. Preliminary results on abdominal T1-weighted and T2-weighted MR images of 20 subjects show that interactive NC and IS are feasible and can significantly improve image quality over automatic methods. Interactive IS for MR images combined with interactive NC can substantially improve numeric characterization of tissues.