This PDF file contains the front matter associated with SPIE Proceedings Volume 12034, including the Title Page, Copyright information, Table of Contents, and Conference Committee listings.

Human-generated, preventable errors, particularly those made intra-operatively, can lead to morbidity and mortality for the patient, poor training outcomes for residents, and high costs for the hospital. Surgical robotic systems could be designed to avoid these errors and improve training outcomes by interpreting, reacting to, and assisting human behavior. This talk will describe some novel data-driven methods to predict, in real-time, surgical style, expertise levels, and task difficulty; as well as present new systems that could be used to assist with surgical intervention or training in a variety of domains.

Transcranial magnetic stimulation is a non-invasive therapeutic procedure in which specific cortical brain regions are stimulated in order to disrupt abnormal neural behaviour. This procedure requires the annotation of a number of cortical point targets which is often performed by a human expert. Nevertheless, there is a large degree of variability between experts that cannot be described readily using the existing zero-mean uni-modal error model common in computer-assisted interventions. This is due to the error between experts arising from a difference of type rather than a difference of degree, that is, experts are not necessarily picking the same point with some error, but are picking fundamentally different points. In order to model these types of localisation errors, this paper proposes a simple probabilistic model that uses the agreement between annotations as a basis, not requiring a ground-truth annotation to be strictly known. This work will allow for the localisation error in transcranial magnetic stimulation to be better described which may spur further developments in clinical training as well as machine learning for cortical point localisation.

Digital subtraction angiography (DSA) is routinely used for measuring the dimensions and characteristics of cerebral aneurysms as a step in planning of interventional treatments. Incorrect sizing of the aneurysm sac puts the patient at the risk of incomplete treatment due to the use of an intrasaccular implant that is too small or too large. In this work, we propose an automatic method to segment the aneurysm sac in 2D DSA images to enable fast and accurate measurements. We use a UNet-like architecture. However, we replace the encoder arm of this network with an EfficientNet architecture, pre-trained on 300 million natural images. We show that this architecture delivers very accurate segmentation of the aneurysm sac on a dataset of 144 DSA images obtained from patients prior to implantation of an intrasaccular device to treat wide-neck bifurcation aneurysms. We report a Dice coefficient of 0.9.

Background: Stereotactic surgery, specifically Deep Brain Stimulation (DBS), has been a well-established intervention for treating neurosurgical disorders, including essential tremor and Parkinson's disorders. Intra- operative MRI-guided DBS implantation has become more a prevalent practice, particularly after stereotactic guidance devices for MRI became available. However, we and others have discovered little to no literature contributions regarding the design and validation of tools to facilitate intraoperative CT-scans for DBS implantation. Objective and Methods: Our goal is to design and validate a guidance device and software to enable CT- guided DBS; specifically, we validated a skull-mounted guidance device integrated for DBS combined with its 3D Slicer software, and tested a hypothesis that the device provides a tool insertion error (Target Point Error) of less than 3 mm. The measurements were done using a skull phantom, with seven clinically relevant targets distributed at variable depths from two different entry points located in the Frontal and Parietal bones. Further analysis was carried out to understand the reason for increased TPE values at certain targets. Results: We found out that our device could produce a TPE of 2.09 ± 0.9 mm for the Frontal bone entry point [p < 0.0001] and 2.52 ± 0.6 mm [p < 0.0016] for the Parietal bone entry point. Additionally, multivariate analysis suggests that depth is the main contributor to larger TPE values when compared to entry points. Implication: These results conclude that our iCT-guided device is capable of replacing DBS tools while enjoying the shorter imaging cycles of CT-scanners. The device proposed may also increase opportunities for patients to receive image-guided DBS since CT-scanners are more accessible to the public than MRI is.

Patient registration enables image guidance by establishing a transformation between patient (physical) space and image space. In this study, we present an automatic patient registration method using intraoperative stereovision (iSV). First, an iSV system attached to a surgical microscope was used to acquired multiple iSV image pairs of the patient's head/face from various angles after the patient is positioned on the operating table but before incision, and the reconstructed iSV surfaces were concatenated to form a composite field-of-view. Second, the composite iSV surface was registered to the surface profile extracted from pre-operative MR (pMR) images as an initial approximation of patient registration. Third, another iSV image pair of the cortical surface was acquired after dural opening, and the reconstructed iSV cortical surface was re-registered to pMR to refine patient registration using automatically segmented surface features such as blood vessels. We evaluated retrospectively the performance of iSV-based patient registration in 6 patient cases in terms of accuracy and computational efficiency. The computational efficiency was ~10 min for the initial registration using skin iSV data and <10 sec for the re-registration using cortical surface. Target registration errors (TRE) were assessed using landmarks (e.g., blood vessels) on the cortical surface that were identifiable in both iSV and pMR, and the average TRE across 6 cases was 1.91±0.61 mm. These results suggest potential OR applications using intraoperative stereovision for automatic patient registration in image-guided open cranial surgery.

Purpose: Neuro-endoscopic surgery requires accurate targeting of deep-brain structures in the presence of deep-brain deformations (up to 10 mm). We report a deep learning-based method to solve deformable MR-to-CBCT registration using a joint synthesis and registration (JSR) network. Method: The JSR network first encodes the MR and CBCT images into latent variables via MR and CBCT encoders, which are then decoded by two branches: image synthesis branches for MR-CT and CBCT-CT synthesis; and a registration branch for intra-modality registration in an intermediate (synthetic) CT domain. The two branches are jointly optimized, encouraging the encoders to extract features pertinent to both synthesis and registration. The algorithm was trained and tested on a dataset of 420 paired volumes presenting a wide range of simulated deformations. The JSR method was trained in a semi-supervised manner and evaluated in comparison to an alternative, state-of-the-art, inter-modality registration method (VoxelMorph). Results: The JSR method achieved Dice of 0.67 ± 0.11, surface distance error (SD) of 0.47 ± 0.26 mm, and target registration error (TRE) of 2.23 ± 0.80 mm in a simulation study – each superior to the alternative methods considered in this work. Moreover, JSR maintained diffeomorphism and exhibited a fast runtime of 2.55 ± 0.03 s. Conclusion: The JSR algorithm demonstrates accurate, near real-time deformable registration of preoperative MRI to intraoperative CBCT and is potentially suitable to intraoperative guidance of intracranial neurosurgery.

This paper describes the design of a modular concentric tube robot actuation system featuring roller gears. We extended prior work in a single-arm system within the literature to a two-arm system. We present mechanical design and initial performance verification experiments, including two-handed manipulation experiments for the proposed system. The overarching contribution is a modular, potentially low cost, approach to the actuation of multi-arm concentric tube robot systems.

Transoral robotic surgery (TORS) has demonstrated improved surgical outcomes with reduced morbidity when compared to traditional open surgical treatments. However, it is more difficult to assess the extent of tumor and localize critical structures due to lack of haptic feedback and the natural orifice approach. Enabling image-guided TORS (igTORS) to compensate for the sensory deficit requires a surgical navigation system that is compatible with both the TORS procedures and the da Vinci surgical system. Previously, the authors developed an imaging compatible oral retractor system for TORS to allow artifact-free intraoperative CT images for use in image guidance. In this work, we developed a surgical navigation system for TORS that utilizes intraoperative images and electromagnetic (EM) tracking. A cadaver experiment simulating a standard TORS procedure was performed to examine system feasibility and accuracy. A da Vinci Bovie instrument was tracked, and its real-time location was visualized in tri-planar CT images and displayed on the surgeon’s console and on the bedside vision cart using the TilePro feature. Target localization error (TLE) was computed to be 3.46±0.77 mm. This was the first time that surgical navigation in TORS was demonstrated with intraoperative image guidance and EM tracking of da Vinci instruments in a cadaver experiment.

Metal artifacts have been a difficult challenge for cone-beam CT (CBCT), especially for intraoperative imaging. Metal surgical tools and implants are often present in the field of view and can attenuate X-rays so heavily that they essentially create a missing-data problem. Recently, an increasing number of intra-operative imaging systems such as robotic C-arms are capable of non-circular orbits for data acquisition. Such trajectories can potentially improve sampling and the degree of data completeness to solve the metal-induced missing-data problem, thereby reducing or eliminating the associated image artifacts. In this work, we extend our prior theoretical and experimental work and implement non-circular orbits for metal artifact reduction on a clinical robotic C-arm (Siemens Artis zeego). To maximize the potential for clinical translation, we restrict our implementation to standard built-in motion and data collection functions, also available on other zeego systems, and work within the physical constraints and limitations on positioning and motion. Customized software tools for data extraction, processing, calibration, and reconstruction are used. We demonstrate example non-circular orbits and the resulting image quality using a phantom containing pedicle screws for spine fixation. As compared with a standard circular CBCT orbit, these non-circular orbits exhibit significantly reduced metal artifacts. These results suggest a high potential for image quality improvements for intraoperative CBCT imaging when metal tools or implants are present in the field-of-view.

During the diagnosis of peripheral pulmonary lesions (PPLs), radial-probe endobronchial ultrasound (RP-EBUS) is often used in combination with bronchoscopy to visualize extraluminal structures and confirm lesion sites. However, due to the steep learning curve, a physician’s ability to use RP-EBUS varies greatly in practice. On another front, image-guided bronchoscopy systems are now commonly used to assist with bronchoscopy planning and guidance.1 A procedure plan is first generated from the patient’s CT scan consisting of airway routes leading to each target lesion. Next, during live bronchoscopy, the system helps navigate the bronchoscope along the preplanned airway route close to the target lesion. However, there is no direct modality linkage and guidance between RP-EBUS and the guidance system during image-guided bronchoscopy. We now propose a multimodal image-guided methodology for guiding RP-EBUS consisting of two parts: 1) a CT-based procedure planning method that enables optimal RP-EBUS invocation, localization, preview, and RP-EBUS video simulation. 2) an intra-operative guidance system tailored to RP-EBUS localization of PPLs. We present results demonstrating the proposed system.

This paper describes a branching level estimation method using the tracking result of the bronchial orifice structure in branches. Since the bronchus has a tree-like structure with many branches, it would be beneficial to physicians if the location of the bronchoscope in branches is provided. Hence, the estimation of branching level is the core work of coarse bronchoscope tracking-based navigation bronchoscopy. Previous method used the changes in the number of the bronchial orifice (BO) and the camera moving direction for the branching level estimation, which cannot observe the changes of each BO region. Therefore, we extract BO regions using a virtual depth image from deep learning and track these regions among real bronchoscope images. The branching level is estimated based on the results of BO tracking. Experimental results showed that the average accuracy of the branching level estimation is 92.1 %.

Endoscopic inspection of the bladder is called cystoscopy, and over 1 million are performed annually in the USA.1 Robotic assistance of the procedure is being considered to overcome healthcare disparities due to geographic distances. To ensure safety, real-time navigation by simultaneous localization and mapping (SLAM) of the cystoscope in the bladder is desired. However, the near-featureless bladder wall and irregular movement of the monocular cystoscope by manual operation has made SLAM-based navigation a difficult challenge. In this work we develop a robot-assisted approach in a bladder phantom using a commercial flexible cystoscope. With this simple remote-controlled robot, we have succeeded in creating a series of real-time 2.5D reconstructions of the bladder interior wall. In post-processing, we combine those 2.5D results and achieve a 3D reconstruction. We compared SLAM performance using the robot and manual control under different scanning speeds. In comparison to manual control using the same cystoscope at 1x, 2x and 3x speeds, the robotic-assistance provides significantly more robust and accurate reconstructions of the camera trajectories and bladder wall locations in real-time.

Purpose: Recent neurosurgical techniques require accurate targeting of deep-brain structures even in the presence of deformation due to egress of cerebrospinal fluid (CSF) during surgical access. Prior work reported Structure-from-Motion (SfM) based methods for endoscopic guidance using 3D reconstruction. We are developing feature detection and description methods for a real-time 3D endoscopic navigation system using simultaneous localization and mapping (SLAM) to for accurate and near real-time registration. Methods: Feature detectors and descriptors were evaluated in SLAM reconstruction in anthropomorphic phantom studies emulating neuroendoscopy. The experimental system utilized a mobile UR3e robot (Universal Robots, Denmark) and ventriculoscope (Karl Storz, Tuttlingen, Germany) affixed to the end effector as a repeatable ventriculoscopy platform. Experiments were conducted to quantify optimal feature detection parameters in scale-space. Neuroendoscopic images acquired in traversal of the lateral and third ventricles provided a rich feature space of vessels and other structures on ventricular walls supporting feature detection and 3D point-cloud reconstruction. Performance was evaluated in terms of the mean number of features detected per frame and the algorithm runtime. Results: Parameter search in scale-space for feature detection demonstrated the dependence on the mean number of features per image and the points of diminishing return in parameter selection (e.g., the number of octaves and scale levels) and tradeoffs in runtime. Nominal parameters were identified as 3 octaves and 9 scale levels, with a mean number of features detected as 492 and 806 respectively. Conclusions: The system for neuroendoscopic guidance based on SLAM 3D point-cloud reconstruction provided a promising platform for the development of robot-assisted endoscopic neurosurgery. The studies reported in this work provided an essential basis for rigorous selection of parameters for feature detection. Future work aims to further develop the SLAM framework, assess the geometric accuracy of reconstruction, and translate methods to clinical studies.

Hand-held stereovision (HHS) has been implemented as an accurate, affordable, and non-ionizing method for image updating, and marker-less registration. Studies using HHS have collected images via individual snapshots. This study compares the accuracy of data acquisition from snapshots compared to a video stream data acquisition. An HHS system was calibrated via a tracked checkerboard to an accuracy of 0:88±0:24mm. A cadaver pig was used to measure accuracy of ducial locations with ducials placed on each spinous, and transverse process of L2 through L6. The system was then used to collect images of the surgical field. The images were collected 5 times. There was one snapshot image acquisition, and 4 video streams. After, the ducial locations were localized via a tracked stylus probe. The error was then computed between the stylus ducial locations and the ducial locations found on the stereovision depth map. The corresponding mean errors for the snapshots and 4 video acquisitions were as follows: 1:03 ± 0:24mm, 1:13 ± 0:34mm, 1:18 ± 0:59mm, 1:19 ± 0:47mm, and 3:23 ± 2:27mm respectively. While collecting the video streams the data were collected at what was perceived a constant speed. Each sequential video stream was collected at a higher speed. These speeds were calculated to be 7:22±1:95mm/s, 12:73±7:75mm/s, 19:19±12:74mm/s, and 30:14±13:33mm/s respectively. These data show that image acquisition via video streams at relatively low speeds have accuracy comparable to that of snapshot image acquisition.

Cochlear implants (CIs) are considered the standard-of-care treatment for profound sensory-based hearing loss. After CI surgery, an audiologist will adjust the CI processor settings for CI recipients to improve overall hearing performance. However, this programming procedure can be long and may lead to suboptimal outcomes due to the lack of objective information. In previous research, our group has developed methods that use patient-specific electrical characteristics to simulate the activation pattern of auditory nerves when they are stimulated by CI electrodes. However, estimating those electrical characteristics require extensive computation time and resources. In this paper, we proposed a deep-learning-based method to coarsely estimate the patient-specific electrical characteristics using a cycle-consistent network architecture. These estimates can then be further optimized using a limited range conventional searching strategy. Our network is trained with a dataset generated by solving physics-based models. The results show that our proposed method can generate high-quality predictions that can be used in the patient-specific model and largely improves the speed of constructing models.

High dose rate (HDR) brachytherapy is a standard radiation therapy for skin, vaginal, cervical, uterine and prostate cancers. Currently, there are few practical approaches to tracking HDR source position in vivo due to dose, shielding, accessibility and workflow challenges, hence internal dosimetry typically cannot be directly verified. Accurate source tracking capabilities are desired to reduce uncertainty in the delivered versus planned treatment dose, which may limit underdose and overdose of the prescribed treatment volume and healthy tissues and improve clinical outcomes. This work investigates the feasibility of using an isocentric C-arm x-ray imaging system to achieve 4D HDR brachytherapy source tracking in vivo using cone-beam tomography and highly constrained reconstruction methods. Monte Carlo simulations using the MC-GPU platform and XCAT anthropomorphic phantom were performed to simulate the imaging workflow for such a system, and investigate the impact of detector, image acquisition and reconstruction parameters on its achievable spatiotemporal resolution. Tradeoffs in the system’s spatial, temporal, and dose characteristics are determined. The results indicate that 4D resolutions on the order of 1 mm and 1-2 second may be achieved, which is acceptable for all current HDR brachytherapy applications.

During cochlear implant (CI) surgical procedures, the surgeon inserts an electrode array into the patient’s cochlea to restore hearing sensation through auditory nerve stimulation. Due to the significant variability in cochlear anatomy from person to person, patient-customized CI surgery planning has the potential to improve the outcome of the CI procedure. For presurgery planning, accurate segmentation of intra-cochlear structures is essential. In this work, we investigate the performance of intra-cochlear segmentation algorithms as a function of variations in image acquisition parameters (i.e., resolution, blurring effect and noise level) that exist in clinical CT images. A dataset of preoperative μCT images of 11 cadaveric temporal bone specimens was used to generate 110 synthetic pseudo-CT images with varying resolution and filter parameters. An active shape model (ASM) based method was evaluated to segment the intra-cochlear structures in those pseudo-CT images. Our results show that the volume of the segmented structures is significantly and strongly correlated with both the resolution and the reconstruction filter strength of the synthetic pseudo-CT images. Recognizing this bias is important for clinicians who use these segmentations or take manual measurements of the cochlea from CT images for pre-surgical planning of CI procedures.

In recent years, the immunotherapy through immunocheckpoint inhibitors significantly improves the survival rate and reduce recurrence risk in metastatic melanoma. Moreover, accurately predicting immunotherapy response is of great importance to improve treatment effectiveness. We are aiming to develop a new automated multi-objective model with hyperparameter optimization (AutoMO-HO) for improving treatment outcome prediction performance. Delta-radiomic features which calculates the difference between pre- and post-treatment radiomic features were used in this study. To obtain balanced sensitivity and specificity as well as higher confidence output, an automated multi-objective model (AutoMO) is applied. However, there are several hyperparameters to be set manually before training, leading to the nonoptimal model performance. As such, Bayesian optimization is introduced to train the model hyperparameter, and a new model termed as AutoMO-HO is developed based on AutoMO. In AutoMO-HO, the training stage consists of two phases, they are Bayesian hyperparameter optimization through the Tree Parzen estimator algorithm and Pareto-optimal model set generation. In testing stage, the evidential reasoning (ER) strategy is used to fuse the output of each Paretooptimal model to obtain more reliable results. Finally, the label with the maximal output confidence is taken as final output label. The experimental results demonstrated that AutoMO-HO outperformed AutoMO and other available methods.

Breast cancer is the most common cancer in women, and surgical resection is standard of care for the majority of breast cancer patients. Unfortunately, current reoperation rates are 10-29%. Uncertainty in lesion localization is one of the main factors contributing to these high reoperation rates. This work uses the linearized iterative boundary reconstruction approach to model patient breast deformation due to abduction of the ipsilateral arm. A preoperative supine magnetic resonance (MR) image was obtained with the patient’s arms down near the torso. A mock intraoperative breast shape was measured from a supine MR image obtained with the patient’s arm up near the head. Sparse data was subsampled from the full volumetric image to represent realistic intraoperative data collection: surface fiducial points, the intrafiducial skin surface, and the chest wall as measured with 7 tracked ultrasound images. The deformed preoperative armdown data was compared to the ground truth arm-up data. From rigid registration to model correction the tumor centroid distance improves from 7.3 mm to 3.3 mm, average surface fiducial error across 9 synthetic fiducials and the nipple improves from 7.4 ± 2.2 to 1.3 ± 0.7, and average subsurface error across 14 corresponding features improves from 6.2 ± 1.4 mm to 3.5 ± 1.1 mm. Using preoperative supine MR imaging and sparse data in the deformed position, this modeling framework can correct for breast shape changes between imaging and surgery to more accurately predict intraoperative position of the tumor as well as 10 surface fiducials and 14 subsurface features.

The parasternal long axis (PLAX) is a routine imaging plane used by clinicians to assess the overall function of the left ventricle during an echocardiogram. Measurements from the PLAX view, in particular left ventricle internal dimension at both end-diastole (LVIDd) and end-systole (LVIDs), are significant markers used to identify cardiovascular disease and can provide an estimation of ejection fraction (EF). However, due to the user-dependent nature of echocardiograms, these measurements suffer from a large amount of inter-observer variability, which greatly affect the sensitive formula used to calculate PLAX EF. While few previous works have attempted to reduce this variability by automating LVID measurements, their models not only lack reliable accuracy and precision, but also generally are not suited to be adapted in point-of-care ultrasound (POCUS) which has limited computing resources. In this paper, we propose a fully automatic, light-weight landmark detection network for detecting LVID and rapidly estimating PLAX EF. Our model is built upon recent advances in deep video landmark tracking with extremely sparse annotations.¹ The model is trained on only two frames in the cardiac cine that contain either the LVIDd or LVIDs measurements labeled by clinicians. Using data from 34,305 patients for our experiments, the proposed model accurately tracks the contraction of left ventricular walls. Our model achieves a mean absolute error and standard deviation of 2:65 ± 2:36 mm, 2:77 ± 2:58 mm, and 8:45 ± 7:43% for predicting LVIDd length, LVIDs length, and PLAX EF, respectively. As a light-weight network with less than 125,000 parameters, our model is extremely accessible for POCUS applications.

Transcranial MRI-guided focused ultrasound (TcMRgFUS) is a therapeutic ultrasound method that focuses sound through the skull to a small region noninvasively under MRI guidance. It is clinically approved to thermally ablate regions of the thalamus and is being explored for other therapies, such as blood brain barrier opening and neuromodulation. To accurately target ultrasound through the skull, the transmitted waves must constructively interfere at the target region. However, heterogeneity of the sound speed, density, and ultrasound attenuation in different individuals' skulls requires patient-specific estimates of these parameters for optimal treatment planning. CT imaging is currently the gold standard for estimating acoustic properties of an individual skull during clinical procedures, but CT imaging exposes patients to radiation and increases the overall number of imaging procedures required for therapy. A method to estimate acoustic parameters in the skull without the need for CT would be desirable. Here, we synthesized CT images from routinely acquired T1-weighted MRI by using a 3D patch-based conditional generative adversarial network (cGAN) and evaluated the performance of synthesized CT images for treatment planning with transcranial focused ultrasound. A dataset of 86 paired CT and T1-weighted MR images were randomly split so that 66 images were used for training, 10 for validation and parameter tuning, and 10 for acoustic testing. We compared the performance of synthetic CT (sCT) to real CT (rCT) images using an open-source treatment planning software, Kranion, and found that the number of active elements, skull density ratio, and skull thickness between rCT and sCT had Pearson's Correlation Coefficients of 0.989, 0.915, and 0.941, respectively, suggesting strong positive linear correlation. Of a total of 990 elements 95:7 ± 1:4% of active and inactive elements overlapped between rCTs and sCTs. Simulations using the acoustic toolbox, k-Wave, resulted in 23:5 ± 6:51% less maximum root-mean-squared (RMS) pressure simulated with sCTs than the corresponding rCT pressure. An average focal shift of 0:96 ± 0:56 mm and 1:07 ± 0:58 mm was observed between the thalamus target and the maximum RMS pressure location in rCTs and sCTs, respectively. Our work demonstrates the feasibility of replacing real CT with the MR-synthesized CT for TcMRgFUS planning.

Percutaneous thermal ablations are promising curative treatment techniques of focal liver tumors, particularly for those patients who are not eligible for surgical resection. Complete coverage of the targeted tumor by the thermal ablation zone and with a safety margin of 5-10 mm is required to ensure that complete tumor eradication will be achieved. 2D ultrasound (US) is a commonly used modality to guide this procedure; however, it has limitations in estimating the ablation tumor coverage due to the difficulty of evaluating tumor coverage using only one or multiple 2D US images. The use of intra-procedural 3D US is a promising approach to solve this unmet need. Although most of current approaches provide reformatted three orthogonal views to better evaluate the tumor coverage, comprehensive volumetric evaluation is rarely available. In this paper, for tumor-visible cases in US, we aim to investigate the ability of 3D US images to visualize the applicators and relevant surrounding structures, then assess the feasibility of evaluating the tumor coverage quantitatively using surface- and volume-based metrics. Using our previously developed 3D US liver ablation system, we collected 10 patients’ 3D US liver images in our clinical trial. The visibility of the applicator and relevant structures were assessed qualitatively. We then evaluated the surface error and volume accuracy of the tumor coverage. Results demonstrated that 3D US images allow visualization of the appropriate anatomical structures and applicators, and our volumetric evaluation can provide systematic knowledge of tumor coverage and an opportunity to correct the ablation applicator position intra-procedurally.

In this study, a strategy to segment the LAA in 3D CT images is presented. This method relies on our recent curvilinear blind-ended model implemented into the B-spline Explicit Active Surface framework for ultrasound images. Starting from a manual identification of the LAA centerline, a curvilinear blind-ended model is initialized and refined to the anatomy through a double-step strategy with fast contour growing and surface refinement. This pipeline was tested in 15 patients, corroborating its high accuracy in terms of segmentation accuracy and narrow limits of agreement for the relevant clinical measurements.

The utilization of point-of-care ultrasound (POCUS) has been on rise in recent years, followed by a growing need for comprehensive, compact obstetrics analysis software systems. The accurate computerized assessment of obstetric ultrasound (US) is a challenging task due to the noisy nature of US images and presence of complex anatomies. In this work, we propose a multi-branch deep learning architecture to identify multiple anatomies in obstetric sonography through segmentation and landmark detection. The multi-task deep model is trained to segment the uterus and gestational sac regions, and to localize the position of landmark points denoting the crown and rump of the fetus. We conduct experiments with varying sizes of models, presenting a trade-off between accuracy and efficiency. Our larger models reach average Dice scores of 90% and 91% for segmenting uterus and gestational sac, respectively, and have 1.64 mm average length error for the fetus crown-rump length (CRL) measurement. Furthermore, we present choices for smaller model sizes suitable for integration in portable POCUS devices with limited computing capacities. The smallest model with the most efficient run-time has only about 167.3k parameters, where compared to the larger models, the Dice score is reduced by 4.1% and 6.7% for uterus and gestational sac segmentation, respectively, and the CRL length error is increased by 0.42 mm. This is while the smallest model has 98.8% reduced model size and could smoothly run on naive mobile devices.

We developed a new method for automated detection of microchannel in intravascular optical coherence tomography images. The proposed method includes three main steps including pre-processing, identification of microchannel candidates, and classification of microchannel. Our method provided excellent segmentation of microchannel with a Dice coefficient of 0.811, sensitivity of 92.4%, and specificity of 99.9%. Our method has great potential to enable highly automated, objective, repeatable, and comprehensive evaluations of vulnerable plaques and treatments. We believe that this method is promising for both research and clinical applications.

While deep learning has shown potential in solving a variety of medical image analysis problems including segmentation, registration, motion estimation, etc., their applications in the real-world clinical setting are still not affluent due to the lack of reliability caused by the failures of deep learning models in prediction. Furthermore, deep learning models need a large number of labeled datasets. In this work, we propose a novel method that incorporates uncertainty estimation to detect failures in the segmentation masks generated by CNNs. Our study further showcases the potential of our model to evaluate the correlation between the uncertainty and the segmentation errors for a given model. Furthermore, we introduce a multi-task cross-task learning consistency approach to enforce the correlation between the pixel-level (segmentation) and the geometric-level (distance map) tasks. Our extensive experimentation with varied quantities of labeled data in the training sets justifies the effectiveness of our model for the segmentation and uncertainty estimation of the left ventricle (LV), right ventricle (RV), and myocardium (Myo) at end-diastole (ED) and end-systole (ES) phases from cine MRI images available through the MICCAI 2017 ACDC Challenge Dataset. Our study serves as a proof-of-concept of how uncertainty measure correlates with the erroneous segmentation generated by different deep learning models, further showcasing the potential of our model to flag low-quality segmentation from a given model in our future study.

Purpose: A recent imaging method (viz., Long-Film) for capturing long-length images of the spine was enabled on the Oarm™ system. Proposed work uses a custom, multi-perspective, region-based convolutional neural network (R-CNN) for labeling vertebrae in Long-Film images and evaluates approaches for incorporating long contextual information to take advantage of the extended field-of-view and improve the labeling accuracy. Methods: Evaluated methods for incorporating contextual information include: (1) a recurrent network module with long short-term memory (LSTM) added after R-CNN classification; and (2) a post-processing, sequence-sorting step based on the label confidence scores. The models were trained and validated on 11,805 Long-Film images simulated from projections of 370 CT images and tested on 50 Long-Film images of 14 cadaveric specimens. Results: The multi-perspective R-CNN with LSTM module achieved 91.7% vertebrae level identification rate, compared to 72.4% when used without LSTM, thus demonstrating the improvement of incorporating contextual information. While sequence sorting achieved 89.4% in labeling accuracy, it failed to handle errors during detection and did not provide additional improvements when applied following the LSTM module. Conclusions: The proposed LSTM module significantly improved the labeling accuracy upon the base model through effective contextual information incorporation and training in an end-to-end fashion. Compared to sequence sorting, it showed more flexibility towards false positives and false negatives in vertebrae detection. The proposed model offers the potential to provide a valuable check for target localization and forms the basis for automatic measurement of spinal curvature changes in interventional settings.

Pediatric Crohn’s disease (pCD) is a chronic relapsing-remitting inflammatory disease of the gastrointestinal tract, where there is a significant need for non-invasive comprehensive markers to accurately target clinical interventions. While Magnetic Resonance Enterography (MRE) is often used to localize pCD in vivo, it is still limited in predicting treatment response and capturing pCD phenotypes. The goal of this study was to identify radiomic features on baseline MRE associated with disease activity and treatment outcomes in pCD, as well as investigate potential associations of radiomics with serum-based pCD subtypes. Baseline MRE scans were acquired from 45 pediatric subjects (including healthy controls and pCD patients) where the latter was further sub-grouped into responders (stable after treatment) and non-responders (required surgery or had active disease 2+ years after treatment initiation). Radiomic features were extracted from the terminal ileum on a per-voxel basis from MRE and evaluated via a multi-stage feature selection scheme for identifying disease presence and patient outcomes separately. A Random Forest (RF) classifier achieved an area under the ROC curve (AUC) of 0.83 in distinguishing diseased patients from healthy subjects and an AUC of 0.85 in distinguishing nonresponders from responders; in leave-one-out cross-validation. Top-ranked Gabor and Laws radiomic features were found to be significantly correlated with serum pCD phenotypes including anemia, inflammation risk, vitamin deficiency, and immune activity. Radiomic features may therefore offer the ability to better characterize pCD phenotypes and predict patient outcomes, which could then be effectively treated via targeted interventions.

Precise placement of needles plays a crucial role in percutaneous procedures as it helps to achieve higher diagnostic accuracy and accurate tumor targeting. C-arm cone-beam computed tomography (CBCT) has the potential to precisely image the anatomy in direct vicinity of the needle. However, exact needle positioning is very difficult due to strong metal artifacts around the needle. In this study, we evaluate the performance of the prior image constrained compressed sensing (PICCS) CBCT reconstruction in presence of metal objects. Our results confirm the high performance of PICCS to reduce needle artifacts using both circular and non-conventional trajectories under kinematic constraints.

Proton therapy can benefit chordoma treatment due to its capability of delivering highly conformal doses to the target area. However, clinical observations show that poor outcomes have been correlated to patients with metallic implants due to artifact contaminated computed tomography (CT) images. This study proposes a deep learning (DL)-based metal artifact reduction (MAR) framework to unsupervisely combine DL and k-means clustering algorithm and to obtain artifact-reduced images to support clinical applications. The framework includes three modules: data preprocessing, DL, and machine learning (ML) modules. An in-house material-based CT simulator was developed to augment the artifact training datasets. The DL module can adopt an unsupervised model to alleviate the need for labeled training data, which increases the clinical applicability. The ML module is used to classify tissue types from implant-free images around the adjacent treatment site, and the results can be used to further correct metal artifacts from DL images. We hypothesize that the artifact-reduced images should have similar CT number distributions to the adjacent CT images without surgical implants at the same treatment site. The Kullback-Leibler divergence (KLD) is used to evaluate the dissimilarity between these two distributions. The results indicate that the proposed method is comparative to Siemens iMAR and the current clinical procedure and can generate images with smaller KLD values than the images obtained by only using a DL model. The proposed method has the potential to reduce the treatment reserved margin that can decrease the radiation dose to normal tissues and accelerate the treatment planning time by revealing clear tumor structures.

Existing clinical C-arm interventional x-ray systems equipped with flat panel detectors (FPDs) can generate fluoroscopic, angiographic, and cone-beam CT (CBCT) images with sufficient volumetric coverage for interventional imaging tasks. However, FPD-CBCT does not provide sufficient low-contrast detectability, resolution, or spectral imaging capability desired for certain interventional procedures. To overcome these limitations, a C-arm photon counting detector (PCD) CT prototype was developed by installing an interchangeable strip PCD on the C-arm gantry. The narrow z width of the PCD reduces detector cost and reduces scatter when paired with a narrow beam collimation. However, it does not provide sufficient volumetric coverage compared to the standard FPD. The purpose of this work was to develop a step-and-shoot data acquisition method to enlarge the effective z-coverage of the C-arm strip PCD-CT system. A total of 10 back-and-forth short-scan C-arm gantry rotations were used with image object translation. By using an Arduino board to process the x-ray-on pulse signals in real-time, a motorized patient table prototype was synchronized with the C-arm system such that it translates the object by the PCD width during the rest time in between gantry rotations. To evaluate whether this multisweep step-and-shoot acquisition mode can generate high-quality and volumetric PCD-CT images, experiments were performed using an anthropomorphic head phantom, and a stent. The multi-sweep step-and-shoot C-arm protocol resulted in volumetric PCD-CT images with lower image noise and improved low-contrast visualization over the FPD-CBCT in the head phantom, as well as improved visibility of small iodinated blood vessels using maximum intensity projections. Under an ultra-high-resolution PCD mode, the fine structures of the stent were visualized more clearly by the PCD-CT than the highest-available resolution provided by the FPD-CBCT. The multi-sweep step-and-shoot acquisition can therefore extend the z-coverage of the C-arm PCD-CT prototype by a factor of 10 to enable high-quality and volumetric C-arm PCD-CT images acquired with a narrow beam-narrow detector setup for image-guided interventions.

Successful estimation of target registration error (TRE) would provide immense opportunities for controlling risks associated with navigation during image-guided surgery. While developed theories exist for predicting spatial distributions of TRE for rigid point-based registration, similar capabilities in the domain of deformable registration are still needed to develop truly reliable image guidance systems for navigation in soft tissue organs. Recently, breakthrough work derived two analytic uncertainty metrics based on the dissipation of constraint energy over distance to measure the susceptibility of elastic deformable registration to errors originating from unknown effects that occur where registration constraints are missing. In this work, these registration uncertainties are leveraged to classify error thresholds for detecting spatial regions that become vulnerable to inaccuracy in sparse data driven elastic registrations. With a large dataset of over 6000 registrations, receiver operating characteristic (ROC) analysis was performed to assess discriminatory performance of these uncertainty metrics to clinically relevant levels of registration error and to identify optimal binary cutoffs for their prediction. Both uncertainty metrics were capable of detecting regions of the organ where deformable registration accuracy exceeded the average magnitude of rigid registration error with AUC above 0.87. Furthermore, both metrics detected regions of the organ with TRE greater than 10 mm with AUC of approximately 0.8. These new capabilities will enhance clinical confidence in image-guided technologies in deforming organs through enabling immediate quantification and communication of navigational reliability and system accuracy during soft tissue surgery.

Respiratory motion is a major contributor to bias in quantitative analysis of magnetic resonance imaging (MRI) acquisitions. Deformable registration of three-dimensional (3D) dynamic contrast-enhanced (DCE) MRI data improves estimation of kidney kinetic parameters. In this study, we proposed a deep learning approach with two steps: a convolutional neural network (CNN) based affine registration network, followed by a U-Net trained for deformable registration between two MR images. The proposed registration method was applied successively across consecutive dynamic phases of the 3D DCE-MRI dataset to reduce motion effects in the different kidney compartments (i.e., cortex, medulla). Successful reduction in the motion effects caused by patient respiration during image acquisition allows for improved kinetic analysis of the kidney. Original and registered images were analyzed and compared using dynamic intensity curves of the kidney compartments, target registration error of anatomical markers, image subtraction, and simple visual assessment. The proposed deep learning-based approach to correct motion effects in abdominal 3D DCE-MRI data can be applied to various kidney MR imaging applications.

Breast conserving surgery (BCS) is a common procedure for early-stage breast cancer patients. Supine preoperative magnetic resonance (MR) breast imaging for visualizing tumor location and extent, while not standard for procedural guidance, more closely represents the surgical presentation compared to conventional diagnostic pendant positioning. Optimal utilization for surgical guidance, however, requires a fast and accurate image-to-physical registration from preoperative imaging to intraoperative surgical presentation. In this study, three registration methods were investigated on healthy volunteers’ breasts (n=11) with the arm-down position simulating preoperative imaging and arm-up position simulating intraoperative data. The registration methods included: (1) point-based rigid registration using synthetic fiducials, (2) non-rigid biomechanical model-based registration using sparse data, and (3) a data-dense 3D diffeomorphic image-based registration from the Advanced Normalization Tools (ANTs) repository. The average target registration errors (TRE) were 10.4 ± 2.3, 6.4 ± 1.5, and 2.8 ± 1.3 mm (mean ± standard deviation) and the average fiducial registration errors (FRE) were 7.8 ± 1.7, 2.5 ± 1.1, and 3.1 ± 1.1 mm (mean ± standard deviation) for the point-based rigid, nonrigid biomechanical, and ANTs registrations, respectively. Additionally, common mechanics-based deformation metrics (volume change and anisotropy) were calculated from the ANTs deformation field. The average metrics revealed anisotropic tissue behavior and a statistical difference in volume change between glandular and adipose tissue, suggesting that nonrigid modeling methods may be improved by incorporating material heterogeneity and anisotropy. Overall, registration accuracy significantly improved with increasingly flexible registration methods, which may inform future development of image guidance systems for lumpectomy procedures.

Volume-to-slice image registration can be used to support image guided surgery, volumetric image reconstruction and image guidance in radiotherapy. However, the volume-to-slice image registration is challenging due to its different dimensionality and lack of slice correspondence. In radiotherapy, it is important to track the tumor motion while delivering the radiation in order to maximize target coverage and spare healthy tissue. 2D MRI real-time imaging has been available for tumor tracking during radiotherapy. To support volumetric tumor tracking, pre-treatment volumetric MRI images can be acquired, annotated and registered to the real-time 2D MRI. In this study, a proof-of-concept study was performed to register 3D MRI to 2D MRI slice that was extracted from the 3D MRI after random rotation and translation. We propose an intentional overfitting deep learning based network (ION) to perform volume-to-slice registration for MRI abdominal images. MRI volumetric datasets from 40 patients were used in this study for the network training and testing. As a proof-of-concept study, we simulated the 2D slices by extracting the middle slice from rigidly transformed 3D MR images. The 2D slice and 3D volume were then input to the network to predict a six degree of freedom (DOF) parameters, including 3 translational and 3 rotational parameters that will align the volume to match the slice. The network was first trained using multiple patients’ datasets and then further optimized by a particular testing patient dataset via intentional overfitting. Tested on 10 datasets, the mean square error (MSE) between the 2D slice and the middle slice of the rigidly transformed MRI volume was reduced significantly from 1924 to 27 after image registration, indicating good registration accuracy.

Image-guided percutaneous thermal ablations are promising techniques for the treatment of focal liver tumors. Conventionally, 2D ultrasound (US)-guidance is used extensively to assist liver tumor treatment. Recently, 3D US imaging has attracted much attention as its provided volumetric information can better help physicians interpret and localize liver structures. However, 3D US imaging still has the same limitation as conventional 2D US imaging in visualizing ultrasonically invisible cases. To address this issue, the current mainstream solution is to provide real-time 2D US-CT/MRI registered images by leveraging external tracking systems, such as electromagnetic (EM) or optical approaches. Due to their inevitable constraints, such as the presence of ferromagnetic structures in the case of EM tracking systems, or the line-of-sight limitation for optical systems, whether these solutions can be readily applied to the clinic has been under investigation. In this paper, we aim to investigate the feasibility of a 3D US-based ablation paradigm using our developed 2D/3D US/CT-guided liver ablation system. To achieve this goal and provide accurate guidance for ablation procedures, we proposed a local-and- global calibration method to track our mechatronic guidance arm. We also used a fiducial-based registration method to align 3D US with diagnostic CT images and implemented the re-slicing function to display the CT image corresponding to the US transducer’s pose. Results demonstrated the feasibility of our system to visualize the complementary information from multiple image modalities in real time. Our calibration method can provide accurate tracking with an unsigned error of 1.6 mm ± 0.4 mm. This work is a step towards providing a system to guide the liver ablation procedure, including cases with ultrasonically invisible or poorly visible tumors.

Image-guided thermal ablations have become an important therapeutic option for patient with cardiac arrhythmia, it is minimally-invasive and provides better and faster patient recovery. However, to enhance the ablation guidance, the therapist needs to link by image registration the intraoperative images to the high-resolution anatomical preoperative imaging, in which the ablation path has been defined. In this work, we present a convolutional neural networks (CNNs) framework for transesophageal ultrasound/computed tomography image registration to solve the problem of high computation time of the classical iterative methods, which is not suitable for a real-time application. We propose the following process: we first pass the input moving and fixed image pairs through a siamese architecture consisting of convolutional layers, thus extracting features of moving and fixed maps analogous to dense local descriptors, then matching the feature maps, and finally pass this correspondence feature map into a registration network, which directly outputs the registration parameters set of the rigid registration. Accuracy of the registration is quantified based on the Target Registration Error (TRE) for specific anatomical landmarks. Results of the registration process show a median TRE of 2.2 mm for all the fiducial points, and the registration computation time was around 3 ms comparing to the classic iterative methods which takes around 70 seconds for one image pair. In our future work we are going to perform our approach on 2D/3D learning-based registration to refine the estimation of the transesophageal probe pose in the 3D preoperative volume.

Optical-based navigation systems are widely used in surgical interventions. However, despite their great utility and accuracy, they are expensive and require time and effort to setup for surgeries. Moreover, traditional navigation systems use 2D screens to display instrument positions causing the surgeons to look away from the operative field. Head mounted displays such as the Microsoft HoloLens may provide an attractive alternative for surgical navigation that also permits augmented reality visualization. The HoloLens is equipped with multiple sensors for tracking and scene understanding. Mono and stereo-vision in the HoloLens have been both reported to be used for marker tracking, but no extensive evaluation on accuracy has been done to compare the two approaches. The objective of our work is to investigate the tracking performance of various camera setups in the HoloLens, and to study the effect of the marker size, marker distance from camera, and camera resolution on marker locating accuracy. We also investigate the speed and stability of marker pose for each camera setup. The tracking approaches are evaluated using ArUco markers. Our results show that mono-vision is more accurate in marker locating than stereo-vision when high resolution is used. However, this comes at the expense of higher frame processing time. Alternatively, we propose a combined low-resolution mono-stereo tracking setup that outperforms each tracking approach individually and is comparable to high resolution mono tracking, with a mean translational error of 1.8±0.6mm for 10cm marker size at 50cm distance. We further discuss our findings and their implications for navigation in surgical interventions.

Phantom evaluation of novel image guided surgery techniques enables low-cost, rapid design iteration prior to (or in place of) studies requiring biological specimens. This work introduces a novel, geometrically-accurate anatomical phantom model of surgical exposure of structures in the upper urinary tract. After segmenting the CT scan of a representative study subject, the kidneys, ureters, and associated vasculature were cast in silicone and loaded with barium to facilitate segmentation of phantom tomography. This deformable silicone model was secured to a rigid replication of the spine and retroperitoneal musculature. An acrylic housing was designed to mimic the abdominal cavity and was filled with wool batting to simulate occluding adipose tissue. Initial evaluation on CT showed good subjective correspondence with clinical tomography, as well as physiologically-relevant 25–30 mm kidney displacement between orientations, with the ability to produce larger displacements as would be expected due to intraoperative manipulation. Stereoendoscopic views of partially occluded structures during simulated dissection with the da Vinci S and Xi Surgical Systems show promise for use in developing and validating image guidance tools for surgical exposure in the abdomen.

The growing demand for radiation therapy to treat cancer has been directed to focus on improving treatment planning flow for patients. Accurate dose prediction, therefore, plays a prominent role in this regard. In this study, we propose a framework based on our newly developed scale attention networks (SA-Net) to attain voxel‐wise dose prediction. Our network ‘s dynamic scale attention model incorporates low-level details with high-level semantics from feature maps at different scales. To achieve more accurate results, we used distance data between each local voxel and the organ surfaces instead of binary masks of organs at risk as well as CT image as input of the network. The proposed method is tested on prostate cancer treated with Volumetric Modulated Arc Therapy (VMAT), where the model was training with 120 cases and tested on 20 cases. The average dose difference between the predicted dose and the clinical planned dose was 0.94 Gy, which is equivalent to 2.1% as compared to the prescription dose of 45 Gy. We also compared the performance of SA-Net dose prediction framework with different input format, the signed distance map vs. binary mask and showed the signed distance map was a better format as input to the model training. These findings show that our deep learning-based strategy of dose prediction is effectively feasible for automating the treatment planning in prostate cancer radiography.

Recently, various deep learning (DL)-based frameworks have been proposed for predicting dose distributions for radiotherapy treatment, which serve as initial dose maps and provide treatment planners with optimization starting points. This can be advantageous for dynamic treatment sites such as pancreatic cancer due to large patient to patient anatomic variations. DL-based methods thus far have used CT and contour maps to train DL networks for predictions of dose distributions. While these inputs provide important information such as density variations within a CT slice as well as spatial locations of various structures within the CT slice, these inputs do not provide radiation dose deposition information, which is typically used in inverse optimization algorithms during treatment planning to reach the desired dose distribution. In this work, we propose a new deep learning-based correction method for generating modulated dose map from non-modulated dose map, which is the dose distribution achieved from uniform beamlet intensities. Our method is consisted of two subnetworks: fusion module and correction module. Fusion module extracts features from CT and contours, including GTV contour and multi-OAR contours. Through hierarchical layers, the extracted feature map is represented as a target activation map, which can well represent the distribution of modulated dose map. To include the geometric information, the target activation map together with non-modulated dose map are then transferred into the correction module to obtain the estimated modulated dose map. Histogram matching loss with traditional losses are used. The goal of histogram matching loss is used to match the distribution of estimated modulated dose to that of ground truth modulated dose map. We performed 3-fold cross-validation on a dataset consisted of 30 patients. Our proposed method generated comparable predictions, compared to the ground truth, for 20/23 clinically relevant dose volume parameters. Overall results demonstrate the feasibility and efficacy of our proposed DL-based method for pancreas SBRT dose prediction.

During endovascular interventions, physicians have to perform accurate and immediate operations based on the available real-time information, such as the shape and position of guidewires observed on the fluoroscopic images, haptic information and the patients' physiological signals. For this purpose, real-time and accurate guidewire segmentation and tracking can enhance the visualization of guidewires and provide visual feedback for physicians during the intervention as well as for robot-assisted interventions. Nevertheless, this task often comes with the challenge of elongated deformable structures that present themselves with low contrast in the noisy fluoroscopic image sequences. To address these issues, a two-stage deep learning framework for real-time guidewire segmentation and tracking is proposed. In the first stage, a Yolov5s detector is trained, using the original X-ray images as well as synthetic ones, which is employed to output the bounding boxes of possible target guidewires. More importantly, a refinement module based on spatiotemporal constraints is incorporated to robustly localize the guidewire and remove false detections. In the second stage, a novel and efficient network is proposed to segment the guidewire in each detected bounding box. The network contains two major modules, namely a hessian-based enhancement embedding module and a dual self-attention module. Quantitative and qualitative evaluations on clinical intra-operative images demonstrate that the proposed approach significantly outperforms our baselines as well as the current state of the art and, in comparison, shows higher robustness to low quality images.

Continuous-sweep limited angle (CLA) fluoroscopy acquires x-ray images during continuous back-and-forth rotation of the C-arm allowing time-resolved 3D reconstruction of curvilinear devices. This method was previously investigated for the navigation of catheters through the airways. The purpose of this work was to extend and evaluate CLA for the guidance of percutaneous needles in the lung. In a porcine study, needles were inserted along anteroposterior, medio-lateral, and cranial-caudal trajectories during CLA imaging. The needle was segmented in the 2D projection images using a deeplearning- based segmentation approach and the corresponding 3D trajectory was reconstructed using an iterative modelbased approach to minimize the distance between a spline model and the 2D segmentations. Needle segmentation accuracies in 2D images were between 99.3 and 99.6%. The 3D reconstruction accuracies were evaluated using projection images corresponding to CLA acquisitions from CBCT scans, where the conventional CBCT reconstruction was used as reference. The highest 3D reconstruction accuracies were observed for needle trajectories parallel to the gantry rotation axis with a median [25th and 75th percentile] root mean squared distance of 0.8 mm [0.6; 1.2] and a tip localization error of 1.7 mm [0.8; 3.0] using an angular range of 10° for device reconstruction. While care must be taken to ensure that the primary extent of the device is in the direction of the C-arm axis of rotation, CLA imaging guidance would address some of the challenges experienced with conventional CT guidance such as patient access, larger coverage in CC direction, and real-time imaging feedback.

Accurate, image-based planning of joint reduction based on intraoperative cone-beam CT forms the basis for precise robotic assistance and quantitative fluoroscopic guidance. The proposed approach combines statistical shape and pose modeling of the ankle joint to: (1) automatically segment individual bones; and (2) identify the target pose for the dislocated fibula to establish a plan for reduction. Leave-one-out analysis of the atlas members demonstrated accurate segmentation with 0.6 mm mean surface distance error and predicted the fibula pose within 1.6 mm and 1.8°. Future work will expand evaluation and analyze the appropriateness of the contralateral ankle as a patient-specific template.

Brachytherapy (BT) combined with external beam radiotherapy (EBRT) is the standard treatment for cervical cancer. Accurate segmentation of the tumor and nearby organs at risk (OAR) is necessary for accurate radiotherapy (RT) planning. While OAR segmentation has been widely studied, showing promising performance, accurate tumor and/or corresponding clinical target volume (CTV) segmentation has been less explored. In cervical cancer RT, magnetic resonance (MR) imaging is used as the standard imaging modality to define the CTV, which is very challenging as the microscopic spread of tumor cells is not clearly visible even in MRI. We propose a two-step convolutional neural network (CNN) approach to delineate CTV from T2-weighted (T2W) MR images. First, a human expert needs to select a seed point inside the CTV region, from which the MR volume is cropped to produce a region of interest (ROI) volume. The ROI volume is then fed to an attention U-Net to produce CTV segmentation. A total of 213 MR datasets from 125 patients was used to develop and evaluate the proposed methodology. The network was trained using 2-dimensional (2-D) slices extracted in the axial direction from 183 MR datasets and augmented using translation operation. The proposed method was tested on the remaining 30 MR datasets and yielded Mean±SD dice similarity coefficient (DSC) of 0.80±0.06 and Hausdorff distance (95th percentile) of 3.30±0.58 mm. The performance of our method is superior to the standard U-Net-based method (pvalue< 0.005). Although the proposed method is semi-automatic, the observer variability coefficient of variation (CV) was reported as 2.86% that demonstrated the high reproducibility of the algorithm.

Typical radiation therapy for head-and-neck cancer patients lasts for more than a month. Anatomical variations often occur along the treatment course due to the tumor shrinkage and weight loss, particularly for head and neck (HN) cancer patients. To maintain the accuracy of radiotherapy beam delivery, weekly quality assurance (QA) CT is sometime acquired to monitor patients’ anatomical changes, and re-plan the treatment if needed. However, the re-plan is a labor intensive and time-consuming process, thus, decisions of re-plan are always made cautiously. In this study, we aim to develop a deep learning-based method for automated segmentation of multi-organ from CT head and neck (HN) images to rapidly evaluate the anatomical variations. Our proposed method, named detecting and boosting network, consists of one pre-trained fully convolutional one stage objection detector (FCOS) and two learnable subnetworks, i.e., hierarchical block and mask head. FCOS is used to extract informative features from CT and locate the volume-of-interest (VOIs) of multiple organs. Hierarchical block is used to enhance the feature contrast around organ boundary and thus improve the ability of organ classification. Mask head then segment organ from the refined feature map within the VOIs. We conducted a five-fold cross-validation on 35 patients’ cases who have multiple weekly CT scans (over 100 QACTs) during their radiotherapy. The 11 organs were segmented and compared with manual contours using several segmentation measurements. The mean Dice similarity coefficient (DSC) values of 0.82, 0.82, and 0.81 were achieved along the treatment course for all the organs. These results demonstrate the feasibility and efficacy of our proposed method for multi-OAR segmentation from HN CT, which can be used for rapid evaluate the anatomical variations in HN radiation therapy.

During breast cancer surgery, there are several modalities that a clinician can choose from to detect and localize cancerous tissue intraoperatively, with one of the most common being non-imaging gamma probes. However, deficiencies in the existing modalities make it difficult for clinicians to completely resect the tumour with clean margins, meaning the patient may have to undergo a revision surgery. We describe a novel, dual-modality image guidance system comprising a non-imaging, focussed gamma probe and an ultrasound transducer that simultaneously acquires molecular and anatomical data for a complete surgical guidance system. The custom-designed focussed gamma probe features a highly convergent collimator that achieves high-resolution in a remote focal region. Monte Carlo simulations show a 3.5 mm full width at half maximum resolution and a maximum sensitivity of approximately 7.8 cps/kBq for a 45 mm focal length collimator. The unique focal point design of the gamma probe enables the region with the highest resolution to be aligned within the ultrasound imaging plane for simultaneous, dual-modality acquisition. The first proof-of-concept computer-aided design for the hybrid imaging system is presented. Simultaneous acquisition was realized using a custom-designed holder that allows the focussed gamma probe and any commercially available ultrasound probe to be connected into a single, hand-held unit. The system also contains a translational stage that allows the focal point of the gamma probe to be adjusted axially within the ultrasound plane for optimal assessment. The development of this novel, dual-modality image guidance system should facilitate more accurate real-time, intraoperative tumour margin assessment.

Quantitative thoracic dynamic magnetic resonance imaging (QdMRI), a recently developed technique, provides a potential solution for evaluating treatment effects in thoracic insufficiency syndrome (TIS). In this paper, we integrate all related algorithms and modules during our work from the past 10 years on TIS into one system, named QdMRI, to address the following questions: (1) How to effectively acquire dynamic images? For many TIS patients, subjects are unable to cooperate with breathing instructions during image acquisition. Image acquisition can only be implemented under freebreathing conditions, and it is not feasible to use a surrogate device for tracing breathing signals. (2) How to assess the thoracic structures from the acquired image, such as lungs, left and right, separately? (3) How to depict the dynamics of thoracic structures due to respiration motion? (4) How to use the structural and functional information for the quantitative evaluation of surgical TIS treatment and for the design of the surgery plan? The QdMRI system includes 4 major modules: dynamic MRI (dMRI) acquisition, 4D image construction, image segmentation (from 4D image), and visualization of segmentation results, dynamic measurements, and comparisons of measurements from TIS patients with those from normal children. Scanning/image acquisition time for one subject is ~20 minutes, 4D image construction time is ~5 minutes, image segmentation of lungs via deep learning is 70 seconds for all time points (with the average DICE 0.96 in healthy children), and measurement computation time is 2 seconds.

PURPOSE: Over 30% of breast conserving surgery patients must undergo repeat surgery to address incomplete tumor resection. We hypothesize that the addition of a robotic cavity scanning system can improve the success rates of these procedures by performing additional, intraoperative imaging to detect left-over cancer cells. In this study, we assess the feasibility of a combined optical and acoustic imaging approach for this cavity scanning system. METHODS: Dual-layer tissue phantoms are imaged with both throughput broadband spectroscopy and an endocavity ultrasound probe. The absorbance and transmittance of the incident light from the broadband source is used to characterize each tissue sample optically. Additionally, a temporally enhanced ultrasound approach is used to distinguish the heterogeneity of the tissue sample by classifying individual pixels in the ultrasound image with a support vector machine. The goal of this combined approach is to use optical characterization to classify the tissue surface, and acoustic characterization to classify the sample heterogeneity. RESULTS: Both optical and acoustic characterization demonstrated promising preliminary results. The class of each tissue sample is distinctly separable based on the transmittance and absorption of the broadband light. Additionally, an SVM trained on the temporally enhance ultrasound signals for each tissue type, showed 82% linear separability of labelled temporally enhanced ultrasound sequences in our test set. CONCLUSIONS: By combining broadband and ultrasound imaging, we demonstrate a potential non-destructive imaging approach for this robotic cavity scanning system. With this approach, our system can detect both surface level tissue characteristics and depth information. Applying this to breast conserving surgery can help inform the surgeon about the tissue composition of the resection cavity after initial tumor resection.

Echocardiography (echo) is one of the widely used imaging techniques to evaluate cardiac function. Left ventricular ejection fraction (EF) is a commonly assessed echocardiographic measurement to study systolic function and is a primary index of cardiac contractility. EF indicates the percentage of blood volume ejected from the left ventricle in a cardiac cycle. Several deep learning (DL) works have contributed to the automatic measurements of EF in echo via LV segmentation and visual assessment,^1-8 but still the design of a lightweight and robust video-based model for EF estimation in portable mobile environments remains a challenge. To overcome this limitation, here we propose a modified Tiny Video Network (TVN) with sampling-free uncertainty estimation for video-based EF measurement in echo. Our key contribution is to achieve comparable accuracy with the contemporary state-of-the-art video-based model, Echonet-Dynamic approach¹ while having a small model size. Moreover, we consider the aleatoric uncertainty in our network to model the inherent noise and ambiguity of EF labels in echo data to improve prediction robustness. The proposed network is suitable for real-time video-based EF estimation compatible with portable mobile devices. For experiments, we use the publically available Echonet-Dynamic dataset¹ with 10,030 four-chamber echo videos and their corresponding EF labels. The experiments show the advantages of the proposed method in performance and robustness.

The quality of neurosurgical planning can become compromised by soft tissue deformations that occur during surgery (i.e., brain shift). Conventional image guidance systems do not consider these intraoperative changes. In recent efforts, model based strategies have been developed to estimate surgical load displacements and modify the patient’s data intraoperatively to account for brain shift. While the efficacy of the model has been previously established, there is also an opportunity to further assist surgical planning with this pipeline. To address this, a mobile application designed for an Android tablet was developed to display simulated brain shifts that would occur during brain tumor surgery. The application has two primary functions to facilitate planning: a patient positioning mode and a simulation mode. The patient positioning mode allows the neurosurgeon to load the patient’s preoperative MR data and create a surgical plan (i.e., head orientation and craniotomy location) for the procedure. The simulation mode then displays both the preoperative data and the model predicted brain shift as a function of the specified orientation from the patient positioning mode. Additionally, to account for positional variations between planning and procedural implementation, the simulation mode also displays solutions with additional perturbations to the planned positioning to estimate shift possibilities. To assess the simulation mode prototype, practicing neurosurgeons were provided a prototype demonstration and interviews were performed to evaluate efficacy and design. Due to computational rendering and 3D rotation shortcomings based on clinical feedback, the prototype was redesigned into a full mixed reality simulation app on the Microsoft HoloLens. Preliminary survey responses show that the prototype could be an impactful surgical planning tool, especially among neurosurgeons with less experience.

PURPOSE: As medical education adopts a competency-based training method, experts are spending substantial amounts of time instructing and assessing trainees’ competence. In this study, we look to develop a computer-assisted training platform that can provide instruction and assessment of open inguinal hernia repairs without needing an expert observer. We recognize workflow tasks based on the tool-tissue interactions, suggesting that we first need a method to identify tissues. This study aims to train a neural network in identifying tissues in a low-cost phantom as we work towards identifying the tool-tissue interactions needed for task recognition. METHODS: Eight simulated tissues were segmented throughout five videos from experienced surgeons who performed open inguinal hernia repairs on phantoms. A U-Net was trained using leave-one-user-out cross validation. The average F-score, false positive rate and false negative rate were calculated for each tissue to evaluate the U-Net’s performance. RESULTS: Higher F-scores and lower false negative and positive rates were recorded for the skin, hernia sac, spermatic cord, and nerves, while slightly lower metrics were recorded for the subcutaneous tissue, Scarpa’s fascia, external oblique aponeurosis and superficial epigastric vessels. CONCLUSION: The U-Net performed better in recognizing tissues that were relatively larger in size and more prevalent, while struggling to recognize smaller tissues only briefly visible. Since workflow recognition does not require perfect segmentation, we believe our U-Net is sufficient in recognizing the tissues of an inguinal hernia repair phantom. Future studies will explore combining our segmentation U-Net with tool detection as we work towards workflow recognition.

Artificial Intelligence (AI) has revolutionized medical imaging by producing state-of-the-art results in various tasks with performance comparable to human readers. However, AI models are often task- and data-specific, requiring input images with specific modalities and sequences. Although DICOM metadata can give provide such information about the data, it is inconsistent, non-standard with many missing values in different fields of metadata. In this paper, we present a deep learning-based unsupervised clustering algorithm to group radiological images solely based on their pixel data. We start by extracting semantically meaningful features through a representation learning approach, and we calculate the nearest neighbors of the images. Next, the image and its neighbors are used as inputs for training an unsupervised clustering algorithm. A human reader then labels each of the clusters by examining the modality, body part, and orientation. At the test time, after using the trained clustering model to identify the group the image belongs to, we can assign consistent and correct metadata labels. Experimental evaluation shows that we can successfully cluster images with 90% accuracy without using any labeled data. We envision our approach to be utilized as a pre-processing step for a variety of tasks that require determining the underlying nature of the data in the DICOM file, even if the metadata is incorrectly formatted.

Purpose: Computer-assisted surgical skill assessment methods have traditionally relied on tracking tool motion with physical sensors. These tracking systems can be expensive, bulky, and impede tool function. Recent advances in object detection networks have made it possible to quantify tool motion using only a camera. These advances open the door for a low-cost alternative to current physical tracking systems for surgical skill assessment. This study determines the feasibility of using metrics computed with object detection by comparing them to widely accepted metrics computed using traditional tracking methods in central venous catheterization. Methods: Both video and tracking data were recorded from participants performing central venous catheterization on a venous access phantom. A Faster Region-Based Convolutional Neural Network was trained to recognize the ultrasound probe and syringe on the video data. Tracking-based metrics were computed using the Perk Tutor extension of 3D Slicer. The path length and usage time for each tool were then computed using both the video and tracking data. The metrics from object detection and tracking were compared using Spearman rank correlation. Results: The path lengths had a rank correlation coefficient of 0.22 for the syringe (p<0.03) and 0.35 (p<0.001) for the ultrasound probe. For the usage times, the correlation coefficient was 0.37 (p<0.001) for the syringe and 0.34 (p<0.001) for the ultrasound probe. Conclusions: The video-based metrics correlated significantly with the tracked metrics, suggesting that object detection could be a feasible skill assessment method for central venous catheterization.

To develop an objective system for perfusion assessment in digital subtraction angiography (DSA), artery-vein (A/V) classification is essential. In this study, an automated A/V classification system in 2D DSA images of stroke patients is proposed.

After preprocessing through vessel segmentation with a Frangi fitler and Gaussian smoothing, a time-intensity curve (TIC) of each vessel pixel was extracted and relevant parameters were calculated. Different combinations of input parameters were systematically tested to come to the optimal set of input parameters. The parameters formed the input for k-means (KM) and fuzzy c-means (FCM) clustering. Both algorithms were tested for clustering into 2 to 7 clusters. Cluster labeling was performed based on the average time to peak (TTP) of a cluster.

A reference standard consisted of manually annotated DSA images of the MR CLEAN registry. Outcome measures were accuracy, true artery rate (TAR) and true vein rate (TVR).

The optimal value for k was found to be 2 for both KM and FCM clustering. The optimal parameter set was: variance, standard deviation, maximal slope, peak width, time to peak, arrival time, maximal intensity and area under the TIC. No significant difference was found between FCM and KM clustering and Both FCM and KM clustering yielded an average accuracy of 76%, average TAR of 74% and average TVR of 80%.

Stereoelectroencephalography (SEEG) is a minimally invasive surgical procedure, used in the treatment of pharmacoresistant epilepsy to precisely locate areas of the brain where seizures originate. An accurate localization of SEEG electrodes is crucial to design a resection plan before surgically removing epileptogenic zone. We propose to train a deep neural network to accurately segment electrode contacts without making any manual adjustments. We trained a 2D and a 3D version of the U-Net^{1, 2} neural network architecture to handle this task, taking postoperative CT scans as input. We evaluated our models on 18 image datasets of patients using different metrics, and provided a comparison of the two approaches. The presented models are robust to electrode bending and do not need any prior information to make quick and accurate predictions. To the best of our knowledge, deep learning has not been used yet for this task.

For opening the blood brain barrier using focused ultrasound (FUS) to treat neurodegenerative diseases, mouse- specific therapy planning is an essential step. For our therapy planning approach based on acoustic simulations we here propose to automatically segment the mouse skull and brain from magnetic resonance imaging, which is usually used in combination with FUS for monitoring purposes. The proposed method consists of (1) pre- processing to enhance the image contrast and remove noise, (2) a rough skull segmentation using morphological operations and adaptive binarization, (3) segmentation of the brain using the established 3D-PCNN method, (4) correction of the skull segmentation using the anatomical information about the brain location and (5) a post-processing to remove obvious errors from the final skull segmentation. The method is evaluated with four in-vivo datasets obtained with different parameters. The median Matthews Correlation Coefficient (MCC) on all slices of four datasets was 0.85 for the brain segmentation, 0.69 for the overall skull segmentation and 0.78 for the skull cap. Finally for showcasing the application an acoustic simulation based on the segmentation is presented, which results in a comparable prediction of the pressure field prediction as our earlier method based on micro-CT, and lines up well with literature estimations of the ultrasound attenuation.

Endovascular interventions have experienced a very important development in recent years. In the particular context of cerebral stroke, mechanical thrombectomy has become a therapeutic of reference. Nevertheless, catheterization involves a technical gesture that can be very difficult or impossible in complex anatomical configurations. Preoperative images can help physicians in planning via visualization of structures of interest and during the intervention via augmented reality navigation. To guide the intervention, segmented navigation structures from pre-operative images are projected onto 2D intraoperative X-ray images. To this aim, common and internal carotids and aortic arch, that correspond to endovascular path, must be quickly and properly segmented. In this this work we propose a method for segmentation of vascular structures corresponding to endovascular path in mechanical thrombectomy from pre-operative images. Cascaded U-net are used to segment the common and internal arteries: a first U-net is used to mask the original MRA and a second U-net is used to segment the arteries. Another U-net is used to segment the aortic arch. The proposed method better highlights vascular structures related to endovascular navigation than a simple 2D U-net. Also, Dice coefficient of segmented vascular structures is equal to 0.851 with a sensitivity equal to 0.922.

Varicose veins are classified as a chronic venous disease of which almost a quarter of the population of the U.S suffers from.1 Although most cases only develop mild symptoms, 6% of the affected women and men between 40 and 80 years develop signs of chronic vein insufficiency like venous ulceration.2 The number of these patients is two million in the U.S. alone. Treatment of varicose veins was mostly composed of surgical interventions until thermal endovenous ablation was introduced3 which resulted in lower cost and faster recovery of the patient.2 A new completely non-invasive method is High-Intensity Focused Ultrasound (HIFU) in which an ultrasound pulse is applied from outside the skin surface in order to thermally ablate the vein and close it permanently.3 This method relies heavily on diagnostic imaging through ultrasound to detect the target vein for ablation and to guide and monitor the procedure. An automated approach to detect and localize the vein during the treatment is rational because of the tedious work to follow the vessel in transversal direction. Previous works in the field of vessel segmentation in ultrasound images with deep learning focus on the frame-wise segmentation of the vessel.4 The possibility of further improvement of this method can be achieved by leveraging the temporal information about the location of the vessel. A previous work proposed by Mathai et. al.5 also features a U-net which implements LSTM-layers in the decoder part of the network and is used for the segmentation of vessels in ultrasound images. The segmentation of ultrasound image sequences can be combined with the prediction of segmentations of future frames to improve the predictive capacity of the model. Zhao et. al. proposed to use a ConvLSTM to predict future frames of ultrasound images for tongue movement,6 which was successful in predicting the next ultrasound image for a sequence of eight frames. In this work we propose a deep learning method for the localization and segmentation of veins in ultrasound sequences in combination with the prediction of future vessel segmentations for the automation of HIFU ablation treatments.

Correct understanding of the geometry of the glenoid (the socket of the shoulder joint) is key to successful planning of shoulder replacement surgery. This surgery typically involves placing an implant in the shoulder joint to restore joint function. The most relevant geometry is the glenoid version, which is the angular orientation of the glenoid surface relative to the long axis of the scapula in the axial plane. However, measuring the glenoid version is not straightforward and there are multiple measurement methods in the literature and used in commercial planning software.

In this paper we introduce SciKit-SurgeryGlenoid, an open source toolkit for the measurement of glenoid version. SciKit-SurgeryGlenoid contains implementations of the 4 most frequently used glenoid version measurement algorithms enabling easy and unbiased comparison of the different techniques. We present the results of using the software on 10 sets of pre-operative CT scans taken from patients who have subsequently undergone shoulder replacement surgery. We further compare these results with those obtained from a commercial implant planning software.

SciKit-SurgeryGlenoid currently requires manual segmentation of the relevant anatomical features for each method. Future work will look at automating the segmentation process to build an automatic and repeatable pipeline from CT or radiograph to quantitative glenoid version measurement.

Motivation: Aim of this project is the automatic classification of total hip endoprosthesis (THEP) components in 2D Xray images. Revision surgeries of total hip arthroplasty (THA) are common procedures in orthopedics and trauma surgery. Currently, around 400.000 procedures per year are performed in the United States (US) alone. To achieve the best possible result, preoperative planning is crucial. Especially if parts of the current THEP system are to be retained.

Methods: First, a ground truth based on 76 X-ray images was created: We used an image processing pipeline consisting of a segmentation step performed by a convolutional neural network and a classification step performed by a support vector machine (SVM). In total, 11 classes (5 pans and 6 shafts) shall be classified.

Results: The ground truth generated was of good quality even though the initial segmentation was performed by technicians. The best segmentation results were achieved using a U-net architecture. For classification, SVM architectures performed much better than additional neural networks.

Conclusions: The overall image processing pipeline performed well, but the ground truth needs to be extended to include a broader variability of implant types and more examples per training class.

Fully supervised learning approaches for surgical instrument segmentation from video images usually require a time-consuming process of generating accurate ground truth segmentation masks. We propose an alternative way of labeling surgical instruments for binary segmentation that first commences with rough, scribble-like annotations of the surgical instruments using a disc-shaped brush. We then present a framework that starts with a graph-model-based method for generating initial segmentation labels based on the user-annotated paint-brush scribbles and then proceeds with a deep learning model that learns from the noisy, initial segmentation labels. Experiments conducted on the 2017 MICCAI EndoVis Robotic Instrument Segmentation Challenge have shown that the proposed framework achieved a 76.82% IoU and 85.70% Dice score on binary instrument segmentation. Based on these metrics, the proposed method out-performs other weakly supervised techniques and achieves a close performance to that achieved via fully supervised networks, but eliminates the need for ground truth segmentation masks.

Deep learning-based pose estimation algorithms can successfully estimate the pose of objects in an image, especially in the field of color images. 6D Object pose estimation based on deep learning models for X-ray images often use custom architectures that employ extensive CAD models and simulated data for training purposes. Recent RGB-based methods opt to solve pose estimation problems using small datasets, making them more attractive for the X-ray domain where medical data is scarcely available. We refine an existing RGB-based model (SingleShotPose) to estimate the 6D pose of a marked cube from grayscale X-ray images by creating a generic solution trained on only real X-ray data and adjusted for X-ray acquisition geometry. The model regresses 2D control points and calculates the pose through 2D/3D correspondences using Perspective-n-Point (PnP), allowing a single trained model to be used across all supporting cone-beam-based X-ray geometries. Since modern X-ray systems continuously adjust acquisition parameters during a procedure, it is essential for such a pose estimation network to consider these parameters in order to be deployed successfully and find a real use case. With a 5-cm/5-degree accuracy of 93% and an average 3D rotation error of 2.2 degrees, the results of the proposed approach are comparable with state-of-the-art alternatives, while requiring significantly less real training examples and being applicable in real-time applications.

Given the prevalence of cardiovascular diseases (CVDs), the segmentation of the heart on cardiac computed tomography (CT) remains of great importance. Manual segmentation is time-consuming and intra-and inter-observer variabilities yield inconsistent and inaccurate results. Computer-assisted, and in particular, deep learning approaches to segmentation continue to potentially offer an accurate, efficient alternative to manual segmentation. However, fully automated methods for cardiac segmentation have yet to achieve accurate enough results to compete with expert segmentation. Thus, we focus on a semi-automated deep learning approach to cardiac segmentation that bridges the divide between a higher accuracy from manual segmentation and higher efficiency from fully automated methods. In this approach, we selected a fixed number of points along the surface of the cardiac region to mimic user interaction. Points-distance maps were then generated from these points selections, and a three-dimensional (3D) fully convolutional neural network (FCNN) was trained using points-distance maps to provide a segmentation prediction. Testing our method with different numbers of selected points, we achieved a Dice score from 0.742 to 0.917 across the four chambers. Specifically. Dice scores averaged 0.846 ± 0.059, 0.857 ± 0.052, 0.826 ± 0.062, and 0.824 ± 0.062 for the left atrium, left ventricle, right atrium, and right ventricle, respectively across all points selections. This point-guided, image-independent, deep learning segmentation approach illustrated a promising performance for chamber-by-chamber delineation of the heart in CT images.

Cochlear implants (CIs) are neural prosthetics used to improve hearing in patients with severe-to-profound hearing loss. After implantation, the process of fine-tuning the implant for a specific patient is expedited if the audiologist has tools to approximate which auditory nerve fiber regions are being stimulated by the implant’s electrode array. Auditory nerves travel from the cochlea where the prosthetic is implanted to the brain via the internal auditory canal (IAC). In this paper, we present a method for segmenting the IAC in a CT image using weakly supervised 3D UNets. Our approach is to train a U-Net with a custom loss function to refine a localization provided by a previously proposed active-shape-model-based IAC segmentation method. Preliminary results indicate that our proposed approach is successful in refining IAC localization, which is an important step towards accurate auditory nerve fiber localization for neural activation modeling.

Proton FLASH therapy can deliver ultra-high dose rate radiation to tumor targets compared to conventional radiotherapy, and it has the potential to enable the paradigm shift of radiation oncology. Due to the inter- and intra-variation of respiratory motion, it is highly desired to accurately localize the target before the FLASH treatment delivery. This proof-of-concept study proposes a novel deep learning-based method to estimate tumor location from two orthogonal 2D kV on-board images. This approach potentially can enable real-time tumor localization and tracking to ensure the effectiveness of FLASH treatment. Our proposed method, central matching networks (CMN), can collect feature maps that represent the probability distribution of tumors, called center-ness maps, from 2D kV images and re-align them to their projections’ angle under the Cartesian coordinate system. Since the center-ness maps are inferred from two orthogonal projections, the depth information is thus obtained. CMN then re-transforms the 2D center-ness maps to the 3D center-ness map via depth inference. We conducted a simulation study with 180 orthogonal 2D kV projections from 4D computed tomography (CT) of 9 patients to test our method. For each 3D CT image set of a breathing phase, we simulated its 2D projections using two orthogonal angles. Then a leave-patient-out method was applied to train CMN to ensure the robustness of models. The model performance is quantified by Euclidean distances of the center-of-mass between the ground truth and predicted tumor locations in 3D space. CMN results in the Euclidean distance of 2.0±0.9 mm for all patients. The results demonstrate the feasibility and efficacy of CMN for 3D tumor localization, which has the potential to achieve in-treatment target localization on the fly.

Currently, 3T hip MRI can be used to estimate femur strength and cortical bone thickness. One of the major hurdles in this application is that objects (osseous structures) are manually segmented which involves significant human labor. In this study, we propose an automatic and accurate algorithm for osseous structure segmentation from hip 3T MRI by using a deep convolutional neural network. The approach includes two stages: 1) automatic localization of acetabulum and femur by using the femoral head as a reference, and 2) 2D bounding box (BB) set up for each object based on the localization information from femoral head followed by a UNet to segment the target object within the BB. 90 3T hip MRI image data sets were utilized in this study that were divided into training, validating, and testing groups (60%:20%:20%), and a 5-fold cross-validation was adopted in the procedure. The study showed that automated segmentation results were comparable to the reference standard from manual segmentation. The average Dice Coefficient for acetabular and femoral (i.e., cortical and medullary bone plus bone marrow) segmentation was 0.93 and 0.96, respectively. Segmentations of acetabular and femoral medullary cavity (i.e., medullary bone plus bone marrow) had Dice Coefficient of 0.89 and 0.95, respectively. Acetabular and femoral cortical bone segmentations were more challenging with lower Dice Coefficient of around 0.7. The proposed approach is automatic and effective without any interaction from humans. The idea of using local salient anatomy to guide object localization approaches is heuristic and can be easily generalized to other localization problems in practice.

Tumor localization is a critical step in head & neck cancer diagnosis and treatment. It allows for treatment planning and accurate projection of tumor spread. In this study, our goal is to develop a fully automated hybrid neural network (HNN) for localization and segmentation of PET/CT images through a combination of Faster-RCNN and U-Net. HNN consists of two deep neural networks, they are (1) Faster-RCNN which is a leading model in object detection is used to localize tumors in PET images, and (2) U-Net which has achieved great success in medical image segmentation is used for tumor segmentation in the HNN. The testing stage consists of two steps (1) tumor localization and segmentation, where each PET slice was fed into the trained Faster-RCNN to produce a class label and bounding box coordinates, and then the CT slice was cropped according to the predicted bounding box and passed to the U-Net for segmentation; (2) the predicted mask is post-processed by applying OTSU thresholding method. The experimental results demonstrated that HNN can obtain promising segmentation performance.

Recent recognition of the poor prognosis of significant tricuspid regurgitation (TR) has resulted in increased indication for tricuspid valve interventions. TR can affect 65% to 85% of the population worldwide and has a 1-year mortality rate greater than 25% in patients with severe regurgitation. A key procedure for patient selection and intraoperative assessment of intervention involves determining the location of the vena contracta width and regurgitant jet area. Manual visualization of the vena contracta (VC) can be time consuming depending on its location. Decreasing the required time for VC visualization would potentially result in decreased time for patient assessment, device deployment and intraprocedural intervention evaluation thus reducing hospital costs. There is currently no commercially available automatic VC detection system. We present a method to automatically localize the VC using 3D intracardiac echocardiography (ICE) on a simplified anthropomorphic phantom as a proof of concept. A beating heart phantom was outfitted with a silicone flange containing a mechanical valve and an orifice to cause a regurgitant jet. We propose an image processing pipeline to segment the regurgitant jet from Doppler ultrasound, as acquired by ICE, to determine the location of the VC automatically. The VC locations output by the algorithm were validated both qualitatively and quantitatively by comparison to manually annotated VC locations. On average, the location of VC detected by the algorithm was within 1.52±35mm to the location of the ground truth VC. We envision that this study will play a major role towards the development of an automated system for VC localization during TV interventions.

Vascular navigation is a prerequisite to transcatheter cardiac interventions. The current standard approach to catheter navigation relies on real-time fluoroscopy, while this technique utilizes ionizing radiation and it places the interventionalist at risk for eye cataracts and cancer. The shielding equipment needed to mitigate these risks is associated with spinal issues and neck and back pain which has led towards the coining of the term “interventionalist’s disc disease”. A proposed alternative is to have an ultrasound-guided vascular navigation system where a catheter-based ultrasound probe scans the vessel and reconstructs the vascular roadmap, which can then be navigated by tracked guidewire or catheter. One of the major challenges here is the segmentation of the vessel lumen from the ultrasound images. In this study, we address this challenge using a deep learning based approach. We acquired inferior vena cava (IVC) images from an animal study performed using a radial, forward-looking Foresight intracardiac echocardiography (ICE) ultrasound probe. The ground truth was established using manual segmentations and validated by an expert clinician. We use the MONAI platform to train a U-net architecture on our dataset to perform vessel segmentation. The images are cropped to retain only the central 300 pixels as the traversed vessel will always appear central to the radial ICE image. Data augmentation was performed to enhance the number of images available for training. After post-processing, the segmentation output, a 90 % accuracy was achieved as indicated by the Dice coefficient. We plan on integrating this vessel segmentation pipeline in an image-guided surgical navigation system.

The laparoscopic treatment of gallstone is a complex procedure requiring multiple skills, notably laparoscopic ultrasounds imaging and laparoscopic suturing. This procedure has many benefits comparing to open surgery, but still fail to be generalised because of these challenging steps. Surgical simulation can provide a useful way to train for the complex skills required by the surgery in a safe environment.

This study focuses on the evaluation of the utilisation of silicone to create a simulator for this procedure. Silicone samples are tested quantitively and qualitatively. The qualitative evaluation compares the ultrasound examination of the silicone samples to the images of ultrasound captured during a real procedure. For the quantitative evaluation, the density, the speed of sound in the material, and the mechanical properties are measured and compared to the properties of the soft tissues.

The qualitative evaluation shows that silicone requires the addition of scattering agents to be visible using ultrasound, especially for the hardest silicones which are the most suitable to simulate the bile duct in term of mechanical properties. The quantitative evaluation shows that the density of silicone and the Young modulus are close to the properties of soft tissues, but the speed of sound is slower in the silicone, which results in deformed images. A solution is to perform image processing, which can lead to a more realistic appearance of the images.

Percutaneous Nephrolithotomy (PCNL), is a minimally invasive surgical procedure for removal of kidney stones typically >1cm. The procedure involves inserting a needle through the patient’s skin into the kidney which is being more commonly performed now using ultrasound (US) guidance. Existing US image-guided needle insertion employed in PCNL faces the challenge in terms of keeping the needle tip visible during the insertion process. We propose a needle insertion mechanism with mirror-integrated US imaging, which provides an intuitive and simple solution to monitor the needle insertion path. This is achieved by using acoustic mirror to change the direction of the US image plane while the needle goes through the spacing in the middle of the acoustic mirror so that the needle path aligns with the US image plane. Both the needle and the acoustic mirror are designed to be rotatable to provide the clinician with the flexibility to search for the optimal needle insertion orientation. According to the law of acoustic wave reflection, the needle should rotate two times the amount of the mirror to keep aligned with the US image plane. A synchronization mechanism consisting of belts and pulleys was designed to achieve this 2:1 rotation ratio. Needle-mirror synchronized rotation is tested using an image-processing-based method. In terms of imaging functionality, US images display point targets inside the gelatin phantom as well as the needle tip clearly. In the needle insertion experiment, a needle is inserted into the gelatin phantom to reach point targets, and results show insertion errors <3mm. Overall, our results demonstrate the potential of using the proposed US image-guided access mechanism in clinical scenarios.

Several novel autonomous lighting systems for illuminating the surgical site (e.g. SmartOT) consist of a large number of swiveling light modules placed on the ceiling instead of two or three movable OR lights. For such a new type of lighting system for operating rooms, the initial placement of the light modules is of great importance, since the light modules cannot be moved during surgery. For this reason, we present a novel approach for optimizing the arrangement of light modules in a new autonomous lighting system that exploits the special characteristics of an operating room and the surgeries that take place there by taking into account occluding geometry (e.g., surgeons and medical staff) via point cloud recordings.

We have implemented a function that provides the brightness at the surgical site for an arbitrary arrangement of light modules at a given time of a point cloud recording. Based on this brightness function, we defined a fitness function which we used for optimization. Using cross-validation, we compared the results of three optimization algorithms (Greedy, SGA, SADE) to naive arrangements of light modules. By performing these optimizations on point cloud recordings of nine real open abdominal surgeries, we obtain arrangements that achieve up to 41% higher minimum brightness during surgery compared to naive arrangements.

Purpose: Radiotherapy (RT) is widely used for cancer management. The standard RT clinical workflow imposes substantial burdens on patients. Multiple image acquisitions for diagnosis and RT planning increase the travel, cost, and wait time before actual RT treatment, which is critical for cancer patients. Diagnostic CT (dCT) is shown to be suitable for RT planning, however different planning CT (pCT) acquisition setup (e.g., table curvature) and motion management procedure (e.g., deep inspiration/active breath control) makes it infeasible. In this study, we present the feasibility of a fully automatic image adaptation method to omit the need for pCT and expedite the treatment course at institutions where on-table adaptive radiotherapy is available.

Methods: We designed a 3D convolutional neural network (3DCNN) to perform the dCT to pCT image adaptation. An automatic table removal algorithm was developed to isolate the patient's body without altering the curvature. The patient body isolated dCT and pCT pair were, then, deformably registered (rCT). The Network was trained on pCTs and the 3D displacement fields (DF). We evaluated the performance of our method using the root averaged sum of squared differences (RASSD), and body contour similarity using Dice similarity coefficient (DSC) and Hausdorff distance (HD).

Results: The generated DFs were applied to dCT to generate the synthetic pCTs (sCT). For test cases, the difference of RASSDs, DSCs, and HD between (rCT-vs-pCT) and (sCT-vs-pCT) were 6 HU, 0.02 and 0.3 mm, respectively.

Conclusion: A novel 3DCNN based method was presented to adapt the dCT to pCT, omitting the need for acquiring multiple scans before treatment delivery, reducing the cost and length of the RT treatment pathway.

Surgical excision for basal cell carcinoma (BCC) is a common treatment to remove the affected areas of skin. Minimizing positive margins around excised tissue is essential for successful treatment. Residual cancer cells may result in repeat surgery; however, detecting remaining cancer can be challenging and time-consuming. Using chemical signal data acquired while tissue is excised with a cautery tool, the iKnife system can discriminate between healthy and cancerous tissue but lacks spatial information, making it difficult to navigate back to suspicious margins. Intraoperative videos of BCC excision allow cautery locations to be tracked, providing the sites of potential positive margins. We propose a deep learning approach using convolutional neural networks to recognize phases in the videos and subsequently track the cautery location, comparing two localization methods (supervised and semi-supervised). Phase recognition was used for preprocessing to classify frames as showing the surgery or the start/stop of iKnife data acquisition. Only frames designated as showing the surgery were used for cautery localization. Fourteen videos were recorded during BCC excisions with iKnife data collection. On unseen testing data (2 videos, 1,832 frames), the phase recognition model showed an overall accuracy of 86%. Tool localization performed with a mean average precision of 0.98 and 0.96 for supervised and semisupervised methods, respectively, at a 0.5 intersection over union threshold. Incorporating intraoperative phase data with tool tracking provides surgeons with spatial information about the cautery tool location around suspicious regions, potentially improving the surgeon's ability to navigate back to the area of concern.

Purpose: Computer-assisted skill assessment has traditionally been focused on general metrics related to tool motion and usage time. While these metrics are important for an overall evaluation of skill, they do not address critical errors made during the procedure. This study examines the effectiveness of utilizing object detection to quantify the critical error of making multiple needle insertion attempts in central venous catheterization. Methods: 6860 images were annotated with ground truth bounding boxes around the syringe attached to the needle. The images were registered using the location of the phantom, and the bounding boxes from the training set were used to identify the regions where the needle was most likely inserting the phantom. A Faster region-based convolutional neural network was trained to identify the syringe and produce the bounding box location for images in the test set. A needle insertion attempt began when the location of the predicted bounding box fell within the identified insertion region. To evaluate this method, we compared the computed number of insertions to the number of insertions identified by human reviewers. Results: The object detection network had an overall mean average precision (mAP) of 0.71. This tracking method computed an average of 4.40 insertion attempts per recording compared to a reviewer count of 1.39 attempts per recording. Conclusions: The difference in the number of insertion attempts identified by the computer and reviewers decreases with an increasing mAP, making this method suitable for detecting multiple needle insertions using an object detection network with a high accuracy.

Accurate identification of tumor boundaries during brain cancer surgery determines the quality of life of the patient. Different intraoperative guidance tools are currently employed during the resection tumor but having several limitations. Hyperspectral imaging (HSI) is arising as a label-free and non-ionizing technique that could assist neurosurgeons during surgical procedures. In this paper, an analysis between in-vivo and ex-vivo human brain tumor samples using HSI has been performed to evaluate the correlation between both types of samples. Spectral ratios of the oxygenated and deoxygenated hemoglobin were employed to distinguish between normal tissue, tumor tissue and blood vessels. A database composed by seven in-vivo and fourteen ex-vivo hyperspectral images obtained from seven different patients diagnosed with glioblastoma Grade IV, metastatic secondary breast cancer, meningioma Grade I and II, and astrocytoma (glioma) Grade II. 44,964 pixels labeled pixels were employed in this work. The proposed method achieved discrimination between different tissue types using the proposed spectral ratio. Comparison between in-vivo and ex-vivo samples indicated that ex-vivo samples generate higher hemoglobin ratios. Moreover, vascular enhanced maps were generated using the spectral ratio, targeting real-time intraoperative surgical assistance.

Transcranial therapy with focused ultrasound under the control of magnetic resonance imaging (tcMRgFUS) enables targeted thermal ablation of brain tissue, for example for movement disorders such as tremor in Parkinson's disease. Fiber tracking can serve as a tool to delineate therapy-relevant pathways in the brain to determine the ablation target and to avoid damage to neighboring structures. We apply a fiber tracking algorithm which relies on the definition of regions of interest (ROIs) used as seed points or waypoints to optimize the tracking precision. It adapts its parameters locally to the current location of the fiber to be reconstructed within a white matter atlas. We propose to fully automate the fiber tracking pipeline using a deep learning-based segmentation of 20 ROIs, trained on T1 images and color-coded direction maps from diffusion tensor imaging (DTI). The training reference data is generated automatically using ROIs registered from an atlas. The resulting U-Nets can segment all required ROIs, also on independent test data, and are more robust than atlas registration for selected ROIs, i.e. the precentral gyrus. The fiber tracts computed using ROIs from DL segmentation versus atlas registration are similar. We expect to further improve the patient-individual ROIs and resulting fiber tracts by using curated ground truth reference data for future trainings.

Background: Successful navigation in spine surgeries relies on accurate representation of the spine’s interoperative pose. However, its position can move between preoperative imaging and instrumentation. A measure of this motion is a preoperative-to-intraoperative change in lordosis. Objective: To investigate the effect this change has on navigation accuracy and the degree to which an interoperative stereovision system (iSV) for intraoperative patient registration can account for this motion. Methods: For six live pig specimens, a preoperative CT (pCT) was obtained of the lumbar spine in supine position and an interoperative CT in the prone position. Five to six iSV images were intraoperatively acquired of the exposed levels. A fiducial-based registration was performed on a navigation system with the pCT. Separately, the pCT was deformed to match iSV surface data to generate an updated CT (uCT). Navigational accuracy of both the commercial navigation and iSV systems was determined by tracked fiducials and landmarks. Change in lordosis Cobb angle between supine and prone positions was calculated representing preoperative-to-interoperative change in spine pose. Results: The preoperative-to-interoperative change ranged from 12 to 41°. Registration accuracy varied by 4.8 and 1.5 mm for the commercial system (6.2+-1.9 mm) and iSV (3.0+0.6 mm) respectively. Rank correlation shows strong association between increased registration error and positional change for the commercial system (correlation of 0.94, P=0.02) while minimal association for iSV (0.09, P=0.92). Conclusion: Change in spinal pose effects navigational accuracy of commercial systems. iSV shows promise in accounting for these changes given its accuracy is uncorrelated with pose change.

Epilepsy is the fourth most common neurological disorder and affects people of all ages worldwide. Deep Brain Stimulation (DBS) has emerged as an alternative treatment option when anti-epileptic drugs or respective surgery cannot lead to satisfactory outcomes. To facilitate the planning of the procedure and for its standardization, it is desirable to develop an algorithm to automatically localize the DBS stimulation target, i.e., Anterior Nucleus of Thalamus (ANT), which is a challenging target to plan. In this work, we perform an extensive comparative study by benchmarking various localization methods for ANT-DBS. Specifically, the methods involved in this study include traditional registration method and deep-learning-based methods including heatmap matching and differentiable spatial to numerical transform (DSNT). Our experimental results show that the deep-learning (DL)- based localization methods that are trained with pseudo labels can achieve a performance that is comparable to the inter-rater and intra-rater variability and that they are orders of magnitude faster than traditional methods

The success of deep brain stimulation (DBS) depends upon the accurate surgical placement of electrodes in the OR. However, the accuracy of pre-operative scans is often degraded by intraoperative brain shift. To compensate for brain shift, we developed a biomechanical brain model that updates preoperative images by assimilating intraoperative sparse data from either the brain surface or deep brain structures. In addition to constraining the finite element model, surface sparse data estimates model boundary conditions such as the level of cerebrospinal fluid (CSF). As a potentially cost-effective and safe alternative to intraoperative imaging techniques, a machine learning method was proposed to estimate surface brain atrophy by leveraging a large number of ventricle nodal displacements. Specifically, we constructed an artificial neural network (ANN) that consisted of an input layer with 9 hand-engineered features such as the surface-to-ventricle nodal distance. The multilayer perceptron was trained using 132,000 nodal pairs from eleven patient cases and tested using 48,000 from four cases. Results showed that in a testing case, the ANN estimated an overall surface displacement of 8.79 ± 0.765 mm to the left and 8.26 ± .455 mm to the right compared to the ground truth (10.36 ± 1.33 mm left and 7.40 ± 1.40 mm right). The average prediction error of all four testing cases was less than 2 mm. With further development and evaluation, the proposed method has the potential of supplementing the biomechanical brain model with surface sparse data and estimating boundary parameters.

We have developed a mobile photogrammetry cart, which can be used to obtain 3D surface models of infant leg anatomy. The purpose is to assess infants with idiopathic clubfoot by providing quantitative data. Traditional methods of 3D scanning are not suitable for infants since they will not stay still for any length of time.

The system consists of 40 Raspberry Pi’s, each equipped with a high-quality camera module. The cameras are mounted on t-slotted framing with custom 3D printed ball joints. The Raspberry Pi’s are connected through an ethernet switch to a laptop that serves as a controller. We modified an Evenflo Exersaucer Activity Seat to safely expose the legs to the cameras in an unobstructed manner. For scaling, registration markers are engraved on a two-toned, laser cut plate mounted underneath the infant’s seat. A custom application controls the Raspberry Pi’s to take synchronized photos, saving them to network attached storage. Post-processing is completed in Agisoft Metashape, which generates an exact 3D model from the 2D images.

We initially tested the system with a healthy, two-month-old infant with the permission of her parents. We successfully obtained 3D surface models of the volunteer’s feet and lower legs, demonstrating the feasibility of the system.

Our next step is to collect clubfoot data from our orthopedic clinic in an Institutional Review Board (IRB) approved study. To date, we have enrolled one two-week-old patient with bilateral clubfoot, and successfully collected a 3D surface model of their condition prior to their first corrective casting.

Percutaneous ablation is a viable treatment option for patients with early-stage hepatocellular carcinoma (HCC) who are not candidates for surgical resection or liver transplantation,1 but the efficacy of this procedure depends heavily on the surgeon’s ability to place the ablation probe accurately into the target tumor. This paper introduces a surgical navigation system for guiding an ablation needle percutaneously, using an electromagnetic tracking system and a mini-stereotactic aiming device. A clamp to attach an EM sensor was designed to attach to the a patient-attached needle guider. The pose information from the guider sensor after calibration was used to project a virtual line that follows the needle guider movements, and shows the applicator path before insertion. A surgical navigation application was developed in the 3D Slicer platform and a pose calibration technique was incorporated to ensure that the virtual needle path that is projected from the needle guider aligns with the physical needle once a needle is inserted into the guider. The estimated ablation zone, based on manufacturer’s guidelines, is displayed at the tip of the virtual path to help the user ensure total tumor coverage. The guidance software allows updating the estimated ablation zone in real-time depending on the treatment specifications. The ablation zone is displayed as a 3D model and as an ellipse overlaid on the 2D ultrasound video. A rotation analysis to obtain the angular error between two virtual lines (guider and a pre-calibrated needle) was performed in two different setups. The first was the ideal setup without anything that could bend the pre-calibrated needle, and the second used a Polyvinyl Alcohol cryogel Phantom. The results for the rotation analysis, mean and STD, for the ideal setup, was (0.33 ± 0.29)°; while using the PVA-c phantom the results (2.70 ± 0.07)°. These results suggest that the disposable mini-stereotactic needle guider has the potential to assist the surgeons in the accurate placement of the ablation needle in the clinical setting.

Ultrasound-guided biopsy is widely used for disease detection and diagnosis. We plan to register preoperative imaging, such as positron emission tomography / computed tomography (PET/CT) and/or magnetic resonance imaging (MRI), with real-time intraoperative ultrasound imaging for improved localization of suspicious lesions that may not be seen on ultrasound but visible on other imaging modalities. Once the image registration is completed, we will combine the images from two or more imaging modalities and use Microsoft HoloLens 2 augmented reality (AR) headset to display three-dimensional (3D) segmented lesions and organs from previously acquired images and real-time ultrasound images. In this work, we are developing a multi-modal, 3D augmented reality system for the potential use in ultrasound-guided prostate biopsy. Preliminary results demonstrate the feasibility of combining images from multiple modalities into an AR-guided system.

Vagus nerve stimulation (VNS), commonly used to reduce seizures in patients with epilepsy, is a promising therapeutic treatment for a number of health issues. Current VNS apparatuses employ a helical electrode design, which stimulates the nerve with no anatomical specificity. The efficacy and breadth of VNS therapy could be improved by targeting stimulation to specific regions of the nerve. A mock electrode was built around a morphologically accurate finite element model of the vagus nerve. Electric currents were injected into the nerve model, and a nodal activating function was used to determine which axons would initiate action potentials. Electrode configurations and stimulation settings were adjusted to target specific fascicles until optimal activation was achieved. Results indicated that small, proximal electrodes could stimulate targeted regions while avoiding activation of off-target axons. Injection of negative current perpendicular to the positive stimulus also proved to refine spatial stimulation, allowing for the activation of deep fascicles with minimal side-effects. While an understanding of the fascicular anatomy of the vagus nerve is still being explored, the preliminary results of this study corroborate the concept of selectively targeting regions of the nerve with electrical stimulation in order to treat specific patient needs. The computational process presented in this work could be employed as a planning tool prior to the geometrical design and surgical implantation of VNS devices.

External beam ablation therapy has the potential to treat cardiac arrhythmias non-invasively by targeting arrhythmogenic myocardial tissue; however, a challenge of treating cardiac tissue with beam ablation therapy is cardiac motion. Currently, cardiac motion is typically compensated by expansion of the target volume which can potentially lead to collateral damage of surrounding healthy tissue. This collateral damage could be minimized by gating the beam delivery to a portion of the cardiac cycle. In prior work, we evaluated cardiac motion using anatomic landmarks in multi-phase cardiac computed tomography volumes of swine hearts across the left atria and ventricles. Other work evaluated left atrial motion using implanted fiducial clips. In the current work, we extend this prior work by quantifying cardiac motion using gold standard implanted fiducial clips across all four chambers of the heart. Cardiac motion varied by chamber, ranging from 2.1 to 7.2 mm in the x direction, 7.2 to 8.1 mm in the y direction, and 3.1 to 9.7 mm in the z direction. In addition, we quantify the reduction in motion if delivery were gated to phases 40% to 90% of the cardiac cycle, which corresponds to treating across 50% of the cardiac cycle. Cardiac motion across 50% of the cardiac cycle ranged from 1.1 to 5.3 mm in the x direction, 4.5 to 5.2 mm in the y direction, and 1.2 to 7.8 mm in the z direction. Percentage reduction in motion for treating during 50% of the cardiac cycle ranged from 18% to 47% in the x direction, 31% to 43% in the y direction, and 11% to 61% in the z direction. These results demonstrate that a substantial improvement in target localization could be achieved by gating the beam to 50% of the cardiac cycle.

Advancements in Head-Mounted-Display (HMD) technology have led to an increasing focus in the development of Augmented Reality (AR) applications in the image-guided surgery field. These applications are often enabled by third-party vision-based tracking techniques, allowing virtual models to be registered to their corresponding real-world objects. The accuracy of the underlying vision-based tracking technique is critical towards the efficacy of these systems, and must be thoroughly evaluated before integration into clinical practice. In this paper, we propose a framework for the purpose of evaluating the technical accuracy of the HMD’s intrinsic vision-based tracking techniques using an extrinsic tracking system as ground truth. Specifically, we assess the tracking accuracy of the Vuforia Augmented Reality Software Development Kit, a vision-based tracking technique commonly used in conjunction with the Microsoft Hololens 2, against a commercial optical tracking system using a co calibration apparatus. The framework follows a two-stage pipeline of first calibrating the cameras with respect to the optical tracker (hand-eye calibration), and then calibrating a Vuforia target to the optical tracker using a custom calibration apparatus. We then evaluate the absolute tracking accuracy of three Vuforia target types (image, cylinder, and cube) using a stand-alone Logitech webcam and the front-facing camera on the Hololens 2. The hand-eye calibration projection errors were 1.4 ± 0.6 pixels for the Logitech camera and 2.3 ± 1.2 pixels for the Hololens 2 camera. The cylinder target provided the most stable and accurate tracking, with mean errors of 12.5 ± 0.6 mm and 10.7 ± 0.0 mm for the Logitech and Hololens 2 cameras, respectively. These results show that Vuforia has promising potential for integration into surgical navigation systems, but the type and size of target must be optimized for the particular surgical scenario to minimize tracking error. Future work will use our framework to perform a more robust analysis of optimal target shapes and sizes for vision-based navigation systems, both independently and when fused with the Simultaneous Localization and Mapping (SLAM) based tracking embedded with the Microsoft Hololens 2.

Magnetic resonance imaging (MRI)-guided prostate focal laser ablation (FLA) therapy shows potential as a minimally invasive treatment method for localized prostate cancer, which minimizes overtreatment of surrounding structures, thereby improving quality of life. We previously developed an MRI-compatible mechatronic guidance system capable of needle positioning within an open-air and in-bore MRI environment. In comparison to open-air testing, an increased error was reported from in-bore experiments, suggesting the effects of image distortion, fiducial localization, and registration error may impact its accuracy. In this paper, we describe the design of an improved registration multi-fiducial for the robust registration of the mechatronic system to MRI, and comparison and validation of MRI-guided needle delivery to virtual targets (simulating localized focal zones) in tissue-mimicking prostate phantoms. The multi-fiducial structure is composed of thirty-six MR-spheres arranged across an extensive volume. Mechatronics-assisted MRI-guided needle delivery (N =10) to virtual targets were evaluated with tissue-mimicking phantoms. 3T MRI images were acquired for registration, the mechatronic system was remotely actuated and needle insertion was performed, then verification images were acquired. The needle tip and needle trajectory error were quantified between the planned and actual trajectories. Our preliminary results show significant improvements in needle targeting with the improved registration fiducial with an FLA ablation region radius of 2.0 mm within 95% confidence. Improvements in robust registration show potential to enable accurate mechatronics-assisted MRI-guided needle delivery for FLA therapy.

Breast cancer is the most dominant cancer type among women. Although digital mammography plays an important role in early breast cancer detection, many cancers cannot be distinguished on mammography only, particularly in individuals with dense breast tissue. Lesions not recognizable on mammography are frequently detected by contrast enhanced magnetic resonance imaging (CE-MRI) of the breast. Based on the suspicious characteristics, these lesions need to be further evaluated with MRI-guided biopsy. However, MRI-guided biopsy is costly, time consuming, and not commonly available. In our earlier work, we proposed a novel method for a matching tool between MRI and spot mammograms using a biomechanical model based registration to match MRI and full X-ray mammograms and an image based registration to align full X-ray mammograms and spot mammograms. In this paper, we focus on developing and evaluating methods for image based registration between full X-ray mammograms and spot mammograms. Results assessed for thirteen patients from the Medical University of Vienna are presented. The median target registration error (TRE) of the image based registration is 21.7 mm and the standard deviation is 9.3 mm.

Purpose. Mobile C-arms capable of 2D fluoroscopy and 3D cone-beam CT (CBCT) are finding application in guidance of transbronchial lung biopsy, but unresolved deformable motion presents challenges to accurate target localization and guidance. We report the initial implementation of a method to resolve deformations via locally rigid / globally deformable 3D-2D registration for motion-compensated overlay of planning data in fluoroscopically-guided pulmonary interventions. Methods. The algorithm proceeds in 3 steps: (1) initialization by 3D-2D rigid registration of CBCT to fluoroscopy (driven by bone gradients); (2) local rigid 3D-2D registration of lung-thresholded CBCT to fluoroscopy within a region of interest (ROI) about each target location; and (3) aggregation of local rigid registrations to estimate global deformation. Several objective functions and optimizers were evaluated for soft-tissue target registration. Phantom studies were performed to determine operating parameters and assess performance with simulated lung deformation. Results. Soft-tissue thresholding and contrast enhancement improved target registration error (TRE) from 10.3 mm for conventional 3D-2D registration (driven primarily by rib gradients) to 3.8 mm using locally rigid 3D-2D registration in regions of interest about each target. The soft-tissue gradient orientation (GO) objective function was found to be superior to alternative similarity measures by de-emphasizing gradient magnitude (in favor of gradient orientation), permitting the algorithm to be better driven by soft-tissue edges. Conclusions. Registration driven by soft-tissue targets is achievable via a novel processing framework to de-emphasize non-target gradients. The proposed method could improve the accuracy of guidance in pulmonary interventions by updating target overlay in fluoroscopy.

Deep inspiration breath hold (DIBH) is a common method for managing respiratory motion in lung radiotherapy (RT) and has shown benefit for significant reduction of cardiovascular and pulmonary toxicity. Cone beam CT (CBCT) is used to capture 3D images of the patient in position for daily treatment, and multiple CBCT scans may be needed to fine tune the patient setup. Furthermore, multiple breath holds are often needed for a single CBCT acquisition (typically ~1 min). Because of differences in the tumor and healthy tissue positions between consecutive DIBHs, the inconsistent anatomy in the 2D projection images degrades the quality of the reconstructed CBCT. Moreover, multiple breath holds during the initial setup will increase the probability that a patient will become tired and less able to reproduce consistent DIBH during treatment delivery when it matters most. To address this important clinical issue, a proof-of-concept study was designed using a novel deep learning-based method to derive a 3D volumetric image from two perpendicular 2D projection images (kV-MV pair), thereby reducing the number of DIBHs needed for imaging. The proposed method – implemented with a feature matching network – derives feature maps from the 2D projections and re-aligns them to their projection angle in a Cartesian coordinate system. The 2D feature maps are rendered in 3D space via depth learning by the feature matching network. Finally, the 3D volume is derived from the 3D feature map. We conducted a simulation study using 10 patient cases (110 CT images). Each patient had a 4D CT scan that was split into 10 phase bins for motion evaluation and a DIBH CT scan during simulation and later received DIBH lung RT at our institution. Ray tracing through each phase-binned CT was used to simulate a 2D kV projection at gantry angle 0° and a MV projection at gantry angle 90°. Orthogonal 2D projections (200 projections) from 10 phases were used to train the network to be patient specific while the DIBH CT was held out for testing. The mean absolute error (MAE), peak signal-to-noise ratio (PSNR) and structural similarity index metric (SSIM) achieved by our method are 93.1 HU and 92.5 HU, 21.7 dB and 15.6 dB, and 0.87 and 0.74 within body and tumor ROI, respectively. These results demonstrate the feasibility and efficacy of our proposed method for 3D imaging from two orthogonal kV and MV 2D projections, which provides a potential solution of fast 3D imaging for daily treatment setup of breath-hold lung RT to ensure treatment accuracy and effectiveness.

Local ultrasonic tissue ablation is induced by a rapid (<3s) rise in temperature in a small ellipsoidal volume (about 13mm3) inside the tissue to a cytotoxic level when exposed to a high-intensity focused ultrasound (HIFU) beam. The aim of this study was to develop a numerical tool to predict the location and extent of a necrotic lesion formed locally inside the ex vivo tissue as a result of exposure to a single or multiple HIFU beam, ensuring the efficacy and safety of destroying solid tumors. The proposed tool was based on modelling the non-linear propagation of acoustic waves and heat transfer in heterogeneous media using the k-wave toolbox. The wave propagation equations were solved for two-layer (water/tissue) media. The source of the acoustic waves was a spherical bowl-shaped transducer with a resonance frequency of 1.08 MHz. The distribution of heat sources was determined from the calculated acoustic pressure distribution in the HIFU beam. The obtained temperature distributions during heating and cooling allowed calculation of the thermal dose and prediction of the extent of the necrotic lesion. The obtained results of numerical simulations were compared with the experimental data from previous studies. The mean difference between the calculated and measured length or diameter of a single exposure induced necrotic lesion was approximately 1 mm. In the case of a necrotic lesion induced by multiple exposures, the mean difference between the measured and calculated cross-sectional area of the planned necrotic lesion covered with necrosis was approximately 11.2 %.

Phantoms are invaluable tools broadly used for research and training purposes designed to mimic tissues and structures in the body. In this paper, polyvinyl chloride (PVC)-plasticizer and silicone rubbers were explored as economical materials to reliably create long-lasting, realistic kidney phantoms with contrast under both ultrasound (US) and X-ray imaging. The radiodensity properties of varying formulations of soft PVC-based gels were characterized to allow adjustable image intensity and contrast. Using this data, a phantom creation workflow was established which can be easily adapted to match radiodensity values of other organs and soft tissues in the body. Internal kidney structures such as the medulla and ureter were created using a two-part molding process to allow greater phantom customization. The kidney phantoms were imaged under US and X-ray scanners to compare the contrast enhancement of a PVC-based medulla versus a silicone-based medulla. Silicone was found to have higher attenuation than plastic under X-ray imaging, but poor quality under US imaging. PVC was found to exhibit good contrast under X-ray imaging and excellent performance for US imaging. Finally, the durability and shelf life of our PVC-based phantoms were observed to be vastly superior to that of common agar-based phantoms. The work presented here allows extended periods of usage and storage for each kidney phantom while simultaneously preserving anatomical detail, contrast under dual-modality imaging, and low cost of materials.

Percutaneous ablation is becoming a viable treatment option for patients with early-stage hepatocellular carcinoma (HCC) who are not candidates for surgical resection or liver transplantation.1 The success of the treatment is measured by the complete coverage, plus a positive margin, of the tumor being contained within the ablation lesion.² In this project, a multi-modality anthropomorphic phantom with simulated tumor and vascular flow was developed. The phantom consists of five different parts: the left and right lobes, internal and external vasculature (part of the Inferior Vena Cava), and the tumors. The geometry of these anatomical features are based on patient-specific CT data. Our anthropomorphic liver phantom is made with PolyVinyl Alcohol cryogel (PVA-c) to serve as an Ultrasound-, MRI-, and CT-compatible tissue-mimicking material. Talcum powder was added to the PVA-c to provide realistic speckle under ultrasound (US) imaging, with the optimal concentration being determined by experiment. The Talcum concentration of the tumors was evaluated by US and CT imaging. To create the closed-loop vasculature flow, positive silicone vasculature molds were inserted into the liver body mold prior PVA-c filling. After the freeze-thaw cycles, the silicone vasculature molds are extracted from the liver body creating a network of canals. To recreate the blood flow, a water pump was connected to the liver phantom vasculature to allow the flowing through the internal canals. Differentiation between the liver tissue, vessels, and simulated tumors was clearly visualized in US and CT imaging. Color Doppler was acquired to test the flow of the closed-loop vasculature. The antropomorphic characteristics and the manufacturing technique makes our liver phantom customizable to work as a sandbox environment for needle puncture procedures (i.e. focal ablation) as well as training and validation of surgical navigation systems for these interventions.

Interactive multimodal surgical simulators are powerful tools that allow efficient and objective assessment of surgical skills. Bilateral Sagittal Split Osteotomy (BSSO) requires precise cutting of the mandible with a motorized saw that cannot afford a high margin of error. Surgeons rely on visual and haptic cues that are hard to train for through existing curricula without training directly on patients. In this paper, we present a new algorithm to achieve low-cost, precise sawing of the bone that is capable of providing realistic force feedback to the user at haptic rates (~1000Hz). Our method is centered around treating the bone surface as a level set that is evolved at interactive rates upon user’s interaction with the virtual saw. The movement of the virtual saw itself is governed by a rigid body solver which is attached to the pose (6 degrees of freedom) of the haptic device via a mass-spring-damper which also supplies haptic force feedback. Our method allows parameterization of the erosion rate, saw speed, and bone density thus making it suitable to any material shaving application including dentistry (e.g., tooth milling) and a variety of surgical osteotomies.

Automated skills assessment of ultrasound-guided needle insertions has previously been explored through 3D motion tracking data. The purpose of this study was to determine the viability of 2D motion tracking data in distinguishing between novice and expert subjects. METHODS: Perspective projection was applied to needle and ultrasound probe time series data. The resulting time series data of 2D points were used to calculate various performance metrics. Using these metrics, classifications between novice and expert were performed by random forest. This procedure was repeated with different camera positions all pointing at the reference point to examine systematically the effect of camera position on assessment. RESULTS: For in-plane needle insertions, mean AUC obtained through 3D data and mean AUC obtained through 2D data were well-matched (0.68 vs. 0.69). For out-of-plane insertions, mean AUC values from 3D and 2D data were more distant (0.86 vs. 0.77), but AUC from the optimal camera angle matched up well (0.85). CONCLUSION: 2D data is comparable to 3D data when used to perform skills assessment of ultrasound-guided needle insertions, and camera placement level with the instruments is optimal. We conclude that videos of needle insertions may be feasible for skills assessment.

Lung cancer is the deadliest form of cancer. A bronchoscopic lymph node staging procedure is used to determine the stage of lung cancer. In this procedure, the physician samples tissue extracted from multiple diagnostic sites in the central chest, including lymph nodes and suspect tumors, using a bronchoscope to examine for cancer spread. Although best practice guidelines recommend a diverse set of sampling locations, most physicians only sample the bare minimum number of nodes during a staging procedure. Image-guided bronchoscopy (IGB) systems aid in guidance by using a patient’s three-dimensional (3D) x-ray computed tomography (CT) image to create a virtual chest model and plan airway routes to chest regions of interest (ROIs). This plan is then executed in follow-on live guidance. Current IGB systems do not create plans for comprehensive nodal staging; they only enable individual ROI guidance and rely on the physician to select the order of ROIs. We now propose a full image-based methodology for guiding comprehensive, multi-destination lymph node staging procedures to span this gap. The complete methodology involves two components: 1) a procedure planning protocol to optimize bronchoscope sampling order; and 2) a guidance strategy and graphical user interface (GUI) designed for multi-destination staging procedures. We present results demonstrating the system and its application.

In this paper, we propose a self-supervised depth estimation with uncertainty-weight joint loss function based on laparoscopic videos. Although self-supervised learning has achieved impressive performance in depth estimation using pose estimation as an auxiliary task, it still shows undesired results for the pose estimation. Different from streetscape datasets, the laparoscope motion is limited by the minimally invasive surgery settings. It is challenging to estimate laparoscopes’ poses with complex rotations from RGB images. To address this issue, we propose an improved self-supervised depth estimation method with relative pose loss for laparoscopic videos. Furthermore, we adopt homoscedastic uncertainty to weigh our loss function to balance each subtask. In addition to the evaluation for known datasets, we also tested the generalization ability of our proposed method using known datasets and unseen datasets. The experimental results showed that our proposed method outperforms baseline for depth estimation and pose estimation on known datasets and had competitive results on unseen datasets. For depth estimation, the proposed method had about 10.5% improvement on RMSE evaluation compared to the baseline.

Minimally invasive image-guided interventions (IGIs) enable better therapy outcomes for patients, but navigation accuracy is highly dependent on the accuracy of the image-/model-to-patient registration. This requires methods to reduce the uncertainty to a level appropriate for the procedure being performed. Since sub-surface tissue landmarks cannot be easily sampled using a tracked stylus and used to perform the patient registration, here we present a method that employs a tracked camera (that mimics a laparoscope) to perform the patient registration or update this registration in case of suspected misalignment within the context of an image-guided renal navigation procedure. We implement and test the method using a simplified patient phantom, which consists of a foam block to which a virtual kidney model featuring both surface and sub-surface landmarks is registered. This setup mimics a situation when a surgeon would navigate a tracked needle to renal landmarks percutaneously, while relying on pre-procedural imaging, optical tracking, and surface video imaging. We conduct several experiments under both optimal phantom registration and purposely altered registration, to not only show the effect of phantom / patient mis-registration on subsequent navigation, but also demonstrate the use of the camera-based registration to restore navigation to an acceptable uncertainty. Our results illustrate that camera-based registration yields a target registration error on the order of 0.4 mm and a subsequent targeting error on the order of 0.6 mm, comparable to the performance achieved following gold-standard landmark-based registration. These results suggest that the proposed method can be used to perform or update the patient registration for image-guided interventions involving sub-surface organs.