This PDF file contains the front matter associated with SPIE Proceedings Volume 11601 including the Title Page, Copyright information, and Table of Contents.

Welcome and Introduction to SPIE Medical Imaging conference 11601: Imaging Informatics for Healthcare, Research, and Applications

This talk presents an overview of most recent advancements in Robotic Imaging, Machine Leaning and Medical Augmented Reality (AR). I will first discuss the particular requirements for intra-operative imaging and visualization. I will then present our latest results in intra-operative multimodal robotic imaging and its translation to clinical applications. I will discuss the impact of recent advancement in machine learning on medical imaging and computer assisted intervention. Finally, I will give a short review of state of art in medical AR and focus on the first two AR applications which made it into real surgery rooms. For video demonstrations see https://www.medicalaugmentedreality.org/ and http://campar.in.tum.de/).

Facing the COVID-19 pandemic, healthcare organizations across the world worked expeditiously to prepare for and manage the outbreak. In the modern era, clinical information systems (CIS) and the electronic health record (EHR) are essential tools that can support patient care and enhance healthcare delivery. Researchers out of the University of California, San Diego, were the first to describe in detail the rapid development and implementation of EHR based tools designed specifically to support the management of COVID-19. Drs. Jeff Reeves and Chris Longhurst will describe how CIS are utilized in response to COVID-19, challenges to successful implementation of informatics tools, and the future of informatics in pandemic management.

In this paper, an ensemble learning framework is proposed for HEp-2 cell images, aiming to making use of both handcrafted features and deep learning-based methods. Firstly, deep unsupervised learning is employed to extract features. Then, a gradient boosting trees-based classifier is trained using both handcrafted features and deep learning-based features. Extensive experiments are conducted on benchmark datasets to test the efficiency and robustness of the proposed framework. Experiment results demonstrate hat the proposed framework yield excellent performances compared with existing deep learning-based models.

The Gut Cell Atlas (GCA), an initiative funded by the Helmsley Charitable Trust, seeks to create a reference platform to understand the human gut, with a specific focus on Crohn’s disease. Although a primary focus of the GCA is on focusing on single-cell profiling, we seek to provide a framework to integrate other analyses on multimodality data such as electronic health record data, radiological images, and histology tissues/images. Herein, we use the research electronic data capture (REDCap) system as the central tool for a secure web application that supports protected health information (PHI) restricted access. Our innovations focus on addressing the challenges with tracking all specimens and biopsies, validating manual data entry at scale, and sharing organizational data across the group. We present a scalable, cross-platform barcode printing/record system that integrates with REDCap. The central informatics infrastructure to support our design is a tuple table to track longitudinal data entry and sample tracking. The current data collection (by December 2020) is illustrated with types and formats of the data that the system collects. We estimate that one terabyte is needed for data storage per patient study. Our proposed data sharing informatics system addresses the challenges with integrating physical sample tracking, large files, and manual data entry with REDCap.

Type 2 Diabetes Mellitus(DM) is a chronic condition that impairs the way the body processes blood sugar(glucose). Over 10 percent of the US population is known to be affected by Type 2 DM as of 2018, and almost quarter of them are unaware or undiagnosed, thus, making early detection and treatment of diabetes an important step in mitigating the associated health risks. Previous studies show that waist circumference and waist-height ratio are found to be better indicators of diabetes than BMI. These measures of central obesity create great interest to study different measures of body fat distribution. The main objective of this study is to provide evidence to prove that variables derived from the DXA(Dual-energy X-ray absorptiometry) analysis including regional fat distribution profiles are better indicators of DM when compared to conventional metrics such as Body Mass Index(BMI). A multi-class classification is performed using Random Forest Classifier to classify patients as ‘Normal’, ‘Prediabetic’ or ‘Diabetic’ across various subsets of data obtained from the NHANES diabetes cohort. Feature selection techniques such as ANOVA/Chi-square, Recursive Feature Eliminations(RFE) and intrinsic feature importance scores from the classifier were used to filter the most important features. It was observed that fat distribution features from DXA can be used as a viable alternative to conventional metrics in the detection of DM. Notably, head fat percentage was proven to be a prominent feature to identify DM. Thus, our study demonstrates the potential of fat distribution variables as a potential standalone or surrogate biomarker for Type 2 DM.

Color, shape (size and volume), and temperature are important clinical features for chronic wound monitoring that could impact diagnosis and treatment. Noninvasive 3D measurement are better and more accurate than those in 2D, but expensive equipment and complexity of the setup prevent their use at hospitals. Therefore, the use of affordable and lightweight devices with straightforward protocol to acquire images for evaluations is fundamental to provide a functional and useful evaluation of the wound. In this work, an automated methodology to generate color and thermal 3D models is presented by using portable devices: a commercial mobile device with a connected portable thermal camera. The 3D model of the wound surface is estimated from a series of color images using structure-from-motion (SfM) while thermal information is overlaid to the ulcer’s relief for multimodal wound evaluation. The proposed methodology contributes with a proof of concept for multimodal wound monitoring in the hospital environment with a simple hand-held shooting protocol. The system was used efficiently with 5 patients on wounds of various sizes and types.

Approximately 37 million falls occur each year worldwide requiring medical attention. Victims are often helpless and not able to call for help, which is a risk for elderly persons living alone. To detect falls at home, several approaches have been proposed. Video cameras are used increasingly. Recently, high accuracy in real-time human pose estimation in videos has been achieved by novel machine learning techniques. In this work, we propose a multi-camera system for video-based fall detection. We augment human pose estimation (OpenPifPaf algorithm) by support for multi-camera and multi-person tracking and a long short-term memory (LSTM) neural network to predict two classes: “Fall” or “No Fall”. From the poses, we extract five temporal and spatial features which are processed by the LSTM. For evaluation of identification and tracking with multiple cameras, we used videos recorded in a smart home (living lab) with two persons walking and interacting. For evaluation of fall detection, we used the UP-Fall Detection dataset and achieve an F1 score of 92.5%. We observed a tendency towards false positive classifications due to lack of activities in publicly available datasets that look similar to falls but stem from normal activity. Moreover, the lack of variation in the activities also results in a higher amount of false positives. This requires the acquisition of more balanced datasets in future work. In conclusion, real-time fall detection from multiple camera inputs and for multiple persons is feasible using a LSTM neural network combined with features obtained via human pose estimation. Source code is available at https://github.com/taufeeque9/HumanFallDetection.

Neurosurgical training is performed on human cadavers and simulation models, such as VR platforms, which have several drawbacks. Head phantoms could solve most of the issues related to these trainings. The aim of this study was to design a realistic and CT-compatible head phantom, with a specific focus on endo-nasal skull-base surgery and brain biopsy. A head phantom was created by segmenting an image dataset from a cadaver. The skull, which includes a complete structure of the nasal cavity and detailed skull-base anatomy, is 3D printed using PLA with calcium, while the brain is produced using a PVA mixture. The radiodensity and mechanical properties of the phantom were tested and adjusted in material choice to mimic real-life conditions. Surgeons find the skull, the structures at the skull-base and the brain realistically reproduced. The head phantom can be employed for neurosurgical education, training and surgical planning, and can be successfully used for simulating surgeries.

The mechanical thrombectomy (MT) efficacy, for large vessel occlusion (LVO) treatment in patients with stroke, could be improved if better teaching and practicing surgical tools were available. We propose a novel approach that uses 3D printing (3DP) to generate patient anatomical vascular variants for simulation of diverse clinical scenarios of LVO treated with MT. 3DP phantoms were connected to a flow loop with physiologically relevant flow conditions, including input flow rate and fluid temperature. A simulated blood clot was introduced into the model and placed in the Middle Cerebral Artery region. Clot location, composition (hard or soft clot), length, and arterial angulation were varied and MTs were simulated using stent retrievers. Device placement relative to the clot and the outcome of the thrombectomy were recorded for each situation. Angiograms were captured before and after LVO simulation and after the MT. Recanalization outcome was evaluated using the Thrombolysis in Cerebral Infarction (TICI) scale. Forty-two 3DP neurovascular phantom benchtop experiments were performed. Clot mechanical properties, hard versus soft, had the highest impact on the MT outcome, with 18/42 proving to be successful with full or partial clot retrieval. Other factors such as device manufacturer and the tortuosity of the 3DP model correlated weakly with the MT outcome. We demonstrated that 3DP can become a comprehensive tool for teaching and practicing various surgical procedures for MT in LVO patients. This platform can help vascular surgeons understand the endovascular devices limitations and patient vascular geometry challenges, to allow surgical approach optimization.

For medical image segmentation, deep learning approaches using convolutional neural networks (CNNs) are currently superseding classical methods. For good accuracy, large annotated training data sets are required. As expert annotations are costly to acquire, crowdsourcing–obtaining several annotations from a large group of non-experts–has been proposed. Medical applications, however, require a high accuracy of the segmented regions. It is agreed that a larger training set yields increased CNN performance. However, it is unclear, to which quality standards the annotations need to comply to for sufficient accuracy. In case of crowdsourcing, this translates to the question on how many annotations per image need to be obtained. In this work, we investigate the effect of the annotation quality used for model training on the predicted results of a CNN. Several annotation sets with different quality levels were generated using the Simultaneous Truth and Performance Level Estimation (STAPLE) algorithm on crowdsourced segmentations. CNN models were trained using these annotations and the results were compared to a ground-truth. It was found that increasing annotation quality results in a better performance of the CNN in a logarithmic way. Furthermore, we evaluated whether a higher number of annotations can compensate lower annotation quality by comparing CNN predictions from models trained on differently sized training data sets. We found that when a minimum quality of at least 3 annotations per image can be acquired, it is more efficient to then distribute crowdsourced annotations over as many images as possible. The results can serve as a guideline for the image assignment mechanism of future crowdsourcing applications. The usage of gamification, i.e., getting users to segment as many images of a data set as possible for fun, is motivated.

In many fields of medical imaging, image segmentation is required as a basis for further analysis and diagnosis. Convolutional neural networks are a promising approach providing high accuracy. However, large-scale annotated datasets are necessary to train these networks. As expert annotations are costly, crowdsourcing has shown to be an adequate alternative. In previous work, we examined how the workforce of a crowd should be distributed for obtaining annotations with an optimal trade-off between quantity and quality. In this work, we present a gamification approach by transforming the tedious task of image segmentation into a game. This approach aims at motivating users by having fun but nevertheless generating annotations of adequate quality. Therefore, this work presents a gamified crowdsourcing concept for medical image segmentation. We give an overview of incentives applied in state-of-the-art literature and propose two different gamification approaches on how the image segmentation task can be realized as a game. Finally, we propose a integrated game concept that combines both approaches with the following incentives: (a) points / scoring to reward users instantly for accurate segmentations, (b) leaderboard / rankings to let users accumulate scores for long-term motivation, (c) badges / achievements to give users a visual representation of their ”strength” in segmentation, and (d) levels to visualize the learning curve of users in performing the segmentation. We give details on a first prototype implementation and describe how the game concept complies with the guidelines from our prior work.

Deep learning in medical imaging typically requires sensitive and confidential patient data for model training. Recent research in computer vision has shown that it is possible to recover training data from trained models using model inversion techniques. In this paper, we investigate the degree to which encoder-decoder like architectures (U-Nets, etc) commonly used in medical imaging are vulnerable to simple model inversion attacks. Utilising a database consisting of 20 MRI datasets from acute ischemic stroke patients, we trained an autoencoder model for image reconstruction and a U-Net model for lesion segmentation. In the second step, model inversion decoders were developed and trained to reconstruct the original MRIs from the low dimensional representation of the trained autoencoder and the U-Net model. The inversion decoders were trained using 24 independent MRI datasets of acute stroke patients not used for training of the original models. Skull-stripped as well as the full original datasets including the skull and other non-brain tissues were used for model training and evaluation. The results show that the trained inversion decoder can be used to reconstruct training datasets after skull stripping given the latent space of the autoencoder trained for image reconstruction (mean correlation coefficient= 0.49), while it was not possible to fully reconstruct the original image used for training of a segmentation task UNet (mean correlation coefficient=0.18). These results are further supported by the structural similarity index measure (SSIM) scores, which show a mean SSIM score of 0.51± 0.14 for the autoencoder trained for image reconstruction, while the average SSIM score for the U-Net trained for the lesion segmentation task was 0.28±0.12. The same experiments were then conducted on the same images but without skull stripping. In this case, the U-Net trained for segmentation shows significantly worse results, while the autoencoder trained for image reconstruction is not affected. Our results suggest that an autoencoder model trained for image compression can be inverted with high accuracy while this is much harder to achieve for a U-Net trained for lesion segmentation.

Due to variations in unique patient populations, imaging hardware, and drift, artificial intelligence (AI) models perform differently in different clinical practices. The validation of externally developed models is a critical step in implementation, but several challenges related to data de-identification, security, and exchange exist. We created a workflow allowing our clinical radiology practice to safely evaluate external AI models. The workflow encompassed four steps: study selection, extraction, inference, and assessment. A commercially available AI model for intracranial hemorrhage (ICH) was used as a proof of concept. Noncontrast head CT cases were collected using both an internal search engine and a neuroradiologist teaching file that contained 16 exams in which ICH had been missed on the original radiology interpretation. These challenge cases were included to enrich the cohort. Our DICOM de-identification and processing pipeline (D2P2) processed the header and stripped identifiable information. The cleaned data was made available to the external party for AI model processing. The processed results are matched against the ground truth, and performance metrics were calculated, including subgroup analyses. The overall precision was 0.98, the recall was 0.75, and the specificity was 0.97, with an F1 score of 0.85. This result was similar to the original radiologist performance for this enriched challenge cohort. The estimated combined performance of the original radiologist with AI improved the recall to 0.91. Subgroup analyses suggested that combined performance was improved over radiologists or AI alone for subtle hemorrhages or those without mass effect. An integrated workflow to handle the assessment and validation of external deep learning models before implementation is feasible and can aid radiology practices seeking to deploy AI products, particularly in assessing the performance in the local setting and clinical value-added scenarios.

We have developed an effective deep learning pipeline to classify brain magnetic resonance imaging scans automatically into 12 subcategories. The classification is performed by a meta classifier which receives level one predictions from Microsoft's Residual Networks (ResNet), Google’s Neural Architecture Search Network (NASNet) and a text-based classifier on DICOM header series description and combine them to get final classification. The overall classifier was trained, validated and tested on 2750 MRI images from multicenter projects. The classifier was packaged using Docker containerization technology and deployed on a local XNAT instance and tested on 3000 independent imaging sessions with 98.5% accuracy.

With the increasing use of deep learning methodologies in various biomedical applications, there is a need for a large number of labeled medical image datasets for training and validation purposes. However, the accumulation of labeled datasets is expensive and time consuming. Recently, generative adversarial networks (GAN) have been utilized to generate synthetic datasets. Currently, the accuracy of generative adversarial networks is calculated using a structural similarity index measure (SSIM). SSIM is not adequate for comparison of images as it underestimates the distortions near hard edges. In this paper, we compare the real DRIVE dataset to the synthetic FunSyn-Net using Fourier transform techniques and show that Fourier behavior is quite different in the two datasets, especially at high frequencies. It is observed that for real images, the amplitude of the Fourier components exponentially decreased with increasing frequency. For the synthesized images, the rate of decrease of the amplitude is much slower. If a linear function is fit to the high frequency components, the slope distributions for the two datasets are completely different with no offset. The average slope in the log scale for DRIVE dataset and FunSyn-Net were 0.0195, and 0.009 respectively. We also looked at auto correlations for the horizontal cut of the Fourier transform and again saw a statistically significant difference between the means for the two datasets. Finally, we also observed that Fourier transforms with real images have higher magnitude squared coherence as compared to the synthesized images. Fourier transform has shown great success for finding differences between real and synthesized images and can be used to improve the synthesized GAN models.

Optical coherence tomography (OCT) images suffer from speckle noise. The presence of noise may degrade the quality of the images which may further make diagnosis difficult. In this work, a wavelet transform based deep generative modeling based method has been proposed to extract multi-scale features to denoise OCT images. The OCT images contain edge information of different retinal layers, to avoid the over-smoothing effect and edge content loss, the Sobel edge detector based loss function has been designed to retain the edge information. The method is compared with other traditional and deep learning based methods in terms of commonly used image quality measures such as peak-signal-to-noise-ratio (PSNR), structural similarity (SSIM) and edge information with the variance of Laplacian.

Deep learning methods for ophthalmic diagnosis have shown considerable success in tasks like segmentation and classification. However, their widespread application is limited due to the models being opaque and vulnerable to making a wrong decision in complicated cases. Explainability methods show the features that a system used to make prediction while uncertainty awareness is the ability of a system to highlight when it is not sure about the decision. This is one of the first studies using uncertainty and explanations for informed clinical decision making. We perform uncertainty analysis of a deep learning model for diagnosis of four retinal diseases - age-related macular degeneration (AMD), central serous retinopathy (CSR), diabetic retinopathy (DR), and macular hole (MH) using images from a publicly available (OCTID) dataset. Monte Carlo (MC) dropout is used at the test time to generate a distribution of parameters and the predictions approximate the predictive posterior of a Bayesian model. A threshold is computed using the distribution and uncertain cases can be referred to the ophthalmologist thus avoiding an erroneous diagnosis. The features learned by the model are visualized using a proven attribution method from a previous study. The effects of uncertainty on model performance and the relationship between uncertainty and explainability are discussed in terms of clinical significance. The uncertainty information along with the heatmaps make the system more trustworthy for use in clinical settings.

The purpose of this study is to develop a machine learning model with the optimal features computed from mammograms to classify suspicious regions as benign and malignant. To this aim, we investigate the benefits of implementing a machine learning approach embedded with a random projection algorithm to generate an optimal feature vector and improve classification performance. A retrospective dataset involving 1,487 cases is used. Among them, 644 cases depict malignant lesions, while the rest 843 cases are benign. The locations of all suspicious regions have been annotated by radiologists before. A computer-aided detection scheme is applied to pre-process the images and compute an initial set of 181 features. Then, three support vector machine (SVM) models are built using the initial feature set and embedded with two feature regeneration methods, namely, principal component analysis and random projection algorithm, to reduce dimensionality of feature space and generate smaller optimal feature vectors. All SVM models are trained and tested using the leave-one-case-out cross-validation method to classify between malignant and benign cases. The data analysis results show that three SVM models yield the areas under ROC curves of AUC = 0.72±0.02, 0.79±0.01 and 0.84±0.018, respectively. Thus, this study demonstrates that applying a random projection algorithm enables to generate optimal feature vectors and significantly improve machine learning model (i.e., SVM) performance (p<0.02) to classify mammographic lesions. The similar approach can also been applied to help more effectively train and improve performance of machine learning models applying to other types of medical image applications.

Breast cancer risk prediction is becoming increasingly important especially after recent advances in deep learning models. In breast cancer screening, it is common that patients have multiple longitudinal mammogram examinations, where the longitudinal imaging data may provide additional information to boost the learning of a risk prediction model. In this study, we aim to leverage quantitative imaging features extracted from prior mammograms to augment the training of a risk prediction model, through two technical approaches: 1) prior data-enabled transfer learning, and 2) multi-task learning. We evaluated the two approaches on a study cohort of 306 patients in a case-control setting, where each patient has 3 longitudinal screening mammogram examinations. Our results show that both two approaches improved the 1-, 2-, and 3-year risk prediction, indicating that additional knowledge can be learned by our approaches from longitudinal imaging data to improve near-term risk prediction.

Identifying “suspicious” regions is an essential process for clinical assessment of digital mammograms in breast cancer screening. Traditional solutions attempt to model malignant lesions directly, necessitating segmentations/annotations for training machine learning models. In this paper, we present a novel approach to identify a suspicion map – a middleware preserving only the suspicious regions in digital mammograms to effectively narrow down the image search space. Our unsupervised method is implemented by modeling normal breast tissue and subsequently identifying tissue abnormal to the model as suspicious. Our method consists of three main components: superpixel-based breast tissue patch generation, deep learning-based feature extraction from normal tissue patches, and breast density-guided one-class modeling of normal patches using the extracted features. Our machine learning approach is able to safely eliminate normal regions of tissue in a digital mammogram. Our normal tissue models were learned from 2,602 normal mammogram images and tested on 180 images (including 90 normal screening mammogram images and an independent set of 90 mammogram images with breast cancer diagnoses). Initial experiments showed that our proposed method can eliminate 97% of normal regions in the normal testing mammograms and 96% of normal regions in the malignant testing mammograms. Our method, based on modeling normal breast tissue, provides a novel and unsupervised scheme to more effectively analyze digital mammogram images towards identifying suspicious regions, and has the potential to benefit a variety of downstream applications for computeraided detection, diagnosis, and triage of breast cancer in mammogram images.

Improvements in resolution, pulse sequences, and access to scanners allow for increasingly subtle changes to be detected in magnetic resonance images, while increasing the volume of data to be interpreted by radiologists. The burden of reviewing this vast amount of information is further exacerbated by a lack of spatial alignment between images as well as the potential presence of multifocal pathologies. In this work, we describe an informatics approach for incorporating difference images into clinical workflows for highlighting 1) changes in anatomy over time, or 2) contrast enhanced regions following administration of gadolinium. These difference images enhance features of interest while suppressing stable features, providing efficient detection of changes over time, whether due to treatment, age, medical intervention, or progression of disease. Our pipeline is initiated by the physician sending images to a DICOM receiver, which then applies spatial transformation, bias-field correction, and intensity normalization before returning the newly generated difference images to the clinical PACS for review. This pipeline has been used to process 2683 neuroimaging data sets for evaluating disease progression and contrast enhancement in patients with brain lesions. We demonstrate that difference images quantitatively improve contrast-to-noise ratio (CNR) of new lesions while reducing clutter, which should translate to improved accuracy and efficiency in radiological interpretations.

Imaging data within the clinical practice in general uses standardized formats such as Digital Imaging and Communications in Medicine (DICOM). Aside from 3D volume data, DICOM files usually include relational and semantic description information. The majority of current applications for browsing and viewing DICOM files online handle the image volume data only, ignoring the relational component of the data. Alternatively, implementations that show the relational information are provided as complete pre-packaged solutions that are difficult to integrate in existing projects and workflows. This publication proposes a modular, client-side web application for viewing DICOM volume data and displaying DICOM description fields containing relational and semantic information. Furthermore, it supports conversion from DICOM data sets into the nearly raw raster data (NRRD) format, which is commonly utilized for research and academic environments, because of its simpler, easily processable structure, and the removal of all patient DICOM tags (anonymization). The application was developed in JavaScript and integrated into the online medical image processing framework StudierFenster (http://studierfenster.tugraz.at/). Since our application only requires a standard web browser, it can be used by everyone and can be easily deployed in any wider project without a complex software architecture.

Tympanic membrane (TM) diseases are among the most frequent pathologies, affecting the majority of the pediatric population. Video otoscopy is an effective tool for diagnosing TM diseases. However, access to Ear, Nose, and Throat (ENT) physicians is limited in many sparsely-populated regions worldwide. Moreover, high inter- and intra-reader variability impair accurate diagnosis. This study proposes a digital otoscopy video summarization and automated diagnostic label assignment model that benefits from the synergy of deep learning and natural language processing (NLP). Our main motivation is to obtain the key visual features of TM diseases from their short descriptive reports. Our video database consisted of 173 otoscopy records from three different TM diseases. To generate composite images, we utilized our previously developed semantic segmentation-based stitching framework, SelectStitch. An ENT expert reviewed these composite images and wrote short reports describing the TM's visual landmarks and the disease for each ear. Based on NLP and a bag-of-words (BoW) model, we determined the five most frequent words characterizing each TM diagnostic category. A neighborhood components analysis was used to predict the diagnostic label of the test instance. The proposed model provided an overall F1-score of 90.2%. This is the first study to utilize textual information in computerized ear diagnostics to the best of our knowledge. Our model has the potential to become a telemedicine application that can automatically make a diagnosis of the TM by analyzing its visual descriptions provided by a healthcare provider from a mobile device.

This work aims at the development of a platform called pathology. Designed to aid learning, with an emphasis on improving diagnoses based on images, it is aimed at medical students and will be available via the Internet. The platform will make possible the interaction between students, teachers and specialists, extending the experience in the classroom to a dynamic based on the interpretation of images, elaboration of diagnoses and comparison with reports previously provided by specialists. Artificial intelligence (I.A.) resources will enable the analysis of images to help compose the knowledge base of the application, developed using cloud’s APIs.

Efficiency and patient safety are top priorities in any surgical operation. One effective way to achieve these objectives is automating many of the logistical and routine tasks that occur in the operating room. Inspired by smart assistant technology already commonplace in the consumer sector, we engineered the Smart Hospital Assistant (SHA), a smart, voice-controlled virtual assistant that handles natural speech recognition while executing a plurality of functions to aid surgery. Simulated surgeries showed that the SHA reduced operating time, optimized surgical staff resources, and reduced the number of major touch points that can lead to surgical site infections. The SHA not only shows its potential in the operating room, but also in other healthcare environments that may benefit from having virtual smart assistant technology.

Purpose: In recent years, endovascular treatment has become the dominant approach to treat intracranial aneurysms (IAs). Despite tremendous improvement in surgical devices and techniques, 10-30% of these surgeries require retreatment. Previously, we developed a method which combines quantitative angiography with data-driven modeling to predict aneurysm occlusion within a fraction of a second. This is the first report on a semi-autonomous system, which can predict the surgical outcome of an IA immediately following device placement, allowing for therapy adjustment. Additionally, we previously reported various algorithms which can segment IAs, extract hemodynamic parameters via angiographic parametric imaging, and perform occlusion predictions. Methods: We integrated these features into an Aneurysm Occlusion Assistant (AnOA) utilizing the Kivy library’s graphical instructions and unique language properties for interface development, while the machine learning algorithms were entirely developed within Keras, Tensorflow and skLearn. The interface requires pre- and post-device placement angiographic data. The next steps for aneurysm segmentation, angiographic analysis and prediction have been integrated allowing either autonomous or interactive use. Results: The interface allows for segmentation of IAs and cranial vasculature with a dice index of ~0.78 and prediction of aneurysm occlusion at six months with an accuracy 0.84, in 6.88 seconds. Conclusion: This is the first report on the AnOA to guide endovascular treatment of IAs. While this initial report is on a stand-alone platform, the software can be integrated in the angiographic suite allowing direct communication with the angiographic system for a completely autonomous surgical guidance solution.

In the current COVID-19 pandemic situation, there is an urgent need to screen infected patients quickly and accurately. Using deep learning models trained on chest X-ray images can become an efficient method for screening COVID-19 patients in these situations. Deep learning approaches are already widely used in the medical community. However, they require a large amount of data to be accurate. The open-source community1 collectively has made efforts to collect and annotate the data, but it is not enough to train an accurate deep learning model. Few-shot learning2 is a sub-field of machine learning that aims to learn the objective with less amount of data. In this work, we have experimented with well-known solutions for data scarcity in deep learning to detect COVID-19. These include data augmentation, transfer learning, and few-shot learning, and unsupervised learning. We have also proposed a custom few-shot learning approach to detect COVID-19 using siamese networks.3 Our experimental results showcased that we can implement an efficient and accurate deep learning model for COVID-19 detection by adopting the few-shot learning approaches even with less amount of data. Using our proposed approach we were able to achieve 96.4% accuracy an improvement from 83% using baseline models. Our code is available on github: https://github.com/shruti-jadon/Covid-19-Detection

With the sudden outbreak of the novel coronavirus pandemic, patient management protocols have varied across countries. Although imaging is not used for screening due to workflow and resource limitations, studies have shown that Computed tomography(CT) has higher sensitivity than RT-PCR kits in detecting COVID-19. However, since CT requires greater acquisition time and in turn more radiation exposure, X-Ray has been the imaging modality more commonly used. We have developed an analysis protocol to monitor COVID-19 positive patients more efficiently and with greater detail. Our proposed method generates a digitally reconstructed 2D radiographic projection (DRR) from the CT scan, performs Lung segmentation on the coronal CT scan and X-Ray for initial co-registration and finally, fine-tunes the registration using Optimization-based techniques. The difference in the infection area as time progresses can then be monitored on the subsequent X-Ray images throughout the patient’s recovery. The proposed method was evaluated on retrospective data of five COVID-19 positive patients from a single hospital. The patients received a CT within the 1st five days of admission and were followed-up with X-Ray through their recovery process. Our generated DRR’s from the CT showed successful registration to the follow-up X-Rays.

The goal of this study is to demonstrate the classification value of the latent encodings of a neural network trained for image segmentation of the lung region. In order to achieve this, the gold standard of semantic segmentation, a 3D U-Net was used to extract the encodings for 20 thoracic CT images (10 COVID-19 and 10 Control), and a random forest classifier was trained based on the encodings developed from two training experiments. Performance was analyzed in terms of the independent classification value of each voxel of the U-Net’s latent encoding layer in distinguishing COVID-19 v/s control images.

We developed and evaluated the effect of deep radiomic features, called U-radiomics, on the prediction of the overall survival of patients with the coronavirus disease 2019 (COVID-19). A U-net was trained on chest CT images of patients with interstitial lung diseases to classify lung regions of interest into five characteristic lung tissue patterns. The trained Unet was applied to the chest CT images of patients with COVID-19, and a U-radiomics vector for each patient was identified from the bottleneck layer of the U-net across all the axial CT images of the patient. The U-radiomics vector was subjected to a Cox proportional hazards model with an elastic-net penalty for predicting the survival of the patient. The evaluation was performed by use of bootstrapping, where the concordance index (C-index) was used as the comparative performance metric. Our preliminary comparative evaluation of existing prognostic biomarkers and the proposed U-survival model yielded the C-index values of (a) extent of well-aerated lung parenchyma: 51%, (b) combination of blood tests of lactic dehydrogenase, lymphocyte, and C-reactive protein: 63%, and (c) U-survival: 87%. Thus, the U-survival significantly outperformed clinical biomarkers in predicting the survival of COVID-19 patients, indicating that the U-radiomics vector of the U-survival model may provide a highly accurate prognostic biomarker for patients with COVID-19.

Oncotype DX recurrence score (RS) is increasingly used to differentiate patients at high risk of recurrence from those who have low risk of recurrence. Despite the promising value, this genetic technology has disadvantage in its expensive and currently not readily available in most of the institute, which limited the clinical applications in management of breast cancer. Dynamic contrast enhanced magnetic resonance imaging (DCE-MRI) is becoming standard-of-care in breast cancer management. It is of vital importance to predict Oncotype DX RS using radiomics from DCE-MRI to perform a noninvasive evaluation of recurrence in breast cancer. To this end, 131 patients were included in the dataset, of which 13 (9.9%) had a low risk of recurrence (RS<18), 74 (56.5%) samples had a moderate risk of recurrence (18≤RS<31), and 44 (33.6%) patients had a high risk of recurrence (RS≥31). Among these, 77 samples were used as the training set and 54 were testing set. Univariate and multivariate regression analyses were performed to evaluate the effectiveness of the radiomics. Specifically, in the multivariate regression analysis, an elastic network regression model was established with a ten-fold cross-validation method to evaluate the prediction performance. A total of 479 features were extracted from DCE-MRI, and 6 clinicopathologic indicators were included. After proper feature selection, 20 features were remained and was used for subsequent analysis. In the univariate linear regression analysis, 11 imaging features and 3 clinicopathologic features (i.e., PR, Ki-67 and molecular classification) were significantly correlated with RS (P<0.05). Multivariate model using 6 radiomic features generated the prediction performance in terms of R square 0.242 (P=0.0352) on the testing set. The prediction model yields an improved performance in terms of R square of 0.308 (P=0.0236) after combing clinicopathological factors. The results showed that DCE-MRI radiomics combined with clinicopathologic indicators would be promising in the risk of recurrence evaluation in breast cancer.

The purpose of this study was to evaluate the ability of a developed computer aided glaucoma screening system to screen for glaucoma using a Food Drug Administration (FDA) Class II diagnostic digital fundus photography system used for diabetic retinopathy screening (DRS). The fundus photos collected from participants underwent a comprehensive eye examination as well as non-mydriatic 45° single photograph retinal imaging centered on the macula. Optic nerve images within the 45° non-mydriatic and non-stereo DRS images (The Retinal fundus Images for Glaucoma Analysis: the RIGA2 dataset) were evaluated by a computer-aided automated segmentation system to determine the vertical cup-to-disc ratio (VCDR). The VCDR from clinical assessment was considered as gold standard, VCDR results from the computer system was compared to that from clinical assessment. The grading agreement was assessed by computing intraclass correlation coefficient (ICC). In addition, sensitivity and specificity were calculated. Among 245 fundus photos, 166 images met quality specifications for analysis. The ICC value for the VCDR between the gold standard clinical exam and the automated segmentation system was 0.41, indicating fair agreement. The specificity and sensitivity for (0.6 VCDR) were 76% and 47% respectively.

The aim of this study is to predict the head and neck squamous cell carcinomas (HNSCs) patient metastasis using PET radiomics with RNA-sequencing data. We performed Gene set enrichment analysis (GSEA) and identified 72 genes have important roles as Epithelial mesenchymal transition (EMT) functional modules by the mount of gene expression pattern during the cancer metastasis. The 47 features were extracted form PET images by local image features extraction. GLZLM_LZHGE and CXCL6, SHAPE_Volume and CLCL6, GLCM_Energy and COL11A1 identified as a high relation by P-value. The test and training value PETr and FEG were 0.45 and 0.50 in LR and 0.75 and 0.83 in GB, respectively.

We compare axial 2D U-Nets and their 3D counterparts for pixel/voxel-based segmentation of five abdominal organs in CT scans. For each organ, two competing CNNs are trained. They are evaluated by performing five-fold cross-validation on 80 3D images. In a two-step concept, the relevant area containing the organ is first extracted by detected bounding boxes and then passed as input to the organ-specific U-Net. Furthermore, a random regression forest approach for the automatic detection of bounding boxes is summarized from our previous work. The results show that the 2D U-Net is mostly on par with the 3D U-Net or even outperforms it. Especially for the kidneys, it is significantly better suited in this study.

A practical method to analyze blood vessels, like the aorta, is to calculate the vessel's centerline and evaluate its shape in a CT or CTA scan. This contribution introduces a cloud-based centerline tool for the aorta, which computes an initial centerline from a CTA scan with two user given seed points. Afterwards, this initial centerline can be smoothed in a second step. The work done for this contribution was implemented into an existing online tool for medical image analysis, called Studierfenster. In order to evaluate the outcome of this contribution, we tested the smoothed centerline computed within Studierfenster against 40 baseline centerlines from a public available CTA challenge dataset. In doing so, we computed a minimum, maximum, and mean distance between the two centerlines in mm for every data sample, resulting in the smallest distance of 0.59mm, an overall maximum distance of 14.18mm, and a mean distance for all samples of 3.86mm with a standard deviation of 0.99mm.

CT colonography (CTC) uses abdominal CT scans to examine the colon for cancers and polyps. To visualize the complete region of the colon without the obstructing residual materials inside the colon, an orally administered contrast agent is used to opacify the residual fecal materials on CT images, followed by virtual cleansing of the opacified materials from the images. We developed a self-supervised 3D generative adversarial network model based on residual blocks (ResBlocks), called 3D-ResNet-GAN, for performing electronic cleansing (EC) in CTC. In this model, the convolution layers of the generator network are implemented with ResBlocks for enhancing the EC performance of the generator network over that of our previously developed U-net-based 3D-GAN EC model. We compared the performance of the proposed selfsupervised 3D-ResNet-GAN EC scheme with that of the previous 3D-GAN EC scheme by the use of an anthropomorphic phantom and a clinical CTC case with submerged polyps in single-energy CTC and dual-energy CTC. Our preliminary quantitative evaluation based on the phantom indicated that the proposed 3D-ResNet-GAN EC scheme can yield a statistically significant improvement over the EC performance of our previous 3D-GAN EC scheme in both single-energy and dual-energy CTC. Our preliminary visual evaluation based on the clinical CTC case indicated that the use of ResBlocks and dual-energy CTC yields the best EC result.