Guest editors Claudia Mello-Thoms, Craig Abbey, and Elizabeth A. Krupinski introduce the Special Series on 2D and 3D Imaging: Perspectives in Human and Model Observer Performance.

Purpose: To clarify whether and to what extent three-dimensional (3D) convolutional neural network (CNN) is superior to 2D CNN when applied to reduce false-positive nodule detections in the scenario of low-dose computed tomography (CT) lung cancer screening.

Approach: We established a dataset consisting of 1600 chest CT examinations acquired on different subjects from various sources. There were in total 18,280 candidate nodules in these CT examinations, among which 9185 were nodules and 9095 were not nodules. For each candidate nodule, we extracted a number of cubic subvolumes with a dimension of 72  ×  72  ×  72  mm³ by rotating the CT examinations randomly for 25 times prior to the extraction of the axis-aligned subvolumes. These subvolumes were split into three groups in a ratio of 8  ∶  1  ∶  1 for training, validation, and independent testing purposes. We developed a multiscale CNN architecture and implemented its 2D and 3D versions to classify pulmonary nodules into two categories, namely true positive and false positive. The performance of the 2D/3D-CNN classification schemes was evaluated using the area under the receiver operating characteristic curves (AUC). The p-values and the 95% confidence intervals (CI) were calculated.

Results: The AUC for the optimal 2D-CNN model is 0.9307 (95% CI: 0.9285 to 0.9330) with a sensitivity of 92.70% and a specificity of 76.21%. The 3D-CNN model with the best performance had an AUC of 0.9541 (95% CI: 0.9495 to 0.9583) with a sensitivity of 89.98% and a specificity of 87.30%. The developed multiscale CNN architecture had a better performance than the vanilla architecture did.

Conclusions: The 3D-CNN model has a better performance in false-positive reduction compared with its 2D counterpart; however, the improvement is relatively limited and demands more computational resources for training purposes.

Purpose: Physicians’ eye movements provide insights into relative reliance on different visual features during medical image review and diagnosis. Current theories posit that increasing expertise is associated with relatively holistic viewing strategies activated early in the image viewing experience. This study examined whether early image viewing behavior is associated with experience level and diagnostic accuracy when pathologists and trainees interpreted breast biopsies.

Approach: Ninety-two residents in training and experienced pathologists at nine major U.S. medical centers interpreted digitized whole slide images of breast biopsy cases while eye movements were monitored. The breadth of visual attention and frequency and duration of eye fixations on critical image regions were recorded. We dissociated eye movements occurring early during initial viewing (prior to first zoom) versus later viewing, examining seven viewing behaviors of interest.

Results: Residents and faculty pathologists were similarly likely to detect critical image regions during early image viewing, but faculty members showed more and longer duration eye fixations in these regions. Among pathology residents, year of residency predicted increasingly higher odds of fixating on critical image regions during early viewing. No viewing behavior was significantly associated with diagnostic accuracy.

Conclusions: Results suggest early detection and recognition of critical image features by experienced pathologists, with relatively directed and efficient search behavior. The results also suggest that the immediate distribution of eye movements over medical images warrants further exploration as a potential metric for the objective monitoring and evaluation of progress during medical training.

Purpose: Conventional stenosis quantification from single-energy computed tomography (SECT) images relies on segmentation of lumen boundaries, which suffers from partial volume averaging and calcium blooming effects. We present and evaluate a method for quantifying percent area stenosis using multienergy CT (MECT) images.

Approach: We utilize material decomposition of MECT images to measure stenosis based on the ratio of iodine mass between vessel locations with and without a stenosis, thereby eliminating the requirement for segmentation of iodinated lumen. The method was first assessed using simulated MECT images created with different spatial resolutions. To experimentally assess this method, four phantoms with different stenosis severity (30% to 51%), vessel diameters (5.5 to 14 mm), and calcification densities (700 to 1100  mgHA  /  cc) were fabricated. Conventional SECT images were acquired using a commercial CT system and were analyzed with commercial software. MECT images were acquired using a commercial dual-energy CT (DECT) system and also from a research photon-counting detector CT (PCD-CT) system. Three-material-decomposition was performed on MECT data, and iodine density maps were used to quantify stenosis. Clinical radiation doses were used for all data acquisitions.

Results: Computer simulation verified that this method reduced partial volume and blooming effects, resulting in consistent stenosis measurements. Phantom experiments showed accurate and reproducible stenosis measurements from MECT images. For DECT and two-threshold PCD-CT images, the estimation errors were 4.0% to 7.0%, 2.0% to 9.0%, 10.0% to 18.0%, and −1.0  %   to −5.0  %   (ground truth: 51%, 51%, 51%, and 30%). For four-threshold PCD-CT images, the errors were 1.0% to 3.0%, 4.0% to 6.0%, −1.0  %   to 9.0%, and 0.0% to 6.0%. Errors using SECT were much larger, ranging from 4.4% to 46%, and were especially worse in the presence of dense calcifications.

Conclusions: The proposed approach was shown to be insensitive to acquisition parameters, demonstrating the potential to improve the accuracy and precision of stenosis measurements in clinical practice.

Purpose: Inverting the discrete x-ray transform (DXT) with the nonlinear partial volume (NLPV) effect, which we refer to as the NLPV DXT, remains of theoretical and practical interest. We propose an optimization-based algorithm for accurately and directly inverting the NLPV DXT.

Methods: Formulating the inversion of the NLPV DXT as a nonconvex optimization program, we propose an iterative algorithm, referred to as the nonconvex primal-dual (NCPD) algorithm, to solve the problem. We obtain the NCPD algorithm by modifying a first-order primal-dual algorithm to address the nonconvex optimization. Subsequently, we perform quantitative studies to verify and characterize the NCPD algorithm.

Results: In addition to proposing the NCPD algorithm, we perform numerical studies to verify that the NCPD algorithm can reach the devised numerically necessary convergence conditions and, under the study conditions considered, invert the NLPV DXT by yielding numerically accurate image reconstruction.

Conclusion: We have developed and verified with numerical studies the NCPD algorithm for accurate inversion of the NLPV DXT. The study and results may yield insights into the effective compensation for the NLPV artifacts in CT imaging and into the algorithm development for nonconvex optimization programs in CT and other tomographic imaging technologies.

Purpose: Photon-counting silicon strip detectors are attracting interest for use in next-generation CT scanners. For CT detectors in a clinical environment, it is desirable to have a low power consumption. However, decreasing the power consumption leads to higher noise. This is particularly detrimental for silicon detectors, which require a low noise floor to obtain a good dose efficiency. The increase in noise can be mitigated using a longer shaping time in the readout electronics. This also results in longer pulses, which requires an increased deadtime, thereby degrading the count-rate performance. However, as the photon flux varies greatly during a typical CT scan, not all projection lines require a high count-rate capability. We propose adjusting the shaping time to counteract the increased noise that results from decreasing the power consumption.

Approach: To show the potential of increasing the shaping time to decrease the noise level, synchrotron measurements were performed using a detector prototype with two shaping time settings. From the measurements, a simulation model was developed and used to predict the performance of a future channel design.

Results: Based on the synchrotron measurements, we show that increasing the shaping time from 28.1 to 39.4 ns decreases the noise and increases the signal-to-noise ratio with 6.5% at low count rates. With the developed simulation model, we predict that a 50% decrease in power can be attained in a proposed future detector design by increasing the shaping time with a factor of 1.875.

Conclusion: Our results show that the shaping time can be an important tool to adapt the pulse length and noise level to the photon flux and thereby optimize the dose efficiency of photon-counting silicon detectors.

Purpose: Our study predictively maps epithelium density in magnetic resonance imaging (MRI) space while varying the ground truth labels provided by five pathologists to quantify the downstream effects of interobserver variability.

Approach: Clinical imaging and postsurgical tissue from 48 recruited prospective patients were used in our study. Tissue was sliced to match the MRI orientation and whole-mount slides were stained and digitized. Data from 28 patients (n  =  33 slides) were sent to five pathologists to be annotated. Slides from the remaining 20 patients (n  =  123 slides) were annotated by one of the five pathologists. Interpathologist variability was measured using Krippendorff’s alpha. Pathologist-specific radiopathomic mapping models were trained using a partial least-squares regression using MRI values to predict epithelium density, a known marker for disease severity. An analysis of variance characterized intermodel means difference in epithelium density. A consensus model was created and evaluated using a receiver operator characteristic classifying high grade versus low grade and benign, and was statistically compared to apparent diffusion coefficient (ADC).

Results: Interobserver variability ranged from low to acceptable agreement (0.31 to 0.69). There was a statistically significant difference in mean predicted epithelium density values (p  <  0.001) between the five models. The consensus model outperformed ADC (areas under the curve = 0.80 and 0.71, respectively, p  <  0.05).

Conclusion: We demonstrate that radiopathomic maps of epithelium density are sensitive to the pathologist annotating the dataset; however, it is unclear if these differences are clinically significant. The consensus model produced the best maps, matched the performance of the best individual model, and outperformed ADC.

Purpose: In vivo optical imaging technologies like high-resolution microendoscopy (HRME) can image nuclei of the oral epithelium. In principle, automated algorithms can then calculate nuclear features to distinguish neoplastic from benign tissue. However, images frequently contain regions without visible nuclei, due to biological and technical factors, decreasing the data available to and accuracy of image analysis algorithms.

Approach: We developed the nuclear density-confidence interval (ND-CI) algorithm to determine if an HRME image contains sufficient nuclei for classification, or if a better image is required. The algorithm uses a convolutional neural network to exclude image regions without visible nuclei. Then the remaining regions are used to estimate a confidence interval (CI) for the number of abnormal nuclei per mm², a feature used by a previously developed algorithm (called the ND algorithm), to classify images as benign or neoplastic. The range of the CI determines whether the ND-CI algorithm can classify an image with confidence, and if so, the predicted category. The ND and ND-CI algorithm were compared by calculating their positive predictive value (PPV) and negative predictive value (NPV) on 82 oral biopsies with histopathologically confirmed diagnoses.

Results: After excluding the images that could not be classified with confidence, the ND-CI algorithm had higher PPV (65% versus 59%) and NPV (78% versus 75%) than the ND algorithm.

Conclusions: The ND-CI algorithm could improve the real-time classification of HRME images of the oral epithelium by informing the user if an improved image is required for diagnosis.

Purpose: Since breast mass is a clear sign of breast cancer, its precise segmentation is of great significance for the diagnosis of breast cancer. However, the current diagnosis relies mainly on radiologists who spend time extracting features manually, which inevitably reduces the efficiency of diagnosis. Therefore, designing an automatic segmentation method is urgently necessary for the accurate segmentation of breast masses.

Approach: We propose an effective attention mechanism and multiscale pooling conditional generative adversarial network (AM-MSP-cGAN), which accurately achieves mass automatic segmentation in whole mammograms. In AM-MSP-cGAN, U-Net is utilized as a generator network by incorporating attention mechanism (AM) into it, which allows U-Net to pay more attention to the target mass regions without additional cost. As a discriminator network, a convolutional neural network with multiscale pooling module is used to learn more meticulous features from the masses with rough and fuzzy boundaries. The proposed model is trained and tested on two public datasets: CBIS-DDSM and INbreast.

Results: Compared with other state-of-the-art methods, the AM-MSP-cGAN can achieve better segmentation results in terms of the dice similarity coefficient (Dice) and Hausdorff distance metrics, achieving top scores of 84.49% and 5.01 on CBIS-DDSM, and 83.92% and 5.81 on INbreast, respectively. Therefore, qualitative and quantitative experiments illustrate that the proposed model is effective and robust for the mass segmentation in whole mammograms.

Conclusions: The proposed deep learning model is suitable for the automatic segmentation of breast masses, which provides technical assistance for subsequent pathological structure analysis.

Purpose: Accurate segmentation of treatment planning computed tomography (CT) images is important for radiation therapy (RT) planning. However, low soft tissue contrast in CT makes the segmentation task challenging. We propose a two-step hierarchical convolutional neural network (CNN) segmentation strategy to automatically segment multiple organs from CT.

Approach: The first step generates a coarse segmentation from which organ-specific regions of interest (ROIs) are produced. The second step produces detailed segmentation of each organ. The ROIs are generated using UNet, which automatically identifies the area of each organ and improves computational efficiency by eliminating irrelevant background information. For the fine segmentation step, we combined UNet with a generative adversarial network. The generator is designed as a UNet that is trained to segment organ structures and the discriminator is a fully convolutional network, which distinguishes whether the segmentation is real or generator-predicted, thus improving the segmentation accuracy. We validated the proposed method on male pelvic and head and neck (H&N) CTs used for RT planning of prostate and H&N cancer, respectively. For the pelvic structure segmentation, the network was trained to segment the prostate, bladder, and rectum. For H&N, the network was trained to segment the parotid glands (PG) and submandibular glands (SMG).

Results: The trained segmentation networks were tested on 15 pelvic and 20 H&N independent datasets. The H&N segmentation network was also tested on a public domain dataset (N  =  38) and showed similar performance. The average dice similarity coefficients (mean  ±  SD) of pelvic structures are 0.91  ±  0.05 (prostate), 0.95  ±  0.06 (bladder), 0.90  ±  0.09 (rectum), and H&N structures are 0.87  ±  0.04 (PG) and 0.86  ±  0.05 (SMG). The segmentation for each CT takes <10  s on average.

Conclusions: Experimental results demonstrate that the proposed method can produce fast, accurate, and reproducible segmentation of multiple organs of different sizes and shapes and show its potential to be applicable to different disease sites.

Purpose: Deep learning (DL) algorithms have shown promising results for brain tumor segmentation in MRI. However, validation is required prior to routine clinical use. We report the first randomized and blinded comparison of DL and trained technician segmentations.

Approach: We compiled a multi-institutional database of 741 pretreatment MRI exams. Each contained a postcontrast T1-weighted exam, a T2-weighted fluid-attenuated inversion recovery exam, and at least one technician-derived tumor segmentation. The database included 729 unique patients (470 males and 259 females). Of these exams, 641 were used for training the DL system, and 100 were reserved for testing. We developed a platform to enable qualitative, blinded, controlled assessment of lesion segmentations made by technicians and the DL method. On this platform, 20 neuroradiologists performed 400 side-by-side comparisons of segmentations on 100 test cases. They scored each segmentation between 0 (poor) and 10 (perfect). Agreement between segmentations from technicians and the DL method was also evaluated quantitatively using the Dice coefficient, which produces values between 0 (no overlap) and 1 (perfect overlap).

Results: The neuroradiologists gave technician and DL segmentations mean scores of 6.97 and 7.31, respectively (p  <  0.00007). The DL method achieved a mean Dice coefficient of 0.87 on the test cases.

Conclusions: This was the first objective comparison of automated and human segmentation using a blinded controlled assessment study. Our DL system learned to outperform its “human teachers” and produced output that was better, on average, than its training data.

Purpose: Foreign body aspiration is difficult to diagnose because many aspirated foreign bodies are low density or radiolucent. Digital radiographs (DR) are poor at detecting radiolucent foreign bodies. Digital tomosynthesis (DTS) has been shown to be ideally suited for applications where DR is insensitive and the increased dose from computed tomography (CT) is not justified. Our objective was to determine if DTS can be a practical alternative to DR and CT in the diagnosis of foreign body aspiration.

Approach: A phantom approximating the densities of a pediatric chest was constructed. Radiolucent foreign bodies were placed in the airways. Seven pediatric radiologists assessed DTS and DR images with and without simulated breathing motion. Two rounds were performed with fixed exposure techniques and then automatic exposure control techniques. Interobserver agreement was evaluated using Fleiss’ kappa.

Results: DTS and DR images using fixed exposure techniques performed very poorly with accuracies of 42% to 60%. DTS with automatic exposure control techniques increased accuracy to 84% for a stationary phantom, but the accuracy dropped to 70% in a phantom with simulated motion. DTS outperformed DR, with DR accuracies of 60% and 63% for stationary simulations and motion, respectively. Interobserver agreement was poor with Fleiss’ kappa of 0.476.

Conclusion: DTS is superior to DR for radiolucent foreign body detection. However, the overall accuracy and interobserver agreement are likely too low for this modality to be clinically useful.

Purpose: Through the last three decades, functional magnetic resonance imaging (fMRI) has provided immense quantities of information about the dynamics of the brain, functional brain mapping, and resting-state brain networks. Despite providing such rich functional information, fMRI is still not a commonly used clinical technique due to inaccuracy involved in analysis of extremely noisy data. However, ongoing developments in deep learning techniques suggest potential improvements and better performance in many different domains. Our main purpose is to utilize the potentials of deep learning techniques for fMRI data for clinical use.

Approach: We present one such synergy of fMRI and deep learning, where we apply a simplified yet accurate method using a modified 3D convolutional neural networks (CNN) to resting-state fMRI data for feature extraction and classification of Alzheimer’s disease (AD). The CNN is designed in such a way that it uses the fMRI data with much less preprocessing, preserving both spatial and temporal information.

Results: Once trained, the network is successfully able to classify between fMRI data from healthy controls and AD subjects, including subjects in the mild cognitive impairment (MCI) stage. We have also extracted spatiotemporal features useful for classification.

Conclusion: This CNN can detect and differentiate between the earlier and later stages of MCI and AD and hence, it may have potential clinical applications in both early detection and better diagnosis of Alzheimer’s disease.

Purpose: We present an original method for simulating realistic fetal neurosonography images specifically generating third-trimester pregnancy ultrasound images from second-trimester images. Our method was developed using unpaired data, as pairwise data were not available. We also report original insights on the general appearance differences between second- and third-trimester fetal head transventricular (TV) plane images.

Approach: We design a cycle-consistent adversarial network (Cycle-GAN) to simulate visually realistic third-trimester images from unpaired second- and third-trimester ultrasound images. Simulation realism is evaluated qualitatively by experienced sonographers who blindly graded real and simulated images. A quantitative evaluation is also performed whereby a validated deep-learning-based image recognition algorithm (ScanNav^®) acts as the expert reference to allow hundreds of real and simulated images to be automatically analyzed and compared efficiently.

Results: Qualitative evaluation shows that the human expert cannot tell the difference between real and simulated third-trimester scan images. 84.2% of the simulated third-trimester images could not be distinguished from the real third-trimester images. As a quantitative baseline, on 3000 images, the visibility drop of the choroid, CSP, and mid-line falx between real second- and real third-trimester scans was computed by ScanNav^® and found to be 72.5%, 61.5%, and 67%, respectively. The visibility drop of the same structures between real second-trimester and simulated third-trimester was found to be 77.5%, 57.7%, and 56.2%, respectively. Therefore, the real and simulated third-trimester images were consider to be visually similar to each other. Our evaluation also shows that the third-trimester simulation of a conventional GAN is much easier to distinguish, and the visibility drop of the structures is smaller than our proposed method.

Conclusions: The results confirm that it is possible to simulate realistic third-trimester images from second-trimester images using a modified Cycle-GAN, which may be useful for deep learning researchers with a restricted availability of third-trimester scans but with access to ample second trimester images. We also show convincing simulation improvements, both qualitatively and quantitatively, using the Cycle-GAN method compared with a conventional GAN. Finally, the use of a machine learning-based reference (in the case ScanNav^®) for large-scale quantitative image analysis evaluation is also a first to our knowledge.

Purpose: While mammography has excellent sensitivity for the detection of breast lesions, its specificity is limited. Adjunct screening with ultrasound may partially alleviate this issue but also increases false positives, resulting in unnecessary biopsies. Our study investigated the use of Google AutoML Vision (Mountain View, California), a commercially available machine learning service, to both identify and characterize indeterminate breast lesions on ultrasound.

Approach: B-mode images from 253 independent cases of indeterminate breast lesions scheduled for core biopsy were used for model creation and validation. The performances of two sub-models from AutoML Vision, the image classification model and object detection model, were evaluated, while also investigating training strategies to enhance model performances. Pathology from the patient’s biopsy was used as a reference standard.

Results: The image classification models trained under different conditions demonstrated areas under the precision–recall curve (AUC) ranging from 0.85 to 0.96 during internal validation. Once deployed, the model with highest internal performance demonstrated a sensitivity of 100% [95% confidence interval (CI) of 73.5% to 100%], specificity of 83.3% (CI  =  51.6  %   to 97.9%), positive predictive value (PPV) of 85.7% (CI  =  62.9  %   to 95.5%), and negative predictive value (NPV) of 100% (CI non-evaluable) in an independent dataset. The object detection model demonstrated lower performance internally during development (AUC  =  0.67) and during prediction in the independent dataset [sensitivity  =  75  %   (CI  =  42.8 to 94.5), specificity  =  80  %   (CI  =  51.9 to 95.7), PPV  =  75  %   (CI  =  50.8 to 90.0), and NPV  =  80  %   (CI  =  59.3  %   to 91.7%)], but was able to demonstrate the location of the lesion within the image.

Conclusions: Two models appear to be useful tools for identifying and classifying suspicious areas on B-mode images of indeterminate breast lesions.

Purpose: Utilization of computer-aided diagnosis (CAD) on radiological ultrasound (US) imaging has increased tremendously. The prominent CAD applications are found in breast and thyroid cancer investigation. To make appropriate clinical recommendations, it is important to accurately segment the cancerous object called a lesion. Segmentation is a crucial step but undoubtedly a challenging problem due to various perturbations, e.g., speckle noise, intensity inhomogeneity, and low contrast.

Approach: We present a combinatorial framework for US image segmentation using a bilateral filter (BF) and hybrid region-edge-based active contour (AC) model. The BF is adopted to smooth images while preserving edges. Then the hybrid model of region and edge-based AC is applied along the scales in a global-to-local manner to capture the lesion areas. The framework was tested in segmenting 258 US images of breast and thyroid, which were validated by manual ground truths.

Results: The proposed framework is accessed quantitatively based on the overlapping values of the Dice coefficient, which reaches 90.05  ±  5.81  %  . The evaluation with and without the BF shows that the enhancement procedure improves the framework well.

Conclusions: The high performance of the proposed method in our experimental results indicates its potential for practical implementations in CAD radiological US systems.

Purpose: Prostate cancer primarily arises from the glandular epithelium. Histomophometric techniques have been used to assess the glandular epithelium in automated detection and classification pipelines; however, they are often rigid in their implementation, and their performance suffers on large datasets where variation in staining, imaging, and preparation is difficult to control. The purpose of this study is to quantify performance of a pixelwise segmentation algorithm that was trained using different combinations of weak and strong stroma, epithelium, and lumen labels in a prostate histology dataset.

Approach: We have combined weakly labeled datasets generated using simple morphometric techniques and high-quality labeled datasets from human observers in prostate biopsy cores to train a convolutional neural network for use in whole mount prostate labeling pipelines. With trained networks, we characterize pixelwise segmentation of stromal, epithelium, and lumen (SEL) regions on both biopsy core and whole-mount H&E-stained tissue.

Results: We provide evidence that by simply training a deep learning algorithm on weakly labeled data generated from rigid morphometric methods, we can improve the robustness of classification over the morphometric methods used to train the classifier.

Conclusions: We show that not only does our approach of combining weak and strong labels for training the CNN improve qualitative SEL labeling within tissue but also the deep learning generated labels are superior for cancer classification in a higher-order algorithm over the morphometrically derived labels it was trained on.

Purpose: Deep learning models are showing promise in digital pathology to aid diagnoses. Training complex models requires a significant amount and diversity of well-annotated data, typically housed in institutional archives. These slides often contain clinically meaningful markings to indicate regions of interest. If slides are scanned with the ink present, then the downstream model may end up looking for regions with ink before making a classification. If scanned without the markings, the information regarding where the relevant regions are located is lost. A compromise solution is to scan the slide with the annotations present but digitally remove them.

Approach: We proposed a straightforward framework to digitally remove ink markings from whole slide images using a conditional generative adversarial network based on Pix2Pix.

Results: The peak signal-to-noise ratio increased 30%, structural similarity index increased 20%, and visual information fidelity increased 200% relative to previous methods.

Conclusions: When comparing our digital removal of marked images with rescans of clean slides, our method qualitatively and quantitatively exceeds current benchmarks, opening the possibility of using archived clinical samples as resources to fuel the next generation of deep learning models for digital pathology.