This guest editorial introduces the special section on Medical Image Perceptions and Observer Performance.

Identifying if participants with differing diagnostic accuracy and visual search behavior during radiologic tasks also differ in nonradiologic tasks is investigated. Four clinician groups with different radiologic experience were used: a reference expert group of five consultant radiologists, four radiology registrars, five senior house officers, and six interns. Each of the four clinician groups is known to have significantly different performance in the identification of pneumothoraces in chest x-ray. Each of the 20 participants was shown 6 nonradiologic images (3 maps and 3 sets of geometric shapes) and was asked to perform search tasks. Eye movements were recorded with a Tobii TX300 (Tobii Technology, Stockholm, Sweden) eye tracker. Four eye-tracking metrics were analyzed. Variables were compared to identify any differences among the groups. All data were compared by using nonparametric methods of analysis. The average number of targets identified in the maps did not change among groups [mean=5.8 of 6 targets (range 5.6 to 6 p=0.861)]. None of the four eye-tracking metrics investigated varied with experience in either search task (p>0.5). Despite clear differences in radiologic experience, these clinician groups showed no difference in nonradiologic search pattern behavior or skill across complex images. This is another viewpoint adding to the evidence that radiologic image interpretation is a learned skill and is task specific.

This study’s purpose was to develop and validate a method to estimate patient-specific detectability indices directly from patients’ CT images (i.e., in vivo). The method extracts noise power spectrum (NPS) and modulation transfer function (MTF) resolution properties from each patient’s CT series based on previously validated techniques. These are combined with a reference task function (10-mm disk lesion with −15 HU contrast) to estimate detectability indices for a nonprewhitening matched filter observer model. This method was applied to CT data from a previous study in which diagnostic performance of 16 readers was measured for the task of detecting subtle, hypoattenuating liver lesions (N=105), using a two-alternative-forced-choice (2AFC) method, over six dose levels and two reconstruction algorithms. In vivo detectability indices were estimated and compared to the human readers’ binary 2AFC outcomes using a generalized linear mixed-effects statistical model. The results of this modeling showed that the in vivo detectability indices were strongly related to 2AFC outcomes (p<0.05). Linear comparison between human-detection accuracy and model-predicted detection accuracy (for like conditions) resulted in Pearson and Spearman correlation coefficients exceeding 0.84. These results suggest the potential utility of using in vivo estimates of a detectability index for an automated image quality tracking system that could be implemented clinically.

Volume of lung nodules is an important biomarker, quantifiable from computed tomography (CT) images. The usefulness of volume quantification, however, depends on the precision of quantification. Experimental assessment of precision is time consuming. A mathematical estimability model was used to assess the quantification precision of CT nodule volumetry in terms of an index (e′), incorporating image noise and resolution, nodule properties, and segmentation software. The noise and resolution were characterized in terms of noise power spectrum and task transfer function. The nodule properties and segmentation algorithm were modeled in terms of a task function and a template function, respectively. The e′ values were benchmarked against experimentally acquired precision values from an anthropomorphic chest phantom across 54 acquisition protocols, 2 nodule sizes, and 2 volume segmentation softwares. e′ exhibited correlation with experimental precision across nodule sizes and acquisition protocols but dependence on segmentation software. Compared to the assessment of empirical precision, which required ∼300  h to perform the segmentation, the e′ method required ∼3  h from data collection to mathematical computation. A mathematical modeling of volume quantification provides efficient prediction of quantitative performance. It establishes a method to verify quantitative compliance and to optimize clinical protocols for chest CT volumetry.

Retinal blood vessels indicate some serious health ramifications, such as cardiovascular disease and stroke. Thanks to modern imaging technology, high-resolution images provide detailed information to help analyze retinal vascular features before symptoms associated with such conditions fully develop. Additionally, these retinal images can be used by ophthalmologists to facilitate diagnosis and the procedures of eye surgery. A fuzzy noise reduction algorithm was employed to enhance color images corrupted by Gaussian noise. The present paper proposes employing a contrast limited adaptive histogram equalization to enhance illumination and increase the contrast of retinal images captured from state-of-the-art cameras. Possessing directional properties, the multistructure elements method can lead to high-performance edge detection. Therefore, multistructure elements-based morphology operators are used to detect high-quality image ridges. Following this detection, the irrelevant ridges, which are not part of the vessel tree, were removed by morphological operators by reconstruction, attempting also to keep the thin vessels preserved. A combined method of connected components analysis (CCA) in conjunction with a thresholding approach was further used to identify the ridges that correspond to vessels. The application of CCA can yield higher efficiency when it is locally applied rather than applied on the whole image. The significance of our work lies in the way in which several methods are effectively combined and the originality of the database employed, making this work unique in the literature. Computer simulation results in wide-field retinal images with up to a 200-deg field of view are a testimony of the efficacy of the proposed approach, with an accuracy of 0.9524.

What are the costs and consequences of interruptions during diagnostic radiology? The cognitive psychology literature suggests that interruptions lead to an array of negative consequences that could hurt patient outcomes and lead to lower patient throughput. Meanwhile, observational studies have both noted a strikingly high rate of interruptions and rising number of interruptions faced by radiologists. There is some observational evidence that more interruptions could lead to worse patient outcomes: Balint et al. (2014) found that the shifts with more telephone calls received in the reading room were associated with more discrepant calls. The purpose of the current study was to use an experimental manipulation to precisely quantify the costs of two different types of interruption: telephone interruption and an interpersonal interruption. We found that the first telephone interruption led to a significant increase in time spent on the case, but there was no effect on diagnostic accuracy. Eye-tracking revealed that interruptions strongly influenced where the radiologists looked: they tended to spend more time looking at dictation screens and less on medical images immediately after interruption. Our results demonstrate that while radiologists’ eye movements are reliably influenced by interruptions, the behavioral consequences were relatively mild, suggesting effective compensatory mechanisms.

The goal of this research was to examine whether search pattern training for central line positioning on chest radiographs (CXRs) improves the ability of healthcare trainees and practitioners to identify malpositioned central venous catheters. Two sets of CXRs with central catheters were shown; half of the images contained catheters that were appropriately positioned, half that were malpositioned. Subjects were asked to: mark the tip of the catheter using the simulated radiology workstations, indicate their confidence in tip localization, and state whether the catheter was appropriately positioned or malpositioned. Subjects were also given a survey assessing their thoughts about the usefulness of search pattern training and the simulated radiology workstation. There was a significant improvement in subjects’ ability to classify a catheter as malpositioned after training, p-value = 0.03. There was no significant difference in localization of the catheter tips or in the confidence for tip localization. Subjects’ responses to the questionnaire were significantly positive for all statements, indicating that they felt search pattern training using a simulated radiology workstation had a positive impact on their education. These results suggest that our knowledge of medical image perception may be useful for developing rational educational tools for image interpretation, and that simulated radiology workstations may be a helpful means of deploying these tools.

Prior research has shown that physicians’ medical decisions can be influenced by sequential context, particularly in cases where successive stimuli exhibit similar characteristics when analyzing medical images. This type of systematic error is known to psychophysicists as sequential context effect as it indicates that judgments are influenced by features of and decisions about the preceding case in the sequence of examined cases, rather than being based solely on the peculiarities unique to the present case. We determine if radiologists experience some form of context bias, using screening mammography as the use case. To this end, we explore correlations between previous perceptual behavior and diagnostic decisions and current decisions. We hypothesize that a radiologist’s visual search pattern and diagnostic decisions in previous cases are predictive of the radiologist’s current diagnostic decisions. To test our hypothesis, we tasked 10 radiologists of varied experience to conduct blind reviews of 100 four-view screening mammograms. Eye-tracking data and diagnostic decisions were collected from each radiologist under conditions mimicking clinical practice. Perceptual behavior was quantified using the fractal dimension of gaze scanpath, which was computed using the Minkowski–Bouligand box-counting method. To test the effect of previous behavior and decisions, we conducted a multifactor fixed-effects ANOVA. Further, to examine the predictive value of previous perceptual behavior and decisions, we trained and evaluated a predictive model for radiologists’ current diagnostic decisions. ANOVA tests showed that previous visual behavior, characterized by fractal analysis, previous diagnostic decisions, and image characteristics of previous cases are significant predictors of current diagnostic decisions. Additionally, predictive modeling of diagnostic decisions showed an overall improvement in prediction error when the model is trained on additional information about previous perceptual behavior and diagnostic decisions.

Frequently, the consensus conclusion after quality assurance conferences in radiology is that whatever mistake was made could have been avoided if more prior images or documents had been consulted. It is generally assumed that anything that was not specifically cited in the report had not been consulted. Is it actually safe to assume that an image or document that is not cited was also not consulted? It is this question that this investigation addresses. In this Institutional Review Board-approved study, one observer watched the board-certified radiologists while they interpreted imaging studies and issued reports. He recorded what type of study was being interpreted [either computed tomography, magnetic resonance imaging, or conventional radiography (x-ray)]. He also recorded the number and type of prior imaging studies and documents that were consulted during the interpretation. These observations were then compared with the signed report to determine how many of the consulted imaging studies and documents were cited. Of the 198 previous imaging studies that the radiologists consulted, 116 (58.6%) were cited in a report. Of the 285 documents consulted, 3 (1.1%) were cited in a report. This difference in citation rate was statistically significant (p  <  0.0001). It cannot be safely assumed that an older radiologic image or medical document was not consulted during radiologic interpretation merely because it is not cited in the report. Radiologists often consult more old studies than they cite, and they do not cite the majority of prior documents that they consult.

The widely used multireader multicase ROC study design for comparing imaging modalities is the fully crossed (FC) design: every reader reads every case of both modalities. We investigate paired split-plot (PSP) designs that may allow for reduced cost and increased flexibility compared with the FC design. In the PSP design, case images from two modalities are read by the same readers, thereby the readings are paired across modalities. However, within each modality, not every reader reads every case. Instead, both the readers and the cases are partitioned into a fixed number of groups and each group of readers reads its own group of cases—a split-plot design. Using a U-statistic based variance analysis for AUC (i.e., area under the ROC curve), we show analytically that precision can be gained by the PSP design as compared with the FC design with the same number of readers and readings. Equivalently, we show that the PSP design can achieve the same statistical power as the FC design with a reduced number of readings. The trade-off for the increased precision in the PSP design is the cost of collecting a larger number of truth-verified patient cases than the FC design. This means that one can trade-off between different sources of cost and choose a least burdensome design. We provide a validation study to show the iMRMC software can be reliably used for analyzing data from both FC and PSP designs. Finally, we demonstrate the advantages of the PSP design with a reader study comparing full-field digital mammography with screen-film mammography.

The ability to correlate anatomical knowledge and medical imaging is crucial to radiology and as such, should be a critical component of medical education. However, we are hindered in our ability to teach this skill because we know very little about what expert practice looks like, and even less about novices’ understanding. Using a unique simulation tool, this research conducted cognitive clinical interviews with experts and novices to explore differences in how they engage in this correlation and the underlying cognitive processes involved in doing so. This research supported what has been known in the literature, that experts are significantly faster at making decisions on medical imaging than novices. It also offers insight into the spatial ability and reasoning that is involved in the correlation of anatomy to medical imaging. There are differences in the cognitive processing of experts and novices with respect to meaningful patterns, organized content knowledge, and the flexibility of retrieval. Presented are some novice–expert similarities and differences in image processing. This study investigated extremes, opening an opportunity to investigate the sequential knowledge acquisition from student to resident to expert, and where educators can help intervene in this learning process.

Mammography is currently the standard imaging modality used to screen women for breast abnormalities, and, as a result, it is a tool of great importance for the early detection of breast cancer. Physical phantoms are commonly used as surrogates of breast tissue to evaluate some aspects of the performance of mammography systems. However, most phantoms do not reproduce the anatomic heterogeneity of real breasts. New fabrication technologies, such as three-dimensional (3-D) printing, have created the opportunity to build more complex, anatomically realistic breast phantoms that could potentially assist in the evaluation of mammography systems. The reproducibility and relative low cost of 3-D printed objects might also enable the development of collections of representative patient models that could be used to assess the effect of anatomical variability on system performance, hence making bench testing studies a step closer to clinical trials. The primary objective of this work is to present a simple, easily reproducible methodology to design and print 3-D objects that replicate the attenuation profile observed in real two-dimensional mammograms. The secondary objective is to evaluate the capabilities and limitations of the competing 3-D printing technologies and characterize the x-ray properties of the different materials they use. Printable phantoms can be created using the open-source code introduced, which processes a raw mammography image to estimate the amount of x-ray attenuation at each pixel, and outputs a triangle mesh object that encodes the observed attenuation map. The conversion from the observed pixel gray value to a column of printed material with equivalent attenuation requires certain assumptions and knowledge of multiple imaging system parameters, such as x-ray energy spectrum, source-to-object distance, compressed breast thickness, and average breast material attenuation. To validate the proposed methodology, x-ray projections of printed phantoms were acquired with a clinical mammography system. The quality of the printing process was evaluated by comparing the mammograms of the printed phantoms and the original mammograms used to create the phantoms. The structural similarity index and the root-mean-square error were used as objective metrics to compare the two images. A detailed description of the software, a characterization of the printed materials using x-ray spectroscopy, and an evaluation of the realism of the sample printed phantoms are presented.

Energy-resolved photon-counting-detector CT (PCD-CT) is promising for material decomposition with multiple contrast agents using two or more energy bins. However, corrections for nonidealities of PCDs are required, which are still active research topics. In addition, PCD-CT is also likely to have a very high cost due to the current lack of mass production capabilities. We proposed an alternative approach to perform multienergy CT (MECT), which is achieved by acquiring triple or quadruple x-ray beam measurements on a dual-source CT scanner. This strategy was based on a “twin-beam” design on a single-source scanner for dual-energy CT. Examples of beam filters and spectra for triple and quadruple x-ray beam were provided. Computer simulation studies were performed to evaluate the noise and accuracy of material decomposition for multiple contrast mixtures using both triple- and quadruple-beam configurations, compared with the performance on a PCD-CT platform. The results demonstrated that the image quality and dose efficiency of the triple-beam configuration in the proposed MECT technique were comparable to that in PCD-CT. The proposed technique can be readily implemented on a dual-source scanner, which may allow material decomposition of multiple contrast agents to be performed on clinical CT scanners with energy-integrating detectors.

Dual-energy microcomputed tomography (DECT) can provide quantitative information about specific materials of interest, facilitating automated segmentation, and visualization of complex three-dimensional tissues. It is possible to implement DECT on currently available preclinical gantry-based cone-beam micro-CT scanners; however, optimal decomposition image quality requires customized spectral shaping (through added filtration), optimized acquisition protocols, and elimination of misregistration artifacts. We present a method for the fabrication of customized x-ray filters—in both shape and elemental composition—needed for spectral shaping. Fiducial markers, integrated within the sample holder, were used to ensure accurate co-registration between sequential low- and high-energy image volumes. The entire acquisition process was automated through the use of a motorized filter-exchange mechanism. We describe the design, implementation, and evaluation of a DECT system on a gantry-based-preclinical cone-beam micro-CT scanner.

Development of advanced preclinical imaging techniques has had an important impact on the field of biomedical research, with positron emission tomography (PET) imaging the most mature of these efforts. Developers of preclinical PET scanners have joined the recent multimodality imaging trend by combining PET imaging with other modalities, such as magnetic resonance imaging (MRI). Our group has developed a combined PET-MRI insert for the imaging of animals up to the size of rats in a clinical 3T MRI scanner. The system utilizes a sequential scanner configuration instead of the more common coplanar geometry. The PET component of the system consists of a ring of 12 liquid-cooled, SiPM-based detector modules (diameter  =  15.2  cm). System performance was evaluated with the NEMA NU 4-2008 protocol. Spatial resolution is ∼1.71  mm 5 cm from the center of the field-of-view measured from single-slice rebinned filtered backprojection-reconstructed images. Peak noise equivalent count rate is 17.7 kcps at 8.5 MBq; peak sensitivity is 2.9%. The MRI component of the system is composed of a 12-cm-diameter birdcage transmit/receive coil with a dual-preamplifier interface possessing very low noise preamplifiers. System performance was evaluated using American College of Radiology-based methods. Image homogeneity is 99%; the ghosting ratio is 0.0054. The signal-to-noise ratio is 95 and spatial resolution is ∼0.25  mm. There was no discernable cross-modality interference.

Magnetic resonance imaging (MRI) provides a number of advantages over computed tomography (CT) for radiation therapy treatment planning; however, MRI lacks the key electron density information necessary for accurate dose calculation. We propose a dictionary-learning-based method to derive electron density information from MRIs. Specifically, we first partition a given MR image into a set of patches, for which we used a joint dictionary learning method to directly predict a CT patch as a structured output. Then a feature selection method is used to ensure prediction robustness. Finally, we combine all the predicted CT patches to obtain the final prediction for the given MR image. This prediction technique was validated for a clinical application using 14 patients with brain MR and CT images. The peak signal-to-noise ratio (PSNR), mean absolute error (MAE), normalized cross-correlation (NCC) indices and similarity index (SI) for air, soft-tissue and bone region were used to quantify the prediction accuracy. The mean ± std of PSNR, MAE, and NCC were: 22.4  ±  1.9  dB, 82.6  ±  26.1 HU, and 0.91  ±  0.03 for the 14 patients. The SIs for air, soft-tissue, and bone regions are 0.98  ±  0.01, 0.88  ±  0.03, and 0.69  ±  0.08. These indices demonstrate the CT prediction accuracy of the proposed learning-based method. This CT image prediction technique could be used as a tool for MRI-based radiation treatment planning, or for PET attenuation correction in a PET/MRI scanner.

Accurate segmentations in medical images are the foundations for various clinical applications. Advances in machine learning-based techniques show great potential for automatic image segmentation, but these techniques usually require a huge amount of accurately annotated reference segmentations for training. The guiding hypothesis of this paper was that crowd-algorithm collaboration could evolve as a key technique in large-scale medical data annotation. As an initial step toward this goal, we evaluated the performance of untrained individuals to detect and correct errors made by three-dimensional (3-D) medical segmentation algorithms. To this end, we developed a multistage segmentation pipeline incorporating a hybrid crowd-algorithm 3-D segmentation algorithm integrated into a medical imaging platform. In a pilot study of liver segmentation using a publicly available dataset of computed tomography scans, we show that the crowd is able to detect and refine inaccurate organ contours with a quality similar to that of experts (engineers with domain knowledge, medical students, and radiologists). Although the crowds need significantly more time for the annotation of a slice, the annotation rate is extremely high. This could render crowdsourcing a key tool for cost-effective large-scale medical image annotation.

Convolutional neural networks (CNNs) have become the architecture of choice for visual recognition tasks. However, these models are perceived as black boxes since there is a lack of understanding of the learned behavior from the underlying task of interest. This lack of transparency is a serious drawback, particularly in applications involving medical screening and diagnosis since poorly understood model behavior could adversely impact subsequent clinical decision-making. Recently, researchers have begun working on this issue and several methods have been proposed to visualize and understand the behavior of these models. We highlight the advantages offered through visualizing and understanding the weights, saliencies, class activation maps, and region of interest localizations in customized CNNs applied to the challenge of classifying parasitized and uninfected cells to aid in malaria screening. We provide an explanation for the models’ classification decisions. We characterize, evaluate, and statistically validate the performance of different customized CNNs keeping every training subject’s data separate from the validation set.

We present a radiomics-based approach developed for the SPIE-AAPM-NCI PROSTATEx challenge. The task was to classify clinically significant prostate cancer in multiparametric (mp) MRI. Data consisted of a “training dataset” (330 suspected lesions from 204 patients) and a “test dataset” (208 lesions/140 patients). All studies included T2-weighted (T2-W), proton density-weighted, dynamic contrast enhanced, and diffusion-weighted imaging. Analysis of the images was performed using the MIM imaging platform (MIM Software, Cleveland, Ohio). Prostate and peripheral zone contours were manually outlined on the T2-W images. A workflow for rigid fusion of the aforementioned images to T2-W was created in MIM. The suspicious lesion was outlined using the high b-value image. Intensity and texture features were extracted on four imaging modalities and characterized using nine histogram descriptors: 10%, 25%, 50%, 75%, 90%, mean, standard deviation, kurtosis, and skewness (216 features). Three classification methods were used: classification and regression trees (CART), random forests, and adaptive least absolute shrinkage and selection operator (LASSO). In the held out by the organizers test dataset, the areas under the curve (AUCs) were: 0.82 (random forests), 0.76 (CART), and 0.76 (adaptive LASSO). AUC of 0.82 was the fourth-highest score of 71 entries (32 teams) and the highest for feature-based methods.

Reducing dose increases noise impacting image quality but can be offset by increasing display luminance. Two contrast detail mammography images were obtained at 26 kV and the same distance between detectors, at 45 and 50 mAs resulting in entrance surface doses of 7.09 and 7.88 mGy, respectively. They were processed to make average gray level of the background independent of the dose level while maintaining original SNR. Eight radiologists viewed the images at 420, 1000  cd  /  m2, and SpotView™ a tool that resulted in an average display luminance of 3138.8  cd  /  m2. Percent correct (PC) for all three luminances was higher for high versus low dose. Performance was always higher with high dose no matter what the luminance. For low dose, PC was highest with SpotView™, and SpotView™ and 1000  cd  /  m2 were significantly higher than 420  cd  /  m2. At high dose, SpotView™ PC was significantly higher than both lower luminances. Average time per image was lower in high dose, and, at both doses, time decreased as luminance increased, with SpotView™ having significantly shorter times. Increasing luminance from 420 to 1000  cd  /  m2 significantly increases target detection by ∼3.0  %   and with SpotView™ by ∼6.2  %  . Increasing display luminance with SpotView™ significantly decreases reading time by 16.0%.

Visual search, the process of detecting and identifying objects using eye movements (saccades) and foveal vision, has been studied for identification of root causes of errors in the interpretation of mammograms. The aim of this study is to model visual search behavior of radiologists and their interpretation of mammograms using deep machine learning approaches. Our model is based on a deep convolutional neural network, a biologically inspired multilayer perceptron that simulates the visual cortex and is reinforced with transfer learning techniques. Eye-tracking data were obtained from eight radiologists (of varying experience levels in reading mammograms) reviewing 120 two-view digital mammography cases (59 cancers), and it has been used to train the model, which was pretrained with the ImageNet dataset for transfer learning. Areas of the mammogram that received direct (foveally fixated), indirect (peripherally fixated), or no (never fixated) visual attention were extracted from radiologists’ visual search maps (obtained by a head mounted eye-tracking device). These areas along with the radiologists’ assessment (including confidence in the assessment) of the presence of suspected malignancy were used to model: (1) radiologists’ decision, (2) radiologists’ confidence in such decisions, and (3) the attentional level (i.e., foveal, peripheral, or none) in an area of the mammogram. Our results indicate high accuracy and low misclassification in modeling such behaviors.

Mammography images undergo vendor-specific processing, which may be nonlinear, before radiologist interpretation. Therefore, to test the entire imaging chain, the effect of image processing should be included in the assessment of image quality, which is not current practice. For this purpose, model observers (MOs), in combination with anthropomorphic breast phantoms, are proposed to evaluate image quality in mammography. In this study, the nonprewhitening MO with eye filter and the channelized Hotelling observer were investigated. The goal of this study was to optimize the efficiency of the procedure to obtain the expected signal template from acquired images for the detection of a 0.25-mm diameter disk. Two approaches were followed: using acquired images with homogeneous backgrounds (approach 1) and images from an anthropomorphic breast phantom (approach 2). For quality control purposes, a straightforward procedure using a single exposure of a single disk was found adequate for both approaches. However, only approach 2 can yield templates from processed images since, due to its nonlinearity, image postprocessing cannot be evaluated using images of homogeneous phantoms. Based on the results of the current study, a phantom should be designed, which can be used for the objective assessment of image quality.

Fluorescence molecular tomography (FMT), as well as mesoscopic FMT (MFMT) is widely employed to investigate molecular level processes ex vivo or in vivo. However, acquiring depth-localized and less blurry reconstruction still remains challenging, especially when fluorophore (dye) is located within large scattering coefficient media. Herein, a two-stage deep learning-based three-dimensional (3-D) reconstruction algorithm is proposed. The key point for the proposed algorithm is to employ a 3-D convolutional neural network to correctly predict the boundary of reconstructions, leading refined results. Compared with conventional algorithm, in silico experiments show that relative volume and absolute centroid error reduce over ∼50  %   whereas intersection over union increases over 15% for most situations. These results preliminarily indicate the promising future of appropriately applying machine learning (deep learning)-based methods in MFMT.

Extracting, harvesting, and building large-scale annotated radiological image datasets is a greatly important yet challenging problem. Meanwhile, vast amounts of clinical annotations have been collected and stored in hospitals’ picture archiving and communication systems (PACS). These types of annotations, also known as bookmarks in PACS, are usually marked by radiologists during their daily workflow to highlight significant image findings that may serve as reference for later studies. We propose to mine and harvest these abundant retrospective medical data to build a large-scale lesion image dataset. Our process is scalable and requires minimum manual annotation effort. We mine bookmarks in our institute to develop DeepLesion, a dataset with 32,735 lesions in 32,120 CT slices from 10,594 studies of 4,427 unique patients. There are a variety of lesion types in this dataset, such as lung nodules, liver tumors, enlarged lymph nodes, and so on. It has the potential to be used in various medical image applications. Using DeepLesion, we train a universal lesion detector that can find all types of lesions with one unified framework. In this challenging task, the proposed lesion detector achieves a sensitivity of 81.1% with five false positives per image.