1.1.

Chest Low-Dose CT Image Analysis Demonstration Application

The methods for dataset documentation are presented in the context of the analysis of LDCT three-dimensional (3-D) images that are acquired for lung cancer screening (LCS). With the recent approval of LCS,³ a large at-risk population of over 8 million patients in the United States is eligible to receive annual LDCT 3-D images. This population is also at high risk for many other diseases in the chest, and the LCS provides an opportunity for annual monitoring. The chest health analysis system (CHAS) system currently evaluates image biomarkers for the lungs, heart, breast, and major vessels and major bone structures. Lung cancer screening CT 3-D images are designed for the detection of pulmonary nodules and have the following characteristics: coverage of the whole lung region, acquired in a single breath-hold, low-dose, and thin slice (1.25 mm or less for current protocols). In LCS, the primary objective is to identify pulmonary nodules. The high image contrast between these nodules and the surrounding lung parenchyma enables radiologists to use a low-dose CT protocol that minimizes radiation exposure to the patient. However, the consequence of using this low-dose protocol is that the image noise, which is inversely related to radiation exposure, is much higher than in typical clinical CT scans; this is especially relevant to nonlung regions of the image.

A database of over 20,000 documented LDCT 3-D images has been created over the last 5 years to aid the development of a fully automated CHAS.^4,5 This system automatically evaluates a set of quantitative image biomarkers and involves multiple region segmentations of LDCT 3-D images. The CHAS was originally designed to identify pulmonary nodule candidates to assist the radiologist in the lung cancer-screening task; it has been extended to report on other diseases and organs that are imaged by these scans. The current CHAS automatically identifies over 100-labeled segmented regions in LDCT 3-D images with a slice thickness of 2.0 mm or less.

1.2.

Fully Automated 3-D CT Image Segmentation Evaluation

Fully automated algorithms must be validated on a very large number of test cases before they can be approved for general clinical use. Evaluation cases must well represent the spectrum of clinical presentations. These requirements are in contrast to most traditional image biomarker studies reported in the research literature. Those studies typically involve very small selected datasets, and, further, they employ semiautomated computer methods that require physician interaction. We review here the evaluation methods and number of images employed for the main studies that are most closely related to our CHAS demonstration application.

A number of fully automated image segmentation systems have been reported in the literature. Segmentation performance is usually reported by subjective post hoc visual evaluation (VE) or by quantitative comparison to a small number of pre-established expert manual markings (QE). For QE, two popular variations that reduce the manual effort required are automated QE (QEa) in which a semiautomated method with manual corrections is used for image markings and sampled QE (QEs) in which only a subset of 2-D slices in a 3-D image are marked and evaluated.

Most work in this area has been performed for a single organ class and has been validated with a limited number of test cases. Very few studies of LDCT 3-D images that meet the recent requirements for LCS have been done.⁶ Studies that involve chest and body CT segmentation are listed in Table 1. To be state-of-the-art, we have included studies that do not conform to the LCS requirements; and to represent multisegmentation studies, we have included CT studies for the abdomen. Studies in other 3-D image modalities typically have similar characteristics. Our main interest is in thin-slice, low-dose thoracic scans that are relevant to LCS; most of the listed studies are thin-slice but not low-dose. In the column for the number of cases, we list any variations from our desired protocol. Since very few studies use LDCT, they are specifically identified; otherwise the study includes regular dose CT 3-D images. The size in mm when mentioned is the slice thickness of the scans. Some studies provide QE for a small number of cases only due to the effort required and VE for a larger number of cases. Only studies with more than 25 CT scans in the evaluation set have been included in Table 1.

Table 1

Anatomical segmentation studies of body regions in CT 3-D images.

Most segmentation studies reported in this chest CT literature are based on less than 100 cases. Other than our own work, VE studies involve a median of 71 cases ($\min =26$, $\max =1000$) and QE studies involve a median of 55 cases ($\min =28$, $\max =212$). Beyond our own work, only three studies address the much more challenging problem of segmenting low-dose 3-D images^9,16,34 and only four studies reported significantly more than 100 cases (Gill et al.¹⁰ 212, Xu et al.¹¹ 400, Haas et al.²⁸ 302, and Zhou et al.³² 1000). For fully automated algorithm design and evaluation, much larger datasets are needed.

One approach to comparing algorithms is through the process of challenges³³ where several algorithms for a task are evaluated on a common dataset. Four of the studies listed in Table 1 are challenges.^33–36 The visceral challenge group has, to date, conducted various organ segmentation challenges using the QE method. They have recently proposed a method for large image documentation^36,37 based on atlas registration to expert annotated reference cases and label fusion from multiple algorithms to improve segmentation quality. A recent paper by Dicente Cid et al.³⁸ involves lung segmentation on a dataset of 12,414 CT scans using the difference between two lung segmentation algorithms to identify problem segmentations. By requiring a 0.95 Dice coefficient agreement between the two algorithms, they rejected 35% of the 3-D images. While this approach allowed preliminary results to be obtained on a large dataset, a large fraction of the data was rejected and it is unknown how many of the accepted segmentations had serious issues since no visual review was performed.

Both the VE and QE approaches suffer from significant drawbacks; the main drawback is the amount of time required to review each image. VE may be applied to a large number of cases, but it is highly subjective and is very costly to repeat (same cost as first evaluation), which must be done for each cycle of evaluation (i.e., for each algorithm modification); further, repeatability is an issue due to the subjective evaluation. QE has the advantage of repeatability. Since a target “ideal” segmentation is created, the evaluation is quantitative and may be automated. However, two issues with this approach are: (a) the time per image to create the target segmentation manually is very large and the “ideal” segmentation has built-in human variation. Frequently QEa and QEs variants are used to reduce this time; however, these techniques introduce new errors due to algorithm bias (QEa) or reduced precision by subsampling the data (QEs).

The NCI sponsored Lung Image Database Consortium³⁹ (LIDC) created a public image database of 2669 pulmonary nodules in 1018 chest CT 3-D images. QEa evaluation was used with four experienced radiologists marking the boundary of each nodule. The very high intermarker variation⁴⁰ illustrates the issues with the QEa method and makes the use of these markings for precision evaluation of computer algorithms challenging. A significant fraction of the multimillion dollar project cost was for the 3-D image marking process, which only addressed documenting a tiny fraction of the pixels in these 3-D images.

1.3.

Image Segmentation Errors

Segmentation errors may be considered to fall into two general classes: precision errors ($E_{\mathrm{P}}$) and catastrophic errors ($E_{\mathrm{C}}$). Precision errors occur due to differences in details among algorithms or between algorithms and the variation of manual image annotations: for this error type the difference among methods for large organs is typically small. Catastrophic errors occur when an algorithm incorrectly identifies or includes a significantly different region with the target region (for example, includes a nearby vessel as part of a lesion); the size of these errors may be very large. A catastrophic error $E_{\mathrm{c}}$ is defined for a 3-D segmentation as a connected incorrect region of a segmentation that involves less than 50% of the surface area of the true image region (typically much less than 50%) and causes a significant change in the overall segmentation volume or spatial extent compared to the true image region. In most studies on segmentation algorithms, the dataset is usually small ($<100$ cases) and of carefully selected images such that the majority of the errors tend to be $E_{\mathrm{P}}$. However, when larger datasets with a wider range of imaging parameters and presentations are considered, the likelihood of $E_{\mathrm{C}}$ errors is significantly increased. In a semiautomated environment where the primary target is image region characterization, $E_{\mathrm{C}}$ errors are rarely an issue since the operator manually corrects them. However, for a fully automated system, the objective has a focus on abnormality detection rather than characterization and $E_{\mathrm{C}}$ errors are a major consideration since they may cause an unacceptable large number of false-positive abnormality detections. Similarly, $E_{\mathrm{C}}$ errors may also adversely affect sensitivity if the whole region is not included in the segmentation. The evaluation criterion we use for image segmentations in this case primarily relates to the $E_{\mathrm{C}}$ error type.

In the large-scale image documentation method, we introduce the concept of 3-D image visualizations for segmentation evaluation, which provides a rapid image review that is primarily sensitive to the $E_{\mathrm{C}}$ error. We also use a simple two- or three-level categorical evaluation that is targeted to biomarker evaluation requirements rather than minimizing $E_{\mathrm{P}}$ errors, which is expected to have high interreader agreement.

Recent studies in large-scale image datasets have had to deal with the significant number of problem cases in these datasets. In the study by Gulshan et al.,¹ 10.1% of the images were excluded for evaluation by poor quality evaluation; in the study by Dicente Cid et al.,³⁸ 35% of the images were excluded by algorithm comparison. In contrast, in our study, less than 0.5% of the cases are rejected for quality issues. Consequently, we have a significant number (our target is $<10\%$) of images for which we do not have acceptable segmentations. One approach would be to use manual methods to correct the algorithm outcomes; such an approach has been implemented by Takx et al.⁴¹ in which they established a database with 1749 validated biomarker outcomes. Experienced physicians manually correct outcome regions for calcium scoring. Although our system has the capability for manual correction, we did not adopt this approach due to the very large burden it would place on human resources for the large image regions we are concerned with and because such a process would introduce human variation into the image documentation.

The image documentation system, presented here, addresses the main shortcomings outlined above. A simple categorical grade related to biomarker evaluation is employed to address the segmentation precision issue. Human variation is addressed by not using any manual markings. Finally, VE evaluation time is minimized (a) through the use of customized visualizations and (b) by minimizing the number of images to review through the use of automated quantitative preselection criteria. This documentation method has been used to create a documented image database with over 20,000 cases and facilitates fully automated algorithm evaluation on multiple segmentations.

1.4.

Image Documentation Implementation

The methods for large dataset documentation presented here have been implemented and evaluated on the system for image management and biomarker analysis (SIMBA).⁵ While image segmentation studies have been ongoing since 2000, the first instance of the formal framework presented here was implemented in 2012. The data documentation system is updated each year by installing the latest version of our image analysis software as the reference documentation method and updating the documentation for all images in the database with this method. There have been five annual revisions of the system, with the outcomes of last two (2016 and 2017) reported here. The SIMBA system⁵ is a mature web-based image management platform for research studies including: study data collection and analysis,⁴² image-based clinical studies,⁴³ public image databases,⁴⁴ and image analysis algorithm development.^17,45,46 The concept of SIMBA is to have all image data and study data in a single web-based management system. It was first used in the I-ELCAP study for data collection and analysis starting in 2000. The system has the capability for full medical image review and annotation and includes the relevant functions of a PACS system through a web interface. For the LIDC study;³⁹ it was the only one of the image annotation tools used by the five participating institutions that was web-based.

1.5.

Overview

The methods section contains three components: the details of the large-scale image documentation system, the CHAS demonstration application, and the documentation evaluations. Key aspects of the method are: a new process for image evaluation, VE with quantitative review (VEQR), and the employment of customized segmentation visualizations. The documentation of over 20,000 3-D images with the SIMBA image documentation system is discussed.

2.1.

Image Documentation System Organization

An overview of the SIMBA system for large-scale image documentation is illustrated in Fig. 1. The database within the system comprises two main components: the image database in which the raw DICOM format images are stored and the data documentation. The data documentation has two components for each image entity: a label map, which contains a label (a unique segment identifier) for every pixel in the corresponding raw image, and a label score set, which contains a categorical grade for every label that has been identified for that image.

Fig. 1

The SIMBA framework for segmentation algorithm development and evaluation on large medical image datasets.

The four main framework operations are (a) addition and evaluation of new image data, (b) automated algorithm evaluation, (c) data documentation updating, and (d) the selection of datasets for algorithm development. The key components of the framework include:

The detailed operations are as follows:

2.1.3.

Customized visualization for segmentation grading

Customized 3-D visualizations are used to minimize the time required for VE. Algorithm outcomes are graded for a score $a(i,s)$ by a two-stage VE process. A 3-D image is first evaluated by a 3-D customized visualization, and, for additional review when necessary, a more traditional 2-D image slice viewing is provided. Efficient VE is especially important for 3-D image modalities since the traditional approach of examining a region on each 2-D slice through the 3-D image is very labor intensive. The 3-D visualization is developed in concert with the early development of the corresponding segmentation algorithm. The visualization web page is developed with experience gained from observation of the most common segmentation errors. A snapshot for the customized 3-D visualization of the breast region in chest CT 3-D images is shown in Fig. 2.

Fig. 2

Snapshot from SIMBA of a 3-D visualization of a breast region segmentation.

In this study, the custom visualizations were developed by the computer algorithm developers following the initial validation of the algorithm by clinical experts. Thus, the 3-D visualizations and accompanying selected 2-D image views were designed to be sensitive to known-algorithm errors. The 3-D visualizations were developed using the custom visualization tools in the V4 VisionX software system⁴⁷ that uses the Visualization Toolkit⁴⁸ for image rendering.

In addition to the 3-D visualization of the breast and the fibroglandular tissue, there are 9 additional 2-D image slices with regions marked as shown on the right side of Fig. 2; these have been found to be useful for resolving many potential issues observed on 3-D visualization. If the reviewer is not satisfied with the first level 3-D review, then a second traditional review is available as shown in the snapshot in Fig. 3. This review permits standard windowing and zooming etc. for a side-by-side comparison of the original 2-D image slices and 2-D image segmentations.

Fig. 3

SIMBA snapshot of a traditional 2-D image slice review with segmentation map.

In the current VEQR method, a region segmented by an algorithm is evaluated by VE into three categories: good (G), acceptable (A), and unacceptable (U). The criterion for acceptable is that, although there is some visible defect in the segmentation, the quality is sufficient for evaluation of related biomarker measurements. An example of an unacceptable segmentation for the lungs is shown in Fig. 4. This is clearly visible in the 3-D visualization alone. (The problem for this case is also visible in the 2-D review, but this is not needed for database documentation).

Fig. 4

Example unacceptable lung segmentation (a) segmentation shown in 3-D, (b) segmentation shown in 2-D, and (c) intensity CT image of the same axial slice.

It is interesting to note that our segmentation and an external segmentation algorithm⁴⁹ had very similar segmentations for this case ($\mathrm{DSC}=0.98$), which illustrates an issue of relying on a method such as label fusion to improve confidence in a segmentation without visual review.

The criteria for grading are based on the ability to precisely compute the quantitative image biomarkers associated with the segmentation. An example of unacceptable and acceptable grades for the same case with two different algorithms is shown in Fig. 5.

Fig. 5

(a) Unacceptable segmentation and (b) acceptable segmentation.

In Fig. 5(a), the left lung has leaked into the bowel. This is sufficient to modify the values for the lung quantitative biomarkers for the left lung: lung volume [2124 mL for (a) and 1972 mL for (b)] and emphysema index [2.33% for (a) and 1.41% for (b)].

For a new 3-D image region or quantitative image biomarker, the development of the 3-D visualization follows the traditional validation of the new algorithm. Then, the 364 images of our development set with this visualization are very carefully reviewed by two experienced graders. The image biomarkers computed from these segmentations are also reviewed.

2.2.

Chest Health Analysis System

The CHAS consists of a number of interdependent anatomy function modules as outlined in Table 2. The models and their dependencies are also shown in Fig. 6. The main anatomy segmentation labels that are managed by each module are also listed in Table 2. The CHAS system employs a divide and conquer approach to the image labeling process in which the most robust labeling algorithms are applied first and subsequent algorithms depend on the existing framework and labeled regions established by earlier algorithms. This approach is appropriate for the LDCT scans in which the high degree of noise makes soft tissue segmentation very challenging.

Table 2

Chest analysis functions: module names, descriptions, and their associated main labels.

Fig. 6

CHAS modules and their dependencies organization. The associated biomarkers are also listed.

In addition to pulmonary nodule detection (usually the primary reason for obtaining the scan), the system also evaluates a number of other organs that are imaged; these evaluations include, for example, coronary artery calcium (CAC) scoring,¹⁸ emphysema measurement,⁵⁹ and breast density.⁵⁸

2.2.3.

Evaluation criteria and customized visualizations

Airway

The airway tree is segmented using a cylinder tracking and 3-D region growing-based method.⁵² A coronal view of the segmented airway (see Fig. 7) is used for the VEQR, where the trachea and the remainder of the airway tree are shown in different colors, thereby enabling the validation of the location of carina at the same time. Airway biomarkers are the airway diameter and wall thickness evaluated for each detectable segment. The grading criteria for the airway algorithm are: G: no visible errors; A: correct trachea and two main bronchi but unable to segment sufficient peripheral branches; and U: visible leakage.

Fig. 7

Customized visualization for airway tree segmentation: trachea to carina, red; bronchi, green. (a) Good segmentation, (b) acceptable segmentation, and (c) unacceptable segmentation.

Lungs

The left and right lungs are segmented using 3-D image filtering, intensity thresholding, and morphological operations.⁶⁶ The left and right lungs are partitioned with a minimum distance path-cutting algorithm. The customized visualization is a 3-D rendering of the two lungs as shown in Fig. 5. Lung region biomarker evaluations include the detection of pulmonary nodules and the measurement of lung health indicators such as the emphysema index. The grading criteria for the lung algorithm are: G: no visible errors and U: major errors, e.g., another region mistaken as lung.

Individual vertebra

The whole spine is first segmented by thresholding and connected component analysis; it is then further divided into individual vertebra by fitting separating planes in the 3-D space. VEQR of the individual vertebra segmentation is performed on the sagittal view of the 3-D segmentation (see Fig. 8), where different colors are used to indicate individual vertebra for the purpose of validating the vertebra labeling. Vertebra biomarker evaluations include the measurement of bone mineral density and the prediction of compression fracture. The grading criteria for the vertebra algorithm are: G: no visible errors: A: minor oversegmentation; and U: major errors including mislabeling or failure to separate vertebra.

Fig. 8

Customized visualization for segmentation of individual vertebra: individual vertebra colored differently. (a) Good segmentation, (b) acceptable segmentation, and (c) unacceptable segmentation.

Cardiac region

Cardiac region segmentation includes the segmentation of aorta, heart region, and pulmonary trunk using anatomical constraint-based methods.^17–19 VEQR of the cardiac region consists of the evaluation of coronal and sagittal views of the segmented aorta, heart region, and pulmonary trunk in different colors (see Fig. 9). Cardiac region primary biomarkers are CAC and aorta diameter profile. The grading criteria for the cardiac algorithm are: G: no visible errors in any of the three regions; A: visible errors but usable for measurement of biomarkers; and U: major errors and not usable for biomarker measurements.

Fig. 9

Customized visualization for cardiac region segmentation: aorta, red; heart, green; pulmonary trunk, blue. (a) Good segmentation, (b) acceptable segmentation, and (c) unacceptable segmentation.

Breast

The whole breast region as well as the fibroglandular tissue and fat tissue contained in the breast region are segmented using an anatomy-oriented algorithm²¹ that separates the pectoral muscles from fibroglandular tissue through the propagation of muscle front along the vertical direction. A coronal view of the breast segmentation as shown in Fig. 10 is used for the VEQR, where the whole breast region and the fibroglandular tissue are shown in different colors. Breast biomarkers are breast density assessment and breast mass detection (ongoing project). The grading criteria for the breast algorithm are: G: no visible errors; A: minor segmentation issue, such as insignificant undersegmentation of fibroglandular tissue or inclusion of muscle into the breast region, which will not influence further biomarker analysis including density assessment and breast mass detection; and U: segmentation errors that may influence further biomarker analysis.

Fig. 10

Customized visualization for breast segmentation: whole breast region (light green), fibroglandular tissue (solid green), and sternum (gray). (a) Good segmentation, (b) acceptable segmentation, and (c) unacceptable segmentation.

2.3.

Large-Scale Image Annotation System Implementation and Evaluation

The large-scale image documentation method has been developed in conjunction with the chest health system application over the last 5 years with the dual goals of improving the robustness of the algorithms and creating a large documented dataset that may be used for the training and evaluation of other computer algorithms. We have used the current CHAS algorithms as the reference algorithm for the image documentation system and have conducted annual updates of first the CHAS algorithms and then the database image documentation. That is, each year algorithms are updated to address issues that caused some of the unacceptable segmentations, then the database is revised using the new CHAS algorithm set, and the reference algorithm used for new image data is set to the new CHAS algorithm set.

Several system parameters for use with the CHAS algorithms were empirically derived. For a visual inspection update to occur, the Dice coefficient difference between the new and reference segmentations must be between 0.75 and 0.95; that is, $T_{\mathrm{DC}\text{low}}=0.75$ and $T_{\mathrm{DC}\text{high}}=0.95$. A two-stage process was used for the introduction of new algorithms/biomarker tasks to the system. First the algorithms were carefully evaluated on a development dataset of 364 3-D images from public datasets, and the customized 3-D review was carefully reviewed. Further, a minimum performance level of 90% acceptable segmentations was required by the new algorithm before it was applied to the full dataset to prevent the generation of too many review events.

3.1.

Algorithm Evaluation Results

The VEQR results in the year 2016 to 2017 are given in Tables 6–9. Table 6 shows the VEQR results for the development dataset (364 CT 3-D images, agreement between two reviewers). Table 7 shows the combined VEQR results for the full 2016 dataset of 7440 cases (the additional 7076 CT 3-D images were reviewed by a single reviewer). In the 2016 year, the breast analysis algorithm was added to the CHAS system and a new cohort LC was added to the database. The results for the breast algorithm are also included in Tables 6 and 7. Table 8 shows the performance of the new cohort (LC, 6752 3-D images). After adding the additional 6557 3-D images in 2017, the complete result is reported in Table 9. For completeness, the results for the review of the two modules without specific biomarker objectives are also included as these were evaluated; they are ribs (24 labels) and skin surface (3 labels).

Table 6

VEQR results for VIA-ELCAP and LIDC 364 development 3-D images (2016/17, development dataset).

Table 7

VEQR results for all datasets, 7440 3-D images (2016, baseline dataset).

Table 8

VEQR results for the new LC dataset of 6752 3-D images (2016 new dataset).

Table 9

VEQR results for all datasets, 20,749 3-D images (2017, full dataset).

3.2.

VE Timing Results

Table 8 shows the time cost for the VE procedure assessed on 200 sequentially selected 3-D images. With regard to the two-level viewing protocol, less than 2% of the module reviews required further inspection beyond the primary 3-D visualization. The second level 2-D visualization for these cases was $\sim 60\mathrm{s}$. This time is included in the average times reported in Table 10.

Table 10

Average time cost per 3-D image for VE on each module.

The comparative update review for the development dataset of 364 3-D images required evaluation of 2548 module outcomes. With the current acceptance thresholds, 26 module outcomes (1%) required side-by-side VE and six module outcomes (0.2%) resulted in database updates. For the FAMRI dataset of 1422 3-D images (9954 module outcomes), 187 (19%) of the module outcomes from 185 3-D images required side-by-side VE and 99 module outcomes (1%) resulted in database updates. The side-by-side comparison between reference segmentation and new segmentation for all 187 FAMRI dataset reviews required 21 min; that is an average of 6.7 s per module outcome review.

3.3.

Interreader Variation

Two readers visually reviewed the development dataset of 364 3-D images independently. The same grading criteria were used for each segmentation, and the differences between the readers are summarized in Table 11. A difference occurs if one reader has assigned a grade of G or A while the other reader has assigned a grade of U to the same segmentation.

Table 11

Difference between two readers (364 development 3-D images).

3.4.

Evaluation Results for the External Lung Segmentation Algorithm

An external lung segmentation algorithm as part of the CIP,⁴⁹ which is an open-source library for quantitative image analysis on lung and airway,⁷⁰ was evaluated through the presented SIMBA system with a subset of 7173 CT scans from the full database. Visual inspection was triggered on 74 scans, including 26 scans in which the DC between the external algorithm and the reference algorithm were in the range $[T_{\mathrm{DC}\text{low}},T_{\mathrm{DC}\text{high}}]$, with $T_{\mathrm{DC}\text{low}}=0.75$ and $T_{\mathrm{DC}\text{high}}=0.95$, and 48 scans for which the reference algorithm had failed to produce acceptable segmentation. No visual inspection was needed for 7099 scans, including 12 scans in which the DC were below $T_{\mathrm{DC}\text{low}}$ and 7087 scans in which the DC were above $T_{\mathrm{DC}\text{high}}$. The unacceptable reference segmentations for 37 scans (0.52% of the total evaluated) were replaced by the new algorithm acceptable outcomes.

4.1.

CHAS Segmentation Algorithm Performance

The results on the development dataset in Table 6 show very high success rates (98% to 100%) acceptable for the more mature modules that have received several cycles of revision compared to the more recent modules. Even the new breast module has 97% acceptable. These modules’ development have all benefited from some extreme cases extracted from the larger full dataset in addition to the public development dataset. In Table 7, the results for the full 2016 dataset of 7440 3-D images show that the acceptable rate is from 93% to 99%, which is within our initial target of at least 90%.

Table 7 shows the initial results on the newly added 6752 3-D images. The ribs and vertebrae modules have an acceptance rate of 82% and 78%, respectively, which is below our 90% target for new algorithms and resulted in low success rates for other modules. Review of a subset of these cases showed that this cohort had scanner parameters that resulted in different noise statistics than our previous cohorts and that the effect was to cause our bone segmentation algorithms to give poor results. After revision, our algorithms now detect this noise characteristic in the image quality analysis module (Fig. 6), and filter the image data prior to bone segmentation. The results for the upgraded system and algorithms for 2017 are shown in Table 9, in which an additional 6557 3-D images were added to the database. The results for 2017 indicate that the performance of the updated CHAS has improved even with the additional image and with only one module (vertebra, 88%) not achieving the 90% target.

4.2.

Visual Image Evaluation Effort

Results given in Table 10 show that the per 3-D image evaluation time for the initial VE is about 25 s. The time for just nodule review by a radiologist for these scans without CAD has been shown to be in the order of 10 min.^71,72 Therefore, the time for initial segmentation review with the SIMBA framework is a tiny fraction ($<5\%$) of the time required for a traditional clinical review of the 3-D image. In addition, update reviews are required when the outcomes from new algorithms are considered. Currently, we are observing about 1% to 2% module reviews that require a review time of about 7 s. This total cost of updating is linearly related to the size of the database; however, as the algorithms mature, it is anticipated that the fraction of triggered events will significantly diminish; note the lower frequency of trigger events for the development dataset compared to more recent FAMRI dataset.

The more mature algorithm modules in CHAS have now received several cycles of updates. For example, when our standard lung module was applied to the LIDC 318 cases, the initial performance was about 70%. With several cycles of algorithm refinements, the performance on that dataset is now 97% to 100%, and attention is directed to failures in the other cohorts.

4.3.

Interreader Variation

The limited study on interreader variation indicated that there were a total of only six 3-D image evaluations out of 2548 (seven algorithms for 364 3-D images), i.e., about 0.24%. Further, the six 3-D images in question were considered to be close to the border of acceptable. This is in line with our design goal of rating the segmentation for the relevant task biomarker evaluation rather than traditional boundary precision. A down side to this approach is that if a new quantitative biomarker is added that requires new segmentation characteristics, then the entire database may need to be re-evaluated.

4.4.

Algorithm Evaluation

The evaluation of the CIP lung segmentation algorithm⁴⁹ demonstrates that the documented database may be used for evaluating different algorithms. Visual review was only required on 74 cases; that is 1.03% of the testing dataset. Although the system does not support the cloud-based approach used by Visceral³⁶ for algorithm evaluation, it is simple to include an external algorithm module in the SIMBA system; further, it is possible to export images for testing and import the resulting segmentations for evaluation.

4.5.

Limitations of Study

The large-scale image documentation method has been demonstrated with a single application and a single validation method (visual inspection with grading into three categories and no manual marking). There are many alternative strategies that could be used with this framework. For example, a cohort of phantom images with a priori known truth can easily be added to the system. A second possibility is to increase inclusiveness by using other methods, such as manual marking to provide good segmentations for the cases graded unacceptable. The tradeoff for this approach is a much higher cost in documentation effort combined with the introduction of human variation into the image documentation. Adding manual documentation for difficult cases may be very useful, especially in scenarios where the database is to be used directly for algorithm training. In addition to the multiregion application discussed here, the framework has also been successfully used on other research projects involving just one or two regions.

Currently, there are some image quality checks within the algorithm modules that indicate algorithm failure and prevent dependent algorithms from executing. In future work, we plan to enhance outcome quality checks such that, for many cases, an automatic unacceptable evaluation grade will be given when a problem is detected. This mechanism will further reduce the number of images that require VE for both new image data and for database updates.

In this work, we have not established that the chosen VE is equivalent or superior to other evaluation methods. A major challenge in evaluating the quality of computer segmentations for medical imaging is that, in general, the true value is unknown and that comparison must be made by either QE or VE methods. Traditionally, radiologists view and evaluate 3-D images by a sequence of 2-D image slices extracted along the axial dimension. More recently with the advent of electronic image viewing systems, other views including sagittal, coronal, MIP projections, and 3-D visualizations are becoming more common. In this work, we take advantage of a well-defined anatomical region possibly being more rapidly evaluated using primarily several 3-D visualizations rather than reviewing every axial 2-D image slice. We have not established any specific precision advantage for any of these approaches; however, we conjecture that different methods may provide acceptable outcomes.

4.6.

Demonstration SIMBA Web System

The image documentation system images and segmentation visualizations have been made available on a demonstration web system.⁵ This allows for access to the ELCAP-VIA public image database of 50 LDCT chest 3-D images and the resulting CHAS image segmentations. In addition, the web system supports interactive viewing of CT 3-D images and the associated customized visualizations for both the initial image review and for the side-by-side update image review for CHAS database documentation.