Dr. David P. Miller Profile

Dr. David P. Miller

at Verily Life Sciences

SPIE Involvement:

Author

Publications (1)

Proceedings Article | 4 May 2004 Paper

Gold standards and expert panels: a pulmonary nodule case study with challenges and solutions

Dave Miller, Kathryn O’Shaughnessy, Susan Wood, Ronald Castellino

Proceedings Volume 5372, (2004) https://doi.org/10.1117/12.544716

KEYWORDS: Gold, Computer aided diagnosis and therapy, Computer aided design, Biopsy, Radiology, Lung, Statistical analysis, Cancer, Receivers

Read Abstract +

Comparative evaluations of reader performance using different modalities, e.g. CT with computer-aided detection (CAD) vs. CT without CAD, generally require a “truth” definition based on a gold standard. There are many situations in which a true invariant gold standard is impractical or impossible to obtain. For instance, small pulmonary nodules are generally not assessed by biopsy or resection. In such cases, it is common to use a unanimous consensus or majority agreement from an expert panel as a reference standard for actionability in lieu of the unknown gold standard for disease. Nonetheless, there are three major concerns about expert panel reference standards: (1) actionability is not synonymous with disease (2) it may be possible to obtain different conclusions about which modality is better using different rules (e.g. majority vs. unanimous consensus), and (3) the variability associated with the panelists is not formally captured in the p-values or confidence intervals that are generally produced for estimating the extent to which one modality is superior to the other. A multi-reader-multi-case (MRMC) receiver operating characteristic (ROC) study was performed using 90 cases, 15 readers, and a reference truth based on 3 experienced panelists. The primary analyses were conducted using a reference truth of unanimous consensus regarding actionability (3 out of 3 panelists). To assess the three concerns noted above: (1) additional data from the original radiology reports were compared to the panel (2) the complete analysis was repeated using different definitions of truth, and (3) bootstrap analyses were conducted in which new truth panels were constructed by picking 1, 2, or 3 panelists at random. The definition of the reference truth affected the results for each modality (CT with CAD and CT without CAD) considered by itself, but the effects were similar, so the primary analysis comparing the modalities was robust to the choice of the reference truth.

View contact details

UPDATE YOUR PROFILE

Is this your profile? Update it now.

Sign into your SPIE.org account

Don’t have a profile and want one?

Create an account on SPIE.org

Keywords/Phrases

Search In:

Publication Years