Interpreting the ROC curve from the radiologist’s point of view: the diagnostician operating choice (DOC) curve

Craig A. Beam; Emily F. Conant M.D.; Elizabeth A. Krupinski; Harold L. Kundel; Edward A. Sickles

doi:10.1117/12.595198

Translator Disclaimer

6 April 2005 Interpreting the ROC curve from the radiologist's point of view: the diagnostician operating choice (DOC) curve

Craig A. Beam, Emily F. Conant M.D., Elizabeth A. Krupinski, Harold L. Kundel, Edward A. Sickles

Author Affiliations +

Proceedings Volume 5749, Medical Imaging 2005: Image Perception, Observer Performance, and Technology Assessment; (2005) https://doi.org/10.1117/12.595198
Event: Medical Imaging, 2005, San Diego, California, United States

Abstract

We introduce an interesting interpretation of the ROC Curve that, subsequently, opens a new research paradigm. We define the "Diagnostician Operating Choice" (DOC) Curve to be the set of all (True Positive Probability/True Negative Probability) or ("skill in diseased population"/"skill in non-diseased population" when considered from the diagnostician's perspective) options made available to a particular radiologist when interpreting a particular diagnostic technology. The DOC Curve is, thus, the choice set presented to the diagnostician by their interaction with the technology. This new paradigm calls for tools that can measure the particular choice set of any particular individual radiologist interpreting a particular technology when applied in a particular clinical setting. Fundamental requirements for this paradigm are for the DOC Curve to be unique to individuals and constant across similar experimental conditions. To investigate constancy, we analyzed data from a reading study of 10 radiologists. Each radiologist interpreted the same set of 148 screening mammograms twice using a modified version of BI-RADS. ROC Curves for each radiologist were computed and compared between the two reading occasions with the CORROC2 program. None of the areas were statistically significantly different (p<0.05), providing confirmation (but not proof) of constancy across the two reading conditions. The DOC Curve paradigm suggests new areas of research focusing on the behavior in individuals interacting with technology. A clear need is for more efficient estimation of individual DOC Curves based on limited case sets. Paradoxically, the answer to this last problem might lie in using large population-based ("MRMC") studies to develop highly efficient and externally validated standardized testing tools for assessment of the individual.

Citation Download Citation

Craig A. Beam, Emily F. Conant M.D., Elizabeth A. Krupinski, Harold L. Kundel, and Edward A. Sickles "Interpreting the ROC curve from the radiologist's point of view: the diagnostician operating choice (DOC) curve", Proc. SPIE 5749, Medical Imaging 2005: Image Perception, Observer Performance, and Technology Assessment, (6 April 2005); https://doi.org/10.1117/12.595198

ACCESS THE FULL ARTICLE