9 May 2002 ROC components-of-variance analysis of competing classifiers
Author Affiliations +
In the last decade in the field of diagnostic imaging, the problem of variability of reader skill as well as patient case difficulty has given rise to a multivariate approach to receiver operating characteristic (ROC) analysis. The multivariate approach is the so-called multiple-reader, multiple-case (MRMC) ROC paradigm in which every reader reads every patient case and, where possible, in each of two modalities under comparison. The present paper demonstrates the isomorphism between patient cases, image readers, and imaging modalities in diagnostic imaging and, respectively, test sets, training sets, and competing discriminant algorithms in the field of statistical pattern recognition (SPR). Thus, the MRMC paradigm can be brought directly across from imaging to SPR. Recent MRMC ROC analytical methods are demonstrated in the context of SPR for the task of analyzing the natural components of variance in that problem involving test sets, training sets, and competing discriminants. Monte Carlo trials reported here indicate that the conventional wisdom that the variance of measures of classifier accuracy comes mainly from the finite test set is only true when assessing a single algorithm in a very limited context. In particular, it is generally not true when comparing competing discriminant algorithms; in that case the variance is dominated by the finite training set.
© (2002) COPYRIGHT Society of Photo-Optical Instrumentation Engineers (SPIE). Downloading of the abstract is permitted for personal use only.
Sergey V. Beiden, Sergey V. Beiden, Marcus A. Maloof, Marcus A. Maloof, Robert F. Wagner, Robert F. Wagner, } "ROC components-of-variance analysis of competing classifiers", Proc. SPIE 4684, Medical Imaging 2002: Image Processing, (9 May 2002); doi: 10.1117/12.467176; https://doi.org/10.1117/12.467176

Back to Top