MRMC analysis of agreement studies

Brandon D. Gallas; Amrita Anam; Weijie Chen; Adam Wunderlich; Zhiwei Zhang

doi:10.1117/12.2217074

24 March 2016 MRMC analysis of agreement studies

Brandon D. Gallas, Amrita Anam, Weijie Chen, Adam Wunderlich, Zhiwei Zhang

Proceedings Volume 9787, Medical Imaging 2016: Image Perception, Observer Performance, and Technology Assessment; 97870F (2016) https://doi.org/10.1117/12.2217074
Event: SPIE Medical Imaging, 2016, San Diego, California, United States

Abstract

The purpose of this work is to present and evaluate methods based on U-statistics to compare intra- or inter-reader agreement across different imaging modalities. We apply these methods to multi-reader multi-case (MRMC) studies. We measure reader-averaged agreement and estimate its variance accounting for the variability from readers and cases (an MRMC analysis). In our application, pathologists (readers) evaluate patient tissue mounted on glass slides (cases) in two ways. They evaluate the slides on a microscope (reference modality) and they evaluate digital scans of the slides on a computer display (new modality). In the current work, we consider concordance as the agreement measure, but many of the concepts outlined here apply to other agreement measures. Concordance is the probability that two readers rank two cases in the same order. Concordance can be estimated with a U-statistic and thus it has some nice properties: it is unbiased, asymptotically normal, and its variance is given by an explicit formula. Another property of a U-statistic is that it is symmetric in its inputs; it doesn't matter which reader is listed first or which case is listed first, the result is the same. Using this property and a few tricks while building the U-statistic kernel for concordance, we get a mathematically tractable problem and efficient software. Simulations show that our variance and covariance estimates are unbiased.

Citation Download Citation

Brandon D. Gallas, Amrita Anam, Weijie Chen, Adam Wunderlich, and Zhiwei Zhang "MRMC analysis of agreement studies", Proc. SPIE 9787, Medical Imaging 2016: Image Perception, Observer Performance, and Technology Assessment, 97870F (24 March 2016); https://doi.org/10.1117/12.2217074

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available