Assessment of computerized algorithms by comparing with human observers in binary classification tasks: a simulation study

Yang Yang; Berkman Sahiner; Zhipeng Huang; Nicholas Petrick; Weijie Chen

doi:10.1117/12.2293807

7 March 2018 Assessment of computerized algorithms by comparing with human observers in binary classification tasks: a simulation study

Yang Yang, Berkman Sahiner, Zhipeng Huang, Nicholas Petrick, Weijie Chen

Author Affiliations +

Proceedings Volume 10577, Medical Imaging 2018: Image Perception, Observer Performance, and Technology Assessment; 1057713 (2018) https://doi.org/10.1117/12.2293807
Event: SPIE Medical Imaging, 2018, Houston, Texas, United States

Abstract

It is generally recognized that recent advancements in computer vision, especially the development of deep convolutional neural networks, has substantially improved the performance of computerized algorithms in medical imaging for classification tasks such as cancer detection/diagnosis.These advancements underscore the importance of the question of how the computer algorithm’s stand-alone performance compares with the performance of physicians. Current literature often uses descriptive statistics or a visual check of plots for the comparison lacking quantitative and rigorous statistical inference. In this work, we developed a U-statistic based approach to estimate the variance of performance difference between an algorithm and a group of human observers in a binary classification task. The performance metric considered in this work is percent correct (PC), e.g., sensitivity or specificity. Our variance estimation treats both human observers and patient cases as random samples and accounts for both sources of variability, thereby allowing for the conclusion to be generalizable to both the patient and the physician populations. Moreover, we investigated a z -statistic method based on our variance estimator for hypothesis testing. Our simulation results show that our variance estimator for the PC performance difference is unbiased. The normal approximation method using our variance estimator for hypothesis testing appears useful for large sample sizes.

Citation Download Citation

Yang Yang, Berkman Sahiner, Zhipeng Huang, Nicholas Petrick, and Weijie Chen "Assessment of computerized algorithms by comparing with human observers in binary classification tasks: a simulation study", Proc. SPIE 10577, Medical Imaging 2018: Image Perception, Observer Performance, and Technology Assessment, 1057713 (7 March 2018); https://doi.org/10.1117/12.2293807

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available

Members: $17.00

Non-members: $21.00 ADD TO CART

PROCEEDINGS
8 PAGES

DOWNLOAD PAPER SAVE TO MY LIBRARY

GET CITATION

RIGHTS & PERMISSIONS

Get copyright permission Get copyright permission on Copyright Marketplace

KEYWORDS

Monte Carlo methods

Statistical analysis

Algorithm development

Binary data

Chromium

Error analysis

Computer simulations

Show All Keywords

Keywords/Phrases

Search In:

Publication Years