Receiver operating characteristic and confidence error metrics for assessing the performance of automatic target recognition systems

David R. Parker; Steven C. Gustafson; Timothy D. Ross

doi:10.1117/1.2042307

1 September 2005 Receiver operating characteristic and confidence error metrics for assessing the performance of automatic target recognition systems

David R. Parker, Steven C. Gustafson, Timothy D. Ross

Author Affiliations +

Optical Engineering, Vol. 44, Issue 9, 097202 (September 2005). https://doi.org/10.1117/1.2042307

Abstract

The ability of certain performance metrics to quantify how well a target recognition system under test (SUT) can correctly identify targets and non-targets is investigated. The SUT, which may employ optical, microwave, or other inputs, assigns a score between zero and one that indicates the predicted probability of a target. Sampled target and nontarget SUT score outputs are generated using representative sets of beta probability densities. Two performance metrics, the area under the receiver operating characteristic (AURC) and the confidence error (CE), are analyzed. The AURC quantifies how well the target and nontarget distributions are separated, and the CE quantifies the statistical accuracy of each assigned score. The CE and AURC were generated for many representative sets of beta-distributed scores, and the metrics were calculated and compared using continuous methods as well as discrete (sampling) methods. Close agreement in results with these methods for the AURC is shown. While the continuous and the discrete CE are shown to be similar, differences are shown in various discrete CE approaches, which occur when bins of various sizes are used. A method for an alternative weighted CE calculation using maximum likelihood estimation of density parameters is identified. This method enables sampled data to be processed using continuous methods.

©(2005) Society of Photo-Optical Instrumentation Engineers (SPIE)

Citation Download Citation

David R. Parker, Steven C. Gustafson, and Timothy D. Ross "Receiver operating characteristic and confidence error metrics for assessing the performance of automatic target recognition systems," Optical Engineering 44(9), 097202 (1 September 2005). https://doi.org/10.1117/1.2042307

Published: 1 September 2005

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available