2 September 2004 Discrimination and confidence error in detector-reported scores
Author Affiliations +
A decision system may compute a score that reflects its confidence in one of its decisions. This paper considers methods for evaluating such scores. There is a class of measures-of-performance (MOPs) for each of the score's two roles: discrimination (how well it separates targets and clutter) and confidence (how well it predicts its own accuracy). Area-under-the-ROC and probability of error are considered as discrimination MOPs. Error in the posterior (EP) and normalized cross entropy (NCE) are considered as confidence MOPs. MOPs for the scores are assessed using Monte Carlo simulations where known score distributions are sampled, allowing comparison of true and estimated MOPs. Classical data-direct ROC estimates are found to be equivalent to those based on explicit distribution estimation using probability mass functions (pmfs). An alternative distribution estimation based on histograms is recommended for empirical ROCs, being accurate and avoiding the unnatural stair-step character of data-direct ROCs. Confidence MOPs are more difficult to estimate than discrimination MOPs and NCE estimates are especially poor. EP may be meaningfully estimated through histogram density estimates and it is recommended as a replacement for the AdaptSAPS binned EP confidence MOP.
© (2004) COPYRIGHT Society of Photo-Optical Instrumentation Engineers (SPIE). Downloading of the abstract is permitted for personal use only.
Timothy D. Ross, Timothy D. Ross, Mark E Minardi, Mark E Minardi, } "Discrimination and confidence error in detector-reported scores", Proc. SPIE 5427, Algorithms for Synthetic Aperture Radar Imagery XI, (2 September 2004); doi: 10.1117/12.542161; https://doi.org/10.1117/12.542161

Back to Top