Information theoretic approach for performance evaluation of multi-class assignment systems

Ryan S. Holt; Peter A. Mastromarino; Edward K. Kao; Michael B. Hurley

doi:10.1117/12.851019

27 April 2010 Information theoretic approach for performance evaluation of multi-class assignment systems

Ryan S. Holt, Peter A. Mastromarino, Edward K. Kao, Michael B. Hurley

Proceedings Volume 7697, Signal Processing, Sensor Fusion, and Target Recognition XIX; 76970R (2010) https://doi.org/10.1117/12.851019
Event: SPIE Defense, Security, and Sensing, 2010, Orlando, Florida, United States

Abstract

Multi-class assignment is often used to aid in the exploitation of data in the Intelligence, Surveillance, and Reconnaissance (ISR) community. For example, tracking systems collect detections into tracks and recognition systems classify objects into various categories. The reliability of these systems is highly contingent upon the correctness of the assignments. Conventional methods and metrics for evaluating assignment correctness only convey partial information about the system performance and are usually tied to the specific type of system being evaluated. Recently, information theory has been successfully applied to the tracking problem in order to develop an overall performance evaluation metric. In this paper, the information-theoretic framework is extended to measure the overall performance of any multiclass assignment system, specifically, any system that can be described using a confusion matrix. The performance is evaluated based upon the amount of truth information captured and the amount of false information reported by the system. The information content is quantified through conditional entropy and mutual information computations using numerical estimates of the association probabilities. The end result is analogous to the Receiver Operating Characteristic (ROC) curve used in signal detection theory. This paper compares these information quality metrics to existing metrics and demonstrates how to apply these metrics to evaluate the performance of a recognition system.

Citation Download Citation

Ryan S. Holt, Peter A. Mastromarino, Edward K. Kao, and Michael B. Hurley "Information theoretic approach for performance evaluation of multi-class assignment systems", Proc. SPIE 7697, Signal Processing, Sensor Fusion, and Target Recognition XIX, 76970R (27 April 2010); https://doi.org/10.1117/12.851019

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available