17 April 2008 A confidence paradigm for classification systems
Author Affiliations +
There is no universally accepted methodology to determine how much confidence one should have in a classifier output. This research proposes a framework to determine the level of confidence in an indication from a classifier system where the output is a measurement value. There are two types of confidence developed in this paper. The first is confidence in a classification system or classifier and is denoted classifier confidence. The second is the confidence in the output of a classification system or classifier. In this paradigm, we posit that the confidence in the output of a classifier should be, on average, equal to the confidence in the classifier as a whole (i.e., classifier confidence). The amount of confidence in a given classifier is estimated using multiattribute preference theory and forms the foundation for a quadratic confidence function that is applied to posterior probability estimates. Classifier confidence is currently determined based upon individual measurable value functions for classification accuracy, average entropy, and sample size, and the form of the overall measurable value function is multilinear based upon the assumption of weak difference independence. Using classifier confidence, a quadratic function is trained to be the confidence function which inputs a posterior probability and outputs the confidence in a given indication. In this paradigm, confidence is not equal to the posterior probability estimate but is related to it. This confidence measure is a direct link between traditional decision analysis techniques and traditional pattern recognition techniques. This methodology is applied to two real world data sets, and results show the sort of behavior that would be expected from a rational confidence measure.
© (2008) COPYRIGHT Society of Photo-Optical Instrumentation Engineers (SPIE). Downloading of the abstract is permitted for personal use only.
Nathan J. Leap, Nathan J. Leap, Kenneth W. Bauer, Kenneth W. Bauer, } "A confidence paradigm for classification systems", Proc. SPIE 6968, Signal Processing, Sensor Fusion, and Target Recognition XVII, 69680U (17 April 2008); doi: 10.1117/12.776755; https://doi.org/10.1117/12.776755

Back to Top