22 May 2015 Evaluation methodology for query-based scene understanding systems
Author Affiliations +
In this paper, we are proposing a method for the principled evaluation of scene understanding systems in a query-based framework. We can think of a query-based scene understanding system as a generalization of typical sensor exploitation systems where instead of performing a narrowly defined task (e.g., detect, track, classify, etc.), the system can perform general user-defined tasks specified in a query language. Examples of this type of system have been developed as part of DARPA’s Mathematics of Sensing, Exploitation, and Execution (MSEE) program. There is a body of literature on the evaluation of typical sensor exploitation systems, but the open-ended nature of the query interface introduces new aspects to the evaluation problem that have not been widely considered before. In this paper, we state the evaluation problem and propose an approach to efficiently learn about the quality of the system under test. We consider the objective of the evaluation to be to build a performance model of the system under test, and we rely on the principles of Bayesian experiment design to help construct and select optimal queries for learning about the parameters of that model.
© (2015) COPYRIGHT Society of Photo-Optical Instrumentation Engineers (SPIE). Downloading of the abstract is permitted for personal use only.
Todd P. Huster, Todd P. Huster, Timothy D. Ross, Timothy D. Ross, Jared L. Culbertson, Jared L. Culbertson, } "Evaluation methodology for query-based scene understanding systems", Proc. SPIE 9476, Automatic Target Recognition XXV, 947605 (22 May 2015); doi: 10.1117/12.2177007; https://doi.org/10.1117/12.2177007

Back to Top