Evaluation of statistical methods for the evaluation of observer trials for the assessment of the effectiveness of signature measures

Patrick Dunau; Daniel Fitz; Karin U. Stein

doi:10.1117/12.2194088

21 October 2015 Evaluation of statistical methods for the evaluation of observer trials for the assessment of the effectiveness of signature measures

Patrick Dunau, Daniel Fitz, Karin U. Stein

Author Affiliations +

Proceedings Volume 9653, Target and Background Signatures; 96530T (2015) https://doi.org/10.1117/12.2194088
Event: SPIE Security + Defence, 2015, Toulouse, France

Abstract

The statistical methods discussed in this paper are drawn from the area of machine learning or data mining as well as from descriptive statistics. These techniques are discussed with focus on their applicability to the results of observer trials in order to evaluate the effectiveness of signature measures. Signature measures aim at the change of the apparent signature of an object, e.g. a vehicle. So signature measures can be camouflage against infrared sensory, or they can be used for deception reasons. In order to evaluate the effectiveness of signature measures, observer trials provide an efficient method. The department of Signatorics of Fraunhofer IOSB developed a software tool named CARPET (Computer Aided inteRactive Performance Evaluation Tool) for the realization of observer trials. The benefit of this system is the reproducibility and uniformity of trials for every observer. The results from this system consist of marks, that were placed at particular times, as well as computer mouse positions recorded for each human observer. Based on the information gathered from these marks together with the known target object positions the statistical treatment can be done. For the statistics it has to be known to which target object the marks belong. The first problem considered in this paper concentrates on the correct labeling of the marks according to the target objects. The labeling is done using an expectation maximization scheme with the k-means clustering algorithm. The next step involves a second labeling. In this step a linear discriminant is used to decide whether a mark should be considered a hit or miss for every particular target object. After these decisions, a receiver-operating characteristics (ROC) analysis is performed in order to evaluate the detectability of each target object. Furthermore the sample mean and sample covariance formulas are used on the so called hit sets in order to approximate Gaussian distributions for every hit set. These Gaussians facilitate the evaluation of the accuracy and the precision of the hit sets. Accuracy and precision offer information about the quality of the marks set by the observers.

Citation Download Citation

Patrick Dunau, Daniel Fitz, and Karin U. Stein "Evaluation of statistical methods for the evaluation of observer trials for the assessment of the effectiveness of signature measures", Proc. SPIE 9653, Target and Background Signatures, 96530T (21 October 2015); https://doi.org/10.1117/12.2194088

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available