Meaningful machine learning robustness evaluation in real-world machine learning enabled system contexts

Ben Hiett; Peter Boyd; Charles Fletcher; Sam Gowland; James Sharp; David Slogget; Alec Banks

doi:10.1117/12.2638492

28 October 2022 Meaningful machine learning robustness evaluation in real-world machine learning enabled system contexts

Ben Hiett, Peter Boyd, Charles Fletcher, Sam Gowland, James Sharp, David Slogget, Alec Banks

Author Affiliations +

Proceedings Volume 12276, Artificial Intelligence and Machine Learning in Defense Applications IV; 1227608 (2022) https://doi.org/10.1117/12.2638492
Event: SPIE Security + Defence, 2022, Berlin, Germany

Abstract

Applied research presented in this paper describes an approach to provide meaningful evaluation of the Machine Learning (ML) components in a Full Motion Video (FMV) Machine Learning Enabled System (MLES). The MLES itself is not discussed in the paper. We focus on the experimental activity that has been designed to provide confidence that the MLES, when fielded under dynamic and uncertain conditions, performance will not be undermined by a lack of ML robustness. For example, to real-world changes of the same scene under differing light conditions. The paper details the technical approach and how it is applied to data, across the overall experimental pipeline, consisting of a perturbation engine, test pipeline and metric production. Data is from a small imagery dataset and the results are shown and discussed as part of a proof of concept study.

Conference Presentation

Citation Download Citation

Ben Hiett, Peter Boyd, Charles Fletcher, Sam Gowland, James Sharp, David Slogget, and Alec Banks "Meaningful machine learning robustness evaluation in real-world machine learning enabled system contexts", Proc. SPIE 12276, Artificial Intelligence and Machine Learning in Defense Applications IV, 1227608 (28 October 2022); https://doi.org/10.1117/12.2638492

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available