On covariance structure in noisy, big data

Randy C. Paffenroth; Ryan Nong; Philip C. Du Toit

doi:10.1117/12.2037882

30 September 2013 On covariance structure in noisy, big data

Randy C. Paffenroth, Ryan Nong, Philip C. Du Toit

Proceedings Volume 8857, Signal and Data Processing of Small Targets 2013; 88570E (2013) https://doi.org/10.1117/12.2037882
Event: SPIE Optical Engineering + Applications, 2013, San Diego, California, United States

Abstract

Herein we describe theory and algorithms for detecting covariance structures in large, noisy data sets. Our work uses ideas from matrix completion and robust principal component analysis to detect the presence of low-rank covariance matrices, even when the data is noisy, distorted by large corruptions, and only partially observed. In fact, the ability to handle partial observations combined with ideas from randomized algorithms for matrix decomposition enables us to produce asymptotically fast algorithms. Herein we will provide numerical demonstrations of the methods and their convergence properties. While such methods have applicability to many problems, including mathematical finance, crime analysis, and other large-scale sensor fusion problems, our inspiration arises from applying these methods in the context of cyber network intrusion detection.

Citation Download Citation

Randy C. Paffenroth, Ryan Nong, and Philip C. Du Toit "On covariance structure in noisy, big data", Proc. SPIE 8857, Signal and Data Processing of Small Targets 2013, 88570E (30 September 2013); https://doi.org/10.1117/12.2037882

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available