30 September 2013 On covariance structure in noisy, big data
Author Affiliations +
Herein we describe theory and algorithms for detecting covariance structures in large, noisy data sets. Our work uses ideas from matrix completion and robust principal component analysis to detect the presence of low-rank covariance matrices, even when the data is noisy, distorted by large corruptions, and only partially observed. In fact, the ability to handle partial observations combined with ideas from randomized algorithms for matrix decomposition enables us to produce asymptotically fast algorithms. Herein we will provide numerical demonstrations of the methods and their convergence properties. While such methods have applicability to many problems, including mathematical finance, crime analysis, and other large-scale sensor fusion problems, our inspiration arises from applying these methods in the context of cyber network intrusion detection.
© (2013) COPYRIGHT Society of Photo-Optical Instrumentation Engineers (SPIE). Downloading of the abstract is permitted for personal use only.
Randy C. Paffenroth, Randy C. Paffenroth, Ryan Nong, Ryan Nong, Philip C. Du Toit, Philip C. Du Toit, "On covariance structure in noisy, big data", Proc. SPIE 8857, Signal and Data Processing of Small Targets 2013, 88570E (30 September 2013); doi: 10.1117/12.2037882; https://doi.org/10.1117/12.2037882

Back to Top