19 December 2002 Statistical methodology for massive datasets and model selection
Author Affiliations +
Abstract
Astronomy is facing a revolution in data collection, storage, analysis, and interpretation of large datasets. The data volumes here are several orders of magnitude larger than what astronomers and statisticians are used to dealing with, and the old methods simply do not work. The National Virtual Observatory (NVO) initiative has recently emerged in recognition of this need and to federate numerous large digital sky archives, both ground based and space based, and develop tools to explore and understand these vast volumes of data. In this paper, we address some of the critically important statistical challenges raised by the NVO. In particular a low-storage, single-pass, sequential method for simultaneous estimation of multiple quantiles for massive datasets will be presented. Density estimation based on this procedure and a multivariate extension will also be discussed. The NVO also requires statistical tools to analyze moderate size databases. Model selection is an important issue for many astrophysical databases. We present a simple likelihood based 'leave one out' method to select the best among the several possible alternatives. The performance of the method is compared to those based on Akaike Information Criterion and Bayesian Information Criterion.
© (2002) COPYRIGHT Society of Photo-Optical Instrumentation Engineers (SPIE). Downloading of the abstract is permitted for personal use only.
G. Jogesh Babu, G. Jogesh Babu, James P. McDermott, James P. McDermott, "Statistical methodology for massive datasets and model selection", Proc. SPIE 4847, Astronomical Data Analysis II, (19 December 2002); doi: 10.1117/12.460339; https://doi.org/10.1117/12.460339
PROCEEDINGS
10 PAGES


SHARE
RELATED CONTENT

Radio data archiving system
Proceedings of SPIE (July 25 2016)
Topic maps as a virtual observatory tool
Proceedings of SPIE (October 31 2001)
Knowledge discovery in astronomical data
Proceedings of SPIE (July 14 2008)
Outlier detection in astronomical data
Proceedings of SPIE (September 15 2004)
CIAO: Chandra's data analysis system
Proceedings of SPIE (June 29 2006)
Building the infrastructure for the virtual observatory
Proceedings of SPIE (October 31 2001)

Back to Top