8 August 2016 RabbitQR: fast and flexible big data processing at LSST data rates using existing, shared-use hardware
Author Affiliations +
Abstract
Processing astronomical data to science readiness was and remains a challenge, in particular in the case of multi detector instruments such as wide-field imagers. One such instrument, the WIYN One Degree Imager, is available to the astronomical community at large, and, in order to be scientifically useful to its varied user community on a short timescale, provides its users fully calibrated data in addition to the underlying raw data. However, time-efficient re-processing of the often large datasets with improved calibration data and/or software requires more than just a large number of CPU-cores and disk space. This is particularly relevant if all computing resources are general purpose and shared with a large number of users in a typical university setup. Our approach to address this challenge is a flexible framework, combining the best of both high performance (large number of nodes, internal communication) and high throughput (flexible/variable number of nodes, no dedicated hardware) computing. Based on the Advanced Message Queuing Protocol, we a developed a Server-Manager- Worker framework. In addition to the server directing the work flow and the worker executing the actual work, the manager maintains a list of available worker, adds and/or removes individual workers from the worker pool, and re-assigns worker to different tasks. This provides the flexibility of optimizing the worker pool to the current task and workload, improves load balancing, and makes the most efficient use of the available resources. We present performance benchmarks and scaling tests, showing that, today and using existing, commodity shared- use hardware we can process data with data throughputs (including data reduction and calibration) approaching that expected in the early 2020s for future observatories such as the Large Synoptic Survey Telescope.
© (2016) COPYRIGHT Society of Photo-Optical Instrumentation Engineers (SPIE). Downloading of the abstract is permitted for personal use only.
Ralf Kotulla, Arvind Gopu, Soichi Hayashi, "RabbitQR: fast and flexible big data processing at LSST data rates using existing, shared-use hardware", Proc. SPIE 9913, Software and Cyberinfrastructure for Astronomy IV, 99131U (8 August 2016); doi: 10.1117/12.2233527; https://doi.org/10.1117/12.2233527
PROCEEDINGS
7 PAGES


SHARE
RELATED CONTENT

ISRO's dual frequency airborne SAR pre-cursor to NISAR
Proceedings of SPIE (May 02 2016)
AVIRIS Ground Data-Processing System
Proceedings of SPIE (January 01 1987)
NOAO data products program
Proceedings of SPIE (January 02 2002)
The Large Synoptic Survey Telescope data challenges
Proceedings of SPIE (July 19 2010)
NRAO VLA archive survey
Proceedings of SPIE (July 12 2008)

Back to Top