Significantly reducing the processing times of high-speed photometry data sets using a distributed computing model

Paul Doyle; Fred Mtenzi; Niall Smith; Adrian Collins; Brendan O'Shea

doi:10.1117/12.924863

24 September 2012 Significantly reducing the processing times of high-speed photometry data sets using a distributed computing model

Paul Doyle, Fred Mtenzi, Niall Smith, Adrian Collins, Brendan O'Shea

Author Affiliations +

Proceedings Volume 8451, Software and Cyberinfrastructure for Astronomy II; 84510C (2012) https://doi.org/10.1117/12.924863
Event: SPIE Astronomical Telescopes + Instrumentation, 2012, Amsterdam, Netherlands

Abstract

The scientific community is in the midst of a data analysis crisis. The increasing capacity of scientific CCD instrumentation and their falling costs is contributing to an explosive generation of raw photometric data. This data must go through a process of cleaning and reduction before it can be used for high precision photometric analysis. Many existing data processing pipelines either assume a relatively small dataset or are batch processed by a High Performance Computing centre. A radical overhaul of these processing pipelines is required to allow reduction and cleaning rates to process terabyte sized datasets at near capture rates using an elastic processing architecture. The ability to access computing resources and to allow them to grow and shrink as demand fluctuates is essential, as is exploiting the parallel nature of the datasets. A distributed data processing pipeline is required. It should incorporate lossless data compression, allow for data segmentation and support processing of data segments in parallel. Academic institutes can collaborate and provide an elastic computing model without the requirement for large centralized high performance computing data centers. This paper demonstrates how a base 10 order of magnitude improvement in overall processing time has been achieved using the "ACN pipeline", a distributed pipeline spanning multiple academic institutes.

Citation Download Citation

Paul Doyle, Fred Mtenzi, Niall Smith, Adrian Collins, and Brendan O'Shea "Significantly reducing the processing times of high-speed photometry data sets using a distributed computing model", Proc. SPIE 8451, Software and Cyberinfrastructure for Astronomy II, 84510C (24 September 2012); https://doi.org/10.1117/12.924863

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available

Members: $17.00

Non-members: $21.00 ADD TO CART

PROCEEDINGS
10 PAGES

DOWNLOAD PAPER SAVE TO MY LIBRARY

GET CITATION

RIGHTS & PERMISSIONS

Get copyright permission Get copyright permission on Copyright Marketplace

KEYWORDS

Image processing

Data processing

Data storage

Charge-coupled devices

Data modeling

Photometry

Calibration

Show All Keywords

Keywords/Phrases

Search In:

Publication Years