The common pipeline library: standardizing pipeline processing

Derek J. McKay; Pascal Ballester; Klaus Banse; Carlo Izzo; Yves Jung; Michael Kiesgen; Nick Kornweibel; Lars K. Lundin; Andrea Modigliani; Ralf M. Palsa; Cyrus Sabet

doi:10.1117/12.551214

16 September 2004 The common pipeline library: standardizing pipeline processing

Derek J. McKay, Pascal Ballester, Klaus Banse, Carlo Izzo, Yves Jung, Michael Kiesgen, Nick Kornweibel, Lars K. Lundin, Andrea Modigliani, Ralf M. Palsa, Cyrus Sabet

Author Affiliations +

Proceedings Volume 5493, Optimizing Scientific Return for Astronomy through Information Technologies; (2004) https://doi.org/10.1117/12.551214
Event: SPIE Astronomical Telescopes + Instrumentation, 2004, Glasgow, United Kingdom

Abstract

The European Southern Observatory (ESO) develops and maintains a large number of instrument-specific data processing pipelines. These pipelines must produce standard-format output and meet the need for data archiving and the computation and logging of quality assurance parameters. As the number, complexity and data-output-rate of instrument increases, so does the challenge to develop and maintain the associated processing software. ESO has developed the Common Pipeline Library (CPL) in order to unify the pipeline production effort and to minimise code duplication. The CPL is a self-contained ISO-C library, designed for use in a C/C++ environment. It is designed to work with FITS data, extensions and meta-data, and provides a template for standard algorithms, thus unifying the look-and-feel of pipelines. It has been written in such a way to make it extremely robust, fast and generic, in order to cope with the operation-critical online data reduction requirements of modern observatories. The CPL has now been successfully incorporated into several new and existing instrument systems. In order to achieve such success, it is essential to go beyond simply making the code publicly available, but also engage in training, support and promotion. There must be a commitment to maintenance, development, standards-compliance, optimisation, consistency and testing. This paper describes in detail the experiences of the CPL in all these areas. It covers the general principles applicable to any such software project and the specific challenges and solutions, that make the CPL unique.

Citation Download Citation

Derek J. McKay, Pascal Ballester, Klaus Banse, Carlo Izzo, Yves Jung, Michael Kiesgen, Nick Kornweibel, Lars K. Lundin, Andrea Modigliani, Ralf M. Palsa, and Cyrus Sabet "The common pipeline library: standardizing pipeline processing", Proc. SPIE 5493, Optimizing Scientific Return for Astronomy through Information Technologies, (16 September 2004); https://doi.org/10.1117/12.551214

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available