16 September 2004 The common pipeline library: standardizing pipeline processing
Author Affiliations +
Abstract
The European Southern Observatory (ESO) develops and maintains a large number of instrument-specific data processing pipelines. These pipelines must produce standard-format output and meet the need for data archiving and the computation and logging of quality assurance parameters. As the number, complexity and data-output-rate of instrument increases, so does the challenge to develop and maintain the associated processing software. ESO has developed the Common Pipeline Library (CPL) in order to unify the pipeline production effort and to minimise code duplication. The CPL is a self-contained ISO-C library, designed for use in a C/C++ environment. It is designed to work with FITS data, extensions and meta-data, and provides a template for standard algorithms, thus unifying the look-and-feel of pipelines. It has been written in such a way to make it extremely robust, fast and generic, in order to cope with the operation-critical online data reduction requirements of modern observatories. The CPL has now been successfully incorporated into several new and existing instrument systems. In order to achieve such success, it is essential to go beyond simply making the code publicly available, but also engage in training, support and promotion. There must be a commitment to maintenance, development, standards-compliance, optimisation, consistency and testing. This paper describes in detail the experiences of the CPL in all these areas. It covers the general principles applicable to any such software project and the specific challenges and solutions, that make the CPL unique.
© (2004) COPYRIGHT Society of Photo-Optical Instrumentation Engineers (SPIE). Downloading of the abstract is permitted for personal use only.
Derek J. McKay, Derek J. McKay, Pascal Ballester, Pascal Ballester, Klaus Banse, Klaus Banse, Carlo Izzo, Carlo Izzo, Yves Jung, Yves Jung, Michael Kiesgen, Michael Kiesgen, Nick Kornweibel, Nick Kornweibel, Lars K. Lundin, Lars K. Lundin, Andrea Modigliani, Andrea Modigliani, Ralf M. Palsa, Ralf M. Palsa, Cyrus Sabet, Cyrus Sabet, } "The common pipeline library: standardizing pipeline processing", Proc. SPIE 5493, Optimizing Scientific Return for Astronomy through Information Technologies, (16 September 2004); doi: 10.1117/12.551214; https://doi.org/10.1117/12.551214
PROCEEDINGS
9 PAGES


SHARE
RELATED CONTENT

ALMA observations during its first early science cycles
Proceedings of SPIE (August 05 2014)
VLT service mode operations at seven years
Proceedings of SPIE (June 29 2006)
Data management for the EVLA
Proceedings of SPIE (September 23 2012)
NRAO VLA archive survey
Proceedings of SPIE (July 11 2008)
From Chile to Europe in minutes handling the data...
Proceedings of SPIE (July 29 2010)
The Chandra X ray Center a combined science and...
Proceedings of SPIE (June 28 2006)

Back to Top