SIP: a modern flexible data pipeline

Pauline Barmby; Zhong Wang; Massimo Marengo; Matthew L.N. Ashby

doi:10.1117/12.550468

16 September 2004 SIP: a modern flexible data pipeline

Pauline Barmby, Zhong Wang, Massimo Marengo, Matthew L.N. Ashby

Proceedings Volume 5493, Optimizing Scientific Return for Astronomy through Information Technologies; (2004) https://doi.org/10.1117/12.550468
Event: SPIE Astronomical Telescopes + Instrumentation, 2004, Glasgow, United Kingdom

Abstract

IRAC, the Infrared Array Camera on the Spitzer Space Telescope, generated well over 150,000 images during the in-orbit checkout and science verification phase of the mission. All of these were processed with SIP, the SAO IRAC Pipeline. SIP was created by and for the members of the IRAC instrument team at the Smithsonian Astrophysical Observatory, to allow short-timescale data processing and rapid-turnaround software testing and algorithm modification. SIP makes use of perl scripting and data mirroring to transfer and manage data, a mySQL database to select calibration data, and Python/numarray to process the image data; it is designed to run with no user interaction. SIP is fast, flexible, and robust. We present some 'lessons learned' from the construction and maintenance of SIP, and discuss prospects for future improvement.

Citation Download Citation

Pauline Barmby, Zhong Wang, Massimo Marengo, and Matthew L.N. Ashby "SIP: a modern flexible data pipeline", Proc. SPIE 5493, Optimizing Scientific Return for Astronomy through Information Technologies, (16 September 2004); https://doi.org/10.1117/12.550468

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available