18 July 2014 Integrating the ODI-PPA scientific gateway with the QuickReduce pipeline for on-demand processing
Author Affiliations +
Abstract
As imaging systems improve, the size of astronomical data has continued to grow, making the transfer and processing of data a significant burden. To solve this problem for the WIYN Observatory One Degree Imager (ODI), we developed the ODI-Portal, Pipeline, and Archive (ODI-PPA) science gateway, integrating the data archive, data reduction pipelines, and a user portal. In this paper, we discuss the integration of the QuickReduce (QR) pipeline into PPA's Tier 2 processing framework. QR is a set of parallelized, stand-alone Python routines accessible to all users, and operators who can create master calibration products and produce standardized calibrated data, with a short turn-around time. Upon completion, the data are ingested into the archive and portal, and made available to authorized users. Quality metrics and diagnostic plots are generated and presented via the portal for operator approval and user perusal. Additionally, users can tailor the calibration process to their specific science objective(s) by selecting custom datasets, applying preferred master calibrations or generating their own, and selecting pipeline options. Submission of a QuickReduce job initiates data staging, pipeline execution, and ingestion of output data products all while allowing the user to monitor the process status, and to download or further process/analyze the output within the portal. User-generated data products are placed into a private user-space within the portal. ODI-PPA leverages cyberinfrastructure at Indiana University including the Big Red II supercomputer, the Scholarly Data Archive tape system and the Data Capacitor shared file system.
© (2014) COPYRIGHT Society of Photo-Optical Instrumentation Engineers (SPIE). Downloading of the abstract is permitted for personal use only.
Michael D. Young, Ralf Kotulla, Arvind Gopu, Wilson Liu, "Integrating the ODI-PPA scientific gateway with the QuickReduce pipeline for on-demand processing", Proc. SPIE 9152, Software and Cyberinfrastructure for Astronomy III, 91522U (18 July 2014); doi: 10.1117/12.2056763; http://dx.doi.org/10.1117/12.2056763
PROCEEDINGS
12 PAGES


SHARE
KEYWORDS
Calibration

Data archive systems

Image processing

Sensors

Decision support systems

Imaging systems

Astronomy

Back to Top