Translator Disclaimer
24 September 2012 J-PAS data management pipeline and archiving
Author Affiliations +
J-PAS survey is going to observe 8000 deg2 in 54 optical narrow band filters plus 3 broad band ones. The survey will produce 1.2 PB of raw data in six years of observations. The treatment of about 1.5 TB per night, coming from the 14 detectors of the JPCam camera in the JST/T250, plus one detector in the JAST/T80 camera, shall be performed during the day after the operations. This contribution presents the software and hardware architecture designed to process and validate the data. The processing of the images is divided in two main stages. The first one, which deals with instrumental correction and data validation, is run daily. The second stage is run when a tile is completed and do the combination of the individual corrected frames and weight maps. To perform the astrometric calibration, image coadding and source extraction the data management pipeline uses software from the community which is integrated through Python. The software uses a database to control the process by storing the operations performed, parameters used and quality checks. This allows fast reprocessing to retrieve intermediate stages of the treatment from the raw data for any data release. This approach saves disk space by avoiding the storage of the processed individual frames. The data archiving and processing will be done in a data center 30 km away from the observatory. It will be equipped with ~2.5 PB of storage capacity to store the raw data and the final mosaics of the 57 filters, and processing power to deal with the incoming data.
© (2012) COPYRIGHT Society of Photo-Optical Instrumentation Engineers (SPIE). Downloading of the abstract is permitted for personal use only.
D. Cristóbal-Hornillos, N. Gruel, J. Varela, A. López-Sainz, A. Ederoclite, M. Moles, A. J. Cenarro, A. Marín-Franch, J. Hernández-Fuertes, A. Yanes-Díaz, S. Chueca, S. Rueda-Teruel, F. Rueda-Teruel, and R. Luis-Simoes "J-PAS data management pipeline and archiving", Proc. SPIE 8451, Software and Cyberinfrastructure for Astronomy II, 845116 (24 September 2012);


Back to Top