Translator Disclaimer
24 September 2012 The MWA archive infrastructure: archiving terabytes of data over dedicated WAN connections
Author Affiliations +
Abstract
The Murchison Wide Field Array (MWA) is being upgraded from 32 tiles to 128 tiles of 16 dual-polarization dipole antennas. In the course of this project the software and the data infrastructure are also undergoing a major overhaul in order to cope with a more continuous and remote operational model; and the substantial increase in data rate (400 MB/s from the correlator plus 160 MB/s from the real time imaging pipeline). During the course of 2012/13 the data collected by the MWA will be transported via a dedicated 40 Gbit WAN network link between the Murchison Radio Observatory (MRO) and Perth (700 km). However, this network will not be available for some time; and until then, the data will be transported using disk arrays instead. Once in Perth, the data will be ingested into a tape library. The archiving process itself consists of various steps executed either at the MRO site or in Perth, and makes use of a modied version of the archiving system from the Atacama Large Millimeter Array (ALMA). This includes the extraction of meta-data from the original raw data and the ingestion and generation of the appropriate data links for the MWA archive. The MWA correlator generates a collection of small les all belonging to the same observation. In order to optimise the network transfer and the storage of those les they are transparently packed into larger containers at the MRO site and subsequently handled as one big le. This paper describes the setup of the MWA archiving infrastructure.
© (2012) COPYRIGHT Society of Photo-Optical Instrumentation Engineers (SPIE). Downloading of the abstract is permitted for personal use only.
Andreas Wicenec, Dave Pallot, Alessio Checcucci, Slava Kitaeff, Kevin Vinsen, and Chen Wu "The MWA archive infrastructure: archiving terabytes of data over dedicated WAN connections", Proc. SPIE 8451, Software and Cyberinfrastructure for Astronomy II, 845118 (24 September 2012); https://doi.org/10.1117/12.925901
PROCEEDINGS
7 PAGES


SHARE
Advertisement
Advertisement
RELATED CONTENT

Phasing up ALMA
Proceedings of SPIE (July 17 2014)
The ALMA software architecture
Proceedings of SPIE (September 14 2004)
NOAO data products program
Proceedings of SPIE (January 01 2002)
The Australian SKA Pathfinder
Proceedings of SPIE (September 16 2012)
The ALMA computing project: initial commissioning
Proceedings of SPIE (July 13 2008)
ALMA: the first year of observations
Proceedings of SPIE (September 12 2012)

Back to Top