6 July 2018 Challenges of real-time processing in HPC environments: the ASKAP experience
Author Affiliations +
The Commonwealth Scientific and Industrial Research Organization (CSIRO) Astronomy and Space Science division operates the Australian Square Kilometre Array Pathfinder (ASKAP), a state-of-the-art radio telescope located at the Murchison Radio Observatory (MRO) in Western Australia. The data rates produced are significant enough that it is necessary to process in real-time in high performance computing (HPC) environment. This presents a number of challenges with respect to resource availability and the performance variability resulting from interactions of many parallel software and hardware components. These issues include: disk performance and capacity, inter-node and intranode parallel communication overheads, and process isolation. Typically, supercomputing facilities are used for batch processing of large established data sources or simulations. Small intermittent processing delays resulting from temporary resource contention issues do not affect these processes as the total compute time is the dominating factor and small wait times are insignificant. However, for real-time processing of high data ingest rates these small and unpredictable contentions for resources can be problematic. In our case, even delays of seconds can have a negative effect on our ability to reliably ingest the data. We have learned much from addressing these challenges and our experience and solutions will provide valuable input to the radio astronomy community, especially for larger projects, like the Square Kilometer Array (SKA), where the same challenges will present at even larger scales.
Conference Presentation
© (2018) COPYRIGHT Society of Photo-Optical Instrumentation Engineers (SPIE). Downloading of the abstract is permitted for personal use only.
Eric Bastholm, Eric Bastholm, Juan C. Guzman, Juan C. Guzman, Maxim Voronkov, Maxim Voronkov, Paulus Lahur, Paulus Lahur, "Challenges of real-time processing in HPC environments: the ASKAP experience", Proc. SPIE 10707, Software and Cyberinfrastructure for Astronomy V, 107070J (6 July 2018); doi: 10.1117/12.2313137; https://doi.org/10.1117/12.2313137

Back to Top