Recent advances in very high resolution (VHR) earth observation (EO) techniques have led to a massive increase in data volumes to be processed. These remote sensing (RS) processing steps are complex and heterogeneous and an optimized use of these algorithms in both High Processing Computer (HPC) and Cloud platforms are still an important and opened study field for RS data providers with actual and future EO missions. The goal of our study is to identify and develop a new architecture to deal with large and increasing quantity of data and versatile production profiles. In this study, we took the example of the actual and complete processing pipeline used in Pleiades production to deliver perfect sensor image. This pipeline is composed of heterogeneous radiometric and geometric processing steps. In the first part, we study five main big data framework solutions. As result of this study, we identify Apache Spark as the best framework to use due to its performance, great development maturity, and data resilience certification. In the second part, and to develop the new processing pipeline, we redesign the processing pipeline with separation of the metadata management and the core processing. These good practices help us to develop and reuse legacy algorithms to an operational processing pipeline compatible with big data paradigm. As result of this development, we successfully identify a generic way to develop new processes and reuse legacy algorithms with large data paradigm and by keeping great performance and, more importantly, gaining platform flexibility.

© (2018) COPYRIGHT Society of Photo-Optical Instrumentation Engineers (SPIE). Downloading of the abstract is permitted for personal use only.

Olivier Melet, Antoine Masse, Yannick Ott, and Pierre Lassalle, "A new architecture paradigm for image processing pipeline applied to massive remote sensing data production," Proc. SPIE 10789, Image and Signal Processing for Remote Sensing XXIV, 107890F (Presented at SPIE Remote Sensing: September 11, 2018; Published: 9 October 2018); https://doi.org/10.1117/12.2325700.