Fast growing space economy pushes for an increased availability of high-resolution Earth Observation (EO) data that can be subsequently used both for scientific and for commercial exploitation. On one hand, there is a growing interest towards data with higher information content thanks to the deployment of hyperspectral sensors (e.g. the Italian Hyperspectral Precursor of the Applicative Mission (PRISMA), the German Environmental Mapping and Analysis Program (EnMAP) satellite mission or the future Copernicus Hyperspectral Imaging Mission for the Environment (CHIME)). On the other hand, there is a need to reduce payload budgets and make them suitable for small platforms in order to reduce costs and also pave the way to the deployment of satellite constellations, thus also improving revisit time to a large extent.
In the whole, this yields to an increasing need to handle large amount of data on board, yet with limited (or very limited, in the case of small platforms) resources, with the consequence that this can often prevent the full exploitation of the acquired data. Additional challenges for the development of EO imagers with high spatial resolution are posed by the limited availability of large focal plane arrays in some spectral regions, like for example the medium infrared spectral range.
Compressive Sensing (CS) has been proposed as disruptive approach that can mitigate some issues related both to handle large amount of data and to use large detector bidimensional arrays for generating high resolution images. The basic idea consists in using a Spatial Light Modulator (SLM) to apply a series of suitable binary coding masks to the image collected by the foreoptics and acquiring a limited number of integrated measurements – each corresponding to a given coding mask applied to the SLM - by using a single pixel detector -. The image is subsequently reconstructed at the ground segment from the series of integrated measurements by means of suitable reconstruction algorithms -. In other words, CS – leveraging on the concepts of sparsity, which can be referred to the characteristics of ‘compressibility’ of many natural signals, and of incoherence, which implies the use of proper coding masks - merges the sampling and compression phases into a single step, providing the possibility to efficiently reconstruct the image from a smaller number of samples than that dictated by the Shannon-Nyquist theorem . Following the development of the first CS based instrument, there were several other studies that addressed the CS architecture for the implementation of instruments in different several application domains -, including space applications -. The latter included not only Earth Observation, but also Planetary Exploration and Space Science . At the same time, SLM technology was further developed – yet with the market still dominated by the commercial DLP® models manufactured by Texas Instruments – and also tested for space environmental conditions -. Another interesting feature of CS data is the possibility to perform information extraction before reconstructing the images: this feature can be exploited for a screening and selection of the the acquired data and the generation of early-warning alarms with very low latency since in this way we can bypass image transmission and full processing. Native encryption is also an additional feature of compressive sensed data that can be addressed at low computational cost. All these aspects, although fascinating for their application to spaceborne instrumentation in which available onboard resources are very limited, require to be investigated in detail and tested for a sound evaluation of actual advantages and drawbacks.
In this paper, we present a demonstrator of a super-resolved compressive imager operating in whiskbroom mode in the Visible-Near Infrared (VIS-NIR) and Medium Infrared (MIR) spectral ranges. The instrument – which is under construction in the frame of the EU H2020 funded SURPRISE project – uses a Digital MicroMirror Device (DMD) as a core element of its architecture and it is inspired by a single-pixel camera in order to overcome the use of large focal plane arrays. The demonstrator has 10 channels in the VNIR and two channels in the MIR and it can reach a superresolution factor from 4 x 4 to 32 x 32. The super-resolution factor represents the ratio between the number of pixels of the image reconstructed at the end of the process and the number of pixels of the detector. Besides, on the grounds of the results obtained by image reconstruction tests on simulated datasets by using Deep Learning based algorithms, data are expected to be natively compressed with a Compression Ratio up to 50%. The study is expected to provide valuable insight for the future development of a novel class of EO instruments with improved performances in terms of ground sampling distance, native compression and on-board processing capabilities.
THE SURPRISE DEMONSTRATOR
The demonstrator under development in the frame of the SURPRISE project is a CS instrument using SLM technology with the aim to shown an enhancement of its performances in terms of number of pixels in the reconstructed image (with respect to the number granted natively by the detector), and also with respect to instrument future capabilities in terms of on-board data processing and native encryption for a novel generation of EO super-resolved CS-based payloads in the VIS-NIR and MIR spectral regions. Although the technical specifications of the demonstrator are simpler than the actual technical features of a possible instrumental counterpart for EO applications, the working principle of the demonstrator and its implementation were conceived to be inspirational for an EO payload counterpart – definitely more complex and costly - working in whiskbroom mode from geostationary platform. In the following sections we describe the working principle of the demonstrator, its main functional blocks and the algorithms used for the reconstruction of the CS images.
Working principle and main technical features
The demonstrator’s working principle is inspired by a single-pixel camera architecture : the core element is represented by a DMD that is used as an SLM to modulate the image of the target on the image plane field stop at the DMD plane. The modulation is done by applying a binary coding mask to the DMD micromirrors. Subsequently, the coded image is spatially integrated by an optical condenser. The spatially integrated signal is finally measured by a single-pixel detector (single-pixel camera). A series of spatially integrated measurements is acquired, each measurement corresponding to a different modulation mask applied to the DMD. The image is finally reconstructed from the dataset of integrated measurements by using appropriate reconstruction algorithms. The reconstructed ‘super-resolved’ image will have a number of pixels dictated by the number of pixels of the modulation mask applied to the DMD. In addition, thanks to the CS paradigm, if we use a set of suitable modulation masks, the number of integrated measurements needed to efficiently reconstruct the image (lossy compression) can be smaller than the number of pixels of the image itself. In other words, by applying the CS paradigm we use a single detector to acquire a compressed N x N pixel image, which in conventional imaging would have required the use of a N x N detector array. More in general, if we use the same architecture with a detector array with M x M elements (instead of a single-pixel detector) and an SLM with N x N modulation elements, we can define as ‘super-resolution factor’ SR the ratio between the number of the SLM modulation elements (which will be equal to the number of pixels in the reconstructed image) and the number of detector elements. In this respect, CS architecture can be said to provide ‘super-resolved’ instruments.
Table 1 shows the main technical features of the SURPRISE demonstrator. The latter has been conceived as an instrument working in whiskbroom mode, with 10 channels in the VIS-NIR spectral region and 2 channels in the MIR spectral region. The SR factor can be set between 4 x 4 and 32 x 32. The SLM used in the demonstrator is a low-cost, commercially available DMD model manufactured by the Texas Instruments Inc.
SURPRISE demonstrator: main technical features.
The overall architecture of the SURPRISE demonstrator is shown in the block diagram of Figure 1, in which mechanical, optical and electronic parts are highlighted in different colors. The three main sub-systems can be identified:
Different types of scenes can be arranged at the TTS, and these are captured by the demonstrator. Movement stages - controlled by the TTS - provide the scan of the scene. The optical section of the demonstrator is arranged on an optical bench: the fore-optics provides the image of the observed portion (target) of the scene on the DMD image plane field stop. Following the coding step applied at the DMD plane by means of suitable binary coding masks, the signal is split by means of dichroic mirrors. There are three different optical sub-sections: the VIS-NIR sub-section with ten channels, the MIR sub-section with two channels and the High- Resolution (Hi-Res) VIS sub-section that is used for validation and calibration procedures. The Master Unit provides overall control and synchronization as well as data handling.
The optical section (Figure 2) is the core of the demonstrator: this is where the image of the target is encoded by the modulation masks and the signal is spatially integrated and finally measured.
A view of the optical section during preliminary tests in the laboratory is shown in Figure 2a: the components are arranged on an optical bench. In the inset, there is a detail of the collection optics, with its primary and secondary mirrors. Figure 2b shows a block diagram with detail of the optical layout of the demonstrator. The collection optics provides the image of the target - placed at the TTS - on the image plane field stop where the DMD is placed and the image is coded. Spectral splitting is applied after the coding stage by means of dichroic mirrors. The next step is the spatial integration by means of optical condensers. The signal is further spectrally filtered (or dispersed by the spectrometer in the VIS-NIR section) and finally measured by the corresponding detectors. An additional sub-section is represented by the Hi-Res panchromatic camera that is used for alignment and for validation procedures.
Target scanning system
The TSS is a motorized mechanical structure for the control and fine movement of several modules representing the scenes to be observed by SURPRISE demonstrator. This structure holds a set of 30x30 mm models that are used as targets. The TTS and its main parts are shown in Figure 3, together with their main functions. The TTS main parts are:
Master unit and communication control
The Master Unit (MU) controls a heterogeneous set of sensors and actuators: the VIS-NIR channels, two MWIR single pixel detectors, the Hi-Res camera, the TSS driver (including illumination sources), the DMD driver. The MU provides the proper physical interface for each of the sensors and actuators. In addition, it provides the synchronization of the sensors and the actuators and it stores the data acquired by the sensors. The Master Unit (MU) consists of the control electronics and software used to drive all the different subsystems of the SURPRISE demonstrator. Since the control of all the subsystems could be done by means of higher-level protocols, it was opted for the use a standard industrial PC instead of a FPGA.
The MU software is coded in C++ compiled with Visual Studio® and it is organized as follows:
The user interacts with the MU either through a command-line interface or a MATLAB script via TCP/IP. The devices are almost all communicating with the Master Unit via USB, except the TSS that communicates via Ethernet. The MU can either be operated locally through a command prompt interface or remotely via Ethernet through a client-server interface.
The recovery of the image by means of CS reconstruction algorithms can be performed given some conditions : nonetheless, the problem is NP-hard and an exhaustive search of the sparsest solution cannot, in fact, be done. Traditional algorithms used to solve the problem rely on different approaches, and include: greedy algorithms, iterative thresholding algorithms, convex relaxation algorithms, non-convex relaxation algorithms -.
Recently, Deep Learning (DL) algorithms have shown very good generalization ability, considerably improving the performance of previous cutting-edge technologies in many sectors, including image processing. As a consequence, CS reconstruction has been investigated also by using DL methods: the basic idea is to make Deep Learning (DL) learn a suitable reconstruction algorithm - optimizing the signal representation - instead of solving a complex numerical problem as that posed by the CS reconstruction. In general, traditional reconstruction algorithm are slower than DL-based ones. Even Total Variation (TV)  and its optimized versions - which can be considered among the fastest traditional algorithms - performs worse than most Deep Neural Network (DNN)-based algorithms, both in reconstruction precision (PSNR) and computational time.
Among different DL methods, we identified (Iterative Shrinkage/Thresholding Algorithm) ISTA-Net  - and its improved version ISTA-Net+ - as particularly suitable for images representing natural scenes. Its framework consists of mapping each classic ISTA algorithm update step into a deep network architecture in which there is a fixed number of phases that correspond to iteration in the traditional algorithm.
Both TV and DL methods were applied to a dataset of simulated images – that already included the main optical parameters of the SURPRISE demonstrator like the Point Spread Function (PSF) - of natural targets and the results compared. In Figure 4, we show the results we obtained in the reconstruction of one of the images of the dataset by applying both traditional and DL methods. Figure 4a shows the original simulated image of a natural target (Travertino stone image), while Figure 4b and Figure 4c show, respectively, its reconstruction by using TV and its reconstruction by using ISTA-Net+. Both reconstructions refer to the case of a 32x32 super-resolution factor and 75% compression ratio. It is apparent that the reconstruction obtained by using the Total Variation method exhibits less detail, and occasionally have significant ‘blockiness’ artifacts, whereas the DL method is successful at reconstructing images of suitable quality (Figure 4c). In general, by evaluating the reconstruction results in terms of Peak Signal-to-Noise Ratio (PSNR), ISTA-Net+ significantly outperformed the TV algorithm (PSNR (dB) with 32x32 super-resolution factor and 75% compression ratio is 33.63 for DL and 29.45 for TV on the ‘Travertino’ sample image). The gain is significant at 25% and 50% compression ratios and becomes even larger at high quality levels. If compression is not applied, the DP method provides an almost perfect image; on the contrary, the TV method shows a quality plateau.
In this paper, we presented a demonstrator of a super-resolved CS instrument working in the VIS-NIR and in the MIR, which is under construction at CNR-IFAC labs in the frame of the EU H2020 SURPRISE project. All the sub-systems of the demonstrator were already assembled and tested whereas their integration is ongoing. The main features of the demonstrator include the availability of 10 channels in the VIS-NIR and 2 channels in the MIR. The demonstrator has a super-resolution factor that can be set up to 32 x 32 to achieve final images reconstructed at the end of the process with an enhanced number of pixels. Its CS-based architecture allows also for native compression of the data at the acquisition stage and intrinsic encryption. Deep learning- based algorithm tests on simulated dataset of natural targets demonstrated an effective reconstruction with Compression Ratio between 50% and 70%. EO optical payloads based on the same approach could benefit not only in terms of native compression and encryption capabilities, but also in terms of increased Ground Sampling Distance with respect to that granted natively by the number of pixels of the detector used in the payload architecture.
The SURPRISE project has received funding from the European Union’s Horizon 2020 research and innovation programme under Grant Agreement No 870390.
Barducci, A., Coluccia, G., Guzzi, D., Lastri, C., Magli, E. and Raimondi, V., “Algorithms and prototyping of a compressive hyperspectral imager,” Compressive Sensing of Earth Observations, ser. Signal and Image Processing of Earth Observations, 329 –350 CRC Press, Chapter 15, pp.2017). https://doi.org/10.1201/9781315154626 Google Scholar
Chen, H., Asif, M. S., Sankaranarayanan, A. C. and Veeraraghavan, A., “FPA-CS: Focal plane array-based compressive imaging in short-wave infrared,” in 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2358 –2366 (2015). Google Scholar
Guzzi, D., Coluccia, G., Labate, D., Lastri, C., Magli, E., Nardino, V., Palombi, L., Pippi, I., Coltuc, D., Barducci, A., Zuccaro Marchi, A., and Raimondi, V., “Optical compressive sensing technologies for space applications: instrumental concepts and performance analysis,” in Proc. SPIE 11180, International Conference on Space Optics — ICSO 2018, 111806B (2018). Google Scholar
Raimondi, V., Magli, E., Coltuc, D., Labate, D., Barducci, A., Baronti, S., Coluccia, G., Garoi, F., Guzzi, D., Iordache, I., Lastri, C., Nardino, V., Palombi, L., Pippi, I., Ravazzi, C., Zuccaro-Marchi, A. and Miranda, M., “Optical compressive sensing technologies for space applications: a pros and cons analysis of application-driven instrumental concepts,” in 5th International Workshop on On-Board Payload Data Compression OBPDC, (ISBN) 978-88-941917-0-7, ESA, (2016). Google Scholar
Raimondi, V., Acampora, L., Amato, G., Baldi, M., Guzzi, D., Lastri, C., Nardino, V., Palombi, L., Magli, E., Bianchi, T., Valsesia, D., Buongiorno, M.F., Romaniello, V., Silvestri, M., Corti, M., Corti, F., Lapucci, T., Scopa, T., “A feasability study for a compressive sensing imager in the medium infrared for hotspot detection,” in Proceedings Vol.11852, International Conference on Space Optics — ICSO 2020, 1185259 (2021). https://doi.org/10.1117/12.2599938 Google Scholar
Coluccia, G., Lastri, C., Guzzi, D., Magli, E., Nardino, V., Palombi, L., Pippi, I., Raimondi, V., Ravazzi, C., Garoi, F., Coltuc, D., Vitulli, R., and Zuccaro Marchi, A., “Optical Compressive Imaging Technologies for Space Big Data,” IEEE Transactions on Big Data, 6 430 –442 (2020). https://doi.org/10.1109/TBigData.6687317 Google Scholar
Lastri, C., Amato, G., Baldi, M., Bianchi, T., Buongiorno, M.F., Corti, C., Corti, F., Corti, M., Franci, E., Guzzi, D.; Magli, E.; Nardino, V.; Palombi, L.; Romaniello, V.; Scopa, T.; De Cumis, Siciliani M.; Silvestri, M.; Valsesia, D.; Raimondi, V., “SISSI Project: A Feasibility Study for a Super Resolved Compressive Sensing Multispectral Imager in the Medium Infrared,” in Engineering Proceedings, 28 (2021). https://doi.org/10.3390/engproc2021008028 Google Scholar
Zamkotsian, F., Lanzoni, P., Grassi, E., Barette, R., Fabron, C., Tangen, K., Valenziano, L., Marchand, L., Duvet L., “Successful evaluation for space applications of the 2048x1080 DMD, Emerging Digital Micromirror Device Based Systems and Applications III,” Proc. of SPIE, 7932 79320A 2011). https://doi.org/10.1117/12.876872 Google Scholar
Travinsky, A., Vorobiev, D., Ninkov, Z., Raisanen, A., Quijada, M.A., Smee, S.A., Pellish, J.A., Schwartz, T., Robberto, M., Heap, S., Conley, D., Benavides, C., Garcia, N., Bredl, Z., Yllanes, S., “Evaluation of Digital Micromirror Devices for use in space-based Multi-Object Spectrometer application,” J. of Astronomical Telescopes, Instruments, and Systems, 3 035003 (2017). https://doi.org/10.1117/1.JATIS.3.3.035003 Google Scholar
Beck, A., and Teboulle, M., “A Fast Iterative Shrinkage- Thresholding Algorithm for Linear Inverse Problems,” SIAM, 2 (1), 183 –202 (2009). Google Scholar