2.1

Selection of a PSoC device for the implementation of embeded multidimensional image processing

The investigations within QualiMess next Generation included performance analysis and evaluation of the available programmable SoC. The aim was to create a systematic of different pSoC components and to identify essential comparison features with special consideration of the application of multidimensional image processing algorithms. For this purpose, market research on the available pSoC systems and a comprehensive hardware evaluation were carried out in 2015.

The tight memory link between PS and PL was defined as an important criterion because access to the shared data (image data) is often the bottleneck for optimal processing. Altera’s pSoC (now Intel) and Xilinx offer the necessary connections combined with powerful PS computing power. Microsemis SoCs, typically in the lower to mid-range range, provide only rudimentary support that is not sufficient for more complex image processing.

As a result, the fully programmable SoC family Xilinx Zynq®-7000 with the Zynq-7020 as the target device was selected. The Zynq family combines an ARM® dual-core Cortex A9 core, programmable logic and essential peripheral functions. It is supported by a large number of IP modules and a tool infrastructure, which also makes it possible to realize very demanding tasks. Versatile prefabricated IP modules (Intellectual Property) and a tool infrastructure, which also enables the implementation of demanding tasks, support the development process. [5]

2.2

Development of a pSoC hardware platform as a versatile research setup

Based on the pSoC selection, a modular target platform was developed. The hardware should allow the connection of one or more sensors of different types (CCD, CMOS, variable bit depths, variable spectral sensitivities). For embedded image processing, an external memory had to be available that supports the required bandwidths for the various applications. In addition to the corresponding logic and computing power in PL and PS, other peripheral devices such as lighting or motion devices had to be connected. For this purpose, appropriate standard interfaces had to be available or be able to be connected via freely programmable IOs. After processing in the system, the images or data should be able to be transferred to display devices or PCs via appropriate image processing interfaces.

In order to accelerate the development process, industrial pSoC modules were used. These modules already deliver critical power supplies as well as impedance-controlled memory connections with correspondingly large memory modules. The IOs can be connected via commercially available connectors. The modules are therefore used together with a baseboard that provides additional peripherals and interfaces. Modules from Trenz GmbH and later from Enclustra were used.

Based on the catalogue of requirements, a block diagram was designed for the implementation of the baseboard. Based on this, a hardware board was created in several iteration steps, which forms the basis for the hardware demonstrators to be created in the project.

3.1

Systematics for separating algorithms that can be implemented by software or hardware

In the previous QualiMess project a general systematic for the separation of host-based and camera-based functions was developed. Since the system is based on pure FPGA structures in the camera system, it must be adapted to the current pSoCs. With pSoCs, the PL can be more closely linked to sequential processing in PS (Table 1), so that algorithms can be partly subdivided into PL and partly into PS and processed (co-processing). The bottleneck is the interface between PS and PL. Algorithms must therefore be critically analyzed for parallelizable components.

Table 1.

Comparative overview between the processing architectures in PL and PS

On this basis, we divide the image processing algorithms into four categories:

3.2

Assessment of the available development environments

For the design entry into the FPGA development process of the PL different approaches have been established, which were examined during the project duration. This includes the classical hardware description languages, high-level synthesis development environment (HLS) and graphical development environments.

Xilinx provides the Vivado development environment for the Zynq-7000 pSoc used in the project. This provides the environment in which the hardware platform is created. Xilinx uses an IP-Core based (Intellectual Property) approach. IP cores are self-contained function modules that can be integrated several times into the design. The cores are already fully specified and tested and can usually be further specified for use via configuration parameters. This approach saves development time and programming work, since large parts of a design are already available in part drafts that can be used in different systems (design re-use). In this way, projects can be granulated into individual function blocks that are reusable and easy to integrate. The IP catalogue provided by the manufacturer can also be extended with 3rd party cores, i.e. purchased IP cores, as well as self-written cores.

The design entry for most IP-Cores is usually done via hardware description languages (HDL) like VHDL or Verilog. HDL are used to describe the structure and behavior of digital circuits. In addition, syntax and semantics of HDL include notations to express time sequences and instantiations required in the hardware module.

Another option is high-level synthesis (HLS), for which Vivado includes the Eclipse-based Vivado HLS development environment. Vivado HLS converts algorithms written in high-level languages (C++, System C) to hardware specifications (e.g. Verilog and VHDL) of the Register Transfer Layer (RTL). The development environment analyzes the code to identify dependencies and parallelism and to map them to hardware.

However, since the input description is abstracted, the hardware optimization is passed from the designer to the HLS compiler, which can lead to inefficient implementations. The parallelization can usually be influenced by providing ‘hints’ (e.g. Pragma statements in C-based HLS) to the compiler. These hints usually also require corresponding design competence at the hardware level.

In the project, the IP-based design process of Xilinx was taken up and an interface concept for all IP-cores for uniform integration and interchangeability was defined. Vivado HLS is used for the creation of IP cores for data-independent, data-linking and partly also data-exchanging algorithms. For time-critical IP cores (e.g. special external interfaces) the IP development is carried out directly in VHDL.

3.3

Design of an framework for the implementation of multidimensional algorithms

As a basis for the following implementations, the design and implementation of a basic infrastructure had to be realized. Infrastructure in this context represents a basic framework for pSoC systems, a architecture for multidimensional image processing applications. This required the implementation of image acquisition and control of image sensors, intermediate storage of image stacks, variable image processing pipelines and output via standard interfaces.

For the development of the architecture different approaches were examined. Typical image processing pipelines in FPGA devices work according to the well-known EVA principle (input processing output). The data is sequentially routed through the individual processing units (or IP cores) and can thus be processed in parallel.

As explained in the introduction, pSoC exchang data between PL and PS via external RAM. In the case of the Xilinx pSoC used, this is done by using prefabricated DMA (Direct Memory Access) or especially in the image processing area via VDMA (Video Direct Memory Access) IP cores. These allow the storage or reading of sequential image data streams into external RAM memories with a management of up to 32 image memory addresses.

In the FPGA part of the pSoC various manufacturers use the AXI bus (Advanced Exensible Interface) as standard for the connection of hardware components. The AXI4 was specially developed for the exchange between digital subsystems and ARM processors and is a widely used on-chip interface standard. The standard defines:

The VDMA IP-Core is equipped with AXI4 stream ports which can be integrated into the image processing chain. The connection to the external memory is established via AXI4 full and the hardware-cast Dynamic Memory Controller. Control is via AXI4 Lite and must be triggered via software routines in the ARM.

With the help of these IP cores and external memory, different processing chains can be realized. A simple image buffer with variable image output can be realized by integrating the VDMA core. With the same resources in different configurations, the processing chain can be converted into a ring structure that allows the same chain to be run through several times starting from the image memory. This allows variable pipeline processing, but at the expense of performance and overall latency. The use of display interfaces such as HDMI, which require fixed frame rates, is also critical in this context.

Therefore a memory decoupling of the actual image processing from the image input output was chosen as approach for the framework. For this purpose, the image processing pipeline is separated and connected to the external memory via VDMAs. The image input / output path contains only basic image restoration and error correction algorithms after image acquisition. The images are then transferred to fixed memory areas in the RAM using the pixel clock of the sensor / camera. At the same time, images are read from an output area and transferred to the host via the interface. At the same time, the system has a scalable number of processing pipelines that buffer their results in RAM. By using AXI4 Stream Interconnect IPs in switch operation, the processing chain and the processing of individual IP cores can also be controlled. The rough overview of the architecture is shown in Figure 1 as a block diagram.

Figure 1.

Simplified schematic representation of the framework underlying the implementation strategy. [3]

The framework was implemented as a block design with the tools of the Vivado Design Suite. Additional hierarchies were set up for the individual parts of the block design to additionally structure the project.

4.1

Calculation of the mean values of the spectral image data (data-independent algorithm)

The functionality was implemented in VHDL as IP-core with an AXI4 Lite interface. The 8 bit grey values of the pixels transmitted one after the other in the data stream of the AXI4 stream are summed in a hardware accumulator (DSP slice). The last pixel triggers an interrupt so that the PS can access this value via AXI4 Lite. In addition, the image dimensions are determined by the synchronization signals themselves and made available via AXI4 Lite registers. Thus, the following calculation section can be dynamically adapted to different acquisition systems.

In contrast to the other calculation steps, the average value of the individual images is not calculated in the processing branch, but directly during data input. The IP core was added directly after image acquisition. The actual mean value is calculated from the transferred pixel sum within the PS. Both divisions and multiplications require valuable hardware resources. Since the determination of the mean value is not time-critical compared to the rest of the calculation, the division is calculated in the PS in a resource-saving way.

The mean values are stored as IEEE 754 coded floating point numbers in a parameter block on the external memory. Here, a memory area was organized in a uniform format that is available as shared memory for the PS and all relevant IP cores of the PL. This means that pSoC-wide work can be carried out on a unified database. However, access rights and times must be observed.

4.2

Calculation of covariances and creation of the covariance matrix (data-linking algorithm)

The calculation of the individual covariances is the most time-consuming part of the algorithm, since the original data must be computed with each other. For this purpose, a hardware IP-core was developed and programmed in Vivado HLS.

The first iteration of the IP-core calculates exactly one covariance between two images. The IP-core reads the image data autonomously from two RAM addresses via an AXI4 full interface. These are calculated pixel by pixel. The data is normalized live during the calculation. The RAM addresses and the mean values are transmitted by the PS via an AXI4-Lite interface. At the end of the calculation, the respective covariance is also read out via AXI4-Lite. The IP core itself is resource-efficient, but requires a high configuration effort and a high memory bandwidth for several calculation runs. The sequential call of the IP core led to only moderate calculation times of 80 ms per core run, which corresponds to a total calculation of 6,240 ms.

In order to achieve reduced computing times, the parallelization in the core was increased and at the same time the multiple readout of single images was minimized. The core was implemented in such a way that as many covariances as spectral channels exist, are calculated simultaneously. Due to the symmetry of the covariance matrix (mirrored into the main diagonal), not all calculations are necessary depending on the filter level. To take this fact into account, a corresponding configuration parameter was implemented.

The RAM addresses of the image data are already transferred during the initial configuration, so that only the respective mean value and the current filter level have to be transferred to carry out the calculation. The calculations are carried out in parallel. Since a fixed number of calculations must be implemented in hardware, a maximum of 12 parallel computing units were realized. This corresponds to the number of spectral channels of the filter wheel camera used as the test setup. The calculations run in parallel and are called in loops according to the number of pixels.

Through optimization, the computing time could be reduced to an average of 58ms per single core call. With 12 core runs required, this corresponds to a total calculation time of 697ms.

4.3

Eigendecomposition of the covariance matrix (data-dependent algorithm)

The Eigendecomposition is the most complex calculation step within the overall algorithm, but can be calculated relatively quickly and requires only the covariance matrix itself. The problem was therefore implemented within the PS. The eigenvalue problem can only be solved after the last covariance has been calculated. For this purpose, an iteration method based on QR decomposition with Householder transformation was implemented in the PS. The calculated eigenvalues and eigenvectors are stored as matrices in the heap memory of the processor. The vectors are sorted by size as feature vectors.

4.4

Projection of the original data, derivation of the aligned main components (data linking algorithm)

The data is derived via a hardware IP-core programmed in Vivado HLS. The original data is read from the external RAM via a full AXI4 interface. For the configuration, the necessary RAM addresses of the original data, the corresponding mean values, the feature vector and the target RAM addresses for the calculated PC are also read from the shared memory via AXI4 full.

In the IP, additional to the image data, the parameter block must be organized internally. In addition to unsigned integer values (RAM addresses), float values (mean values and eigenvectors) are also required, which are stored within the RAM as 32-bit values. This means that, in addition to memory access, a reconversion according to IEEE 754 is also required. For this configuration, an additional AXI4 full interface was implemented, which enables burst access to the RAM. The internal memory was implemented using block RAM, which was segmented according to parameter ranges using compiler pragmas. The individual block RAM areas operate as dual-port RAMs. This enables parallel access to the individual registers, which minimizes computing time.

A parallel calculation of several PCs requires an enormous amount of resources, especially look-up tables. For the implementation in the filter wheel camera, where the Core would have to calculate 12 PCs, the utilization of the Core alone would amount to over 80 percent of the pSoC’s resources. Since the first main components already contain most of the information of the spectral range, only a parallel calculation of three PCs was carried out. Parallel to the calculation, the mean value of the pixel values is calculated for each resulting image. The average values can be used as an abort criterion for several runs of the core.