Hyperspectral imaging (HSI) is by now a well-established technique. It evolved from use in airborne remote sensing for environmental monitoring and detection of natural resources into a method employed for process analysis.1 Fields of application are vast and cover nearly every industrial sector. Generally, all processes that require 100% real-time inspection of produced goods and/or raw materials, waste materials or other constituents such as solvents, exhausts and tools can successfully employ HSI to replace or supplement other inspection systems that are less reliable, time consuming or costly and do not provide full and constant control. Even non-industrial applications have a lot to gain from HSI. Agriculture and forestry, medicine or art inspection are cases in point. Lab-based investigations can profit from high-speed 100% analysis as well, using automated test systems based on HSI.
The use of RGB (Red-Green-Blue) cameras and associated software tools for feature recognition in industrial applications has become a standard during the last 30 years.2 These inspection systems that mimic the view of the human eye and the recognition and response capabilities of the human brain can provide a huge range of shape and colour information. However, they reach their limits whenever the desired information is a) not in the visible range, or b) it is not sufficient to classify the emitted radiation as a combination of RGB intensities in three very narrow spectral regions, but a continuous assessment of a wide wavelength range with good spectral resolution is necessary.
One branch that has gained much from the application of HSI is the recycling business.3 Polymers give rise to distinct spectral features and thus enable straightforward separation of plastics using a near infrared (NIR) HSI camera. (Please note that here, “NIR” denotes the spectral range from 780-3000 nm that is sometimes separated into a NIR and a shortwave infrared, SWIR, region). This has evolved into a mature technique and today many companies offer ready sorting machines either for retraction of complete packaging units (e.g. bottles), or for separating out valuable polymer types after disposal and shredding into flakes. Another comparably straightforward application is quantification of moisture in different media, owing to high absorbance of NIR radiation by water. In addition to these, below some potential or established applications of HSI are given. Applications from different areas and in different wavelength ranges were chosen.
The data given below originates from feasibility studies that were carried out by the company inno-spec GmbH, Germany. The HSI systems used for these tests were the following of inno-spec’s products: RedEye (NIR HSI camera, wavelength range 950-1700 nm), GreenEye (visible and NIR HSI camera, 400-1000 nm) and BlackEye (medium infrared (MIR) camera, 2900-4200 nm).
CLASSIFICATION USING HYPERSPECTAL IMAGING
HSI data is analysed by chemometric tools that use either classification or quantification algorithms.4 In classification, characteristic spectral features are used to assign either each image pixel or recognized objects to predetermined classes. This is the basis for applications such as automated sorting, elimination of contaminants or reject, detection of desired materials in a mixture or discrimination of properties. Spatial information of the data combined with its class assignment can then be utilized for e.g. ejection of undesired parts, separation of different materials or the control of process parameters in a feedback loop.
Plastic separation in recycling
Plastic separation is one of the classical examples for the application of NIR HSI. The growing demand for recycling of plastic waste and the increased readiness to use recyclates in production that leads to the need of very pure recycled materials has boosted this technology over the last 15-20 years. While the practice of industrial countries to sell their waste to developing countries overseas is widely criticized and met with decreasing acceptance, the future demand for these systems is growing to enable effective waste treatment within the countries of origin.
Plastic separation can be regarded as a textbook example for NIR HSI use and thus is included in this overview. Apart from political and social motivations, there are physical reasons why it features so prominently in application examples. The NIR region is especially suited to display absorption spectra with very characteristic features that enable identification of different polymers. Figure 1 shows spectra of common packaging polymers such as polyethylene terephthalate (PET), polyethylene (PE), polypropylene (PP) and polyvinyl chloride (PVC). All show prominent peaks around 1200 (CHn), 1400 (CHn and ROH) and 1700 nm (CHn).5 Peak shifts and multiplet structures give polymers a characteristic fingerprint that enables straightforward class assignment and separation.
Black plastic – medium infrared HSI
While polymers can be easily identified by their NIR spectra irrespective of the material’s colour, this is different for polymers containing black pigments. Black polymers are often encountered in technical plastic parts used in housings of electrical devices such as computers, phones or keyboards or in vehicle trims. Also, consumer products are sometimes packaged in black bottles or bowls to create a more valuable or masculine effect. Figure 2 shows commonly encountered black plastic waste that is separated out of the polymer sorting stream as it is not identifiable.
The black pigment used in plastics is often carbon black, pure carbon obtained by combustion of coal in an oxygen deficient atmosphere. This C-C structure is highly absorptive for electromagnetic radiation in a broad wavelength range covering the visible (hence encountered as black by the human eye), but also the NIR region. NIR hyperspectral cameras are basically blind for black materials, as the radiation used to excite the samples (typically halogen light) is nearly completely absorbed. It is thus not possible to differentiate black polymers by their NIR spectra.
Hyperspectral cameras that operate in the medium infrared (MIR, typically the wavelength range between 3-50 μm) are a feasible way to overcome that restriction, as MIR radiation is absorbed to a lower extent by carbon black. The system used for the tests here operates in the 2.9-4.2 μm wavelength range (inno-spec BlackEye). It thus avoids the spectral region around 4.5 μm where absorbance bands of CO2 appear. Even for the low CO2 content of the atmosphere this peak is quite intense – more than ten times higher than the peaks observed for the C-H and R-OH bands of organic materials such as polymers. Thus, even low changes in atmospheric CO2 content will introduce dominant spectral variations that interfere with interpretation by the analysis software and will lead to misclassification.
Figure 3 gives spectra of some black polymers commonly used for black casing materials. While spectral features of e.g. PP differ clearly from those of other polymers and thus allow straightforward classification and separation, other spectra (compare e.g. ABS and PA) are quite close. Powerful chemometric models need to operate in the background of separation systems based on very tiny spectral variances.
In Figure 4, the prediction results for some pieces of black plastic waste are shown. The materials are colour coded according to their pixel-by-pixel material classification. Overall, prediction results are quite good, but the image also illustrates some commonly encountered issues when analysing black plastic materials. Several misclassified areas are visible. Misclassification arises for several reasons. One common problem is specular reflection on glossy material surfaces. While in NIR technology, such problems can be tackled by diffuse sample irradiation, such systems are not standardly available for MIR where irradiation sources are radiant heaters. Apart from that, peak intensity is commonly low so that the overall setup and recording conditions need to be optimized such that the combination of intensity and signal to noise ratio is at an optimum. While this is true for any sensing technique, it is especially critical and especially challenging for MIR hyperspectral imaging.
Another issue is the lower sample penetration depth of MIR radiation as compared to NIR radiation, which results in the fact that surface contaminations such as dirt or coatings will make up a higher proportion of the spectral response and hence will introduce errors. Furthermore, technical plastics often contain several additives, e.g. fire-retardant agents based on halogenated compounds, plasticizers and agents increasing structural rigidity. Many of these compounds give rise to spectral features, which again introduces a factor of uncertainty to analysis results. Overall, black plastic spectra interpretation and analysis is more complex than the respective approach in the NIR region, and more elaborate software tools are necessary for useful results.
Food industry – sorting pears by ripeness and detecting defects
A branch with a huge potential for HSI application is the food industry.7 Baked goods, dairy products, grains, fruits and vegetables all contain multiple parameters that can be monitored by HSI as a means of either sorting or controlling process parameters. As one example of this section, a study on pears is presented here.
The two pears shown in Figure 5 (left) were investigated by NIR HSI. They differ in ripeness and softness. While the pear to the left is overripe, very soft and shows some damaged areas, the pear to the right is hard and good to eat. As a first step, the HSI data of both pears were analysed by a principal component analysis (PCA). The right-hand side image in Figure 5 displays the HSI data of both pears. It is colour coded by the 2nd PCA component with a colour scale from blue to red, with blue corresponding to high and red corresponding to low values. Simple visual inspection of this image illustrates clearly that there are determined spectral differences between both pears that likely makes NIR HSI a suitable method for sorting.
In addition, the PCA scatter plot for the data is given in Figure 6. The data points for the overripe (green) and the hard (blue) pear separate well into two clusters, which is also an indicator that spectral separation is feasible.
The spectral image of the overripe pear is investigated in more detail for more further analysis. In Figure 7, the HSI data of this fruit is displayed, coloured coded by PCA results using several data preprocessing steps and different PCA components for colour coding. Please note that the HSI images are mirrored in comparison to the photo. For all HSI images, it is possible to spot the cut in the lower half of the pear and also the brown dots close to it. This indicates that these damages of the fruit give rise to clear spectral variations in several regions of the utilized wavelength range that can serve to identify such faulty regions.
Glues and coatings – examples from wood industry
In this chapter, a quick overview is given over the potential use of HSI in inspection of glue used in wood processing. This example serves to illustrate a much wider range of HSI uses in processes where coatings, paint and glue are applied in all kinds of industries. The three examples given here are the detection of glue on wood, monitoring the curing of a UV cured glue and monitoring the mixing of a two-component adhesive.
For all manufacturing processes where glue or other coatings are applied automatically, it can be of interest to monitor the application to make sure it is complete and not interrupted due to a blocked nozzle or an empty reservoir. For the example shown in Figure 8, three lines of glue were applied manually to a wooden board. The presence of glue can nicely be monitored in the HSI image, as the spectral data deviates sufficiently from that of uncoated wood. The PCA scatter plot indicates the same result, as the green data points that represent glue spectra are located outside of the cluster formed by the blue data points that represent wood spectra. In addition, the amount of glue can be estimated from the HSI image – to the top, were application of glue was started, the film is thicker as can be deduced by the colour change towards the bottom, where the glue stripes are smeared out and thinner.
Figure 9 shows that it is possible to monitor the curing of a glue cured by UV irradiation. The green spectrum is recorded on the freshly applied glue, while the red spectrum is recorded after a curing time of 30 sec. The distinct peak at ∼1620 nm is a good indicator for the curing grade. While being very prominent for the fresh glue, it has nearly vanished after 30 sec of curing time.
In the 3rd example, a two-component adhesive is investigated. After or prior to application, a proper mixing of both components is mandatory to ensure a durable fusion of materials. For the tests shown here, both components of the two-component adhesive were first applied as two parallel separate lines and NIR HSI data was recorded. Both glue lines were then mixed in consecutive steps by stirring, while further data was recorded. Figure 9 shows the HSI images of the adhesive, colour coded by the 1st PCA component. The first image to the left shows the separate lines of both adhesive components. Due to their very different chemical composition, they give rise to clearly distinguishable spectra and hence to a very different colour (red and blue). Towards the right, HSI images are shown after the consecutive mixing steps. One can follow the increasing degree of mixing by the colour change from blue and red to the light greenish-blue tint in the center of the adhesive spot. It is obvious that after the first stirring step, both pure components are still present to a large extent at the edges of the mixed mass, perceptible by still red and blue areas. These regions of pure components decrease in size after repeated mixing, but still small portions of both are present towards the rim in the image furthest to the right after 10 times stirring.
This is actually not a typical classification example, as analysis is not based on the three different classes “pure component A”, “pure component B” and “mixed component”. Applications like these can be solved by utilizing a partial least squares (PLS) model that is commonly applied in quantification, where the amount of a compound in a mixture is identified. For examples like the one given here, it is advantageous to treat the analysis situation like that of a compound containing different amounts of the same substance. Both pure compounds are given arbitrary values, e.g. 0 and 10, that correspond to the pseudo amount of the component. The degree of mixing can then be derived from the intermediate value that is given as the PLS prediction result.
Colour fidelity – visible HSI
A last example for classification concentrates on the visible range. The HSI system used for this investigation (inno-spec GreenEye) operates in the wavelength range from 400-1000 nm and hence detects visible and short wave NIR radiation. This is especially suitable to investigate colour variations in all kind of materials such as fabrics, printing products, or items for car interior furnishing. In the example given here, the emission of light emitting diodes (LEDs, see Figure 11) is investigated. In the last decade, LEDs have become the most common illumination media used in devices ranging from highly intense spotlights to small indicator lights in electronic devices.
For quite some applications, the sole purpose of the LED device is to emit light that is perceived as white by the human eye, which is tantamount to only low demands on the device’s emission spectrum. For many others, however, it is important that LEDs give rise to an identical colour impression for reproducible and exchangeable devices. HSI comes into play whenever several LEDs implemented into one device are to be tested in a single quality control step.
The human eye is able to perceive colour differences that correspond to a wavelength shift of a few nm only. Hence, RGB cameras that use a combination of pixels that are sensitive to red, green and blue of one certain wavelength only for colour determination are not able to mimic the perception of a human being and to offer a sufficient quality assessment for these narrow wavelength variations.
Figure 12 shows spectra of the emission of green (left) and white (right) LEDs that come from the same production line and should be identical. The peaks resulting from the green LEDs show a very narrow maximum which superimposes quite well for the three investigated devices. The peaks’ full width at half maximum (fwhm) is around 40 nm for all, and only slight deviations can be detected. For the white LEDs, one narrow peak can be seen in the blue range, and a broad structure that covers the rest of the visible wavelength range. Again, the spectra of the three different devices coincide quite well. It is furthermore possible to monitor the intensity of emitted light as a quality criterion. For example, the device giving rise to the green spectrum in the right hand side graph (white LED) shows a 20 % lower intensity as compared to the blue and red spectra and is sorted out.
QUANTIFICATION USING HYPERSPECTRAL IMAGING
In a last example, the potential of HSI in quantification is demonstrated. It is possible to quantify all kinds of parameters in materials or mixtures of compounds. For the example given here, quantification of moisture content is demonstrated. NIR HSI is especially suited for this application, as water strongly absorbs NIR radiation in determined spectral regions. Coarse paper used for cardboard manufacturing was investigated. To generate different moisture contents, the paper was first stored under ambient conditions and then kept in a humidity box for 2 hours. The moisture content was determined as 6 % for the paper kept at ambient, and as 15 % for the paper stored in a humid environment by monitoring the weight increase when compared to the completely dried paper sample.
To the right of Figure 13, example spectra of the paper at both moisture contents (6 and 15 %, red and green, resp.) are given. The most prominent changes take place at around 1450 nm, a region well known to be associated with NIR absorption by water. This peak clearly increases for the paper at higher moisture content. It is straightforward to correlate these spectral changes to the water content of the samples.
A PLS model that is often employed for quantification purposes in chemometrics was compiled from several samples at different moisture contents. Four different contents (4, 6, 12 and 14 %) were used as training data to generate a robust model (not shown). The HSI images of the paper are given to the left of Figure 13. They are colour coded by the water content predicted by the PLS model. The actual values are predicted quite well, and it is also obvious that there are variations across the paper. The rims of the paper are drier than the rest of the paper. This variation towards the rim is also commonly encountered in the raw material used in cardboard manufacturing. These rims of deviating moisture content lead to distortions in the final product. Cardboard industry thus has a lot to gain from the application of quantitative NIR HSI in raw material inspection.
Multiple application examples were given where hyperspectral imaging (HSI) in the visible, near or medium infrared (NIR, MIR) wavelength range can be used in process analysis. The systems used for these studies were the HSI cameras of inno-spec, namely the GreenEye for the visible and short wave NIR (400-1000 nm), the RedEye for the NIR (950-1700 nm) and the BlackEye for the MIR (2900-4200 nm) range. Different classification and quantification examples were outlined.
The well-known example of plastic sorting with NIR is included here as the most prominent application. Clear differences in the different polymers’ NIR spectra enable straightforward separation, as long as the material is not black. For black plastic, a similar separation can be enabled by MIR, as MIR radiation is absorbed to a lesser extent by the black pigment (mostly carbon black). Separation is not as straightforward as with NIR, and it is very important to select a suitable wavelength range with no disturbances from e.g. atmospheric gas spectra and to increase signal intensity and irradiation homogeneity.
The food sector also has multiple potential fields for HSI application. Pears are shown as one example where ripeness and softness can be determined. Imperfections such as cuts, breaks or spots can also be detected.
Different examples to monitor either the application of glue, the curing of glue or the mixture of a two-component adhesive outline potential applications in wood or construction materials processing.
For the visible range, very accurate colour determination is possible with HSI systems that RGB cameras are not capable of. Emission spectra of LEDs were investigated for quality control to ensure identical emission characteristics for one batch of devices.