Hyperspectral imaging using RGB color for foodborne pathogen detection

Abstract. This paper reports the development of a spectral reconstruction technique for predicting hyperspectral images from RGB color images and classifying food-borne pathogens in agar plates using reconstructed hyperspectral images. The six representative non-O157 Shiga-toxin producing Escherichia coli (STEC) serogroups (O26, O45, O103, O111, O121, and O145) grown on Rainbow agar plates were used for the study. A line-scan pushbroom hyperspectral imaging spectrometer was used to scan full reflectance spectra of pure non-O157 STEC cultures in the visible and near-infrared spectral range from 400 to 1000 nm. RGB color images were generated by simulation from hyperspectral images. Polynomial multivariate least-squares regression analysis was used to reconstruct hyperspectral images from RGB color images. The mean R-squared value for hyperspectral image reconstruction was ∼0.98 in the spectral range between 400 and 700 nm for linear, quadratic, and cubic polynomial regression models. The accuracy of the hyperspectral image classification algorithm based on k-nearest neighbors algorithm of principal component scores was validated to be 92% with the test set (99% with the original hyperspectral images). The results of the study suggested that color-based hyperspectral imaging would be feasible without much loss of prediction accuracy compared to true hyperspectral imaging.


Introduction
Detection and identification of foodborne pathogens are increasingly important for development of intervention and verification strategies for the food industry and regulatory agencies. Traditional culture-based direct plating methods are still the gold standard for presumptive-positive pathogen screening in many microbiology laboratories, where agar media are routinely used for isolation, enumeration, and detection of pathogenic bacteria. In practice, highly skilled technicians visually screen and manually select presumptivepositive colonies by trial and error for microscopic, biochemical, serological, and molecular confirmation tests whose results may or may not be obtained rapidly. Therefore, the culture methods are labor intensive and prone to human subjective errors. Another challenge with direct plating is that competitive microflora often grow together with target microorganisms on agar media and can appear morphologically similar.
Hyperspectral image classification algorithms in the visible and near-infrared (VNIR) spectral range from 400 to 1000 nm have been previously developed for automated screening of pathogen colonies on agar plates, which included Campylobacter, Salmonella, and Shiga toxinproducing Escherichia coli (STEC). [1][2][3][4][5] The key idea of using hyperspectral imaging was to find spectral and spatial features unique to the bacterial colonies on agar and utilize the spectral and/or spatial features for detection and classification so that relatively simple classification algorithms, such as k-nearest neighbor (kNN) and quadratic discriminant analysis, could be applied. Although hyperspectral imaging using full spectra demonstrated the good sensitivity and specificity to detect pathogen colonies and identify their types, the high cost of a hyperspectral imaging system hinders its routine use as a screening tool in microbiology laboratories. RGB color (or multispectral) imaging techniques can be a cost-effective solution, although a commercial multispectral imaging solution is still much more expensive than color imaging. In addition, either a color-based or multispectral imaging solution for pathogen detection, such as non-O157 STECs, requires the development of a new classification algorithm. Therefore, an idea was to use the already developed hyperspectral image classification algorithm(s) while using color (or multispectral) imaging.
The use of multispectral imaging to recover reflectance spectra has been extensively studied in the color science field typically with color charts or targets with known spectra. 6 There are many techniques to recover reflectance spectra from multispectral (typically color) information. 6,7 Some of the hyperspectral estimation techniques to recover reflectance spectra include Wiener estimation, multiple regression analysis, Maloney-Wandell method, Imai-Berns method, and Shi-Healey method. [6][7][8][9][10][11][12] Hyperspectral images have also been used to develop a hyperspectral estimation method from color and/or multispectral images for spectral estimation of paint. 8 A different group of techniques for hyperspectral estimation is based on sparse signal representation and sampling, such as compressive sensing. Sparse representation methods were developed to increase the spatial resolution of hyperspectral images with spectral unmixing and L1 minimization. 13,14 Compressive sensing techniques were applied to reconstruct hyperspectral images from undersampled fluorescent microscopic images in biomedical imaging 15 and from a few noisy multispectral images in remote sensing. 16 Active spectral illumination has been studied to quantify complex spectral responses of tissue oxygenation with sparse sampled multispectral images. 17 But the use of color to recover hyperspectral images is still an underexplored research area, especially to the hyperspectral imaging researchers in the field of agriculture and food science and engineering.
In this paper, polynomial multivariate regression analysis was used to recover reflectance spectra from RGB color images and to reconstruct full VNIR hyperspectral images that would be applied to a hyperspectral image classification algorithm specifically developed for detection of non-O157 STEC colonies on agar plates. Thus, the overall objective of the study was to investigate the potential of using color to recover reflectance spectra in the form of hyperspectral images and to evaluate the performance of the hyperspectral image classification algorithm with the recovered hyperspectral images.

Pathogenic Bacteria
The pathogenic bacterial cultures were obtained from a culture collection at the Eastern Laboratory of U.S. Department of Agriculture-Food Safety and Inspection Service (FSIS) according to the FSIS protocol. 18 A total of six non-O157 STEC strains were chosen from each of the big six O-serogroups (O26, O45, O103, O111, O121, and O145). The details on how to prepare the non-O157 STEC bacteria are provided in our previous publications. [3][4][5] Three replicated experiments were carried out to grow the bacterial colonies on Rainbow agar plates. Each experiment generated two sets of six agar plates inoculated with two cell concentrations of the pure cultures, where only one type of bacteria was grown in an agar plate. Serial dilutions of each cell suspension were prepared in ∼50 and 100 colony forming units (CFU) (50 and 100 μL aliquots of 10 3 CFU mL −1 dilutions) that were inoculated onto Rainbow agar plates by a spread plating technique. All inoculated plates were incubated at 37°C for 24 h. A total of 36 agar plates (2 dilutions × 3 replicates × 6 serogroups) were prepared.

Hyperspectral Images
Hyperspectral image acquisition was done with a pushbroom line-scan hyperspectral imaging system (Themis Vision Systems, Richmond, Virginia) in the VNIR spectral range from 400 to 1000 nm. The imaging system, shown in Fig. 1, consisted of a 12-bit CCD camera with 1376 × 1040 pixels (SensiCamQE, The Cooke Corporation, Romulus, Michigan), an imaging spectrograph (ImSpector V10E with 30 μm slit, Specim-Spectral Imaging Ltd., Oulu, Finland), a C-mount objective lens (APO-Xenoplan 1.8∕35 mm, Schneider Optics), a motion control (Newark, California), a custom sampleholder, and a computer. The front objective lens was moved to scan one spatial line for one scan such that the dispersed full spectra along the line were recorded on the CCD detector. Further details on the system are provided in our previous publications. [3][4][5] The average wavelength interval was 1.27 nm. The dimensions of a raw hyperspectral data cube after 2 (spatial) by 2 (spectral) hardware binning on camera were 688 ðWÞ × 500 ðHÞ × 520 ðwavelengthsÞ from 380 to 1000 nm. Measured reflectance values were calibrated (i.e., normalized) to relative reflectance R with a 40% reflectance Spectralon® target (13 × 13 cm, SRT-75-050, Labsphere, North Sutton, New Hampshire), as described in Refs. 7 and 8. The spectral dimension of each image was reduced to 473 spectral bands ranging from 400 to 1000 nm by removing extreme wavelength bands. Thus, the resulting dimension of each hyperspectral data cube became 688 ðWÞ×500ðHÞ×473ðwavelengthsÞ. Also, spectral noise was reduced by a Savitzky-Golay smoothing filter (window size: 25; order of moment: 4) at each pixel position. After that, the calibrated hyperspectral images (relative reflectance) were stitched together into a single image mosaic. The images from the same dilution were added to each column of the mosaic from left (less cells) to right (more cells). Figure 2 shows two colorcomposite image mosaics (enhanced by 2% linear stretch for display purpose only) obtained from 36 calibrated hyperspectral reflectance images divided into training and test sets with 18 images for each set. The dimensions of each mosaic were 2064 × 3000 × 473 pixels with floating-point values. The file size of each mosaic was ∼11.7 GB. The 18 hyperspectral images in each mosaic were arranged in six rows and three columns, where rows consisted of images with pure cultures of O26, O45, O103, O111, O121, and O145 from top to bottom, and columns referred to replicates and different cell dilutions.

RGB Color Images
The color images (called camera responses) were obtained from the calibrated hyperspectral images by simulation. Mathematically, the camera responses c i at each color channel and at each image pixel location ðx; yÞ were calculated by the following: where rðx; y; λÞ and β i ðλÞ are a reflectance spectrum at ðx; yÞ and the spectral sensitivity of the camera's i'th channel, respectively. A total of 473 wavelengths, λ's, were defined at nonuniform intervals from 400 to 1000 nm, whose average interval was 1.27 as previously mentioned. The spectral sensitivities were resampled at the same wavelengths as the hyperspectral data. Three spectral sensitivity functions were compared in order to determine the best method to extract color images from hyperspectral images. The first method, assuming the synthetic unit-impulse functions for a camera's spectral sensitivities, was to use three spectral band images obtained at three discrete wavelengths, 455, 535, and 610 nm for blue, green, and red color channels, respectively. The second method, assuming the synthetic rectangular functions for the sensitivities, was to use three spectral ranges of nonoverlapping wavelengths at 435 to 485 (blue), 505 to 575 (green), and 595 to 660 (red) nm. Average reflectance values computed in each wavelength range were assigned to the corresponding RGB values. The third method, which was more practical, was to use the spectral sensitivities of a CCD sensor (ICX445AQ, Sony Corp.) used in a commercial RGB color camera (Prosilica GC1290C, Allied Vision Technologies GmbH, Stadtroda, Germany), whose spectral sensitivities

Region of Interest
Both hyperspectral and RGB images of agar plates were masked by binary plate masks in Fig. 4 in order to suppress the unwanted background outside of each agar plate. The plate masks were created manually by a drawing tool in the ENVI software (Exelis Visual Information Solutions, Boulder, Colorado).
The ground-truth colony regions of interest (ROIs) representing the identity (i.e., STEC serogroup) of each colony were obtained by an automatic colony segmentation algorithm utilizing the characteristics of colony morphology and response to lights. 19 As shown in Fig. 5, the colony segmentation images were stitched into two mosaics for training and testing in the same way as the hyperspectral image mosaics. The choice of the ROIs was studied to determine whether there is a sampling bias caused by different ROI (i.e., sampling).
Seven different sample sets derived from the original colony ROIs and agar background regions in the training set were compared to find the best sampling size for hyperspectral data reconstruction from RGB color data. Figure 6 shows the examples of the defined ROI sets. Each sample set consisted of (1) all pixels in each plate image (excluding the outside of agar plate) defined by a binary plate mask (MASK), (2) randomly selected 5% of total pixels in each plate mask (MASK5%), (3) all pixels in each colony ROI plus randomly selected agar pixels (COLONY), (4) four neighboring pixels from the center position of each colony ROI plus the center position pixel (a total of five colony pixels) and randomly selected agar pixels (COLONY5p), (5) one mean spectrum from each colony ROI plus randomly selected agar pixels (COLONYa), and finally (6) all pixels in morphologically dilated (i.e., spatially extended) colony ROIs and randomly selected agar pixels (COLONYd). The number of randomly selected agar pixels from (3) to (6) was ∼30% of the total number of pixels in each ROI set, which was heuristically determined.

Reconstruction of Hyperspectral Data
Given the reflectance spectra and RGB values of n observations, which were obtained from the n pixels on the previously mentioned ROIs, the hyperspectral data reconstruction problem was formulated as a system of linear equations as follows:  (2) where R, C, and T are a hyperspectral reflectance data matrix of n row vectors each with 473 reflectance values, a camera response matrix of n row vectors each with an RGB color 3-tuple, and the transform matrix (unknown) to be estimated, respectively. There are several solutions available in the literature. [6][7][8][9][10][11][12] In this paper, polynomial multivariate leastsquares regression (PMLR) was used to estimate the transform matrix T and thus the hyperspectral data. The polynomial regression model assumes that there is a nonlinear relationship between the predictors (color image) and the dependent variables (hyperspectral image). The degree of a polynomial is an unknown control parameter that determines the behavior of the independent variables (predictors) to fit to the hyperspectral data. The optimal degree was determined in a heuristic way, for which the regression performance was compared. A total of five PMLR models from firstto fifth-degree polynomials was compared to find the best estimation method. The camera response matrix C in Eq. (2) was expanded to include higher-order polynomial terms as follows: E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 3 ; 6 3 ; 1 where C e ¼ ½c 1 ; : : : ; c n is an expanded camera response matrix of n row vectors c i , i ¼ 1; 2; : : : ; n. For example, a row vector c i consists of ½c 1 ; c 2 ; c 3 ; c 1 × c 1 ; c 2 × c 2 ; c 3 × c 3 ; c 1 × c 2 ; c 1 × c 3 ; c 2 × c 3 ; : : : ; higher-order terms. The Moore-Penrose pseudoinverse method was used to solve Eq. (3) and to obtain the estimated transform matrixT as follows: E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 4 ; 3 2 6 ; 4 2 9T Then, given the RGB color values, the hyperspectral data were estimated by the transform matrix by plugging into Eq. (3). The goodness of fit (GOF) of hyperspectral reconstruction was measured by the coefficient of determination (R 2 ) and the root mean square deviation (RMSD, also called root mean square error), which were defined as follows: E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 5 ; 3 2 6 ; 3 2 8 where r k andr k are the measured and estimated reflectance values at the k'th wavelength band, respectively. The number of wavelength points was N λ (N ¼ 473). The R 2 value is a statistical measure of how well the predicted data agree with the measured data. An R 2 of 1 refers to a perfect fit of a regression line to the measured data. The RMSD represents the sample standard deviation of the differences between predicted values and measured values.

Hyperspectral Image Classification Algorithm
A hyperspectral image classification algorithm for non-O157 STEC colony classification was based on chemometric preprocessing of spectral data, including the transformation of measured reflectance spectra to absorbance spectra, standard normal variate and detrending for correction of light scattering, first derivative, and moving average smoothing. [3][4][5] Then, the preprocessed spectra were transformed by principal component analysis. The classification for the six STEC classes was performed by the kNN algorithm (with k ¼ 3) of scores in the principal component subspace spanned by six principal components. [3][4][5] Note that the reflectance spectra at each pixel were transformed to absorbance log 10 ð1∕RÞ for classification, whereas the prediction of hyperspectral data from color data was done with reflectance spectra. When extracting spectral data from pixel locations defined by the various ROIs, the data were unfolded into an n × 473 spectral data matrix X whose values were associated with n observations (number of samples in pixels) in rows and 473 independent variables (wavelengths) in columns. A response vector of class labels from 1 to 6 was also created. The training set for classification consisted of 916 ROIs (i.e., colonies) with 63,136 pixels (i.e., observations or samples), whereas the test set for validation of classification models consisted of 1250 colonies with 77,473 pixels. The classification model was trained with both original and reconstructed hyperspectral data and validated with the training and test sets. All spectral data were treated in the same way as the aforementioned data preprocessing techniques and unfolded into a data matrix. The final decision-making rule was applied at colony level by the winner-take-all strategy (simple majority voting) of prediction results at pixel level. Prediction results of each sample in the training or test set were projected back onto the corresponding image domain to create a hyperspectral image.

Spectral Sensitivity Functions
The RGB color images obtained by simulation of a camera sensitivity function (Sony ICX445AQ CCD) are shown in Fig. 7. The other RGB images are not shown because they were visually similar to each other. The displayed color images in Fig. 7 were enhanced by 2% linear stretch for display purpose.
The GOF of the regression models in relation to spectral sensitivity functions is summarized in Table 1.
The average R 2 and RMSD values in Table 1 were obtained from the validation of the regression models with all samples in the test set. The statistical significance of R 2 and RMSD values varying among different spectral sensitivity functions was tested by the analysis of variance (ANOVA). The p values resulting from the ANOVA test were 0.33 (p > 0.05) and 0.34 (p > 0.05) for R 2 and RMSD, respectively. Hence, the selection of a particular spectral sensitivity function would not be statistically significant. Nonetheless, when the three different spectral sensitivity functions were compared, the spectral sensitivity functions emulating them of a commercial color camera with the broad range of wavelengths produced the best fit spectra, especially over the spectral range between 400 and 700 nm. Therefore, the spectral sensitivity functions of a real camera covering the broad spectral ranges were used for the development of the regression models later on.
In this study, there was no need to solve an image registration problem to find the corresponding points or regions between the RGB and hyperspectral images, because RGB color values used to compute the transform matrix were created by the simulation of three different camera sensitivity functions from the defined ROI sample locations on hyperspectral images. However, in real-life applications of hyperspectral estimation from color, it would be important to calibrate a color camera with measured camera responses (color values) and their corresponding spectra measured by a hyperspectral image sensor, which will be a topic for our future research.

Sample Sizes
The size of samples (i.e., the number of pixels) of each ROI set in the training set is summarized in the first row of Table 2. The sample sizes selected from the training set with the six different ROI sets were evaluated and compared on the basis of the regression performance with the test set. The ANOVA test of the GOF values (R 2 and RMSD), similar to the comparison of the spectral sensitivity functions, showed that the differences in sample size were statistically significant (p < 0.001). Therefore, it was important to find the best sample size for the regression models. The GOFs of each different sample size are summarized in Table 2. The average R 2 and RMSD values in Table 2 were computed with the samples (5% random selection, ROI set MASK5%) in the test set and for all regression models.
The average R 2 values tended to increase as sample size increased. The 5% random selection of pixels in the training set for training the regression models showed the best prediction (or called regression) performance, closely followed by all pixels in plate masks. The RMSD values <1 could mean <1% difference in percent reflectance. Note that this small difference in reflectance did not mean uniform performance regardless of regression model, sample location, material type (serogroup), and wavelength.

Hyperspectral Image Reconstruction
The prediction performance of the five regression models was evaluated by the fitting accuracy of the reconstructed hyperspectral data to the measured hyperspectral data in terms of GOFs. Figure 8 shows the average R 2 and RMSD values of the five regression models, where each model was trained with all samples (ROI set MASK, N ¼ 3;058;576) in the training set and tested with all samples (ROI set MASK, N ¼ 3;121;509) in the test set. Overall, linear, quadratic, and cubic polynomial regression models showed better performance than the higher-order polynomial regression models. Figure 8(b) shows that the spectral data in the spectral range below 700 nm would be better predicted by all five regression models than the spectral range above 700 nm, although the quintic (p ¼ 5) polynomial model was the worst. The prediction performance of the regression models measured by α, not shown here, was also similar to the performance measured by R 2 and RMSD. The cubic (p ¼ 3) polynomial regression model showed the least prediction errors in terms of the GOFs when measured with all samples in the test set.
The prediction performance of the regression models was also evaluated in relation to colony identities (STEC serogroups in our case). Figure 9 shows the mean reflectance spectra obtained from the colony ROIs on the original and the regressed hyperspectral images, when the regression models were trained with all pixels on the plate masks (ROI set MASK, N ¼ 3;058;576) in the training set and tested with all colony ROIs (1250 colonies with 77,473 pixels) in the test set.
The analysis of the mean reflectance spectra showed overall better fit of all regression models in the wavelengths below 700 nm. Large variations in mean reflectance were observed at wavelengths longer than 700 nm. The cause of the large variations in the longer wavelength region was not  studied in this paper. Thus, we only speculated that first, the simulated RGB values did not contain the spectral information in the wavelength and/or second, the variation in the original data was too large to model with the regression models. Nonetheless, the first three-order polynomial regression models were better than the higher-order polynomial models. The linear regression model showed the best prediction performance for O26 and O103, whereas the quadratic (p ¼ 2) polynomial regression model was the best (or close to the best) for O45, O111, and O121, and the cubic (p ¼ 3) model was the best for O121 and O145. Figure 10 shows the mean reflectance spectra obtained from the colony ROIs on the original and the regressed hyperspectral images, when the regression models were    trained with samples corresponding to the colony ROIs and background agar (ROI set COLONY) in the training set and tested with all colony ROIs (1250 colonies with 77,473 pixels) in the test set. As shown in Fig. 10, when the regression models were trained with the ROI set COLONY (colony ROIs plus random agar pixels), the overall prediction performance was greatly improved over all wavelengths. Especially, the large variation at the longer wavelengths seen in Fig. 9 was reduced. This experiment confirmed that the camera sensitivity functions were indeed not statistically significant, but the difference in same size about how to sample the data in the training set was statistically significant.
The linear regression model showed the best (or close to the best) prediction performance in terms of spectral analysis for all serogroups except O111. Although sampling only with the ROI set COLONY showed better regression performance than sampling with the entire population in the training set, the performance of the developed classification model may or may not be directly correlated with these findings because the classification models including image segmentation and chemometric preprocessing were optimized with the original hyperspectral images. Figure 11(a) shows RGB color-composite images obtained from the reconstructed hyperspectral images, and Fig. 11(b) shows pseudo-color images derived from the normalized sum of differences between the measured and predicted hyperspectral images over all wavelengths, both using the linear regression model trained with all pixels in the plate masks. The normalized sums of reflectance differences at each image pixel were mapped to a color according to a scale between 0 (small error: blue at the bottom of the color bar) and 1 (large error: red at the top of the color bar). The regression errors were more prominent at the colonies. The centers of the colonies tended to produce more regression model errors. From Fig. 11(b), it was implied that the spectra of the O145 colonies were the most difficult to predict using the regression model.

STEC Classification Results
The performance of the pathogen classification algorithm was measured in terms of overall classification accuracy. The performance of the classification algorithm depended greatly on the characteristics of the samples to train, which was similar to the case of the regression models mentioned in this paper. Due to this issue, three different ROI sets, including MASK (all pixels on plate masks), COLONYa (mean spectra plus random agar pixels), and COLONY5p (five pixels near the colony center), were compared for the performance evaluation of the classifier. The linear regression model was used for the reconstruction of hyperspectral images and trained with the ROI set MASK (all samples) in the training set. The linear regression model was applied to predict (i.e., reconstruct) hyperspectral images from the color images in both training and test sets. The classification algorithm was retrained with the reconstructed hyperspectral images in the training set. For validation, the classification algorithm was applied to all pixels of the ROI set MASK in both original and reconstructed hyperspectral images and validated on colony ROIs for performance assessment. Table 3 shows the overall classification accuracy of the classification algorithm trained with three different sets of samples in the training dataset.
The training accuracy varied from 66 to 99%, where the best performance was achieved by the classification model trained with all pixels in the colony ROIs (99%), whereas the best test performance was achieved by the model trained with mean colony spectra (92.3%). When compared with the results from the original hyperspectral images, the classification accuracy based on the reconstructed hyperspectral images varied more greatly than the case of the original images. One reason is because the classification algorithm was not optimized with the reconstructed hyperspectral images. It is known that there are very subtle spectral differences among different STEC classes. Therefore, any small error in reconstruction may have caused unexpected large errors and variations in classification results. Nonetheless, the reason for this large variation in performance needs to be studied in the future. Also, the classification algorithm developed with the original hyperspectral images may not be optimal for the reconstructed hyperspectral images. Therefore, a new hyperspectral image classification algorithm for STEC detection may need to be developed in the context of different regression models in the future.

Conclusions
This study showed a potential of the use of RGB color to reconstruct hyperspectral data and its application to classification of non-O157 STEC colonies on agar media using a hyperspectral image classification algorithm. Contrary to the most previous work using color charts or targets with known spectra, the current work used the hyperspectral images of the actual targets (microbial colonies on agar media) to obtain the spectra of the real targets of interest for reconstructing the spectra from color images by polynomial multivariate least-squares regression. A limitation of the current study was the use of simulated color images for reconstructing hyperspectral images by the regression analysis. Polynomial multivariate least-squares regression models using up to the fifth-order polynomial were evaluated in relation to the camera's spectral sensitivity functions and sample sizes. The regression performance measured by GOF measures (coefficient of determination and RMSD) was similar among the linear, quadratic, and cubic polynomial regression models. The effect of different spectral sensitivities on the regression performance was not statistically significant, but that of different sample sizes was statistically significant. The spectra in the wavelengths shorter than 700 nm were better predicted by the regression models, but the spectra at the wavelengths longer than 700 nm were not well predicted. The analysis of mean reflectance spectra revealed the potential of linear, quadratic, and cubic regression models. The classification algorithm trained with the reconstructed hyperspectral images achieved up to 92% classification accuracy, whereas the same algorithm with the original hyperspectral images achieved 99% classification accuracy. The classification results indicated that the GOF measures were not directly matched with the classification accuracy, which needs to be further investigated later. In conclusion, although there were mixing results and still unanswered questions, there is still room for improvement of hyperspectral image regression and classification accuracy for non-O157 STEC detection using a color camera.