2.1.

Thermal Dataset

A total of 252 volunteers were enrolled in an Institutional Review Board approved prospective study aimed at testing the ability of thermal videos to detect SARS-CoV-2. The study protocol was approved by the Ethics and Research Committees at Escuela de Medicina y Ciencias de la Salud, Tecnológico de Monterrey. The participants’ ages ranged from 18 to 75 years. The number of participants per gender with their height, weight, and PCR results are summarized in Table 1. The study recorded a set of measurements from participants regarding PCR-results, demographics, vital signs, participant activities, medications, respiratory symptoms, and a thermal video session in which the volunteers performed, in a secluded cabin, simple breath-holds during the video capturing in four different positions—front, back, left, and right—as seen in Fig. 1. The room temperature was maintained in the vicinity of 25°C, and among the 55 participants registered as having fever, the highest body temperature registered was 37.7°C; therefore, it was not expected that the presence of droplets of sweat could exert any influence on the recording. Thermal images were recorded in video mode, mostly at five frames per second, using a Digital Thermal Imaging Camera TI-128 from Omega Engineering Inc. (800 Connecticut Ave. Suite 5N01, Norwalk, CT 06854). The camera was connected via a USB to a laptop computer running a Windows^® operative system from Microsoft^® as suggested by the provider instructions. The acquisition software Omega TI Analyzer version 4.1.8.6875 was used. Drivers and acquisition software were obtained and installed following provider instructions. Table 2 shows the patient demographics and vital signs at the time of video capture; significant differences were assessed using a $t$-test. Table 3 shows the patient symptoms and medical conditions; significant differences were calculated with a comparison of frequencies using a chi-square test. Regarding the medical conditions, 11 subjects presented diabetes and 15 presented arterial hypertension. In cases of diabetes and arterial hypertension, it is expected that the entities may have disordered sleep breathing.^28,29 However, in the case of hypertension, only the pulmonary arterial hypertension was previously correlated to this disorder; therefore, we do not expect breathing anomalies during the video capture due to this condition.

Table 1

Number of enrolled subjects per sex and PCR results. A total of 251 subjects (after removing the sole participant with PCR results that were not reported) were considered for further analyses.

Fig. 1

Thermal imaging session. A single continuous take of thermal video was done to capture 10 s of patients breathing and 10 s holding their breath.

Table 2

Characteristics of the enrolled subjects stratified by sex and PCR result. Significant differences between positive and negative results are highlighted.

Table 3

Symptoms and conditions of the enrolled subjects. Significant differences between positive and negative results are highlighted.

2.2.

Video Preprocessing

The video acquisition protocol included breathing in four positions, as shown in Fig. 1. Each thermal video was visually inspected to determine all of the frames for these body positions. Each infrared spectrograph video segment was then converted into a standardized black-and-white MPEG-4 video with gray levels that were standardized between 62°F and 102°F. Therefore, four MPEG-4 were generated and labeled with front, left, back, and right corresponding to each body position.

After that, all four MPEG-4 video clips were standardized and corrected for sampling differences, temperature-calibration variations, blur, and large motions. Then, the Lucas Kanade optical flow method was used to estimate the frame by frame skin motion for each one of the videos.³⁰ Motion correction based on rigid transformations was used to integrate the first 25 frames of each video clip and generate an HR thermal image.³¹ After optical flow and super resolution, a set of six images were generated. The first image was the HR thermal image. The second image consisted of the pixel-by-pixel temperature variance observed during the breath-hold. The third and fourth images consisted of the average optical flow in the horizontal and vertical directions. The last two images were created by estimating the frame-wise variance of the optical flow. In summary, the derived image set consisted of two temperature-driven images and four motion-derived (optical flow) images. Figure 2 shows the video processing steps used for this research work.

Fig. 2

(a) Processing: a thermal video is oriented, calibrated, framerate standardized, and converted into a set of six images. (b) The first 25 frames of each video are integrated into an HR image. (c) Segmentation process: eight labeled HR images are used to vote for the definitions of the human body regions of interest (ROIs). (d) ROI quantitation: each ROI is used to mask a specific body area to be quantitated.

2.3.

Image Segmentation and Body Regions of Interest Definitions

Image segmentation and regions of interest (ROI) definition were done using two different approaches. The first approach of image segmenting based on the Otsu method identified the entire upper body and removed background information.³² The first segmentation process was applied to each of the six images separately, and all segmented images were reviewed and manually refined to avoid loss of information in missed segmented body regions due to the presence of hair, necklaces, masks, and underwear.

The second approach for image analysis was the identification of the ROI. The identification of the head, chest, and back was done using an atlas-based approach.^33,34 HR images from eight subjects (four males and four females) were manually labeled with head, chest, neck, and legs. Figure 2 shows an example of the labeled images used for the atlas-based segmentation approach. Once all subjects were labeled, an affine registration method was used to match each one of the atlas subjects with the HR thermal image for each subject. Once we have the eight segmentations per subject, we use the majority-vote approach to get the final label of each pixel. The majority-vote segmentation was then refined by a morphological closing operation that corrected segmentation errors in the neck and breast regions. All video preprocessing and image segmentation was done using MATLAB. (2010). version 7.10.0 (R2010a). (The MathWorks Inc., Natick, Massachusetts)

2.4.

Image Feature Extraction

Heat and motion patterns were quantified by extracting image features from the global segmentation and the extracted ROIs. From the global segmentation-based approach, we extracted textural descriptors and signal moments from all pixels inside the segmentation mask and for each one of the six images at three different image resolutions. The textural features quantified the spatial signal heterogeneity using the gray level co-occurrence matrix (GLCM) and local binary patterns (LBP). Details about the mathematical definitions of textural features can be found in previous literature.^35,36 Descriptive statistics of the signal distribution inside the segmentation were also collected using the moment features. Finally, fractal dimension formulation was used to characterize the association of the scale dependent features to changes in resolution

Table 4

Summary of the features extracted from each of the six summary images [Fig. 2(a)] after segmentation.

2.5.

Data Conditioning, Features Transformation, and Demographic Adjustments

Figures 3(a) and 3(b) show the set of features used for each ROI and for the global segmentation, respectively. Regarding Fig. 2(a), the 1990 features extracted per subject were obtained considering 82 features (14 signal moments, two shape moments, and 66 signal texture) from each of the six images and each of the four views (chest, face, back, and side) and the 22 symptoms and clinical data ($82\times 6\times 4+22=1990$). Missing data were imputed using the nearest neighborhood approach.³⁷ Regarding data conditioning, the power transformation, Yeo-Johnson transformation, or pseudo log transformation was applied to highly skewed features. To mitigate the effect of age, gender, height, and size of ROI, all image features of the training set were adjusted using a robust fitting model from the FRESA.CAD R package version 3.3.0. For adjustment purposes, asymptomatic or subjects with only one symptom that were not taking any temperature modifying medication (non-steroidal anti-inflammatory drugs) were considered to be reference control. Once adjusted, all features were decorrelated selecting a set of basis vectors using the feature decorrelation function of FRESA.CAD.³⁸ Finally, all transformed, adjusted, and decorrelated features were $z$ normalized.

Fig. 3

Summary of the features extracted on (a) each ROI per view for a total of 1990 features for a specific subject and (b) the global segmentation for a total of 132 when using all views together.

2.6.

Machine Learning

To ascertain the role of thermal imaging and ML in COVID-19 diagnosis, we explored the data set using two different approaches. First, we explored a small set of ML methods using the whole-body segmentation to research the impact of ML methods in COVID-19 assessment using demographics, vital signs, symptoms, and thermal image data. Second, we tested the hypothesis that different parts of the body have different powers in detecting COVID-19 from thermal videos with and without the aid of vital signs and symptoms.

For the first data exploration, we split the data into 70% training and 30% testing sets, and we considered five classifiers: support vector machine (SVM), AdaBoost, Random Forest, Naïve Bayes, and $K$-nearest neighbors (KNN).^39–43 200 repetitions of cross-validation were applied.

Image segmentation, feature extraction, and transformation processes were done using Python programming language (2019) version 3.7.6 (Python Software Foundation⁴⁴). Adjustment and classification were executed using FRESA.CAD in R Core Team (2020) version 4.0.2. (R: A language and environment for statistical computing, R Foundation for Statistical Computing, Vienna, Austria⁴⁵).

For the second set of experiments, the subjects were split as in Fig. 4. We selected Naïve Bayes as the ML classifiers, and we varied the ROI used for the detection of COVID-19, as well as the set of features to be selected for COVID-19 prediction under two scenarios. The first scenario used all subjects (symptomatic and asymptomatic), for the second scenario, only symptomatic subjects were used for training and validation. The analyzed ROIs were chest, face, back, and the combination of left and right views. For this set of experiments 500 leave two out cross-validations without class balance were used to document the performance of COVID-19 detection.

Fig. 4

Splitting of the subjects for conditioning and normalization before fighting the ML model.

2.7.

Statistical Analysis

All of the test results of the ML experiments were aggregated for consensus prediction. The consensus predictions (median of all test-results per subject) were analyzed for balanced error rate (BER):

3.1.

Global Segmentation Experiments

3.1.3.

Female and male classification

As in the last classification experiments, here we used the 132 thermal features with a sample of 251 (Table 1) subjects stratified by sex, with the not reported subject being removed, and ending with a sample of 93 for the female classification and 158 for the male classification. We used AdaBoost to compare results among female and male patients of individual views. Results from the front view are reported here, for outcomes from different views did not present significative differences. Although both performances are almost similar, male patients tended to be classified with slightly better accuracy than females (Fig. 5).

Fig. 5

ROC for COVID-19 classification, comparison among female and male patients.

3.1.4.

Classification including vital signs and symptoms

Vital signs, such as temperature (°C), systolic, diastolic, heart rate, and oxygen saturation; symptoms, such as sore throat, diarrhea, vomit, anosmia, ageusia, shivering, headache, myalgia, and arthralgia; and the total number of symptoms were also included for classification. Similar to the previous experiment, the results reported here were obtained by taking features extracted from the front view. The resulting areas under the curve are shown in Fig. 6. We show results when including image features in three different cases: coupled with symptoms—142 features (132 thermal features and 10 symptoms), with vital signs—137 features (132 thermal features and 5 vital signs), and with symptoms and vital signs together—147 features (132 thermal features, five vital signs, and 10 symptoms). We also present baselines, without thermal features, in the same three cases for comparison.

Fig. 6

ROC for COVID-19 classification using thermal features, vital signs, and symptoms.

3.2.

Regions of Interest Experiments

The segmentation results of the ROI-based experiments were visually inspected to avoid including in the analysis segmentation results that failed to correctly identify the correct body component or that had other issues affecting the quality of the thermal data, such as beard, face mask, and large motion artifacts. The quality inspection indicated that one subject from the chest ROI had to be excluded. A total of 18 subjects were removed from the face ROI, 15 from the back ROI, and five from the left-right ROI analysis.

The ML results of the second set of experiments are shown in Figs. 7 and 8 for the chest ROI. Figure 7(a) shows the ROC analysis for the combinations of features set: symptoms, vital-signs, and thermal features. Figure 7(b) shows the boxplot of the testing results distribution for each subject and sorted by the predicted probability of COVID-19. Figure 8 shows a detailed comparison of the ML models for the different feature combinations and data stratification: all, males, and females.

Fig. 7

(a) ROC of the different feature combinations. (b) Predictions of each one of the 500 repetitions per subject for the all-feature model.

Fig. 8

Comparison of key metrics across all models for the different feature combinations and stratified by sex.

Figure 9(a) shows the ROC analysis of the symptomatic only subjects from the chest ROI; Fig. 9(b) shows the box plots of the 500 test results per subject, ordered by the predicted COVID-19 probability. Figure 10(a) shows the heatmap of the top features associated with COVID-19 when using all subjects. Figure 10(b) shows the heatmap of the features associated with COVID-19 when using only symptomatic subjects. Supplementary material in Appendix A shows the detailed results of the ROI experiments for chest, back, left-right, and face experiments.

Fig. 9

(a) ROC analysis of only symptomatic subjects, for combinations of symptoms and thermal imaging data. (b) Comparison of the AUC for the different combinations of view and stratified by sex.

Fig. 10

Heatmaps of the top selected features associated with the presence of COVID-19. (a) Features for symptomatic and asymptomatic subjects. (b) Top features that separate COVID-19 positive and negative results on the set of symptomatic subjects.