1.1.

Sensors and Target Selection in New IRTs

Newly released IRTs commonly use IR sensors with much lower resolution than FDA-cleared IRTs. For example, Heimann sensors are a widely used component “temperature tablet” IRTs, which are available in resolutions ranging from $16\times 16$ to $80\times 64\text{pixels}$; sensors with a resolution of $32\times 32\text{pixels}$ or 1024 pixels in total, are the most common.¹² By comparison, the FDA-cleared FLIR E54 (a control device in this study) uses a sensor of $320\times 240\text{pixels}$ or 75 times as many pixels at 76,800 pixels. Per manufacturer specifications, the distance of the subject from the IRT may be as far as 1 m ($\sim 3\mathrm{ft}$). The low resolution and long screening distance may yield potentially unreliable readings as they may not capture the medial canthus.

The new devices also differ from conventional thermography with respect to target selection. Conventionally, IRTs target the medial canthus (a.k.a., inner canthus) for human temperature measurement as the evidence suggests this area most closely approximates the surface temperature best correlated to core temperature. In contrast, the new devices typically target either the warmest part on the entire face irrespective of location or the forehead region. Notably, the studied devices in this article apply a compensating algorithm only upon detection of a human head, suggesting that whatever purpose the algorithm is intended to serve is human-specific.

1.2.

Approach to Estimating Human Core Body Temperature

Human temperature varies depending on the part of the body that is measured. The range of human temperatures is usually defined in terms of core body temperature or oral temperature in clinical settings. Core temperature is usually estimated with oral temperature, tympanic temperature (ear), or surface temperature (skin temperature), among others. Devices measuring oral, tympanic, or surface temperatures for diagnostic purposes either report raw measurements or report an estimate of core body temperature computed with an on-board algorithm. The relationship between the accuracy of core body temperature estimates may vary depending on the location measured and the device design for computing an estimate.

All IRTs measure the surface temperature of the face, with edge-case exceptions. Some devices utilize the surface temperature input to provide a reading that estimates body temperature; however, the conversions vary by device. The approach of recently introduced IRTs is poorly understood because the manufacturer specifications do not include the conversion algorithm, and to our knowledge, no independent assessment has been published to date. Nevertheless, it is well established that surface temperature is lower than core body temperature and lower than oral temperature.^13–15 Therefore, the IRT estimates of core body temperature would be expected to be higher than the surface temperature measured.

2.1.

Devices

The devices selected included five “temperature tablets” (a.k.a., temperature kiosks, temperature terminals) and two “bullet camera” IRTs. (Table 1) The temperature tablets were as follows: Bems Temperature Terminal, TVT TD-E2128-TM, Certify SnapXT Pro, Meridian Clear 2, and the ZKTeco 8” SF1008+. The bullet camera IRTs were the Dahua DH-TPC-BF5421-T and the Hikvision DS-2TD2636B-13/P.

All fit the first criterion in that they were introduced in 2020. None have undergone the 510(k) clearance process, our second criterion. Regarding the third criterion, Hikvision and Dahua are the largest and second largest video surveillance manufacturers in the world, respectively. For both, there is evidence of worldwide sales of IRT temperature screening products.^17–19 In marketing materials, Certify claims to be utilized in 75% of NFL stadiums.²⁰ Meridian states they are used by numerous school districts²¹ and has been purchased by the U.S. Department of Veterans Affairs, which administers hospitals across the United States.²² Bems and TVT supply temperature tablets for rebranding to multiple companies worldwide.^23,24 The ZKTeco device was selected because it is one of the world’s largest access control and biometrics manufacturers, with clients including Walmart and Carrefour.²⁵

The two control devices were the Extech IR200, a handheld NCIT, and the FLIR E54 EST, an IRT. Both were FDA 510(k)-cleared.

2.2.

Approach

As stated previously in Sec. 1, IRT measurement capability is negatively affected by various factors, namely poor environmental conditions and improper subject preparation. The laboratory setup was designed to ensure our observations were representative of true device performance; that is, with minimal impact on performance by interfering factors.

Specifically, we sought to replicate conditions under which IRTs have been tested in literature⁹ and expected to perform normally, including following FDA guidelines, the IEC 80601 standards, and manufacturer specifications. Testing was conducted in a windowless indoor laboratory with concrete walls. The area was temperature controlled, and ambient temperature and humidity were verified for each trial with an Extech RH520A Humidity+ Temperature Chart Recorder with Detachable Probe (FLIR Systems, Oregon, United States of America). All devices, inclusive of all IRTs, BBs, and NCITs were turned on to warm-up 30 min prior to the start of testing (in accordance with manufacturer guidelines). To minimize airflow, we ensured there were no open doors, or active air-conditioning, heating, or fans. IRT optics were checked for lens contamination (e.g., dust, debris, and fingerprints) that could skew readings. If devices had calibration settings, they were calibrated per manufacturer guidelines. Three of the IRT devices, the Dahua, Hikvision, and Meridian, and the Extech NCIT had methods for manual calibration, which was performed with the Dahua BB reference device set at 37°C. While the human body core temperature is roughly 1°C above forehead temperature, calibration at 37°C was appropriate to avoid introducing bias. In addition, a fixed offset will not impact trends, variance, or correlation. The Meridian was given a fixed offset of $\pm 0.2°\mathrm{C}$, the Hikvision and Dahua IRTs were each given an offset of $-1.6°\mathrm{C}$; these offsets calibrated the IRTs such that when the reference device was set at 37°C, the IRT read 37°C. Finally, testing was conducted with a low reflective background, mounted (fixed) and on a horizontal plane directly in front of the subject.

Since IEC, FDA, and manufacturer guidelines are intended for use with human subjects, our procedure differed by employing a non-human subject. We used a human face printed on generic $9.5\mathrm{in}\times 11\mathrm{in}$. white printer paper attached to a paperboard to ensure a flat surface (all test devices only display temperature readings upon detection of a human face). An opening in the paper face and paperboard of 5 cm width by 2.5 cm height was made over the paper face’s forehead. We used a BB of known emissivity and accuracy as the measurement target, the Sunell SN-TH01. The paperboard, with the printed face, was placed over the BB device such that its radiant area protruded from the cutout to allow direct measurement by the IRTs/NCITs [Figs. 1(a) and 1(b)]. This setup, the BB, and printed face are hereafter referred to as the “subject.” All IRT devices measured the subject’s radiant area, which was the highest thermal yield displayed in the field of view. The exposed surface area of the subject was $12.5{\mathrm{cm}}^2$, providing a large target with uniform temperature and high emissivity, conditions favorable to IRT performance [Figs. 1(a) and 1(b)]. By comparison, the surface area of the human face from which febrile temperatures can be observed is considerably smaller and less uniform, ensuring that the test process was conservative (Fig. 2). This setup is thought to be more favorable to IRT performance than real-world testing environments, because the subject is uniformly heated, still, and unobstructed, and various environmental factors likely to affect performance were eliminated [Figs. 3(a) and 3(b)].

Fig. 1

Testing configuration for simulated human thermographic reading. (a) Regular image. (b). Thermal image. Subject was mounted at 168 cm from the floor. Manufacturer specifications were followed in the placement of the IRTs relative to the subject. All tablets were mounted at a height of 155 cm and placed at a distance of 50 cm from the subject. The FLIR IRT was mounted at a height of 157.5 cm and placed 70 cm from the subject.

Fig. 2

IRT image of (a) male with normal temperature and (b) male with elevated temperature from IEC 80601.³

For trials of the Dahua and Hikvision bullet IRTs, we used an accompanying external reference device per manufacturer instructions. This was the Dahua JQ-D70Z BB, selected for its compatibility with these products. The reference device was placed 1 m apart from the subject to avoid interference, and each was 3 m away from the bullet IRT being tested. The reference device was placed at a height of 1.8 m, the subject was placed at a height of 2 m, and the bullet IRT was also placed at a height of 2 m. Both the subject and reference device were within the camera’s field of view. We calibrated the bullet IRTs with the reference device after the 30-min warm-up. The FLIR IRT was at a height of 157.5 cm and 70 cm from the subject. The reference device was not in the FLIR IRT’s field of view [Figs. 4(a) and 4(b)].

Fig. 3

Testing environment. (a) Side view. (b) Top view.

Fig. 4

Testing environment for devices requiring BB calibration with subject and reference device placed side-by-side. (a) Side view. (b) Top view.

The readings for each of the seven tested devices were recorded between 35°C and 40°C at 0.25°C increments, yielding 21 readings per device. For each increment, we took a measurement with the IRT being tested, one with the FLIR E54 EST, one with the Extech NCIT in surface mode, and two with the Extech NCIT in body mode. Two measurements were taken with the NCIT in body mode to confirm that the subject’s BB was measured, not the paper; these results were averaged for the subsequent analyses.

3.1.

Analysis of 510(k)-Cleared Control Devices

The control devices were consistent and accurate, recording small differences from the subject’s actual BB temperature setting over all 21 measurements from 35.0°C to 40.0°C and across all seven rounds of testing, indicating that nothing about ambient conditions or the testing environment produced an inaccurate reading (Table 2). The two Extech NCIT surface mode readings never differed from each other by more than 0.2°C, and all readings were within 0.1°C of the subject. The FLIR performed similarly, with readings never varying by more than 0.2°C and all readings within 0.15°C of the subject. The Extech NCIT body mode averages were consistent but differed more from the subject setting than in surface mode. This is to be expected because the device uses an algorithm to estimate body temperature from surface temperature. In body mode, the averaged Extech NCIT varied by 0.45°C or less. These results suggest that the test environment was conducive to accurate IRT and NCIT performance.

3.2.

Analysis of Tested IRTs

In contrast to the FLIR and Extech control devices, all seven tested IRTs recorded temperatures that deviated from the subject’s temperature setting, systematically reducing temperatures above 37.0°C and increasing those lower than 36.0°C. Specifically, the devices tended to record temperature deviations toward a human mean of 36.0°C to 37.0°C, with more marked reductions found as temperature increased above the mean (Fig. 5). In contrast, the 510(k)-cleared control devices demonstrated near-perfect correlations with the subject; the variance in readings was uniformly dispersed across the range of temperatures measured.

Fig. 5

Bland–Altman plot of seven tested devices.

Following the calculation of descriptive statistics as described in Table 2, we constructed Bland-Altman plots to visually inspect the limits of agreement between the tested devices and the subject (Fig. 6). We found strong evidence of variable proportional bias in the tested devices but no such bias in the control devices. While the FLIR had nearly zero bias, which appeared to be dispersed randomly, tested devices deviated from the subject and had biases that were proportional to the mean temperature.

Fig. 6

Recorded readings for tested IRT reading versus subject.

Fig. 7

Bland–Altman plot of Hikvision DS-2TD2636B-13/P Results.

Fig. 8

Bland–Altman plot of Dahua DH-TPC-BF5421-T.

Fig. 9

Bland–Altman plot of Meridian.

Fig. 10

Bland–Altman plot of ZKTeco.

Fig. 11

Bland–Altman plot of Certify.

Fig. 12

Bland–Altman plot of TVT.

Fig. 13

Bland–Altman plot of Bems.

Specifically, we found that the tested IRTs each had proportional biases that decreased upon approach from hypothermic to the human mean, elevating the temperature. As temperature increased past the human mean into the febrile range, the bias then changed direction from positive to negative, thereby reducing the temperature. While the degree of bias varied with each specific device, it was clearly present and proportional in the tested devices. For comparison, the 510(k)-cleared FLIR demonstrated virtually zero bias across the readings whereas the Bems demonstrated significant bias ($-0.829$), nearly always recording a near-normal temperature regardless of the subject’s temperature setting (Fig. 5).

In addition, all tested IRTs were not as strongly correlated with subject temperature as would be expected for a device that is intended to accurately measure febrility. None had a regression coefficient above 0.8 with subject temperature (a coefficient of 1 would indicate a perfect correlation, and a slope of 1 on a graph, as in the dotted line in Fig. 6); and only the Bullet IRTs regression coefficients were above 0.6 (Figs. 6Fig. 7Fig. 8Fig. 9Fig. 10Fig. 11Fig. 12–13). This shows changes in surface temperature do not lead to equivalent increases in IRT output. An accurate device free of bias or smoothing would generate a regression coefficient closer to 1, as seen with FLIR and Extech (Table 2). In other words, one might expect a consistent offset (e.g., a straight line with slope of 1) but not a systematic and variable norming to the human mean.

An inaccurate device could have random or systematic error; however, these seven devices had deviations in readings, which suggest a systematic error that is correlated with the subject temperature, driving the readings closer to the human mean (Fig. 14). All seven tested devices systemically overestimated low temperatures and underestimated high temperatures. The error is highly correlated with temperature in a linear relation, demonstrating a clear pattern of correction to the human mean (Fig. 15).

Fig. 14

Disparity between control (FLIR) and tested IRTs: magnitude and direction of deviation from BB reading. (a) Difference between device and subject versus subject. (b) Absolute value of difference between device and subject versus subject.

Fig. 15

Pooled differences between tested devices and BB temperature.

Fig. 16

Results from control devices, FLIR E54-EST and Extech IR200.

The tested IRTs exhibited much smaller variances than the subject and the 510(k)-cleared control devices, another sign of normalization. An accurate device would be expected to demonstrate variance comparable to that of the subject, whereas a device that adjusted results toward the mean would have lower variance. In our results, the tested devices demonstrated very little variance that was nonuniformly distributed across the measured temperatures, resulting in a normalization to the human mean (Table 3). The least reliable device (Bems) consistently recorded temperature near the human mean, regardless of the BB setting, resulting in a low variance, 0.005°C. In contrast, the FDA-cleared devices had variances similar to the subject (2.4°C); the bullet cameras had notably lower variances (1.4°C to 1.6°C) and the temperature tablets had significantly lower variances (0.005°C to 0.95°C) (Table 3). The IRTs had means between 36.7°C and 37.8°C, with all devices except the Meridian having a lower mean than the BB. This analysis of variance further suggests a normalization toward the human mean in the seven tested IRTs.

Table 3

Variance across measured temperatures by control and tested devices.

Table 4

Laboratory humidity and temperature conditions in each trial by device.

4.1.

Comparison to Previous Research

We are unaware of any papers analyzing new IRT devices for signs of normalization. This could simply be that researchers use FDA (or international equivalent) cleared devices or other purportedly accurate devices, whereas we focused on popular consumer grade devices. The tested devices were all recently marketed in 2020; if normalization in IRTs is a recent phenomenon, this may also explain why normalization has not been observed in prior research. There has been extensive research into the accuracy of NCITs and IRTs as a tool for core body estimation and for screening devices for fever or influenza like illnesses. In one recent meta-analysis of thermography sensitivity and specificity, the authors

found good pooled sensitivity and specificity for IRTs and NCITs, in excess of 0.8 and 0.9, respectively.¹⁰ However, these studies have largely focused on analyzing whether IRTs or NCITs could be used for accurate estimation of core temperature for clinical purposes, and therefore the selection of tested devices may have been different than our review of newly released thermography screeners. One study analyzing several NCIT devices found that the accuracy of two of the three devices tested fell outside the manufacturer specified range.²⁶

4.2.

Limitations

In our test environment, temperature conditions were aligned with manufacturer and FDA recommendations, but humidity was aligned only with manufacturer recommendations. Our ambient humidity ranges were in excess of 53% (no greater than 70%) for all tests, whereas the FDA-recommended humidity range is 10% to 50% (Table 4). Given that the test environment was uniform across all devices and within manufacturer recommendations, it is unlikely that our results were biased to reflect poorly on the tested devices and more highly on the control devices. In addition, the Extech on surface mode and FLIR IRT were accurate across every round of testing, which indicated ambient conditions did not have an impact on either device. Further, it should be expected that real-world scenarios would include high-humidity conditions, thus making it important that testing occur uniformly across all devices but in close approximation to real-world settings.

In addition, we selected one IRT in each of the seven series; follow-on research studies should consider testing multiple IRTs to detect within-device variation in the same series. This was beyond the scope of our budget, and we reflect real-world scenarios in which facilities or individuals are limited to the devices on the market and cannot purchase multiple devices for testing prior to use.

We did not use human subjects. This was considered impractical given the ongoing pandemic of communicable disease and because a properly controlled study with human subjects is more experimentally complicated. Future research could assess these devices’ performance on humans; however, it may not provide any benefit vis-à-vis our goal of assessing compensating algorithms, which is assisted by artificially controlled conditions as in our methods.