2.1.

Heart Activity

Heart activity can be monitored either in visible spectrum (VIS), NIR, or IRT imaging. These techniques are based on different principles and thus methods used to estimate HR. In VIS/NIR videos, this vital parameter can be extracted using one of the following approaches: motion-based (presented by Balkrishnan et al.¹⁷ and Li et al.¹⁸) or color-based (presented by Poh et al.¹⁹).

The former uses optical flow to measure the weak vibrations of the body caused by the beating heart. In short, feature points (around the ROI) are identified in the first frame of the video and tracked over time (applying, e.g., a template-based point tracker). Afterward, their trajectories are processed to extract the chest motion related to the cardiac function. This step consists in filtering the trajectories of the feature points to remove noise outside the expected signal frequency range. Using blind source separation, based either on principal or on independent component analysis, the trajectories are decomposed into subsignals (principal components) to isolate the heart-related component. The principal component that most closely resembles the heart signal is selected; its main frequency in the frequency spectrum corresponds to the HR.¹⁸

The latter approach (color-based) uses color or brightness changes of the skin due to blood flow to quantify heart activity (it is an extension of the classical reflective PPG). Using this method, an ROI, enclosing the subjects’ skin, must be identified in the first frame of the video and tracked over time for motion compensation. Thereafter, the ROI should be separated into the three RGB channels and spatially averaged to obtain the raw RGB traces (i.e., the mean color pixel intensity of the ROI). Blind source separation (independent component analysis) is also used in this approach in order to decompose the traces into three independent source signals. Finally, the component containing the heart signal is selected; the frequency with the highest power of the spectrum—within the operational frequency band—corresponds to the pulse frequency.¹⁹

Both approaches were primarily developed for human medicine and have recently been applied in animal research. Regarding thermal videos, only the motion-based approach can be applied. Table 1 presents a detailed overview of the imaging systems and their properties, applicable algorithms (whether color-based or motion-based), ROIs, and light conditions necessary to extract HR from four animal species: mice, rats, pigs, and primates. Some of the commercially available systems capable of detecting this vital parameter are listed in Table 2.

Table 1

Detailed overview of the imaging systems and properties to extract HR from four animal species: mice, rats, pig, and primates. Algorithmic approach (color-based or motion-based), ROIs, frame rate, spatial and thermal resolution, as well as required light conditions are specified.

Table 2

Examples of imaging systems.

2.2.

Respiration

Respiration can be assessed by measuring the displacement of the chest or abdomen using one of three imaging systems: VIS,²³ NIR,¹⁸ or IRT.²⁴ For this purpose, the motion based-approach, described in the previous section, can be used.

In thermal videos, temperature variation around the nostrils during the respiratory cycle also offers insights into this physiological process. This results from the inhalation of cold air from the environment and exhalation of warm air from the lungs.²⁵ In this approach, the nose (ROI) is segmented in the first frame of the thermal video. Afterward, a rough tracking overtime is performed to compensate motion. The filtered mean temperature of the ROI, which represents the respiratory waveform, is computed. Finally, the respiratory frequency can be extracted either by selecting the frequency with the highest power of the spectrum or using breath-to-breath estimation algorithms.

In addition to RR, these techniques allow us to monitor breathing patterns²⁶ (such as tachypnea, Kussmaul breathing, Cheyne–Stokes respiration, and apnea phases) and tidal volume.²³ As in the previous section on heart activity, respiration assessment with imaging systems has primarily focused on human medicine. In the last years, these new techniques have been translated to animal research.^7,11,20,27 Table 3 presents a detailed overview of the imaging systems and their properties, relevant algorithms, ROIs, and light conditions necessary to extract the RR from four animal species: mice, rats, pigs, and primates. For this application, the cameras presented in Table 2 can be used. However, if the focus is only on respiration, cameras with lower frame rates and resolutions are an option as well, as the amplitude of the thorax movement is prominent and the frequency smaller than the HR. Two examples are the webcam Logitech C270HD (Logitech International S.A., Lausanne, Switzerland)—RGB, and the FLIR T640 (FLIR Systems, Wilsonville, Oregon)—IRT.

Table 3

Detailed overview of the imaging systems and properties to extract RR from four animal species: mice, rats, pig, and primates. Algorithmic approach (color-based or motion-based), ROIs, frame rate, spatial and thermal resolution, as well as required light conditions are specified.

2.3.

Oxygen Saturation Monitoring

In addition to heart activity and respiration, NIR imaging also permits the assessment of arterial oxygen saturation (${\mathrm{SpO}}_2$) in blood by adapting the principles of conventional pulse oximetry. As well-known, oxyhemoglobin (${\mathrm{HbO}}_2$) and deoxyhemoglobin (HHb) have different absorption spectra, whereas HHb absorbs more light in the red band of the electromagnetic spectrum (600 to 750 nm), ${\mathrm{HbO}}_2$ absorbs more light in the infrared band (850 to 1000 nm). Therefore, conventional pulse oximeters contain two LEDs of different wavelengths (usually red and infrared). By capturing a PPG at two suitable wavelengths, the relative concentrations of ${\mathrm{HbO}}_2$ and HHb can be derived, and consequently, the ${\mathrm{SpO}}_2$. In fact, this can be defined as the ratio of ${\mathrm{HbO}}_2$ to the total hemoglobin (sum of ${\mathrm{HbO}}_2$ and HHb) as given by

The measurement of ${\mathrm{SpO}}_2$ using NIR cameras is again an extension of the classical pulse oximetry method. In addition to the monochromatic/RGB camera, a trigger-controlled wavelength LED-array is required. Alternatively, two or more cameras with different optical filters can be used. The application of NIR imaging for ${\mathrm{SpO}}_2$ measurement was primarily tested on human subjects.^29,30 In the last years, this technique has also been being used in animal research to measure cortical blood oxygenation.³¹ For this application, publications reported the use of monochromatic/RGB cameras (sensitive also in the infrared region of the electromagnetic spectrum) with a resolution of $1024\times 768\text{pixels}$ or higher, a dynamic range of 12 bits, and a frame rate of 30 Hz (e.g., AD-080GE, Jai A/S, Denmark).^28,31 The illumination unit should be composed of at least two LED arrays with two appropriate wavelengths (e.g., red and NIR). However, Shao et al.²⁸ demonstrated that the combination of orange/infrared LEDs is the best choice considering the PPG signal-to-noise ratio. Finally, the measuring system should include a control unit to trigger LEDs and camera, since the red/orange and infrared LEDs must flash alternatingly.^28,31 It is important to note that ${\mathrm{SpO}}_2$ assessment is based on the color-based approach (presented in Sec. 2.1). Thus to extract the signal of interest, a portion of skin or tissue must be visible and illuminated.

2.4.

Motion and Activity

In addition to vital parameters, camera-based techniques are frequently used for monitoring of activity and exploratory behavior, mostly in rodents. As is well-known, movement and location of animals, e.g., during open field tests, are major markers for anxiety, stress, and pain.

Motion activity can be remotely assessed using one of three imaging systems: VIS, NIR, or IRT. Although all systems are qualified for this application, IRT has some additional advantages. First, this technique does not depend on any light source. In contrast, visible and NIR systems require a light source (whether artificial and natural light). This property plays a major role for circadian rhythm monitoring. Second, IRT identifies defecation and urination (which can also be affected by stress or fear) as hot (and later cold) spots. Third, animals can be easily released from the background, which is significantly colder (animals are hot spots in IRT videos). In addition, the color of the animals does not have any influence on the detection/segmentation algorithm. Despite all advantages, IRT is still an expensive technology in comparison with visible and NIR imaging.

For motion analysis, there are two main steps that may influence the performance of the system: the detection algorithm and the tracking algorithm. In fact, robust detection and tracking are the key factors of a reliable software. From tracking, several parameters can be assessed, including acceleration, position, velocity, total distance travelled, and measurement of the time spent in different regions of the open field, among others. Furthermore, for visualization of the exploratory behavior, motion heat maps can be estimated.

Regarding the measurement setup, the camera should be aligned perpendicularly (90 deg) to the cage or open field. When using an IRT camera, it should have a resolution of $1024\times 768\text{pixels}$ or higher as well as a frame rate of at least 30 Hz (e.g., FLIR T1020, FLIR Systems, Wilsonville, Oregon; or VarioCAM head 800, InfraTec GmbH, Dresden, Germany). Visible or NIR cameras should have similar properties: ANY-maze 60516 (Dublin, Ireland), Logitech C270HD (Logitech International S.A., Lausanne, Switzerland) and Sony DCR-SR60 (Sony Corporation, Tokyo, Japan) are two examples used for these applications.³² Equally important is the calibration of the camera, which must be calibrated to remove lens distortion.

2.5.

Wound Monitoring

The four cardinal symptoms of inflammation include rubor (redness), calor (heat), dolor (pain), tumor, and functio laesa (disturbance of function).³³ Unfortunately, these signs are hard to diagnose by visual inspection alone. Fur and thick skin complicate the diagnosis even further, because they hide rubor and tumor. Touching the wound area can be useful to detect calor, dolor, and tumor, but should be avoided since it can be painful for the animal and for hygienical reasons.

The usage of various cameras with subsequent image processing can contribute to a contactless monitoring of the cardinal symptoms (rubor, calor, and tumor) that are not sensitively assessable by existing (clinical) methods. IRT can display the temperature distribution around the wound area, which can be examined for homogeneity, size, and shape. A normally healing wound is expected to show a homogeneous warm area around the wound, with margins equidistant from the wound origin. The presence of localized hot areas or cold spots can imply problems in the healing process. Visual- and depth-sensing cameras can be used as a replacement for manual visual inspection. Using computer algorithms for wound analysis, the results can be made quantifiable and more objective. Those cameras can be used to detect rubor (visual cameras) and tumor (depth cameras), e.g., by comparing successive images taken of a wound.

For wound monitoring, we use two classes of IRT cameras: high-end and entry level devices. All camera systems contain additional integrated visual cameras. The use of depth sensing camera technology is currently a subject of research. An exemplary entry-level IRT camera comprises a $160\times 120\text{pixel}$ image resolution, a horizontal field of vied (HFOV) of 50 deg, and a minimum object distance of 30 cm (e.g., FLIR One Pro, FLIR Systems, Wilsonville, Oregon). Thus a maximum resolution of $\sim 6\text{pixel}/\mathrm{cm}$ can be achieved. Given the assumption that four pixels are enough, structures of around 0.7 cm can be represented. High-end cameras deliver the most flexibility, as they feature high image resolutions, e.g., $464\times 384\text{pixels}$, and changeable lenses (e.g., FLIR E95, FLIR Systems, Wilsonville, Oregon). Resolutions of 22 pixels/cm and the detection of structures larger than 0.2 cm are easily possible.

3.1.

Heart and Respiratory Rate

Tables 4 and 5 present the performance of the imaging systems for monitoring of HR and RR in different animal species. Since the application of such techniques in animal research is very recent, these tables cannot be filled completely since studies are still lacking. Visible imaging was the first system analyzed. To date, its feasibility has merely been tested in rodents (mice and rats) and nonhuman primates. For HR and RR, promising results were obtained, the root-mean-square errors (RMSEs) averaged $<4\text{beats}/\min$ and 1.5 breaths/min, respectively. Although in rodents vitals were derived using the motion of the thorax/abdomen, in primates HR was estimated by measuring variations in blood volume with consecutive changing skin color (color-based approach). Blanik et al.²¹ were, to the best of our knowledge, the only researchers studying the application of NIR imaging in animal research. In their study, the authors assessed HR and RR in the snout of pigs using the color-based approach and motion, respectively. Unfortunately, instead of quantitative data analysis (comparison with a gold standard method), only a qualitative analysis was carried out. According to the publication, a good agreement between NIR and the gold standard was obtained. Finally, regarding IRT, there are more studies applying this technique in animals, especially for assessment of RR.^7,11,27 For the selected studies, the mean absolute error ($\overline{{\epsilon }_{\mathrm{abs}}}$) was $<3.5\text{beats}/\min$ and 5 breaths/min for HR and RR, respectively. Note that HR was only assessed in thermal videos of pigs. Figure 1(a) shows a Bland–Altman plot comparing the HR estimated with visual imaging—using the motion-based approach—and the gold standard (ECG). These results were presented by Kunczik et al.³⁴ As illustrated, a great agreement between ECG was obtained with the limits varying between $-3.2$ and 1.7 beats/min. Figure 1(b) illustrates the regions of the rodent (rat) used to extract HR; regions with a signal-to-noise ratio smaller than a specific threshold were not used to compute this vital parameter.

Table 4

Accuracy of three imaging systems for assessment HR.

Table 5

Accuracy of three imaging systems for assessment RR.

Fig. 1

(a) Bland–Altman plot comparing the HR estimated with visual imaging—using the motion-based approach—and the gold standard (ECG).³⁴ The limits of agreement range from $-3.2$ to 1.7 beats/min. The bias averages $-0.7\text{beats}/\min$. (b) Regions of the rat used to extract HR.

Figure 2(a) shows, in turn, a Bland–Altman plot comparing the RR estimated with IRT and the gold standard (ECG-derived RR). In this example, a good agreement between both techniques is also visible, with the limits of agreement ranging from $-0.8$ and 0.6 breaths/min. Figure 2(b) illustrates the thermogram and the points used for motion tracking.

Fig. 2

(a) Bland–Altman plot comparing the RR estimated with IRT and the gold standard (ECG-derived RR).¹¹ The limits of agreement range from $-0.8$ to 0.6 breaths/min. The bias averages $-0.1\text{breaths}/\min$. (b) Thermogram of a rat and feature points used for motion tracking.

3.2.

Oxygen Saturation

Van Gastel et al.³⁰ conducted a study with human subjects to compare two methods of camera-based ${\mathrm{SpO}}_2$ extraction algorithms: the classical ratio-of-ratio (ROR) algorithm, used in pulse oximeters, and their own method (“APBV”), which improves the raw signal quality through signal fusion and a priori knowledge.³⁰ Four minutes of videos were recorded with static and moving subjects. The contactless ${\mathrm{SpO}}_2$ was extracted from the subjects’ heads. A classical pulse oximeter was used as reference.

For the static experiment, the RMSE averaged 3% in case of the ROR algorithm and 2% for the APBV algorithm, respectively. The RMSE increased to 27% for the ROR algorithm, once free head movements were allowed. On the other hand, the ABPV approach managed to contain an RMSE of 3% in this case.³⁰

In a rat model, changes of cerebral blood flow and oxygen consumption were examined during a 16-s electrical forepaw stimulation. By thinning out the bone to translucency, a cranial window was prepared enabling to record image sequences. Although blood flow increased by about 15%, deoxyhemoglobin concentration dropped and oxyhemoglobin concentration rose by the same amount. Not only the sum level but spatially resolved blood flow and oxygenation maps could be obtained (Fig. 3).

Fig. 3

Superficial tissue imaging system for combined RGB reflectometry and laser speckle contrast analysis [superficial tissue imaging system (STIS) provided by Kohl-Bareis and Steimers, Biomedical Optics Laboratory, Rheinahrcampus Remagen, Germany]. Blood flow and blood oxygenation responses to 16-s electrical forepaw stimulation were recorded through a cranial window preparation (bone thinned to translucency) in isoflurane plus fentanyl anaesthetized rats. Laser speckle contrast, ${\mathrm{HbO}}_2$, and HHb-images were online calculated from raw images of speckle pattern and red, green, and blue LED-illumination images using STIS and stored at $\sim 0.75\mathrm{Hz}$ (LabVIEW 2010, National Instruments Inc.) for later offline calculation of cortical blood flow from speckle contrast.

3.3.

Motion Activity

Several approaches for assessing motion activity have been published.^11,32,36,37 In these studies, the results of the tracking performance were evaluated qualitatively. They demonstrated the ability to accurately detect and track the animal during different behavioral tasks, including open field test, Y-maze task, and Barnes maze, among others. Note that the animal detection was performed automatically. Since automated identification is still a challenging task (especially when the animal and background have similar colors), in some processing algorithms, the position of the animal must be defined manually by the user. As demonstrated in Fig. 4(a), this task is simplified with IRT, as the animal is a hot spot in the thermal video (the background is much colder than the animal). Figure 4(b) shows a motion heat map, which is usually used to demonstrate the cumulative time spent in different parts of the arena; yellow denotes more time and blue, less or no time.

Fig. 4

(a) Thermogram of the open field arena. The animal is the hot spot in the image. (b) Motion heat map.¹¹

3.4.

Wound Monitoring

Figure 5 shows the exemplary results for contactless wound monitoring. Both upper images show visual images of laparotomy wounds of two different, male Wistar rats. In addition, from the incision, no redness can be observed. The right wound image seems to show better healing, as the visible remainder of the incision is much smaller.

Fig. 5

(a) and (b) Images of two laparotomy wounds. The lower images present, in addition, an IRT overlay showing the temperature distribution around the wound area in “false colors.”

Below, the same images are presented with an IRT overlay, which shows the temperature distribution around the wound area in “false colors.” Here, Fig. 5(b) shows high-temperature fluctuations around the wound, whereas Fig. 5(a) presents a much more homogeneous temperature distribution. Also large cold spots can be observed in Fig. 5(b), which imply the secretion of exudate.