The monitoring of basic vital signs, like heart rate (HR) or respiration rate (RR), is a fundamental task in clinical care. For this purpose, various techniques have been established and are available in ICU monitors, anesthesia devices, etc. For continuous long-term monitoring, or for frequently repeated measurements, an unobtrusive way of acquiring data becomes increasingly important. However, even when unobtrusive, most methods currently available still require direct contact between a sensor element and a specific body region. Although in many cases direct skin or tissue contact is feasible, for some fields of application, skin contact is either undesirable or is not possible. For example, in patients with extensive burns or lesions that involve large areas, adhesive or mechanically attached sensors (like ECG electrodes or finger clip pulse oximeters) cannot be applied in or on the wounded areas. A similar situation arises with the use of sensors in neonatal intensive care. Because of the immature skin (particularly in preterm infants), direct skin contact with sensors may cause irritation or skin injury, thereby increasing the risk of infection.1,2
To prevent the possible harmful side-effects of contact monitoring methods, a noncontact, camera-based alternative for continuous monitoring is proposed. This involves two camera systems: (1) photoplethysmography imaging (PPGI) for active optical measurement of skin perfusion and (2) passive infrared thermography imaging (IRTI) for measurement of surface temperature. In addition, the spatial resolution of the cameras allows one to assess the local distribution of vital parameters. With these compact and user-friendly cameras, the measurement setup can easily be adapted to a wide range of applications.
Therefore, the basic principles of the two camera-based systems are described and the results of preliminary experiments are presented to illustrate the possibilities with these techniques. Finally, a combination of contactless measurements based on space-resolved IRTI and simultaneous PPGI is proposed for a novel hybrid camera-based monitoring system.
The application of camera-based monitoring systems offers an ideal opportunity to realize contactless recognition of vital signs, as well as a spatially resolved functional assessment and interpretation of the obtained data. The two optical systems are described below.
The PPGI technique is an advancement of the classical and well-established photoplethysmography (PPG).34.–5 Nowadays, PPG is widely used in many medical disciplines and is appreciated by physicians because of its simple design and relatively low cost per examination. PPG systems with optical sensor technology in the near-infrared () light range can noninvasively detect changes in blood volume (related to arterial and venous hemodynamics) in the dermal blood vessel network by registering the optical attenuation of transilluminated tissue, according to the Beer-Lambert law.6
In this active optical sensing system, one or more selective light sources and one or more light detectors are integrated into the optical sensor unit, which can work in reflection or transmission mode.
Vital signs that can be observed include HR and RR (and their variability), which may provide information on the autonomic nervous system, perfusion phenomena with frequencies below respiration, venous reflux, venous refilling rates, and other diagnostically relevant physiological indexes or signals.78.9.–10 Reflection mode sensor technology (rPPG) can be applied to practically all body areas, thus overcoming the limitation of transmission (tPPG) sensors, which can only be applied to thinner body parts, such as the fingertip or ear lobe.
In rPPG, distances between light source and light detector are in the order of a few millimeters [Fig. 1(b)], and typical penetration depths of light into the tissue are around a few millimeters, depending on the wavelength and sensor geometry.7
Using two or more wavelengths at the same time also allows one to measure arterial oxygen saturation in skin, for example, using the pulse oximetry method, see, e.g., Ref. 11.
In PPGI, the classical reflective-mode, skin-contact PPG sensor is replaced by a highly sensitive camera detector and an external illumination unit consisting of multiple light-emitting diodes.1213.14.15.–16 In particular, to detect the weak light modulation caused by blood hemodynamics, our setup17 uses the Pike210B camera (Allied Vision Technologies Ltd., Hod-Hasharon, Israel) chosen due to its high dynamic range of 96 dB and its high readout speed of up to 31 full frames per second. The imaging sensor is a Kodak KAI-2093 type 1 progressive scan CCD (diag. 15.3 mm) with a pixel resolution of and a spectral range of 320 to 950 nm with a quantum efficiency of at 480 nm.
The setup shown in Fig. 2 allows one to measure arbitrary parts of the skin surface and to simultaneously assess dermal perfusion in different skin regions. The PPGI system can assess local changes in perfusion patterns and create a functional mapping of the perfusion status. After recording a video sequence of the skin surface, the operator can select different regions of interest (ROIs; virtual sensors) for which the backscattered light intensity is calculated. The resulting signals contain the same information as the classical PPG signals. Of considerable importance is the possibility to obtain measurements even in wounds, transplanted skin, or on a neonate’s sensitive immature skin.
The PPGI camera is connected to a high-performance PC via FireWire (IEEE1394b). The PPGI software was developed to control the camera settings, capture the image sequence, and perform postprocessing and analysis. Currently, the PPGI software is capable of acquiring the video in online mode and saving it for offline processing. In online mode, the live video can be displayed dynamically. The maximal sampling rate amounts to 31 fps at full resolution, but it can be increased by reducing sensitivity, acquiring only a subarray, or by binning. The file size of a typical PPGI sequence with spatial resolution of , 15 Hz sampling frequency, and 1-min duration is . At full frame and maximal readout speed, 1-min recordings will produce of data. Hence, to prevent extensive storage consumption, the recording parameters should be limited according to the analyzing requirements: the necessary image detail and sampling rate. Inside the PPGI frames, different sizes and positions of ROIs can be selected so that virtual PPG sensors of variable measuring window can be freely selected at different places. Here, typical sizes of these ROIs range from 10 to 50 pixels edge length. By calculating the average pixel values of the ROIs, the reflected optical signal (the PPG signal), which corresponds to blood volume changes, can be monitored. This signal describes the skin hemodynamics inside the ROI in the time domain.
For offline processing, first, the video is calibrated. Calibration includes equally intervalled resampling and motion compensation. Motion artifacts may occur due to movements of the patient/infant, or the patient’s body part in relation to the camera. Calculating differences relevant to subsequent images provides a first indication of the occurrence of movement. At points in time with motion artifacts, the calculated differences increase considerably, especially in regions with high contrast. Tracking of easily detectable edges in the images allows conclusions to be drawn about, e.g., an infant’s activity or, when monitoring the rise and fall of the chest contour, the registration of breathing activity.
For evaluation of perfusion information from PPGI sequences, motion artifacts are generally disturbing. On one hand, a clear spatial assignment of detected signals to certain body regions is no longer possible. On the other hand, the detected signal itself is corrupted by coupled frequency components of the movement artifact. Depending on the movements, the detected signal may be totally disrupted. To avoid this, depending on the type of expected movement (e.g., slight, strong, translational, or rotatory), the computational time available, and required accuracy of the result, various compensatory techniques can be applied.15 In our research, we mainly use block-matching algorithms, hierarchical motion approximation with the modified Newton algorithm, or calculation of motion constraints. The temporal effort for these methods depends on the image resolution, the size and number of tracked ROIs as well as the required accuracy. On modern PCs, the movement compensation of a single ROI with our experimental setup can be computed in quasi-real-time. For IRTI, these motion detection and compensation methods are applied in an analogous way.
Infrared Thermography Imaging
Similar to PPGI, IRTI is a contactless but passive imaging technique for spatial and temporal registration of an object’s surface temperature.
To understand the basic principle of IRTI, it is essential that any physical object with a temperature above the absolute zero point (0 K) emits radiation. Physically, the specific spectral radiance of ideal black bodies is described by Planck’s law.18
In this equation, is the black body’s temperature, is the Planck constant, is the speed of light in a vacuum, is the wavelength, and is the Boltzmann constant. For selected temperatures, the Planck spectra are shown in Fig. 3.
The maximal emitted intensity for a fixed-size wavelength interval is strongly dependent on the body’s temperature. In fact, according to Stefan-Boltzmann’s law, the complete emitted radiation power in the whole spectrum is a function proportional to the body’s temperature to the power of four.19
A shift of the maximum to longer wavelengths for lower temperatures can also be observed. This effect is described by Wien’s displacement law.4
For typical skin temperatures (ca. 35°C), this maximum shifts to lying in the infrared C-band (IR-C). To draw conclusions about an object’s temperature by measuring its passive radiant emittance, the sensing device should be most sensitive in this wavelength region.
To record thermal image sequences, the VarioCam hr head (InfraTec GmbH, Dresden, Germany) is used. It possesses a geometric resolution of and a sampling frequency up to 50 Hz.20 It is sensitive in the long-wave infrared band (7 to 14 μm) and most sensitive at 9.5 μm. Its dynamic range of 72 dB allows measurement of temperatures from to with best temperature resolution at 30°C (better than 0.03 K). Recorded data may be transferred via FireWire to a PC. The file size of a typical thermal video sequence of 1-min duration at 50 Hz is .
The recorded signal intensity correlates with the incident radiation weighted with the sensor’s sensitivity characteristic. With respect to internal calibration of the camera with regard to damping of the object lens, sensor, and camera temperature, and duration of exposure, this intensity can be correlated to the black body temperature. The distance of the observed object relative to the camera does not affect the detected temperature because the radiation density of a fixed-sized object decreases with the square of the distance, but at the same time, the size of an object observable with a single sensor pixel increases with the square of the distance.
However, to obtain precise measurements an additional calibration is required because in this way only temperatures of ideal black bodies can be measured exactly. However, objects of interest generally act like gray bodies or selective radiators. Their specific radiation is less than the specific radiation of an equivalent black body .
The emissivity coefficient is dependent on the wavelength of the object’s material and its thickness. In the wavelength interval of the sensor’s sensitivity range, can be defined unambiguously. Typical values of for different materials are provided in reference tables or by the camera manufacturer.20,21 Table 1 presents emissivity constants for a selection of different materials valid in the IR-C band (8 to 14 μm).
Emissivity coefficients ε in the infrared C band (8 to 14 μm).20,21
|Silver (polished)||0.02 to 0.03|
|Aluminum (oxidized)||0.2 to 0.31|
|Iron (polished)||0.14 to 0.38|
|Iron (dark gray)||0.31|
|Iron (rusted red)||0.61|
|Glass||0.85 to 0.95|
|Concrete||0.85 to 0.92|
|Water||0.95 to 0.96|
|Ice||0.95 to 0.98|
Although the aberration of skin from an ideal black body is relatively small (0.97 versus 1), its influence is not negligible for accurate measurement. According to Eq. (5), an aberration of 0.03 of the detected radiance intensity leads to a nontolerable error of 2 to 3 K for measured skin temperature. However, knowing the emissivity coefficient, the equivalent black body radiation and thus the temperature can be estimated.
An emissivity coefficient close to 1 also indicates almost black body behavior. This means that nearly all incident radiation is absorbed. Hence, the penetration depth of electromagnetic radiation into the subject is close to 0. In reverse, all measured radiation from a black body has its origin at its surface. For example, radiation from deeper skin layers is already absorbed by the overlying layers. As a consequence, the measured temperature corresponds to the upper surface skin temperature.
Besides calibration according to the type of material involved, the measuring section itself can influence the recorded temperatures and therefore needs additional calibration and compensation steps. Figure 4 shows the potential sources of errors depending on the measurement geometry.22
On one hand, total radiation coming from the measured surface is generally a sum of three separate components. The first is the emitted radiation of the body (which we are interested in), the second is the transmitted radiation from the background through the body, and the third component is the radiation reflected on the object’s surface from the environment. On the other hand, total radiation detected by the camera differs from the sum of the radiation coming from the measured surface depending on the media in the path of the rays. Here, part of the radiation might be absorbed, or even new components might be emitted toward the camera, according to the media’s material property, density, and temperature.
For medical IRT applications, the influence of transmitted radiation through tissue can be neglected because the human body is not transparent in the IR-C band (Table 1). Reflections at the surface are strongly suppressed because of the nearly ideal black body behavior of skin, absorbing incoming radiation; however, minor parts are reflected toward the camera. The percentage of this reflected radiation of the total radiation is a function of the environmental radiation intensity. For example, monitoring of neonates under an IR heating unit requires compensation of the reflected components. Disturbing media in the pathway are not totally transparent objects, like windows or even air moisture.23 According to Eq. (1), their effect on measurement increases when increasing the measuring distance or thickness of the material.
As is usual, the environmental and geometric conditions of IRTI measurements are assumed to be constant, or at least changing only slowly (e.g., room temperature or monitoring of a calm patient). Therefore, even disturbing influences will not change during measurement. This allows one to either calculate the offset in the temperature measurement if all environmental factors are known or compensate for their influence by additional calibration measurement with known references (e.g., measurement on dummies with known temperature and emissivity).
Preliminary Experiments and Results
The compact and nonintrusive nature of the camera-based monitoring techniques allow multiple and flexible applications. The two systems have been tested and their feasibility confirmed in preliminary studies with volunteers or patients.
Results from PPGI
For the PPGI system, the extraction of vital parameters is comparable to the classical PPG. Due to the spatial resolution, multiple time series (instead of a single one) need to be analyzed. Time series can be obtained either from the mean time-varying gray values inside the ROIs or for every pixel (or pixel array) separately. Algorithms used for extraction of vital signs in classical PPG are applicable one-to-one on these time series. Therefore, vital parameters (like HR or HR variability) can also be recovered from the PPGI sequence. Depending on the observed perfusion frequency band, in addition to rather, other rhythmic phenomenon (like breathing pattern or vasomotor activity) can also be monitored.
Figure 5 shows a typical PPGI recording with the corresponding time series of three ROIs. The sequence was recorded on the left hand with a small wound on the middle finger [Fig. 5(a); red ROI). For each ROI, the time signal can be calculated by estimating the mean gray value inside the ROI for each frame successively. Hence, the absolute perfusion amplitudes may feature strong inter- and intrapersonal variability; the signals were normalized by dividing them through their dc components to improve comparability. In all three time series, a signal component that strongly correlated with the heartbeat is clearly visible at [Fig. 5(b)]. However, the perfusion patterns from healthy skin and from the wound on the middle finger show significant differences. When examining only the heartbeat, its amplitude is slightly increased inside the wound, whereas the slower rhythms at are markedly reduced. More detailed analysis of signal components can be obtained by applying band-pass filters to the signals. Depending on the selected cut-off frequencies, for example, either rhythms correlated to the respiration frequency band [Fig. 5(c)] or vasomotor rhythms [Fig. 5(d)] around 0.1 Hz can be extracted. Please note that minor inaccuracies in the first and last few seconds of the band-pass filtered sequence are due to filter characteristics. In this example, although the amplitude of the respiration correlated signal is much weaker than the heartbeat or 0.1-Hz rhythm, phase synchronous patterns can be monitored on any region on the skin. In contrast, (vasomotor) rhythms differentiate between wounded and healthy skin. Inside the wound, the slow rhythms have almost vanished. In healthy skin, even in adjacent ROIs, local variations are clearly visible.
A similar behavior can be monitored with PPGI recordings on the forehead of a relaxed, seated subject. Here, spatial rhythm fluctuations in dermal perfusion are visible. This phenomenon is best observed in actual video sequences of the PPGI recordings as blood volume clouds, which move across the forehead in an organized but complicated pattern.
Figure 6 aims to illustrate this with screenshots of the recording; the actual video is available as an electronic supplement on the website of the Journal of Biomedical Optics.
An alternative approach to analyzing the frequency components of recorded time series (superposition of all rhythmic phenomena, of venous and arterial origin, in the dermal perfusion) is visualization of these phenomena by analyzing beat-to-beat distances of subsequent heartbeats (Fig. 7). It is assumed that the rhythmic changes of HR correlate with the respiratory sinus arrhythmia. For this kind of arrhythmia, the HR increases during inspiration and decreases during expiration. A combination of these video signal postprocessing strategies may provide new insight into perfusion dynamics.
Results from IRTI
Following approval from the Medical Ethics Committee of RWTH Aachen University Hospital (EK032/09 August 19, 2009), the development of temperature distribution in seven neonates inside an incubator was monitored for 10 min. The IRT setup [Fig. 8(a)] was calibrated taking into account environmental humidity, radiant heat sources of the incubator unit, and the optical parameters of the window inside the incubator. For the latter, the side door of the incubator was replaced by a thin polyethylene cling film, which seals the incubator’s microclimate from the environment but has several orders of magnitude higher IR transparency (0.92 to 0.94) than the usual acryl (Plexiglas) doors (). A compensation routine was performed with a reference measurement, which was repeated every 2 min.
In Fig. 8, the middle and right images are a typical freeze-frame image of an IRT sequence and the temperature development for the selected ROI. Even inside the incubator, slight differences in skin temperature can be detected. For this image the extremities are ca. 0.5°C cooler than the head and cooler than the abdomen due to good temperature insulation of the mattress and of the bent left arm.
It is interesting to take a closer look at the recordings of the infant’s nasal region. According to the breathing air flow, the surrounding skin is either cooled or heated by warm exhaled air or cooler inhaled air (Fig. 9). These temperature changes are clearly visible in the IRT recordings as long as the nasal region is in the field of view of the camera and the spatial and/or zooming ability of the camera offers sufficient image resolution of the small nasal region. Although temperature differences inside the incubator are relatively small compared to measurements on adults, with lower common environmental room temperatures of , fluctuations of 0.3°C are still visible. In the example presented in Fig. 9, the recorded RR was confirmed by additional neonatal bedside monitoring (IntelliVue MP30, Philips, Eindhoven, the Netherlands).
In all recordings, the distribution of skin temperature of all not-covered skin regions could be monitored clearly. As long as the nasal regions of the neonates were visible in the IRTI image section and the neonates were calmly lying in the incubator (no turning of head or heavy struggling movements), the RRs could be calculated from analyzing the nasal region for all seven neonates at least in sections.
Discussion and Future Trends for Hybrid Imaging
Both of these camera-based monitoring techniques have potential for continuous patient monitoring. They are particularly suited for neonatal care because the methods used are contactless, thus protecting sensitive skin. The feasibility result presented in this work demonstrated the high potential and beneficial effect of the steady and well-controlled environmental conditions (as in a neonatal incubator) for high-quality measurements.
PPGI and IRT complement each other by covering different vital parameters. PPGI allows one to extract HR, heart rate variability (HRV), and RR, as well as even slower rhythms around 0.1 Hz from recorded perfusion data by analyzing their frequency bands. IRT monitors the distribution and development of skin surface temperature. Also, by analyzing the nasal region, the respiration rhythm can be extracted from the changing skin temperature according to the alternating cooling and warming of skin due to exhaled and inhaled air.
Thus, a combination of both systems is desirable. To create such hybrid imaging, both camera systems must be synchronized for data recording and the lines of sight of the cameras must be defined separately. To achieve this, three basic geometric setups are differentiated (Fig. 10).
For the first setup, the positioning of the cameras can be chosen arbitrarily. Here, both cameras are usually focused on different body parts from different points of view. Therefore, it is unlikely that exactly the same body site will be selected for both recordings. However, for both systems this allows one to choose the most ideal perspective on the most relevant anatomic region separately. If selection of the same body site is required for both recordings, one of the following setups must be used instead.
The second setup is a stereoscopic-like positioning of the camera. Here, the cameras are placed close to each other with almost the same view on the object. In contrast to classical optical stereoscopy, two different camera types are used. Therefore, direct matching of identical image regions is not possible due to the different image content. Nevertheless, the two recordings are not totally different. For example, by applying block-matching algorithms on edge-filtered image data, corresponding points in both recordings can be detected. This allows a reconstruction of the recording geometrics,25,26 interpolation of the remaining image sections, and calculation of the object’s camera distance. This reconstruction process can be simplified and becomes less susceptible if the exact positioning of the camera is known or calibrated previously.
The third setup uses a beam splitter that allows measurements from the same perspective with the two systems. Though the distance information can no longer be calculated, the complex stereoscopic image processing is no longer required. A simple way of realizing a beam splitter is to use a polished glass plate. For PPGI in visible and near-infrared wavelengths, glass is almost perfectly transparent, and for IRT in the IR-C band it is sufficiently reflective. However, the glass plate and its temperature must be integrated in the IRT calibration routine. This arrangement requires simple postprocessing: first, image flipping to compensate for the mirroring effect at the glass plate and, second, image stretching and (if necessary) interpolation to compensate for different imaging properties and image resolution.
Together, both camera systems cover a broad range of monitorable vital parameters. Improvements can be achieved by utilizing redundancies, as in detection of respiration rhythms. Also, the systems are rather complementary in monitoring the effects of human thermoregulation. As shown above, PPGI allows one to detect 0.1-Hz rhythms in skin perfusion. Hence, this rhythm has its origin not in a central oscillator (like the heart or respiratory rhythm), but this effect is locally controlled. In Ref. 27 it is assigned to the vasomotor rhythms controlling heat distribution and heat transfer by rhythmically dilating and contracting different local segments of the vascular tree. It is assumed that, thereby, blood flow and its associated convective heat transfer to single body parts are regulated. In particular, this involves skin perfusion because here heat exchange with the environment as a central element in body temperature regulation takes place. Thus, a correlation between vasomotor rhythms and skin temperature can be assumed. For verification and further analysis of this phenomenon, the proposed hybrid monitoring technique is recommended. Separately, both techniques are well suited to monitor temperature or vasomotor rhythms, and both have the required spatial resolution to capture position-dependent effects. With this complementary usage of both camera systems, a new experimental access for interpretation of thermoregulatory effects is created.
Both PPGI and IRTI are contactless alternatives for long-term measurement of vital parameters. PPGI is able to detect parameters correlated with skin perfusion. Detected signals are modulated by the heartbeat allowing calculation of HR or beat-to-beat analysis, such as estimation of HRV. Besides these pulse-synchronous rhythms, the measurements contain other lower-frequency components. Analysis of the PPGI recordings and references from the literature indicate that respiratory rhythms are also extractable from the signal, as are even slower rhythms.
With IRT, skin temperature behavior, in general, can be recorded. The derived respiration rhythms can be monitored by analyzing temperature changes in the nasal region of the patient.
A combination of the two optical sensing systems (active PPGI and passive IRT) allows one to cover a broad range of vital parameters in the field of medical diagnostics and remote patient monitoring. Potential redundancies between the systems can be used to increase the stability and reliability of the obtained values. The unobtrusive, spatially resolved, and contactless technique allows a wide range of applications, even in case of highly sensitive body areas.
Nikolai Blanik received his Dipl.-Ing. degree in electrical engineering in 2010 from RWTH Aachen University, Aachen, Germany, where he is currently working toward his Dr.-Ing. (PhD) degree in the Philips Chair of Medical Information Technology. He also is a research assistant in Philips Chair of Medical Information Technology, RWTH Aachen University. His research interests include research of photon–tissue interaction with Monte-Carlo simulation and photoplethysmographic imaging.
Abbas K. Abbas received his MSc degree in biomedical engineering in 2004 from Al Nahrain University, Baghdad, Iraq. He is currently working toward his Dr.rer.medic. (PhD) degree in the Philips Chair of Medical Information Technology at RWTH Aachen University. His research interests include research of neonatal infrared thermography imaging and developing an intelligent monitoring solution for neonatal intensive care units.
Boudewijn Venema received his Dipl.-Ing. degree in electrical engineering in 2009 from RWTH Aachen University, Aachen, Germany, where he is currently working toward his Dr.-Ing. (PhD) degree in the Philips Chair of Medical Information Technology, where he is also a research assistant. His research interests include the study of photon–tissue interaction and long-term measurement of vital signs using photoplethysmography.
Vladimir Blazek received his Dipl.-Ing. degree in electrical engineering from the Technical University, Brno, Czech Republic, 1969, the Dr.-Ing. degree from RWTH Aachen University, Germany, 1979, and the Venia Legendi degree from the Czech Technical University, Prague, 1993. Since 2011, he is with the Philips Chair of Medical Information Technology, RWTH Aachen University. His research interests include optoelectronics in medicine, biomedical sensors, functional imaging techniques, and tissue optics.
Steffen Leonhardt received his MS degree in computer engineering from the State University of New York, Buffalo, the Dipl.-Ing. degree in electrical engineering and the Dr.-Ing. degree in control engineering both from the Technical University of Darmstadt, Germany, and the MD degree in medicine from J.W. Goethe University, Frankfurt, Germany. Since 2003, he has been a professor and head of the Chair of Medical Information Technology, RWTH Aachen University.