Significance: Mueller matrix (MM) polarimetry is a promising tool for the detection of skin cancer. Polarimetric in vivo measurements often suffer from misalignment of the polarimetric images due to motion, which can lead to false results.
Aim: We aim to provide an easy-to-implement polarimetric image data registration method to ensure proper image alignment.
Approach: A feature-based image registration is implemented for an MM polarimeter for phantom and in vivo human skin measurements.
Results: We show that the keypoint-based registration of polarimetric images is necessary for in vivo skin polarimetry to ensure reliable results. Further, we deliver an efficient semiautomated method for the registration of polarimetric images.
Conclusions: Image registration for in vivo polarimetry of human skin is required for improved diagnostics and can be efficiently enhanced with a keypoint-based approach.
Imaging of the human skin with polarimetric techniques has gained importance in recent years, as the incidence of melanoma, which is the deadliest type of skin cancer, is rising throughout all skin types.1 Beyond the classical skin cancer screening done by a dermatologist with a dermoscope,2 new modalities are being implemented for the early detection of melanoma: polarimetric imaging devices such as Stokes3 or Mueller polarimeters4,5 and machine learning-based computer-aided diagnosis systems6 are among the recently reported approaches. Furthermore, the early detection of melanoma is increasingly profiting from automated registration and analysis.7 In addition, adaptive optics such as adjustable liquid lenses allow for the implementation of autofocus in imaging systems without moving parts.8 However, because the up to 36 images necessary for Mueller matrix (MM) determination9 are typically obtained in acquisition times of 20 to 30 s, depending on the type of the setup,10 unintended body movements of the patient can become a significant limitation. Patient motion can lead to misalignment of the polarimetric images.11 Further, it can induce motion blur, in particular because laser power levels have to be reduced due to laser safety and therefore relatively high exposure times are needed to acquire the MMs. In addition, the motion can add up in cases in which a summation process of the polarimetric data is needed to enhance the signal-to-noise ratio (SNR).11 For example, the registration of scanned polarimetric images for ophthalmology was investigated by Nourrit et al.11 The above-mentioned problems in the MM acquisition can be partly overcome by hardware upgrades, e.g., camera and lens upgrades. However, such upgrades typically rely on complex and costly components.12 Therefore, software solutions represent an alternative approach and promise improved results.
In this work, we investigate the need and potential benefits of registering polarimetric images for application in vivo for dermoscopy. Registration of such data is particularly complex because the skin rarely shows sufficient key points with strong contrast (except for mole evaluation13). In addition, the contrast of key points usually changes considerably between the polarization states of the lighting and the analyzer. This issue and the possibility of key points moving out of the field of view make it difficult to detect the same set of key points within the up to 36 images typically acquired in MM polarimetry (MMP). To evaluate the potential benefits of this method, we determine the MM and its polarimetric parameters with and without the polarimetric image registration for the case of a printed melanoma phantom, healthy skin with a drop of honey, and a benign nevus, respectively. The instrumentation and registration schemes are described in Sec. 2. The results of registration and polarimetric analysis are presented in Sec. 3. Section 4 concludes the study.
Instrumentation and Experimental Procedure
With MMP, the information about the polarization changing properties of a sample can be obtained.14 The MM combines the complete polarization properties of a sample within a matrix.15 The sample is illuminated at different polarization states. The MM can be calculated from the intensity of the reflected or transmitted light.16 For in vivo skin imaging, only the reflection modality is relevant. Mathematically, the MM is a transformation matrix for Stokes vectors that describe the polarization state of light. The polarization state of the light after interaction with a sample is calculated from the Stokes vector of the incident light and the MM entries . of the sample as17
These polarization states are described using Stokes vectors18 as follows:
The indices of the intensity values stand for the polarization states as shown in Table 1.
Explanation of indices and the correlated polarization states.
The Stokes vector changes after the interaction of incident light with the sample, which is recorded to measure the MM. For the used setup, either 16 (needed states: , , , and ) or 36 (needed states: , , , , , and ) different images are acquired.8 Although the acquisition of the MM based on 16 images is faster and therefore results in less motion error, the usage of 36 images improves the SNR. More detailed descriptions of the formalism can be found in the literature.17 To reduce calibration measurement errors, in this work, we use six polarization states for the acquisition of the MM from the required 36 intensity measurements to increase the measurement accuracy and the SNR in comparison with the calculation of the MM from 16 images only.19 The calibration of the polarimeter is tested by measuring samples of known MMs such as commercially available polarizers, retarders, and diffusors as well as air. In general, the calibration of our polarimetric device involves two main steps. First, the relative orientation of the optical elements (i.e., linear polarizers and liquid crystal retarders) is set. Second, the physical behavior of the optical elements (i.e., the phase-shift of the liquid crystal retarders) needs to be calibrated by employing quarter- and half-wave-plates. The detailed calibration steps can be found in the literature.20 The results of the calibration procedures performed in this work are very consistent with the expected matrices. An increase in measurement time to improve the SNR needs to be carefully considered, especially in vivo measurements in which movement disorder can change the result.
The experimentally obtained MM entries do not show a direct relation to physical properties of the sample. Therefore, for a more detailed interpretation of the measurements, a polar decomposition is usually carried out. In this work, we use the common polar decomposition of Lu and Chipman21 in the pixel-by-pixel way with the rows of the pixel array indexed as and the columns indexed as :
In this decomposition is the experimentally obtained MM, and , , and represent the pure depolarizer, retarder, and diattenuator properties, respectively.21
The key parameters resulting from the polar decomposition are , the depolarization power; , the total retardance; , the diattenuation; and , the total polarizance. These physical properties of the sample are calculated from the following equations:21
The distance sensor (DT35-B15851, Sick AG, Waldkirch, Germany) measures the distance from a point close to (but outside) the polarimeter’s field of view to avoid interference with its illumination, as visualized in Fig. 2.
With the liquid lens, it is possible to adjust the focus in real-time based on the distance information provided by the distance sensor and on a function derived from the calibration procedure, which relies on the collection of value pairs of working distance and the liquid lens current. In detail, the liquid lens current is adjusted for different working distances until the imaged plane object is in maximum focus.23 The automatic focus works well for most skin parts within a theoretical working distance range of 30 cm to 8 m.8 However, the working distance is limited by the illumination intensity (and therefore noise) and the desired image resolution. With the automatic focus, it is possible to fully open the imaging aperture and accept the reduced depth of field, as larger openings reduce the acquisition time due to shorter possible exposure times.
To minimize patient motion, we designed a mobile MM acquisition system based on an ergonomic arm (Ergotron LX ARM 45-241-026, Ergotron, St. Paul, Minnesota) that allows patients to lie on a patient bed. Figure 3 shows a CAD design of the complete measurement system.
The mobile MM acquisition system allows the operator to image essentially all parts of the human skin. For better visualization, the appropriate laser safety measures are not shown in this representation.
Polarimetric datasets are obtained with the MMP only under unintentional movement (e.g., breathing and shaking). In the following, we describe the different image registration techniques used in this work to correct for the misalignment caused by that motion. First, registration assessment is conducted on the basis of subjective criteria. In addition, an objective criterion is applied with the comparison of the numerical values of each polarimetric parameter, respectively.
Image Registration of Polarimetric Images
Image registration is the process of aligning two or more images of the same scene.24 It is often used as a precondition for other image processing applications.25 Determining an effective approach to image registration depends on the application case. Careful selection of a point transformation model is required to provide reference points between the images. In addition, a method for comparing information to identify the parameters necessary to correctly align images is needed. In general, there are two well-known methods for automatic image registration: feature-based and intensity-based registration algorithms.26 By contrast, manual image registration relies on control point mapping registration algorithms.27
Local features and their descriptors constitute the basis for many computer vision algorithms.28 Their applications include image registration,29 object detection30 and classification,31 tracking,32 motion estimation,33 and content-based image retrieval.34 Local features refer to a pattern or distinct structure that exists in an image, such as a point, edge, or small patch of the image. They are usually associated with an image patch that differs from its immediate environment in texture, color or intensity. Although the distinction from its environment is important, the representation of the content of the characteristic is usually not relevant. Examples of local features are blobs, corners, and edge pixels.24 Due to the lack of matching points in skin images, feature matching is not a reliable method for batch image processing alone.
The Image Processing Toolbox™ by MATLAB (MATLAB, 2021. version 9.11.0 (R2021b), Natick, Massachusetts: The MathWorks Inc.) provides tools for point mapping to determine the required transformation parameters to align images. In point mapping, the user selects points in a pair of images that identify the same reference. Then a geometric mapping is deduced from the positions of these control points.24 In selecting control points, a high level of accuracy is required. To align the target image, we need to select at least two pairs of matching points between the target image and the reference image. More pairs of matching points improve the registration result. This increases the processing time for batch image processing.
We propose a semiautomated key point-based registration method for polarimetric images to take into account the inevitable movement during acquisition. We combine the registration of both feature-based (automatic) and control point mapping (manual). Figure 4 shows a visual representation of this process.
The process starts with an algorithm based on KAZE35 features. In the case of insufficient pairs of matching points, it will switch to the manual mapping algorithm of the control points for this particular image. The Computer Vision Toolbox™ provides several methods for detecting corner and blob features and includes various descriptors. In this method, feature detection selects regions of an image with unique content, such as blobs. The feature detection finds possible points for subsequent processing that do not necessarily refer to physical structures. Finding features that remain locally invariant so that they are identifiable even in the presence of rotation or scaling is essential to feature detection.
Feature extraction involves the calculation of a descriptor, which is typically done on regions centered around detected features. By the means of image processing, descriptors transform a local pixel neighborhood into a compact vector representation. This new representation allows for comparison between neighborhoods, regardless of changes in scale or direction. We chose the best feature detector and descriptor from the perspective of our application criteria and the nature of our data by testing the available feature detector and descriptor combinations. Table 2 shows a comparison of the number of matching pairs between two images for different blob detectors and descriptors.
Comparison of the number of matching pairs between two images for different blob detector and descriptor combinations. We exemplarily show the values for two images of the polarimetric raw data of a nevus (compare with Fig. 15) and a honey drop on healthy skin (compare with Fig. 13).
Note: The detector and descriptor combinations employed in this article are highlighted in bold.
If there are sufficient valid matches, the false matches are removed by selecting only the strongest among them. This work is based on the M-estimator sample consensus (MSAC),36 a variant of the random sample consensus (RANSAC)37 algorithm. This method finds a geometric transform, separates the correct matches from the spurious matches, and then applies the computed geometric transform to the image being compared.36 For this method, a minimum of two matching pairs is required. If the requirement is not met, then it allows the operator to manually select the two matching pairs between the references and the particular target image. The manual selection of control points is shown in Fig. 5.
After alignment of all images, the MM is calculated. However, the edges of the spatially resolved MM entries are usually not valid and show distracting extreme values. To achieve results without distracting edges, we crop all images to the reference overlap zone of all aligned images or the area of interest.
We implement our approach on healthy skin with hair and a mole with surrounding skin, respectively. In addition, we apply a tiny drop of honey to the healthy skin as a phantom for small skin patches with varying optical activity.38 Aligning these samples is particularly important due to their strong contrast resulting from the hair and the nevus. The clinical and dermoscopic images serve as reference for the polarimetric skin imaging.
Registration of Polarimetric Data for In Vivo Skin Imaging
First, we show that in polarimetric data the visibility of features in the skin is usually polarization sensitive, as shown in Fig. 7.
From the comparison of the two images from the polarimetric raw data in Fig. 7, it is obvious that the visibility of features in human skin can depend on the combination of the state of the polarization state generator and the PSA.
In the following, we demonstrate that movement during data acquisition can lead to false MMs and derived polarimetric parameters. To do this, we use an icon as shown in Fig. 8 printed on paper as a lesion phantom.
The phantom design is based on the ABCD criteria for melanoma diagnosis. It is asymmetric and has irregular borders and a diameter . The phantom is moved throughout the acquisition. The translational and rotational movement within the focal plane, realized by manually moving the target on a rail with one additional rotational degree of freedom throughout the acquisition, leads to an error of alignment as shown in Fig. 9.
In the following the spatially resolved polarimetric parameters retardance, polarizance, depolarization power, and diattenuation are calculated. In the remainder of this work, we refer to our feature-based registration method as alignment (a) and the standard intensity-based image registration method as alignment (b). As displayed in Fig. 10, the results differ largely depending on whether the polarimetric data were registered or not and which registration method was applied.
The previous comparison shows that registration has two immediate positive effects. First, image alignment leads to an improved image quality, as the features of the phantom are more clearly visible. Second, alignment corrects spatially resolved polarimetric parameters. The trend of retardance, polarization, and diattenuation of the phantom is reversed if the images are not aligned. The aligned and cropped data parameters indicate that the alignment has been achieved, but that the values on the phantom edges are still different compared with the static reference sample, as seen in the polarizance panels in Fig. 10, third and fourth rows. As shown in Fig. 10, it is also obvious that the polarimetric parameters from the raw data that have been registered with the feature-based method shows a greater agreement with the polarimetric data from the data at rest (ground truth) compared with the application of the standard intensity-based method. Further, the feature-based method is less prone to artifacts on the edges of the phantom.
As shown in Table 3, it is obvious that the feature-based registration leads to results that are closest to the ground truth for , , and . For , both alignment methods give a similar numerical value that does not match with the ground truth.
Comparison of the numerical values of the determined polarimetric parameters from the center of the phantom.
Note: Significant values are highlighted in bold.
Next, a polarimetric in vivo measurement of a honey drop on healthy skin of a human arm with hair, as shown in Fig. 11, is carried out.
The diameter of the honey drop is . Figure 12 shows the degree of unintentional motion during data collection by comparing the position of detected features in two images of the polarimetric data.
Figure 12 shows the degree of unintentional motion and the need for registration of the polarimetric data. The spatially resolved polarimetric parameters for a drop of honey on healthy skin with hair are displayed in Fig. 13 for the raw data as well as after feature-based and intensity-based alignment.
Feature-based registration of polarimetric images has an immediate effect on spatially resolved polarimetric parameters. After registration, the hair can be distinguished from the skin, and the range of values has changed. In addition, the tiny drop of honey is most distinguishable from the surrounding skin in the case of feature-based registration, especially in the -patch, showing that such polarimetric changes in small skin patches can only be restored with a suitable image registration method.
Subsequently, the influence of in-focal plane motion on the polarimetric parameters for in vivo nevus assessment was investigated. An in vivo measurement with our MMP is performed on a benign nevus of a volunteer. The degree of misalignment is obvious from Fig. 14, as the matching feature points are detected at slightly different positions.
It is clear from Fig. 14 that the features of the nevus do not overlap in the polarimetric images due to motion and that the polarimetric data must therefore be registered. In Fig. 15, the polarimetric parameters of a nevus of unaligned and aligned data are shown.
The feature-based registration of polarimetric data allows for the in vivo assessment of nevi. By contrast, unaligned or intensity-based aligned polarimetric data lead to artifacts in the polarimetric parameters that interfere with dermatological interpretation. The inconsistencies of polarimetric parameters at the edge of the nevus, as obvious from the retardance patch in the second row in Fig. 15, suggest that the image registration can be further improved. It is necessary to examine, in future work, whether the inconsistencies at the edges of the nevus are caused by the occurrence of biological and structural changes or by a slight residual misalignment.
Autofocus for Polarimetric In Vivo Skin Imaging
Finally, we investigated the effect of movement in the direction of the optical axis of the polarimetric system and the associated changes in the polarimetric parameters. For this, the focal plane was changed during the acquisition of the polarimetric parameters displayed in Fig. 16.
In comparison with Fig. 10, first row, it is clear that the change of the focus plane during acquisition leads to a bad focus on some of the images of the polarimetric data, which results in artifacts on the edges of the phantom. Consequently, an automatic focus system is required to take into account the skin movement in the direction of the optical axis during acquisition.
We implemented a semiautomatic feature-based registration method and studied the impact of different registration methods on polarimetric analysis. In addition, we found mapping functions that are suitable to reliably detecting the corresponding skin features as key points in polarimetric data. The body movement can induce motion blur because of relatively high exposure times. Furthermore, the movement of the skin in the 36 images acquired leads to false polarimetric parameters, preventing reliable diagnosis. In rare cases, some key points leave the field of view due to movement. To avoid false MM results, the characteristics of the 36 images are aligned using image processing techniques. In addition, the MMP is mounted on a movable arm that enables the system to be placed easily in the skin area studied while the patient movement is minimized with the patient bed setting. The results indicate that our methodology is a good solution to the problem. Further research into the feature detection technique is needed to enable a fully automated and reliable registration of in vivo skin polarimetry data. The results emphasize the importance of adequate image registration techniques for in vivo skin polarimetry. Thus, equipped with image registration, MM polarimetry could be a valuable asset for dermatology. The performance of image registration is most important when the skin area of interest is particularly small, as characteristics of polarimetric activity can be lost without data registration. Our proposed method is suitable for the alignment of polarimetric images overcoming the problems of intensity and threshold matching. In the future, we will use the approach for in vivo measurements of inflammatory skin diseases and melanoma skin cancer.
This research was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany’s Excellence Strategy within the Cluster of Excellence PhoenixD (EXC 2122, Project ID 390833453). This work has been supported by iToBoS (Intelligent Total Body Scanner for Early Detection of Melanoma), project funded by the European Union’s Horizon 2020 research and innovation program, under grant agreement no. 965221
Lennart Jütte received his BS and MS degrees in nanotechnology from Leibniz University Hannover in 2019 and 2021, respectively, where he is currently pursuing his PhD in applied optics at the Hannover Centre for Optical Technologies. His current research interests include dermoscopy, biophotonics, and polarimetry.
Bernhard Roth received his PhD in atomic and particle physics from the University of Bielefeld, Bielefeld, Germany, in 2001 and his State Doctorate (Habilitation) degree in experimental quantum optics from the University of Duesseldorf, Duesseldorf, Germany, in 2007. From 2002 to 2007, he was a Research Group Leader at the University of Duesseldorf. Since 2012, he has been the scientific and managing director at the Hanover Centre for Optical Technologies, and since 2014, he has been a professor of physics at Leibniz University Hanover, Hanover, Germany,. His scientific activities include applied and fundamental research in laser development and spectroscopy, polymer optical sensing, micro- and nanooptics fabrication, and optical technology for illumination, information technology, and the life sciences.