Translator Disclaimer
25 July 2019 Method for coregistration of optical measurements of breast tissue with histopathology: the importance of accounting for tissue deformations
Author Affiliations +

For the validation of optical diagnostic technologies, experimental results need to be benchmarked against the gold standard. Currently, the gold standard for tissue characterization is assessment of hematoxylin and eosin (H&E)-stained sections by a pathologist. When processing tissue into H&E sections, the shape of the tissue deforms with respect to the initial shape when it was optically measured. We demonstrate the importance of accounting for these tissue deformations when correlating optical measurement with routinely acquired histopathology. We propose a method to register the tissue in the H&E sections to the optical measurements, which corrects for these tissue deformations. We compare the registered H&E sections to H&E sections that were registered with an algorithm that does not account for tissue deformations by evaluating both the shape and the composition of the tissue and using microcomputer tomography data as an independent measure. The proposed method, which did account for tissue deformations, was more accurate than the method that did not account for tissue deformations. These results emphasize the need for a registration method that accounts for tissue deformations, such as the method presented in this study, which can aid in validating optical techniques for clinical use.



Optical technologies can potentially be used for tissue characterization in, for example, the detection of positive resection margins during cancer surgery. These technologies are based on the principle that light that has undergone interaction with the tissue contains information regarding the composition, morphology, and microvasculature of the tissue which can be used to discriminate different tissue types. Validation of these optical measurements is done by comparing the experimental results with the gold standard.1 Currently, the gold standard for tissue characterization is an assessment of microscopy sections, which are stained with hematoxylin and eosin (H&E), by a pathologist. However, the tissue in the H&E sections, used for histopathological assessment, is profoundly deformed compared to the appearance of the tissue during the optical measurements.24 These deformations are introduced when the tissue is processed into H&E sections as explained and shown in Fig. 1. Part of these deformations is related to tissue shrinkage due to formalin fixation and tissue compression and stretching when the tissue is embedded into paraffin blocks. Consequently, overlaying digitalized H&E sections over images that reflect optical measurements of the fresh, unfixed tissue is prone to registration errors.58

Fig. 1

Histopathological processing of the tissue. After arrival at the pathology department, the margins of the specimen are colored and the tissue is dissected into tissue slices (3  mm thick). (a) The white light image of the tissue slice that is selected for optical measurements. After these measurements, the slice is fixed in formalin and embedded in paraffin. (b) The white light image of the paraffin-embedded tissue slice. From this paraffin-embedded tissue slice, 3-μm thin sections are cut, which are stained with H&E stain. (c) The H&E section of the tissue slice, which is available a few days after surgery. Due to the histopathological processing of the tissue, (c) is deformed in comparison to (a).


In our research group, we use diffuse reflectance spectroscopy (DRS) for tissue characterization by means of a camera [hyperspectral imaging (HSI)] or fiber-optic probe (point measurements). With this optical technology, diffusely reflected light is measured after it has undergone multiple absorption and scattering events in the tissue. Therefore, a DRS measurement will contain information regarding the composition, morphology, and microvasculature of the tissue.

Registration errors between a DRS measurement and histopathology could negatively affect the assessment of the accuracy of this technology. The problem becomes even more relevant when developing classification algorithms, in which adding the correct histopathology label to each measurement is crucial. Accurate correlation between the optical measurements and histopathology is acknowledged as an important challenge encountered during clinical validation of optical technologies.1,9,10 However, there is only limited literature focusing on correlating optical measurements with histopathology,1113 and when a method for correlation the optical data to histopathology is described, the reported registration methods do not seem to correct for tissue deformation.1416

In this article, we demonstrate the importance of accounting for tissue deformations when correlating optical measurements with histopathology. For this, we use the histopathologic information that becomes available after the standard histopathologic protocol. No additional tissue was processed into H&E sections outside the standard sampling procedure to prevent impairing clinical diagnosis: embedded material might be needed in a later stage for clinical diagnostic purposes. In this study, we propose a registration method that considers tissue deformations parallel to the tissues’ surface and show that for an accurate correlation, tissue deformations should be taken into account when correlating optical measurements with histopathology.


Materials and Methods


Study Design

Data were obtained from fresh ex vivo breast tissue slices: the optical measurements (both hyperspectral images and fiber-optic probe measurements) and a white light image were acquired simultaneously. In addition, microcomputed tomography (μCT) data were obtained right after the optical measurements. Therefore, we assumed that the tissue did not deform between the acquisition of the optical measurements, the white light image, and the μCT data. As such, both the optical measurements and μCT data were registered to the white light images. A few days after the optical measurements, histopathological information (H&E section) was available. Our proposed method consists of correlating the H&E section to the white light image using a combination of affine and deformable registration. We compared the registration of our proposed registration method with the registration after using only an affine registration method, which does not account for tissue deformations. We quantified this accuracy using the μCT data as an independent modality. Finally, we compared the registered H&E sections using both registration methods with our optical measurements.


Ex Vivo Tissue Specimens

Breast specimens were obtained from patients undergoing breast surgery, either breast-conserving or mastectomy, at the Antoni van Leeuwenhoek Hospital. The study was approved by the Institutional Review Board of the Netherlands Cancer Institute/Antoni van Leeuwenhoek, and according to Dutch law (WMO) no written informed consent was required. The specimens were brought to the pathology department immediately after surgery. There, after coloring of the resection margins, the specimens were frozen and sliced in a bread-loafed manner. One tissue slice was placed in a macrocassette. Black rubber was placed underneath the tissue to ensure that measured spectra only originated from the tissue.


White Light Images

Prior to the DRS measurements, an overview white light image of the specimen in the macrocassette was acquired. This image was used as a reference since the tissue remained in the macrocassette throughout the measurements.


DRS Measurements


Hyperspectral imaging

Hyperspectral images were obtained with a push-broom HSI setup (VLNIR CL-350-N17E, Specim, Spectral Imaging Ltd., Finland) that captures light in the near-infrared (900 to 1700 nm, 256 wavelength bands, 5 nm increments) with an InGaAs sensor (320×256  pixels). The setup [shown in Fig. 2(a)] and measurement calibration are described in a prior publication.17 In short, the tissue is placed under the camera and imaged line-by-line through moving the imaged scene. Thereby a three-dimensional (3-D) hypercube is obtained that contains multiple two-dimensional (2-D) images of the tissue at different wavelengths, i.e., each pixel in the 2-D image contains a full diffuse reflectance spectrum [Fig. 2(b)]. Raw hyperspectral data obtained from the tissue were normalized to a diffuse reflectance percentage relative to Spectralon® (SRT-99-100, Labsphere, Inc., Northern Sutton, New Hampshire).

Fig. 2

Optical measurement setups. The tissue was measured with (a) the hyperspectral camera to (b) obtain a 3-D hypercube, in which each pixel contains a diffuse reflectance spectrum. Next, (c) the tissue was measured with the fiber-optic probe to obtain (d) a diffuse reflectance spectrum.



Fiber-optic probe

Fiber-optic point measurements were acquired with a measurement setup [shown in Fig. 2(b)] including a broadband light source, a spectrometer for the visual wavelength region (Andor Technology DU420A-BRDD), and a spectrometer for the near-infrared region (Andor Technology, DU492A-1.7). A fiber-optic probe was attached to the measurement setup to measure diffuse reflectance spectra between 400 and 1600 nm [Fig. 2(d)].18,19 The distance between the illuminating and collecting fiber at the tip of the probe was 1 mm. Before measuring, the setup was calibrated by acquiring a white reference measurement of Spectralon® (SRT-99-100, Labsphere, Inc., Northern Sutton, New Hampshire) in a calibration cap.

To acquire a DRS measurement, the probe was brought in contact with the tissue. During the probe measurements, a custom-made grid with holes [consisting of two pieces and shown in Fig. 2(c)] was placed on top of the tissue to gently fixate the tissue in the cassette and allow correlation between the measurement locations and H&E section afterward. The grid was 3-D printed and transparent. In addition, the grid stabilized the probe while ensuring contact between the probe and the tissue and restraining it from moving during measurements.


Microcomputed Tomography (μCT) Data

After the optical measurements, the grid was removed and the μCT data were acquired. During the optical measurements and μCT imaging, the tissue remained in the macrocassette and the black rubber was not removed. With the SkyScan® 1275B (Bruker, Kontich, Belgium), μCT data were acquired using either 40-kV source voltage and 250-μA source current, or 50-kV source voltage and 200-μA source current and a 1-mm aluminum filter. The exposure time varied between 39 and 105 ms, and one scan consisted of 1801 (0.2 deg increments) to 901 (0.4 deg increments) projections. Reconstruction of the data was performed with 30 to 50  μm resolution using SkyScan’s NRecon software.


H&E Sections

After measuring, the tissue slice was brought back to the pathology department and processed in H&E sections according to standard protocol. The side of the specimen used for the optical measurements was marked with methylene blue to ensure that the H&E section was taken from the same side. After digitalizing the H&E sections with the Aperio® ScanScope AT2 (Leica Biosystems, Wetzlar, Germany), tumor tissue was labeled in the H&E images by a pathologist. The remaining tissue in the H&E sections, fat tissue, and connective tissue, was labeled by thresholding the RGB channels of the H&E images since fat tissue is transparent and therefore white in H&E sections and connective tissue colors pink.


Registration of H&E Sections

The registration of H&E sections to the optical measurements required multiple steps, which are shown in Fig. 3. Prior to the registration, we confirmed that the H&E section was obtained from the measured surface of the tissue, in other words, the correct side, using the μCT data. All image registrations were made using MATLAB 2018a® (MathWorks, Natick, Massachusetts). To prevent possible bias, the H&E image without the tissue labels was used throughout the process. Subsequently, after finishing the registration, the same registration steps were applied to the H&E image with the tissue labels. First, the background of both the H&E image and the white light image was manually removed (Fig. 3, step 1). Second, the H&E image was rescaled using the shape of the tissue in both images such that the size of the tissue was matching the size of the tissue in the white light image (Fig. 3, step 2).

Fig. 3

The steps required to register the H&E image to the white light (WL) image.



Affine registration

To compensate for rotation between the H&E image and the white light image, an affine registration was made between the two images. This was accomplished by using the shape of the tissue on both images and cubic interpolation (Fig. 3, step 3). The registration error, which was minimized using the least squares nonlinear solver in MATLAB, was defined as the sum of the difference between the shape of the tissue on the white light image and the registered H&E image.


Affine + deformable registration

For the affine + deformable registration, an additional step was applied after the affine registration (Fig. 3, step 4). First, control points were manually selected in both the H&E image (moving image) and the overview white light image (fixed image). Thereby, each control point in the H&E image had a corresponding point in the white light image. Two persons manually selected control points independently to make two registrations. In addition, the control points of both observers were combined to form the basis of one mean registration.

Figure 4 shows the control points selected by observer 1 in one representative example. First, control points were placed on distinctive features in both the H&E image and the white light image [Figs. 4(c) and 4(d)]. After assigning points to these features, additional points were placed in large areas of the images without any control points. These additional control points included four points in the corners of the images and a number of points surrounding the tissue. Without the additional control points, the algorithm is unable to accurately perform the local weighted mean transformation. For the deformable registration, a nonrigid local weighted mean transformation with 12 and 24 neighboring control points for the individual and mean registration, respectively, was applied using the built-in MATLAB function “fitgeotrans.m.”

Fig. 4

Representative example of control points (green and blue dots) selected by observer 1 in (a) the H&E image and (b) the white light image. The yellow squares in (a) and (b) correspond to the magnified squares, (c) and (d).



Registration of optical measurements to white light images

The optical measurements were registered to the white light image to bring both histopathological and optical data into the same coordinate system. In the case of the hyperspectral images, a simple affine registration based on the shape of the tissue in both the hyperspectral image and white light image sufficed.

For the fiber-optic probe measurements, a more extensive registration was required. To retrieve the probe measurement locations in the overview white light image, we used (I) a schematic image of the grid, (II) a white light image of the tissue with the grid on top, and (III) the overview white light image (Fig. 5). First, the schematic image of the grid was registered to the white light image including the grid using a projective registration. This was done by aligning the measurement locations in the four corners of the grid in both images. Second, to account for small deformations to the tissue caused by the grid, the white light image including the grid was registered to the overview white light image using a deformable registration (similar to the registration method presented in Sec. 2.7.2). The calculated transformation matrices from both registrations were applied to the schematic image of the grid. With the final result (Fig. 5, IV), each measured probe location can be defined in relation to the overview white light image. All registrations were made using MATLAB 2018a® (MathWorks, Natick, Massachusetts).

Fig. 5

Measurement locations of the fiber-optic probe measurements. (a) A schematic image of the under grid (I) was registered to a white light image of the tissue with the grid on top (II) using a projective registration. (b) Subsequently, the white light image with the grid (II) was registered to the overview white light image (III) with a nonrigid (deformable) registration. The result after both registrations is the overview white light image with a projection of the measurement locations (IV).



Data Analysis

To quantify the registration performance after affine registration and affine + deformable registration, two measures were used. First, the shape of the tissue in the H&E image with respect to the shape of the tissue in the white light image was analyzed using the Dice similarity coefficient (DICE).20 DICE is a measure for comparing the similarity of two images by measuring the overlap of the shape of the contour. Second, the tissue composition based on the H&E image was compared with the tissue composition in the μCT data by using the point biserial correlation coefficient (rpb).21 This is a special case of the Pearson’s correlation coefficient (Pearson’s r) suitable for calculating the correlation between a continuously measured variable (μCT) and a dichotomous variable (H&E section). An absolute rpb of 1 indicates a perfect linear correlation, whereas an rpb of 0 indicates no correlation. In addition, we compared the composition of the tissue in the H&E image with the composition estimated with the optical measurements using the Pearson’s r. To compare the composition of the tissue in all modalities, we focused on discriminating fat from nonfat tissue since these tissue types can be well discriminated in all modalities.


Fat percentage in μCT data

Even though with μCT data no actual fat percentage could be obtained, a lower μCT gray value is related to a higher fat percentage.22 Since the 3-D μCT data will be correlated to 2-D data (the 2-D H&E image and the 2-D estimated fat percentage of the optical measurements), the 3-D data were reduced to a 2-D image. For this purpose, first, the height of the tissue in the 3-D μCT volume was determined to obtain the superficial cross-section. To prevent imaging artifacts at the transition of nontissue to tissue, the superficial cross-section was defined at 0.15 mm underneath the determined tissue height as shown in Fig. 6(a). Subsequently, a second deeper cross-section was defined at 1 mm underneath the first cross-section. Third, over this 1 mm distance, an average was calculated per pixel. Thereby, a 2-D image was obtained that represents the average μCT gray value over that 1-mm tissue thickness. An example of such a 2-D image is shown in Fig. 6(b).

Fig. 6

Determination of the average fat percentage within 1 mm of the surface of the tissue. (a) The side view of a μCT image with two red lines that indicate the superficial cross-section, located 0.15 mm underneath the tissue height, and the deeper cross-section, located 1 mm underneath the superficial cross-section. (b) The side view corresponds to the dotted line in the 2-D image that represents the average gray value of the tissue within the two cross-sections.



Fat percentage in H&E image

With the labels of the H&E image, the tissue in the H&E image was binary divided into fat and nonfat tissues in MATLAB. The latter group contained tumor, connective tissue, and normal glandular ducts. The entire binary image was compared to the 2-D image of the μCT data. In addition, for comparing the fat percentage in the H&E image with the estimated fat percentage from the optical measurements, the average fat percentage of the measurement locations of the probe was calculated. This percentage was the percentage of fat pixels within a circle (diameter=2  mm) around the centroid of the measurement location.


Fat percentage in optical measurements

Optical measurements were spectrally fitted with an analytical model that was derived from diffusion theory using a Levenberg–Marquardt nonlinear inversion algorithm in MATLAB. For the fiber-optic probe, each measurement location was fitted and for the hyperspectral measurements, each pixel in the image that contained tissue was fitted. For the optical probe, this analytical model was described elsewhere.18,19 For the hyperspectral camera, the analytical model used is given by23

Eq. (1)

where α=μsμs+μa and k=1+rd1rd. rd is the internal reflection coefficient for diffuse light and depends on the refractive index of the sample, which was calculated as nsample=nwater+0.14·[lipid]. Note that μs and μa are the scattering and absorption coefficients, respectively, and given by

Eq. (2)

μs=α(λ1197  nm)b,

Eq. (3)

where α is the reduced scattering at 1197 nm and b is the scatter power. Here, υWL is the fraction of water and lipid in the tissue and assumed to be 100% in the near-infrared wavelength region because water and lipid are the dominant absorbers in this wavelength region. Note that μa,water and μa,lipid are the absorption coefficients of water and lipid, respectively, and [lipid] and [water] correspond to the concentration of lipid and water, respectively.




Acquired Data

Optical measurements and μCT data were acquired of 13 specimens. One specimen was excluded from further analysis because the slice was not thick enough to obtain a reliable 2-D μCT image. Table 1 lists the characteristics of the tissue slices.

Table 1

Characteristics of included specimens.

SpecimenSize (mm) (height ∗ width)Slice thickness (mm) (depth)
162 ∗ 383.5
243 ∗ 322.5
360 ∗ 433.0
464 ∗ 466.6
550 ∗ 332.7
635 ∗ 282.8
748 ∗ 253.4
845 ∗ 463.8
945 ∗ 173.2
1046 ∗ 354.5
1147 ∗ 435.5
1260 ∗ 353.2


Registration of H&E Section


Shape of the tissue: DICE

Figure 7 shows an example of an H&E image after affine registration and after affine + deformable registration (mean registration of both observers) and the effect of the transformations. The affine registration only scaled and rotated the H&E section and hardly any deformations are observed between the striped H&E section before [Fig. 7(c)] and after affine registration [Fig. 7(d)]. The DICE was 0.93 for this registration. After affine + deformable mean registration [Fig. 7(e)], however, a large transformation can be observed which increased the DICE from 0.93 [Fig. 7(g)] to 0.97 [Fig. 7(h)].

Fig. 7

Example of one of the tissue slices after (a) affine registration and (b) affine + deformable registration. (c) The second column shows the shape of the original H&E section with a striped pattern, and (d), (e) its transformation after both registrations. (f) The third column shows the white light image, (g), (h) which was used to calculate the Dice coefficient. (i) The fourth column shows the average μCT intensity between the cross sections and (j), (k) the rpb coefficient with the H&E sections after both registrations. The arrow indicates a structure that is easy to distinguish on both the μCT and the H&E image that, respectively, did and did not match after affine and affine + deformable registration. The values between brackets in (g), (h), (j), and (k) correspond to the Dice and rpb coefficient, respectively.


Table 2 shows the DICE after both registrations, given as the average of all samples and the standard deviation (std). For all samples, this DICE was high after both registrations. On average, the highest DICE was obtained using the affine + deformable registration.

Table 2

Registration accuracy of the histopathology section using two registration methods; measured using Dice coefficient (DICE) and point biserial correlation (rpb). #cp stands for the mean±std of paired control points used.

Shape: DICEComposition: rpb
Affine registration0.910.030.360.20
Affine + deformable registration
 Mean (#cp=233±35)0.940.020.470.18
 Observer 1 (#cp=157±41)0.940.020.460.18
 Observer 2 (#cp=80±27)0.930.030.470.17


Composition of the tissue: rpb

In addition to the Dice coefficient, we used the rpb coefficient to compare the composition of the tissue in the H&E image with the composition of the tissue in the μCT data. As stated in Sec. 2.8, an absolute rpb of 1 indicates a perfect linear correlation, whereas an rpb of 0 indicates no correlation. Inspecting the structures in the tissue that are easy to distinguish [orange arrows in Figs. 7(i)7(k)] reveals that the composition of the tissue in the H&E section did not match the composition of the tissue in the μCT data after only affine registration. This was corrected for after registering the H&E section with both the affine + deformable algorithm [Fig. 7(k)]. The rpb coefficient increased from 0.24 to 0.53.

As can be seen in Table 2, averaged over all specimens, the rpb increased substantially from the affine registration to the affine + deformable registration. Furthermore, when comparing the affine registration with the affine + deformable registration, on average over all specimens, 23% of the pixels had a different label (fat versus nonfat). Of these pixels, before the deformable registration step, 49% were labeled as nonfat and 51% were labeled as fat.


Practical consequence of the proposed registration method: required time and operator dependence

The required time for registration depended on (1) the size of the tissue and (2) the number of distinctive features in both images when selecting the control points. On average, the time required for removing the background in both the H&E image and the white light image was 10  min. Rescaling the H&E image to the white light images and applying the affine registration was fast and took less than 1 min. The most time-consuming part of the registration was the selection of control points in both images. This took around 15 min per observer. The registration of the optical measurement to the white light images required less than 1 min for the hyperspectral images and 15  min for the fiber-optic probe measurements.

In this study, two observers, both experienced in registering H&E sections to white light images, selected control point pairs to perform the deformable registration. To assess the operator dependence of the affine + deformable registration method, the registrations of the two observers separately were compared. As shown in Table 2, the number of control point pairs varied between the registration of both observers and the mean registration (in which the control points of the two observers were combined). However, the differences between the mean registration, the registration of observer 1, and the registration of observer 2 are minimal for both the DICE and rpb.


Correlation with optical measurements

The fat percentage derived from the H&E sections was compared to the fat percentage derived from the optical measurements to assess the effect of accounting for tissue deformations in the correlation between histopathology and optical measurements. A clear distinction was made between “unaltered” and “altered” points. In the altered points, the averaged fat percentage derived from the H&E section varied with more than 20% before and after deformable registration. The unaltered points are the remaining points. In case of a perfect correlation between optical measurements and histopathological information, the fat percentages in both modalities would be equal and therefore the Pearson’s r will equal 1.

In the representative tissue slice shown Fig. 8, 7 out of 17 probe measurements are “altered” points (cyan circles). Of these seven locations, the fitted fat percentage of the optical measurements was plotted against the fat percentage according to the histopathology after affine registration [Fig. 8(d)] and after affine + deformable registration [Fig. 8(e)]. The fit results of both optical modalities have small confidence intervals and are similar. Figures 8(d) and 8(e) show that the linear correlation between the optical and the histopathological fat percentage improved substantially after adding the deformable registration for both the probe and the camera. The Pearson’s r improved from 0.08 to 0.93 and from 0.01 to 0.93, for the probe and the camera, respectively.

Fig. 8

Comparison of registered H&E sections with optical results in one representative specimen. The H&E section after (a) affine registration and (b) after affine + deformable registration is displayed. In (c), the fitted fat percentage derived from the HSI (whole image) and the probe measurements (colored circles) is shown. The color of the circles in (a) and (b) indicates that the ratio of fat and nonfat tissue between the two registration methods differed less than 20% (black) or more than 20% (cyan). The fat percentage obtained with the optical measurements [as demonstrated in (c)] of the cyan-colored locations is plotted against the fat percentage obtained after (d) affine registration and (e) affine + deformable registration. The error bars represent the standard deviation of the fitted fat percentages. The dotted lines represent the linear fitted line through the measurements.


Of all 234 locations that were measured with the fiber-optic probe, we compared the fat percentage of the optical measurements with the fat percentage according to the histopathology after affine registration and after affine + deformable registration (Table 3). We did this for the fitted fat percentages derived from both the probe and the hyperspectral measurements. Even though these fat percentages were not equal, they were similar and highly correlated with a Pearson’s r of 0.87.

Table 3

Comparison of the fat percentage according to the optical measurements and the histopathology after both registrations using the Pearson correlation coefficient (Pearson’s r). The altered points represent points in which the averaged fat percentage derived from the H&E section varied with more than 20% before and after deformable registration.

Unaltered pointsAffine registration0.900.86
Affine + deformable registration0.910.89
Altered pointsAffine registration0.360.36
Affine + deformable registration0.660.61

For the 186 “unaltered” points, the Pearson’s r is not different before and after adding the deformable registration. For these points, using only affine registration, the Pearson’s r was 0.90 and 0.86, for the probe and HSI data, respectively. Adding the deformable registration step produced a Pearson’s r of 0.91 for the probe measurements and 0.89 for HSI data. Thus, as the deformable registration resulted only in a slightly higher Pearson’s r, correcting for tissue deformations was considered not necessary for these measurement locations. For the 48 “altered” points, however, adding the deformable registration strongly enhanced the correlation between the optical measurements and histopathological information. The Pearson’s r increased from 0.36 to 0.61 for the HSI measurements and from 0.36 to 0.66 for the probe measurements. These findings confirm the importance of accounting for tissue deformations when correlating optical measurements with histopathology.



Coregistration between optical measurements and histopathology is extremely important to validate an optical diagnostic technology. Due to deformation of the tissue during histopathology processing, an accurate correlation between optical measurements and the gold standard requires registration of H&E images to the optical measurements. Especially in tissue with an inhomogeneous character, such as breast tissue, this correlation is crucial for validating the optical measurements with the current gold standard. In this paper, we highlight the importance of accounting for tissue deformations when correlating optical measurement with histopathology. As such, we present a methodology for making a registration between optical measurements and the gold standard using an affine + deformable registration method. In this process, the H&E images were registered to a white light image that was taken simultaneously to the optical measurements. This registration approach was compared to a simple registration method using only an affine registration that does not account of tissue deformations. The performance of both registration methods was quantified using a different independent imaging modality, μCT. The registrations were assessed by first, comparing the shape of the tissue by calculating the DICE for each registration, and second, by comparing the composition of the tissue by calculating the rpb. Although both registration methods have high Dice coefficients (>0.90), the affine + deformable registration provides the highest registration accuracy when evaluating both the shape (DICE) and the composition of the tissue (rpb).

H&E sections after affine registration and after affine + deformable registration were compared to μCT data to see which registration method results in the smallest registration errors between histopathology and optical measurements. We showed that accounting for tissue deformations is important for reducing the registration error between optical measurements and histopathology. It should be noted, however, that in all cases the H&E section remains a 2-D image of one cell layer thickness whereas the optical measurements represent the optical properties of a volume of several mm3. Therefore, differences in depth will not be expressed in the H&E section, although such differences will affect the optical measurements. This phenomenon can be seen in Fig. 6 and has also been reported by Ma et al., who showed that the tissue architecture may change between adjacent tissue sections.24 In addition, the thickness of the tissue within a tissue slice varies, as can be seen in Fig. 6. In practice, the pathology technician will keep slicing sections from the paraffin-embedded tissue until a section is obtained that encloses the complete shape of the tissue slice. Therefore, the H&E section does not necessarily represent exactly the surface of the tissue as it was measured with the optical techniques. To account for this, we compared the registered H&E sections with μCT data over a tissue thickness of 1 mm underneath the tissue’s surface. The authors realized that the fact that the H&E sections are only a cell layer thick and the height of the tissue differs over the surface are limitations in obtaining a fully accurate correlation between the H&E sections and the optical measurements. These circumstances also explain why a registration accuracy of 100% can never be obtained. However, as we use human material that is processed for diagnostic purposes, we were only able to obtain the histopathologic information that becomes available after the standard histopathologic protocol. Therefore, methods such as 3-D histology as described by Kartasalo et al. and Pichat et al. could not be implemented.25,26

A limitation of the affine + deformable registration method as currently described is the manual input that was required, which makes it time-consuming. In addition, the method can be operator-dependent as the observers have to manually place points in both the H&E image and the overview white light image based on distinctive features that they recognize in both images.

In this study, registrations were made by two observers, with the same experience in registering H&E images to white light images. We showed that, even though the number of selected control point pairs varied between both observers, the registration accuracy was similar (Table 2). Therefore, registering the white light image with the H&E section using only the input of one of the two observers would also have resulted in an accurate registration. However, care should be taken when observers are less experienced. Also, we did not investigate how the number of neighboring control points that were used in the local weighted mean transformation impacted the registration. Optimization might be possible by using more or less neighboring points.

To reduce processing time, Naranjo et al. presented several registration methods to register infrared images with H&E-stained sections, with the ultimate goal to automatically segment regions in the infrared data.11 However, in their study, H&E-stained sections needed to be registered to sections of the tissue samples that are only a few cell layers thick, whereas in our study the tissue samples are 3  mm thick. Furthermore, the total area of tissue that was registered to an H&E section in Naranjo’s research was much smaller than the surface that was registered in this study. Due to these differences, their automatic key point selection method, which was based on finding the centroids of cells, was not applicable to our data. In their study, the projective transformation is reported to provide the best registration. This generalization of an affine transformation is a much less complex registration method compared to the affine + deformable method as proposed in this research. The projective transformation was sufficient in their study probably because the tissue slices were much smaller and therefore less deformation was present.

As shown in Table 2 and Fig. 7, the rpb increased substantially from affine registration to affine + deformable registration. Also, when comparing the labels of the H&E after affine registration and the H&E section after affine + deformable registration, on average 23% of the pixels obtained a different label. This result is confirmed by comparing the Pearson’s r of only affine registration or affine + deformable registration of the altered points (Fig. 8 and Table 3). However, for the altered points, the Pearson’s r remained lower compared to the unaltered points. This might be explained by the fact that the altered points are more likely to be located at a tissue transition. Therefore, there is an increased chance that the estimated tissue composition is different from the composition optically measured. Nevertheless, using a different registration method leads to different labeling of a substantial part of the optical measurements. This potentially has a strong positive effect on the maximum classification performance that can be obtained with the optical techniques.

Although, the fit results of the two optical modalities used are not exactly the same, the linear correlation between the fat percentage derived from the optical measurements and the histopathological fat percentage improved for both optical modalities after adding the deformable registration method to the affine registration. Differences between the fit results of the optical techniques can be explained by a difference in the measurement configuration: the distance between the illuminating and collecting fiber determines the measured volume with the fiber-optic probe, whereas for the hyperspectral camera the volume is mainly depended upon the tissue properties. Therefore, both configurations might measure a different volume, resulting in a difference between the obtained fat percentages. Nevertheless, the linear correlation between the fat percentages derived from the probe and the camera measurements is high with a Pearson’s r of 0.87.

In this study, we used μCT as an independent 3-D modality to quantify both registration methods. Abe et al.27 have reported on using MRI for imaging of pathology slices and showed good image quality. In our study, we did attempt to quantify the registration methods with MRI using the mDIXON-Quant sequence. This sequence was chosen as it can make excellent discrimination between fat and nonfat tissues.2830 However, the resolution, although acquired with a relatively high-resolution voxel size of 0.6×0.6×0.9  mm, proved to be too low for quantifying the registration methods. In the study of Abe et al., an animal MRI with a higher magnetic field (9.4T) was available than the clinical human wide-bore 3T MRI available in this study. For this reason, in this study, MRI data could not be used for quantifying the coregistrations and was thus not included in the analysis.

A disadvantage of using only μCT data was that we could only discriminate fat from nonfat tissue. In which the latter included both connective tissue and tumor tissue. One of the reasons was the low contrast between tumor and healthy fibroglandular breast tissue in the μCT scans. This has been reported by previous research as well.31 Other studies indicate that this differentiation should be possible.22 However, in that specific study, biopsy specimens were imaged with a much higher resolution than the resolution that could be obtained in this study. Nevertheless, the μCT data in this study functioned as an independent measure to discern areas of fat and nonfat. The use of μCT was not intended to replace the current gold standard but allowed us to quantify the registration accuracy of H&E images.

It should be noted that this research was conducted mainly to improve the validation of optical measurements based on the current gold standard. As previously explained, the H&E section remains a 2-D image of one cell layer thickness and will, therefore, no matter which registration method is used, never completely represent the volume measured with the optical modalities. In addition, histopathology can also suffer from other potential flaws arising from factors such as intraobserver variation,3234 interobserver variation,35,36 under sampling,37,38 and specimen handling and reporting.3942 Nevertheless, there is no way to validate the technology without considering the current gold standard. Therefore, once established as a robust diagnostic technology, the true potential of optical measurements in clinical practice should be assessed with evaluation of clinical end-points like, for breast cancer surgery, a decrease in positive resection margins or decrease in excised specimen volumes.

In conclusion, for an accurate correlation between histopathological information and optical measurements, tissue deformations should be taken into account in the registration. The proposed registration method in this study, which does account for these tissue deformations, shows a better correlation with the optical measurements. Although it is labor intensive, adding the deformable registration step can aid in validating the optical measurements with the current gold standard.


The authors declare no potential conflicts of interest.


The authors gratefully acknowledge the financial support of this research by the Dutch Cancer Society (Grant No. KWF 10747). We also thank Bruker MicroCT for providing the microCT scanner. The authors thank N.N.Y. Janssen for help with acquiring μCT data, O. Ivashchenko for help in developing the registration algorithm and the NKI-AVL core Facility Molecular Pathology & Biobanking (CFMPB) for supplying NKI-AVL biobank material. We also thank all students that participated in this research for their time and effort, and all surgeons and nurses from the Department of Surgery and all pathologists and pathologist assistants from the Department of Pathology for their assistance in collecting specimens.



W. A. Wells et al., “Validation of novel optical imaging technologies: the pathologists’ view,” J. Biomed. Opt., 12 (5), 051801 (2007). JBOPFO 1083-3668 Google Scholar


B. Pritt et al., “The effect of tissue fixation and processing on breast cancer size,” Hum. Pathol., 36 756 –760 (2005). HPCQA4 0046-8177 Google Scholar


J. Unger et al., “Method for accurate registration of tissue autofluorescence imaging data with corresponding histology: a means for enhanced tumor margin assessment,” J. Biomed. Opt., 23 (1), 015001 (2018). JBOPFO 1083-3668 Google Scholar


E. McInnes, “Artefacts in histopathology,” Comp. Clin. Path., 13 100 –108 (2005). Google Scholar


W. K. Jerjes et al., “The future of medical diagnostics: review paper,” Head Neck Oncol., 3 38 (2011). Google Scholar


M. N. Gurcan et al., “Histopathological image analysis: a review,” IEEE Rev. Biomed. Eng., 2 147 –171 (2009). Google Scholar


V. Krishnaswamy et al., “Scanning in situ spectroscopy platform for imaging surgical breast tissue specimens,” Opt. Express, 21 (2), 2185 –2194 (2013). OPEXFF 1094-4087 Google Scholar


M. J. Piling et al., “Infrared spectral histopathology using haematoxylin and eosin (H&E) stained glass slides: a major step forward towards clinical translation,” Analyst, 142 1258 –1268 (2017). ANLYAG 0365-4885 Google Scholar


B. C. Wilson, M. Jermyn and F. Leblond, “Challenges and opportunities in clinical translation of biomedical optical spectroscopy and imaging,” J. Biomed. Opt., 23 (3), 030901 (2018). JBOPFO 1083-3668 Google Scholar


G. Lu et al., “Framework for hyperspectral image processing and quantification for cancer detection during animal tumor surgery,” J. Biomed. Opt., 20 (12), 126012 (2015). JBOPFO 1083-3668 Google Scholar


V. Naranjo et al., “Stained and infrared image registration as first step for cancer detection,” in IEEE-EMBS Int. Conf. Biomed. and Health Inf. (BHI), 420 –423 (2014). Google Scholar


M. Halicek et al., “Deformable registration of histological cancer margins to gross hyperspectral images using demons,” Proc. SPIE, 10581 105810N (2018). Google Scholar


G. Lu et al., “Hyperspectral imaging for cancer surgical margin delineation: registration of hyperspectral and histological images,” Proc. SPIE, 9036 90360S (2014). Google Scholar


J. E. Phipps et al., “Automated detection of breast cancer in resected specimens with fluorescence lifetime imaging,” Phys. Med. Biol., 63 015003 (2018). PHMBA7 0031-9155 Google Scholar


A. M. Laughney et al., “Automated classification of breast pathology using local measures of broadband reflectance,” J. Biomed. Opt., 15 (6), 066019 (2010). JBOPFO 1083-3668 Google Scholar


A. M. Laughney et al., “Scatter Spectroscopic imaging distinguishes between breast pathologies in tissues relevant to surgical margin assessment,” Clin. Cancer Res., 18 6315 –6325 (2012). Google Scholar


E. Kho et al., “Hyperspectral imaging for resection margin assessment during surgery,” Clin. Cancer Res., 25 (12), 3572 –3580 (2019). Google Scholar


R. Nachabé et al., “Estimation of lipid and water concentrations in scattering media with diffuse optical spectroscopy from 900 to 1600 nm,” J. Biomed. Opt., 15 (3), 037015 (2010). JBOPFO 1083-3668 Google Scholar


R. Nachabé et al., “Estimation of biological chromophores using diffuse optical spectroscopy: benefit of extending the UV-VIS wavelength range to include 1000 to 1600 nm,” Biomed. Opt. Express, 1 (5), 1432 –1442 (2010). OPEXFF 1094-4087 Google Scholar


L. R. Dice, “Measures of the amount of ecologic association between species,” Ecology, 26 (3), 297 –302 (1945). ECGYAQ 0094-6621 Google Scholar


G. V. Glass and K. D. Hopkins, Statistical Methods in Education and Psychology, 3rd Ed.Allyn & Bacon, Needham Heights, Massachusetts (1995). Google Scholar


H. Gufler et al., “Fine structure of breast tissue on micro computed tomography: a feasibility study,” Acad. Radiol., 18 230 –234 (2011). Google Scholar


S. T. Flock et al., “Monte Carlo modeling of light propagation in highly scattering tissues. I. Model predictions and comparison with diffusion theory,” IEEE Trans. Biomed. Eng., 36 (12), 1162 –1168 (1989). IEBEAX 0018-9294 Google Scholar


Z. Ma et al., “Data integration from pathology slides for quantitative imaging of multiple cell types within the tumor immune cell infiltrate,” Diagn. Pathol., 12 (1), 69 (2017). DMPAES 1052-9551 Google Scholar


K. Kartasalo et al., “Comparative analysis of tissue reconstruction algorithms for 3-D histology,” Bioinformatics, 34 (14), 3013 –3021 (2018). BOINFP 1367-4803 Google Scholar


J. Pichat et al., “A survey of methods for 3D histology reconstruction,” Med. Image Anal., 46 73 –105 (2018). Google Scholar


H. Abe et al., “Comparing post-operative human breast specimen radiograph and MRI in lesion margin and volume assessment,” J. Appl. Clin. Med. Phys., 13 (6), 267 –276 (2012). Google Scholar


A. Mashhood et al., “Reproducibility of hepatic fat fraction measurement by magnetic resonance imaging,” J. Magn. Reson. Imaging, 37 1359 –1370 (2013). Google Scholar


Y. Kise et al., “Clinical usefulness of the mDIXON Quant method for estimation of the salivary gland fat fraction: comparison with MR spectroscopy,” Br. J. Radiol., 90 20160704 (2017). BJRAAP 0007-1285 Google Scholar


G. M. Kukuk et al., “Comparison between modified Dixon MRI techniques, MR spectroscopic relaxometry, and different histologic quantification methods in the assessment of hepatic steatosis,” Eur. Radiol., 25 2869 –2879 (2015). Google Scholar


S.-Q. Qiu et al., “Micro-computed tomography (micro-CT) for intraoperative surgical margin assessment of breast cancer: a feasibility study in breast conserving surgery,” Eur. J. Surg. Oncol., 44 (11), 1708 –1713 (2018). Google Scholar


S. L. Jackson et al., “Diagnostic reproducibility: what happens when the same pathologist interprets the same breast biopsy specimen at two points in time?,” Ann. Surg. Oncol., 24 1234 –1241 (2017). Google Scholar


P. Boiesen et al., “Histologic grading in breast cancer: reproducibility between seven pathologic departments,” Acta Oncol., 39 (1), 41 –45 (2000). Google Scholar


S. J. Schnitt, “Dilemmas in breast disease. Evaluation of microscopic margins in patients with invasive breast cancer: technical and interpretive considerations,” Breast J., 4 (3), 204 –208 (1998). BRJOFK 1075-122X Google Scholar


K. H. Allison et al., “Understanding diagnostic variability in breast pathology: lessons learned from an expert consensus review panel,” Histopathology, 65 (2), 240 –251 (2014). HISTDD 1365-2559 Google Scholar


J. G. Elmore et al., “Diagnostic concordance among pathologists interpreting breast biopsy specimens,” J. Am. Med. Assoc., 313 (11), 1122 –1132 (2015). Google Scholar


G. M. Clarke et al., “Whole-mount pathology of breast lumpectomy specimens improves detection of tumour margins and focality,” Histopathology, 69 35 –44 (2016). HISTDD 1365-2559 Google Scholar


N. Houssami and M. Morrow, “Margins in breast conservation: a clinician’s perspective and what the literature tells us,” J. Surg. Oncol., 110 2 –7 (2014). JSONAU 0022-4790 Google Scholar


S. Persing et al., “Variability in the quality of pathology reporting of margin status following breast cancer surgery,” Ann. Surg. Oncol., 18 (11), 3061 –3065 (2011). Google Scholar


R. Emmadi and E. L. Wiley, “Evaluation of resection margins in breast conservation therapy: the pathology perspective-past, present and future,” J. Surg. Oncol., 2012 180259 (2012). JSONAU 0022-4790 Google Scholar


S. J. Schnitt and J. L. Connolly, “Processing and evaluation of breast excision specimens: a clinically oriented approach,” Anat. Pathol., 98 (1), 125 –137 (1992). Google Scholar


S. P. Povoski et al., “Standardized and reproducible methodology for the comprehensive and systematic assessment of surgical resection margins during breast-conserving surgery for invasive breast cancer,” BMC Cancer, 9 254 (2009). BCMACL 1471-2407 Google Scholar


Lisanne L. de Boer achieved her MSc degree in technical medicine in 2013 from the University of Twente. Recently, she finished her PhD at the Netherlands Cancer Institute—Antoni van Leeuwenhoek Hospital, Amsterdam, The Netherlands. This project was focused on the use of fiberoptic DRS for detecting positive resection margins during surgery to improve the surgical treatment of breast cancer.

Esther Kho is a PhD candidate at the Netherlands Cancer Institute, The Netherlands. She received her MS degree in biomedical engineering from the University of Groningen in 2014. Her research involves the development, testing, and clinical evaluation of a hyperspectral imaging system, specifically for the assessment of tumor-positive resection margins during breast-conserving surgery.

Jasper Nijkamp is a senior postdoc image-guided oncology at the Netherlands Cancer Institute. He obtained his PhD in 2012 studying the uncertainties in radiotherapy for rectal cancer. Subsequently, he moved to image-guided surgery, where he designed and developed a surgical navigation system for mobile tumors. The navigation system is currently evaluated in pelvic lymph node dissections, colorectal, hepatic, and head and neck surgery. Jasper also works on surgical margin evaluation in breast cancer using microCT.

Koen K. Van de Vijver is a surgical pathologist with interest in diagnosis and treatment of cancer of the female genital tract and the breast. He received his PhD in 2007 at Antwerp University and Leiden UMC. In 2008, he became an assistant professor at Maastricht University. After working as a consultant pathologist in the NKI-AvL, he became a professor of gynaecological pathology at Ghent University Hospital in 2018.

Henricus J. C. M. Sterenborg has been active in biomedical optics research since 1987 as a staff member of the Medical Laser Centre of the Amsterdam UMC. In 1998, he (co) founded the Centre for Optical Diagnostics and Therapy at the Erasmus Medical Center and in 2008 he became a professor of photodynamic therapy. Since 2013, he has held a joint position at the Department of Biomedical Engineering and Physics at the Amsterdam UMC and the Department of Surgery at the NKI-AvL.

Leon C. ter Beek is an MR physicist at the Netherlands Cancer Institute since 2014. Before he worked as a research scientist at GE Healthcare in Japan and Philips Healthcare in the Benelux, collaborating with Academic Hospitals in the field of MR imaging and spectroscopy. He holds a PhD in chemistry on NMR spectroscopy and liquid crystals from the University of British Columbia in Vancouver, Canada.

Theo J. M. Ruers is an oncological surgeon and head of the Department of Surgery at the Netherlands Cancer Institute. In 2010, he became a professor at the Faculty of Science and Technology at the University of Twente. He has extensive experience in oncologic surgery and the implementation of techniques in cancer care. Special focus is on using optical (imaging) techniques for diagnosis and treatment of cancer.

© The Authors. Published by SPIE under a Creative Commons Attribution 4.0 Unported License. Distribution or reproduction of this work in whole or in part requires full attribution of the original publication, including its DOI.
Lisanne L. de Boer, Esther Kho, Jasper Nijkamp, Koen K. Van de Vijver, Henricus J. C. M. Sterenborg, Leon C. ter Beek, and Theo J. M. Ruers "Method for coregistration of optical measurements of breast tissue with histopathology: the importance of accounting for tissue deformations," Journal of Biomedical Optics 24(7), 075002 (25 July 2019).
Received: 29 March 2019; Accepted: 9 July 2019; Published: 25 July 2019

Back to Top