2.1.

Benchtop Prototype

Our benchtop PSE system was constructed with four light sources mounted around a camera with a fish-eye lens [Fig. 1(b)]. The size of the housing was $30\times 30{\mathrm{mm}}^2$, and the four sources were oriented at equal angles about a circle with a 35 mm diameter. We used a dragonfly®2 remote head camera with a $1/3$ in color, 12 bit, $1032\times 776\text{pixel}$ CCD (Point Grey Research, Inc., Richmond, BC, Canada). The image was created with a 145 deg FOV board lens (PT-02120, M12 Lenses). The FOV of this system was partially occluded by the imaging head. We used white LEDs for illumination (Mightex FCS-0000-000), coupled to 1 mm diameter, 0.48 NA multimode fibers. Sources were synchronized to the camera frame rate of 15 Hz. A holographic light shaping diffuser was placed at the end of each source to efficiently spread illumination light (80 deg Circular Diffuser, Luminit). Linear polarizers were placed in front of the sources and objective lens in a cross-configuration to minimize specular reflection. Images in raw data format were processed with a de-mosaicing interpolation algorithm to provide full resolution RGB images from Bayer-patterned raw images. The pixel intensities were then estimated by a weighted average of the three color channels.

2.2.

Modified Commercial Endoscope

To generate images with smaller light, camera, and object separations, we modified a commercial endoscope to use added light sources [Fig. 1(c)]. We used a Pentax EG-2990K gastroscope with a Pentax EPK-1000 video processor. For illumination, we used fibers with an integrated light diffuser ($\mathrm{NA}=0.5$, $D=1\mathrm{mm}$, Doric Lenses Inc.), and no polarization filters. The four fibers were secured at equal angles in a 12-mm diameter circle around the endoscope tip, resulting in an external diameter of 14 mm. Components were held together with tape wrapping. We acquired uncompressed video in NTSC format at 8 bit, $720\times 486\text{pixel}$ resolution, and 29.97 interlaced frames per second using a video capturing device (Blackmagic Intensity Shuttle). Light sources were alternated at 60 Hz, synchronized with the video signal to deinterlace a sequence of RGB frames captured with only one light source active at a time. Frames were then interpolated in every other horizontal line to obtain full resolution images. The image intensity was estimated as the weighted average of the three color channels.

2.3.

PSE Algorithm

Our algorithm relies on the approximation that the light remitted from our sample surface follows the Lambertian reflectance model. For each pixel, the apparent brightness or luminous intensity ($I$) is determined by the surface irradiance ($I_0$), the cosine of the angle between the unit vector normal to the surface ($\widehat{n}$) and the unit vector indicating the direction to the illumination source ($\widehat{s}$), and a constant amplification factor ($A$) that includes the reflection coefficient or albedo:

Neglecting cast shadows and specular reflections, with the light source $i$ coming from the direction ${\widehat{s}}_i$, the measured intensity $m_i$ at one pixel can then be represented as

Fig. 2

Example of PSE workflow. The algorithm includes acquiring a series of images of the same FOV under different illumination (a–d), calculating the surface orientations using conventional photometric stereo, which results in severe topography artifacts (e), and performing additional spatial frequency filters (f). The resulting topography can be integrated and 3-D rendered without texture (g) and with textured color (h). Surface orientation maps in (e) and (f) show color coded RGB maps, where normalized $x$, $y$, and $z$ components in the interval $(-1\dots 1)$ have been mapped to R, G, and B values in the interval $(0\dots 1)$.

Under PS assumptions, a sequence of three measurements of the same sample can be written as

This is a linear system of equations that can be solved for $\stackrel{\rightharpoonup }{n}$ if the light sources matrix is nonsingular. This condition is equivalent to requiring that the three light sources and the sample do not lie in the same plane. If more than three measurements are acquired, the normal vector can be estimated by minimizing the residual error given the measurements and the source directions:

Our PSE approach uses the conventional framework of PS imaging as described above, but includes an additional filtering of lower spatial frequency components of the surface normal data. This is necessitated by close placement of the sources, camera, and imaging object. Specifically, the traditional PS assumption that ${\widehat{s}}_i$ is constant for all pixels in the image becomes especially inaccurate when the FOV is large relative to the distance from the object to the light sources. We empirically found this effect to induce only a low frequency error on the reconstructed normals, on the order of 1 cycle/FOV. This result is expected, given that the source directions change slowly across the FOV. As the motivation for PSE is to obtain useful information about the lesions and texture present in an endoscopic setting, which are often high frequency topographies, we can preserve these features while filtering out the inaccurate information by applying a high-pass filter to our reconstructed normal map.

Assuming a continuous sample that can be described as $z=f(u,v)$ with $z$ the distance from the objective to the sample and $(u,v)$ the pixel coordinates, its directional gradients can be obtained by scaling the direction normal to the surface:

Because both the spatial frequency filter and the differentiation are linear operations on $(u,v)$, these operations are interchangeable, and the high-pass filter of $-\partial f/\partial u$ is equivalent to the gradient in direction $u$ of the high-passed surface. Thus, by high-pass filtering each of the directional gradients, we can obtain the gradients of the high frequencies of the shape. For each directional gradient, we applied a high-pass filter by subtracting the low frequency component of the signal, which was calculated as a convolution of the original gradient with a Gaussian kernel with $\sigma =40\text{pixels}$ in image space. This kernel’s full width at half maximum value was approximately 8 cycles/FOV. Applying this filter to the normal map in Fig. 2(e) results in the normal map as shown in Fig. 2(f). In this work, we represent normal maps using a color coded RGB image, as is commonly used in computer vision,¹² where the surface normal vector is normalized and its $x$, $y$, and $z$ components are mapped to values of red, green, and blue, respectively.

To calculate height maps, the filtered gradients were integrated using a multigrid solver¹³ for the Poisson equation¹⁴ that minimizes integration inconsistency errors [Fig. 2(g)]. To visualize both the color information and the acquired topography, we can overlay the color image on the calculated height map [Fig. 2(h)].

2.4.

Tissue Models

3.1.

PSE Imaging of Anatomic Shapes: Phantom Imaged by Benchtop Prototype

We performed PSE imaging on several regions of the silicon anatomical phantom with the benchtop prototype. The expected surface normals were recovered across the FOV, as shown in the frontal view of the cecal wall presented in Fig. 3(a). The flat regions of the cecum generated regions with normal components pointing primarily in the $z$-direction, and bumps and ridges yielded normals such that they were correctly reconstructed after integration. We emphasize that the topographical data presented in the surface normal map and the 3-D rendering are complementary to the color information in the conventional image—this topography cannot be reconstructed from the conventional image alone. Three diminutive bumps that range in height from 0.5 to 1 mm were correctly identified as elevations in our reconstruction [white arrows, Fig. 3(a)], though it is difficult to appreciate based on the conventional color image alone.

Fig. 3

(a) Conventional view, PSE normal map, and topography estimation in a silicon colon phantom. The surface normal directions and 3-D rendering of a cecum view captures the orientation of slopes and curvature, which are not contained in a conventional color image. Three diminutive bumps that are 0.5 to 1.0 mm in height are correctly identified as elevations on the normal map (white arrows) (Video 1, QuickTime, 6.3 MB) [URL: http://dx.doi.org/10.1117/1.JBO.18.7.076017.1]. (b) In a tubular region of the transverse colon, the high-frequency morphology shows the details of features at different working distances contained in the FOV. Cast shadow artifacts consistently exaggerate slopes from the feature generating the shadow.

The illumination intensity that reached the sample from the light sources was strongly affected by the working distance, which varied significantly within the FOV. For example, when imaged down a tubular shape, the pixels in the center of the image received much less light than those at the periphery. However, accurate normal reconstruction in PSE relies on intensity differences for each pixel in a series of images, and lighting changes that are consistent across the PSE image series should primarily affect the SNR of the measurements rather than introduce topographical bias. This was demonstrated in a PSE image of the transverse colon presented in Fig. 3(b). Though the light intensity reaching the surface down the tube was much lower than that illuminating the adjacent wall, the high frequency details in surface orientations of the object were still acquired.

There are several sources of topographical error in PSE that become amplified when imaging in a tubular geometry. The assumption that the source vectors are constant across the FOV becomes worse as the distance between each point in the object and the light source changes. Furthermore, any portion of the object that casts a shadow creates a nonlinear artifact; the region that is cast in shadow is reconstructed to have a surface normal that points more perpendicularly than it should to the direction of the light source that shadows the region. This artifact exaggerates slopes facing away from the lights. Qualitatively, this effect emphasizes ridges and sharp features, which may actually be helpful for the purpose of increasing lesion contrast. This effect was observed in the shadows cast by the muscular features and haustra of the colon phantom, shown in Fig. 3(b).

3.2.

PSE Imaging of Tissue Surfaces: Ex Vivo Human Tissue Imaged by Benchtop Prototype

We performed PSE on ex vivo human gastrointestinal tissue in order to evaluate performance on samples with heterogeneous optical properties, reflective surfaces, and clinically relevant lesions. Figure 4(a) presents the topography obtained from a right colectomy with a tattoo applied next to a benign ulcer that resulted from a polypectomy. Here, our normal map correlated to the visible folds in the conventional image. The ulcer, identified by a gastroenterologist at the time of the imaging, was reconstructed as a prominent indentation in the tissue, indicated with a dashed arrow in Fig. 4(a). However, the tattoo, which left a concentrated point of indigo color at the site of the injection, did not register as a topographical change (solid arrow). This demonstrated that the PSE was able to separate a pixel’s surface normal vector from its albedo.

Fig. 4

PSE morphology estimation of ex vivo human tissue with heterogeneous optical properties. (a) The reconstructed morphology of a post-polypectomy benign ulcer and surrounding tissue folds in formalin fixed colon tissue correlates with the folds that are visible in the conventional image (Video 2, QuickTime, 8.1 MB) [URL: http://dx.doi.org/10.1117/1.JBO.18.7.076017.2]. As expected, an injection site with remnant ink (solid arrow) produces color contrast but not morphologic contrast, while the ulcer (dashed arrow) produces both color and morphologic contrast. The plateau shape of a benign precancerous sessile serrated polyp in the fixed ex vivo right colon tissue (b), and a metastatic melanoma lesion in fresh ex vivo small bowel tissue (c) are both prominent in the estimated morphology.

Next, we imaged a benign precancerous sessile serrated polyp that was identified after a right colectomy [Fig. 4(b)]. In this experiment, the light source in the bottom right of the FOV did not diffuse as well as the other three sources. As a result, the image with this light source saturated the bottom right of the FOV and the topographies were poorly reconstructed in that region. Nonetheless, the sessile lesion encircled with a dashed oval in Fig. 4(b) clearly influenced the normal map. In the surface rendering that was generated from the normal map, the lesion had a plateau-like topography that is characteristic of a sessile lesion, and that was observed during this experiment.

Finally, we imaged a metastatic melanoma that was present in fresh ex vivo human small bowel tissue [Fig. 4(c)]. This feature was also identifiable in the normal map and reconstructed height profile. Note again that here PSE was able to distinguish between color changes of the tissue and actual folds that were present in the sample.

Because the ex vivo human tissue was wet, specular reflection was more prominent than what was observed in the silicon phantom. This led to artifacts in our surface normal reconstructions. Specifically, pixels that had specular reflections were reconstructed to have surface normals that pointed more toward the source that generated the specular reflection than they actually should.

3.3.

Translation of PSE Imaging to Endoscopic Size: 3-D Printed Phantom Imaged by Modified Endoscope

These previous results show the ability of PSE to obtain high frequency topography in geometrical settings where traditional PS assumptions do not hold, and demonstrate that PSE topography records changes in surface orientation of tissues with heterogeneous optical properties. However, these results have been obtained using a light source separation distance larger than what fits in an endoscope.

To qualitatively confirm the scalability of PSE to endoscopic size, we compared results of imaging a simple object with the benchtop prototype and the modified endoscope. Because our modified endoscope setup was sensitive to specular reflections and had an inaccessible video digitizing system, we used a nonspecular 3-D printed phantom with a single bump and compared with PSE reconstructions from our two prototypes. PSE consistently estimated the morphology of bumps and depressions with both systems. In all estimations, the surface normal directions correctly showed the elevation or depression as a region in which border surfaces were oriented outwards for elevations and inwards for depressions.

When using a small source separation of 12 mm and a large working distance of 35 mm with the modified endoscope, PSE was still able to recover surface orientations of a small bump. Figure 5 shows results comparing the performance of PSE in the benchtop prototype at an 87 mm working distance [Fig. 5(a)–5(c)] and the modified endoscope at a 35 mm working distance [Fig. 5(d)–5(f)]. Figure 5(a) and 5(d) display the full FOV of each system, while the remaining insets show a $5\times 5{\mathrm{mm}}^2$ region of interest. The conventional images [Fig. 5(b) and 5(e)] were insufficient to discriminate the feature as an elevation or a depression, while their morphology was revealed in the surface orientations [Fig. 5(c) and 5(f)]. As noticeable in Fig. 5(c), the topography imaged at 87 mm distance resulted in an increased level of noise, possibly due to the low illumination intensity at that distance and a lower number of pixels covered by the $5\times 5{\mathrm{mm}}^2$ area.

Fig. 5

PSE enabled topographical identification of a diminutive bump with the benchtop prototype at an 87 mm working distance [panels (a) to (c)] and the modified endoscope at a 35 mm working distance [panels (d) to (f)]. (a) and (d) show the full FOV of each system. The conventional images in (b) and (e) were insufficient to discriminate the feature as an elevation or a depression, while their morphology was revealed in the surface orientations (c) and (f). Insets (b), (c), (e), and (f) cover a $5\times 5{\mathrm{mm}}^2$ region of interest.

Noticeable artifacts present in these estimations included measurement noise, gradients amplitude scaling, discretization of the shape, and albedo variations. The albedo nonuniformities were caused by an uneven layer of paint and differences in surface orientation. The amplitude of the gradients decreased with the working distance. Discretization of object was noticeable and is attributable to the small portion of the FOV that the sample covered. For example, a 1-mm wide feature imaged at a 35 mm working distance covered approximately 9 pixels across the images acquired with the modified endoscope.