Fiber bundle fluorescence endomicroscopy is an effective method for in vivo imaging of biological tissue samples. Line-scanning confocal laser endomicroscopy realizes confocal imaging at a much higher frame rate compared to the point scanning system, but with reduced optical sectioning. To address this problem, we describe a fiber bundle endomicroscopy system that utilizes the HiLo technique to enhance the optical sectioning while still maintaining high image acquisition rates. Confocal HiLo endomicroscopy is achieved by synchronizing the scanning hybrid-illumination laser line with the rolling shutter of a CMOS camera. An evident improvement of axial sectioning is achieved as compared to the line-scanning confocal endomicroscopy without the HiLo technique. Comparisons are also made with epifluorescence endomicroscopy with and without HiLo. The optical sectioning enhancement is demonstrated on lens tissue as well as porcine kidney tissue.
Optical fiber bundles represent an important element for implementing endomicroscopy for flexible maneuvering along a curved lumen and providing microscopic imaging of a living tissue.1–3 Fiber bundles have been used in various single-photon imaging modalities, such as epifluorescence endomicroscopy and confocal fluorescence endomicroscopy.1–5 The epifluorescence endomicroscopy acquires signal from all pixels in parallel to provide the highest acquisition speeds. However, for optically thick specimens, light from different tissue layers contributes to image formation and out-of-focus light cannot be rejected.5 Confocal fluorescence endomicroscopy provides an effective means of eliminating signals from out-of-focus light by focusing laser illumination to a single point and the use of a small pinhole in an optical plane conjugate.3,6 It allows optical sectioning of cells and tissue and thus has found a wide range of clinical applications, including in vivo imaging of the urinary tract as well as the human airway,7,8 and diagnosis of colorectal cancer.9
The low frame rate (10 to 20 Hz) due to the point scanning mechanism and the trade-offs between resolution, field of view, and acquisition speed has motivated the development of alternative confocal endomicroscopy techniques.10 A detector slit, together with a scanning laser line, has been used in confocal endomicroscopy to provide much higher frame rates compared with the point scanning system but with reduced optical sectioning.11 Recently, Hughes and Yang12 developed a high-speed line scan confocal endomicroscopy system where the slit detection was realized using the rolling shutter of a CMOS scanner as a virtual detector slit. With such a system, image acquisition rates of up to 120 fps were achieved. Although images are acquired at much higher frame rates, the optical sectioning with line-scanning techniques is poorer than conventional point-scanning confocal systems. A subtraction technique was used in this study to improve optical sectioning by subtracting a second image with an offset virtual slit, but it also introduces noise and bit depth loss in the images.
An alternate approach to achieve confocal-like optical sectioning is structured illumination microscopy (SIM).13–15 In conventional SIM, three phase-shift grid illuminated images are sequentially acquired and computationally combined to synthesize an optical-sectioned image.13 Although SIM is effective in optical sectioning, its imaging rate is only one-third of the raw frame rate and it is highly susceptible to artifacts due to imprecise grid translations and sample motion.16 A related method called HiLo microscopy has been developed, in which optical sectioning is achieved with only two images, one with structured illumination and one with uniform illumination.17,18 In HiLo, when the predefined structured illumination is projected onto the sample, only in-focus information is modulated due to rapid attenuation of the high-spatial frequency pattern of defocused areas.18 Thus the final optical-sectioned image is formed by combining high- and low-spatial frequency information from the uniform and structured illuminated images.19 The advantage of such hybrid-illumination HiLo technique is that its speed is 1.5 times that of conventional SIM when the raw frame rate remains the same. Further, HiLo is more resilient to artifacts such as intensity streaking than SIM due to use of uniform illumination.16 However, widefield HiLo technique in thick fluorescent samples is still a challenging task.
Recently, line scanning light with physically suppressing out-of-focus light has been used with SIM to improve the modulation and the quality of the illumination pattern.20 This line-scan SIM demonstrated an optical-sectioning capability increment and a reconstruction artifact reduction compared with conventional SIM. But its low frame rate and the bench-top setting limit the applications in in vivo imaging of living tissues. Line-scan light with confocal slit detection has also been applied in light-sheet microscopy to increase contrast and optical sectioning capability by combining the benefits of selective plane illumination and out-of-focus light suppression.21,22
In this paper, we present an implementation of a high-speed fiber bundle fluorescence endomicroscope that demonstrates enhanced optical sectioning by combining the benefits of HiLo technique and confocal slit detection. A spatial light modulator (SLM) is used to enable rapid control of the pattern illumination, allowing higher imaging rates and increased flexibility.23,24 Synchronizing the camera rolling shutter with a scanning hybrid-illumination laser line results in a fast HiLo endomicroscopy with confocal line detection. The superior grid modulation and optical sectioning of the endomicroscopy are demonstrated on different nonbiological and biological samples.
A schematic of the proposed imaging system is shown in Fig. 1. The output beam from a CW solid-state laser diode (Vortran Stradus, , ) is expanded by a telescope to a beam diameter of . Then it is projected onto the SLM via a polarizing beam splitter (Thorlabs, PBS 251). The SLM incorporates a phase-only ferroelectric liquid crystal on silicon device (Forth Dimension Displays, QXGA-3DM) with and pixel pitch. The SLM operates in a binary imaging mode, where pixels in the ON state reflect light with its polarization state changed and pixels in the OFF state reflect the light without changing the polarization. With the help of the beam splitter, the SLM operates in an amplitude-modulation mode when its pixels are in ON and OFF states. An arbitrary amplitude pattern can be applied on the beam by computer control via a software (MetroCon from Forth Dimension Displays). In this experiment, the SLM toggles between two illumination patterns, grid and uniform. The grid pattern has a user-defined pitch that can be controlled by the MetroCon software.
A half-wave plate is placed between the SLM and the beamsplitter to rotate the polarization state of the light for maximal light modulation in the grid pattern, similar to the design proposed in Ref. 25. Further, a linear polarizer is applied after the beamsplitter to remove the unwanted polarization light. The modulated beam is reflected off a galvanometer scanning mirror (Thorlabs, GVS001), then focused by a cylindrical lens () and an achromatic doublet (). The laser line is then reflected by a dichroic mirror (Thorlabs, MD498) onto a microscope objective (Thorlabs, ) which focuses it on the proximal face of a fiber bundle.
The imaging fiber bundle transfers the laser line to the tissue and collects emitted fluorescence. For this study, the Cellvizio Gastroflex UHD probe (Mauna Kea Technologies, France) is used which has 30,000 cores, with a useful image diameter and a interspacing. The distal end of the fiber bundle is equipped with a miniaturized micro-objective with a demagnification, yielding a field of view of and an imaged core separation in the sample of . The miniaturized micro-objective has a nominal working distance of and a numerical aperture of 0.8.
The fluorescence emission is isolated with a notch filter (Thorlabs NF488-15) and an emission filter (Thorlabs FEL0500) and imaged onto a CMOS camera (Point Grey Flea 3, pixel size ) via an achromatic doublet (). The camera provides 440 pixels across the fiber bundle, or per core spacing, which is sufficient to allow individual cores to be visualized in the images. The camera is operated in a rolling shutter mode at a full frame rate of 120 Hz for the experiments reported below. The rolling shutter of the CMOS camera offers a versatile electrically controllable detection slit. The slit width, which corresponds to the active rows on the camera sensor, can be changed by varying the exposure time of the camera. If the virtual slit width is reduced sufficiently, most of the out-of-focus light can be rejected. The scanning mirror needs to be synchronized with the rolling shutter to retain most beam light while rejecting scattered photons along the direction of beam scanning. The synchronization details have been explained in our previous papers.12,26 The physical slit width in a unit of micrometers, as projected onto the proximal face of the bundle, can be calculated by the multiplication of the exposure time, the camera line rate, the pixel size, and the magnification factor between the camera and the bundle.12
Hybrid illumination across the fiber bundle is achieved by sequentially projecting uniform and grid illumination patterns which are generated using the SLM. Since the maximum frame rate of the SLM is about 1 kHz, the image acquisition rate of the proposed system is limited by the camera frame rate, which is 120 fps. A custom designed Labview code is developed to acquire the endomicroscopy images and processed offline to reconstruct a HiLo image using the following steps. First, a circular window is applied on each frame of the data to crop the edges of the bundle. Two adjacent frames with uniform and grid illuminations are processed by the HiLo ImageJ plugin developed by Lim et al.18 for producing one modulated image frame. The theoretical background of the HiLo microscopy has been described in detail in Refs. 1718.–19. Basically, both of the uniform illuminated image and the structured illuminated image contains the in-focus and out-of-focus components, where is the lateral position vector.19 In-focus high-spatial frequency components are obtained by applying a high-pass filter HP to directly:
For the structured illuminated image , only in-focus information is modulated by the grid pattern due to rapid attenuation of pattern in the out-of-focus area. Thus normalized and can be subtracted to calculate the grid contrast . can be used to weight the in-focus portion of . Then in-focus low-spatial frequency components are obtained by applying a low-pass filter LP to the weighted :
The final optically sectioned image is simply the addition of the high- and low-spatial frequency components with a scaling factor to ensure a seamless fusion:
A homogeneous fluorescence reference slide (green, AG2273-G, Agar Scientific) is imaged first to test the structure modulation ability of the system. Representative images of uniform and structure illuminated beams on the fluorescence reference slide are shown in Figs. 2(a) and 2(b). The measured laser line width at the reference slide is around . The illumination beams are swiped across the reference slide at the perpendicular direction to form uniform and structure illuminated images. Figures 3(a) and 3(b) show raw structure illuminated images of the reference slide imaged at different slit widths. The slit width is set to its maximum of in Fig. 3(a) and in Fig. 3(b). Though smaller slit width is possible, is chosen to minimize the trade-off between confocal performance and signal strength of the grid pattern. As have been demonstrated in Ref. 12, out-of-focus background largely remains at slit width and imaging at this slit width can be considered as the widefield slit detection. The out-of-focus background is largely reduced in and imaging at this slit width can be considered as the confocal slit detection. By comparing Figs. 3(a) and 3(b), the contrast ratio between the ON and OFF pixels is evidently improved from 0.6 dB at the widefield slit detection to 2.5 dB at the confocal slit detection, demonstrating an increased structure modulation with the confocal slit detection.
To further evaluate the line-scan HiLo endomicroscopy, a sample comprising lens paper stained with 0.02% acriflavine solution is imaged. The resulting images illustrated in Fig. 4 are imaged at different slit widths of 120, 18, and . Figures 4(a)–4(c) show raw images illuminated with a uniform pattern. As the slit width is decreased, optical sectioning of the endomicroscopy system gradually increases, in which out-of-focus blur becomes weaker. However, even with the confocal slit detection, there is still some background left as shown in Fig. 4(c), demonstrating the need to further reduce the background light. Figures 4(d)–4(f) show raw images illuminated with a grid pattern at different slit widths. As have been demonstrated in the uniform fluorescent plane, the contrast between the ON and OFF pixels is improved as the slit width decreases. Figures 4(g)–4(i) represent combined HiLo images using uniform and structure illuminated images at the corresponding slit width. A clear reduction in fluorescence background is observed for the combined HiLo images at all the three slit widths and the optical sectioning increases as the slit width becomes smaller. By comparing the nine images in Fig. 4, the best optical sectioning performance is achieved when both the HiLo technique and the confocal detection are applied.
The optical sectioning performance of the system is compared for four different cases: widefield, widefield HiLo, confocal, and confocal HiLo. As have been explained, the uniform illuminated images at 120- and slit widths are considered to be the widefield and confocal slit detections. The widefield HiLo and confocal HiLo results are combined from the uniform and structure illuminated images at the corresponding slit width. To determine the optimal spatial frequency of grid patterns, four different grid patterns with periods of 40, 30, 16, and (measured at the sample plane) are tested. Average intensities of small regions in the dark (OFF) and bright (ON) areas are calculated. It is found that the contrast ratio between the ON and OFF pixels is 3.56 dB for period, 2.33 dB for period, 1.47 dB for period, and 0.53 dB for period, respectively. Based on these results, a grid pattern with period is chosen for the remaining experiments in this study as it provided a reasonable comprise between grid pattern contrast and optical sectioning capacity. Despite a smaller grid period leads to stronger axial sectioning in theory, it also leads in practice to a decreased grid pattern contrast in the sample.10 Figures 4(a) and 4(c) represent the widefield and confocal images without using the HiLo, whereas Figs. 4(g) and 4(i) represent the corresponding widefield and confocal results with the HiLo technique. Intensity profiles along the lines indicated in these images are plotted in Fig. 4(j), where peak A indicates the out-of-focus background while peak B is the in-focus signal. The intensities of the in-focus signal in all cases are very similar at peak B. When comparing the intensities at peak A, it is obvious that the confocal HiLo gives the lowest value while the widefield has the highest value, the confocal only or the HiLo only has a value in between. To further quantify the effect of contrast enhancement, the mean pixel values from regions of signal and background areas are used to calculate the signal-to-background ratio (SBR). It is found that the SBR is 1.82 dB in Fig. 4(a), 3.33 dB in Fig. 4(c), 3.56 dB in Fig. 4(g), and 7.59 dB in Fig. 4(i). Thus both the HiLo technique and the confocal slit detection can reduce the background while the largest background reduction happens when both the confocal and the HiLo are applied. Noise of the imaging results in four different cases is also compared by calculating the standard deviation of intensities over areas. It is found that the noise of confocal HiLo images is 1.08 times of confocal images while the noise of widefield HiLo images is 1.32 times of widefield images. Thus confocal imaging detection also reduces the amplification of noise in the HiLo process.
To further compare the optical sectioning performance for different configurations, a steup of a fluorescence reference slide mounted on a motorized translation stage is used. The fiber bundle tip is initially placed in contact with the reference slide and then moved away at a speed of up to a range of . The acquired image is normalized before taking the mean pixel value from a region of interest as the intensity at that depth position. Axial sectioning profiles are compared by plotting normalized intensities at different configurations as a function of the stage movement, as shown in Fig. 5. The sectioning profiles are only plotted from the peak value (focus position of the micro-objective, from the fiber bundle tip) to a farther position to show near contact-based imaging. From Fig. 5, the half-width at half-maximum (HWHM) of the sectioning profile is calculated as for the confocal HiLo as opposed to for the confocal, for the widefield HiLo and for the widefield. The results demonstrated that the optical sectioning ability of the HiLo-confocal endomicroscopy is largely improved when compared to other configurations. Despite using different grid period, the HWHM of the proposed confocal HiLo endomicroscopy is also evidently reduced compared with a previous illustration of HiLo endomicroscopy ( HWHM).17
Imaging of Biological Samples
The optical sectioning performance can degrade in biomedical tissue due to its increased level of scattering than a reference target. In order to demonstrate the optical sectioning performance of the proposed system in a biologically relevant example, perirenal fat and renal cortex tissues from a porcine kidney are imaged. Imaging of perirenal fat and renal cortex plays an important role in staging for renal cell carcinoma which is the most common primary malignancy of the kidney.27 The kidney is stained with 0.02% acriflavine solution for 1 min and then washed by saline. The thickness of the kidney tissue used in this study is and the light power delivered to the sample is around 0.5 mW. Figure 6 shows images of perirenal fat and renal cortex performed without and with the use of HiLo technique at the confocal slit width of . All images have been scaled so that the maximum pixel value is 255. In Figs. 6(a) and 6(d), there is still some background left since only confocal detection is used to reject out-of-focus background from the thick tissue sample. When the HiLo technique is applied combined with the confocal detection, the out-of-focus background is significantly reduced and tissue features become more apparent as shown in Figs. 6(b) and 6(e). To quantify the improvement, Fig. 6(c) shows intensity plots along the horizontal lines in Figs. 6(a) and 6(b) and Fig. 6(f) shows intensity plots along the lines in Figs. 6(d) and 6(e). Compared to only use confocal slit detection, the background intensity between the features can be reduced to half when both HiLo and confocal techniques are used. The SBR is also calculated in Fig. 6 using the same method as in Fig. 4 to further quantify the contrast enhancement. It is observed that the SBR increases from 6.6 dB in Fig. 6(a) to 14 dB in Fig. 6(b) and the ratio increases from 4.8 dB in Fig. 6(d) to 10.6 dB in Fig. 6(e), demonstrating an average twofold contrast enhancement in confocal HiLo images as compared to that of confocal-only images.
A hybrid line-scanning fluorescence endomicroscopy is developed in this study by combining the benefits of HiLo technique and confocal slit detection. Confocal line detection is realized using the rolling shutter of a CMOS camera as a virtual detector slit synchronized with scanning laser line. HiLo technique combines two images sequentially acquired with uniform and structured illumination to provide an effective means of eliminating signals from the out-of-focus background.
The system proposed in this study is specifically intended to reduce out-of-focus background. An advantage of the proposed system is that it can be used in different configurations, including widefield, widefield HiLo, confocal, and confocal HiLo, by digitally controlling the illumination pattern and detector slit width. The imaging performance on these different configurations is compared using stained lens tissue. The results presented above demonstrate that the performance of combined HiLo technique and the confocal slit detection is better in removing the out-of-focus background as compared to only HiLo, only confocal, or wide-field conditions.
Another advantage is that the virtual slit width can be adjusted to suitable for imaging of different kind of tissues. For example, a large slit width is needed in imaging weakly fluorescent samples because the signal-to-noise ratio is low at small slit width. This is similarly noted in our previous work presented in Ref. 12.
Benefiting from the fast speed of the SLM and the CMOS camera, the imaging acquisition rate of the proposed system is as high as 120 fps equivalent to a maximum pixel sampling rate of 23 MHz. Considering that the final image is formed by combining two images, the effective frame rate is 60 Hz (this is equivalent to the imaging speed of a typical endoscope, Karl Stoze Image 1 HD). For these proof-of-concept experiments, a standard desktop PC is used, and the HiLo algorithm is performed offline at a speed of around 20 Hz. It is expected that using parallel computing via FPGA or GPU, this processing time can be minimized and HiLo confocal imaging at the full frame rate of 60 Hz could be achieved.
Imaging of the perirenal fat and renal cortex tissues from a porcine kidney demonstrates that the proposed system can enhance depth discrimination in thick biological samples. Invasion of the perirenal fat together with tumor size is key factors in staging for renal cell carcinoma.27 The size of the fiber bundle can be further reduced to be used in the working channel of a urethroscope, showing its potential application in renal cell carcinoma staging.
A hybrid line scanning, fiber bundle fluorescence endomicroscopy is demonstrated using the HiLo technique and the confocal slit detection to produce images with enhanced optically sectioning. The confocal detection is realized using the rolling shutter of a CMOS scanner as a virtual detector slit. The results show that the system has an improved structure modulation at the confocal slit detection. When both the confocal slit detection and the HiLo technique are applied, an increment in optical sectioning capability was measured compared with using only confocal or only HiLo or widefield endomicroscopy. The system can realize full image acquisition rate of 120 fps, which is much higher than a point-scanning laser confocal microscopy. The slow offline HiLo combination rate can be improved in our future work using a GPU to combine the frames. To our knowledge, the results represent the first demonstration of fiber bundle fluorescence HiLo endomicroscopy with confocal slit detection, which we hope will have a broad impact on the biomedical community.
The authors acknowledge EPSRC for financial support through EP/N019318/1 (REBOT: Robotic Endobronchial Optical Tomography).