1.

Introduction

The examination of a surgically excised specimen using a microscope has long been the gold-standard for disease diagnosis. In recent years, there is an upsurge in worldwide attention on whole-slide imaging (WSI) for pathology analysis.¹ Instead of manually inspecting the specimen using a microscope, the WSI system converts the entire tissue slide into a digital image that can be viewed, managed, and analyzed on computer screens. A typical WSI system employs a 0.75 numerical aperture (NA) objective lens to acquire high-resolution images of the sample. The images are then aligned and stitched to form a large image of the entire slide. Catalyzed by the rapid development of high-density solid-state detector technology and affordable terabyte-scale data storage, WSI is currently experiencing a period of exponential growth. In the medical realm, a major milestone was reached in 2017 when the US Food and Drug Administration (FDA) approved Philips’ WSI system for primary diagnostic use in the US.² The emergence of artificial intelligence in digital pathology promises further growth of this field in the coming decades.

One important development of the WSI system is to integrate microscopic imaging with spectroscopy to obtain both the spatial and spectral information. Thanks to different spectral signatures of biochemical constituents, spectroscopy can be used for better histopathological analysis of labeled tissue sections.^3–7 In particular, acquiring whole-slide images with spectral information at each pixel permits the use of multiplexed antibody labeling and allow for the measurement of cellularly resolved information about pathways, cell fates, and cell types.⁸ Existing solutions include spatial-scan methods by integrating a spectrometer to a microscope platform and wavelength-scan methods via variable filters.^9–12 Snapshot hyperspectral solutions have also been reported for imaging dynamic samples.^13–15 However, snapshot solutions may not be able to achieve diffraction-limited performance due to the need of encoding the spectral information in spatial measurements.

Here, we report the development of a hyperspectral WSI platform based on prism dispersion and slit-array detection/projection. We note that the general idea of using prism dispersion for spectroscopy is well known.^10,16,17 Different from the previous demonstrations, the key considerations for our design are threefold. First, the system needs to achieve diffraction-limited spatial resolution for digital pathology applications. Second, the system needs to be computationally efficient due to the acquired terapixel dataset. Different from many snapshot hyperspectral solutions, we cannot afford the computational cost of L1/L2 norm regularization. Third, the system needs to be in high-throughput and the acquisition time for the entire terapixel dataset is better to be limited by the data transfer link of the camera instead of the optomechanical hardware.

To address these considerations, we demonstrate a slit-array detection scheme for absorption-based measurements and a slit-array projection scheme for fluorescence-based measurements. The use of slit-array mask allows us to perform parallel spectral measurement and significantly shorten the acquisition time for WSI. The contribution of this paper can be summarized as follows. First, the spectral resolution and spectral range in the reported system can be adjusted by simply changing the orientation of the slit-array mask. The capability of adjusting spectral resolution and spectral range in the reported system provides a flexible solution to multilabeling schemes chosen at the users’ discretion. Second, the reported design is compatible with most existing WSI systems. The slit-array detection and projection systems can be integrated as add-on modules attaching to the imaging port or the epi-illumination arm of an existing microscope platform. Third, to the best of our knowledge, the slit-array projection scheme for fluorescence spectroscopy is unique and has not been reported before. In this scheme, all emitted photons can be detected and the induced phototoxicity to the sample is minimized. This is different from conventional filter-based implementations that only detect photons at a certain wavelength while discarding all other valuable photons. The projection scheme may also provide insights for developing structured illumination imaging or confocal imaging systems.

2.

Slit-Array Detection Scheme

Figure 1(a) shows the slit-array detection scheme for absorption-based measurements (Video 1). In this scheme, we placed two LEDs at the back focal plane of the condenser lens to perform focal plane detection.^18,19 The rest of the setup is based on a Nikon Ti Eclipse microscope platform with a $20\times$, 0.75 NA objective lens, a $0.55\times$ reduction lens adapter, and a motorized stage (prior 101A). We fabricated a slit-array mask using laser direct writing procedures. The period of the slit array is $138\mu \mathrm{m}$ and the linewidth is $3.45\mu \mathrm{m}$. The period is chosen for a spectrum range of $\sim 200\mathrm{nm}$ and the linewidth is chosen to match the pixel size of the image sensor. The slit-array mask was placed on the image plane of the microscope platform. We then used a $4f$ system to relay the slit-array mask to a 5-megapixel camera with a $3.45\text{-}\mu \mathrm{m}$ pixel size (Imaging Source DMK 33UX250 camera with a Sony IMX 250 image sensor, 72% quantum efficiency, and 71 dB dynamic range). This $4f$ system consists of a $2\times$, 0.1 NA object lens (Nikon APO $2\times$ lens), and a 200-mm tube lens (Thorlabs ITL 200). A 4-deg wedge prism (Thorlabs PS881-A) was placed at the Fourier plane of the $4f$ system.

Fig. 1

(a) The slit-array detection scheme of the proposed hyperspectral WSI platform (Video 1). (b) The experimental prototype setup. The captured images of the slit-array mask without (c1) and with (c2) the wedge prism at the Fourier plane (Video 1, MP4, 9213 KB [URL: https://doi.org/10.1117/1.JBO.23.6.066503.1]).

Figure 1(b) shows the prototype setup, where the slit-array mask is placed at the image plane of the camera port. Figure 1(c1) shows the captured image of the slit-array mask without placing the wedge prism at the Fourier plane. The sample is an empty slide and the light source is a halogen lamp with a bandpass filter (450 to 660 nm). Figure 1(c2) shows the captured image with the wedge prism at the Fourier plane. We can clearly observe the effect of spectral dispersion in this case. We also note that placing the slit-array at the image plane for blocking transmission light is not a problem for brightfield microscopy, which has a sufficient photon budget for spectral measurements.

The spectral measurement range of the reported setup is determined by the separation between the adjacent slits. A larger separation results in a larger spectral measurement range or a higher spectral resolution (using a larger deflection-angle prism). One key advantage of the reported setup is that we can adjust the spectral measurement range by simply rotating the slit-array mask. In Fig. 2, we used a mercury lamp as the light source to calibrate the wavelengths of the captured images. In Fig. 2(a1), the dispersion direction of the slit-array mask is along the $x$-axis and the spectral measurement range is 205 nm. In Fig. 2(a2), we rotate the slit-array mask by 45 deg to increase the spectral measurement range from 205 to 290 nm, corresponding to an improvement factor of 1.414 (square root of 2). The line traces of the spectra are shown in Fig. 2(b), where we can see that the 405-nm line appears in the red trace thanks to the change of the mask orientation (the dispersion direction remains the same). We also note that a large spectral measurement range by rotating the slit-array mask also leads to a longer acquisition time for acquiring the spatial–spectral dataset.

Fig. 2

The captured spectra of a mercury lamp. The captured images with the slit-array mask aligned at the 0 deg (a1) and 45 deg (a2). (b) The line traces of the spectra. The measured spectral range increases from 205 to 290 nm by rotating the orientation of the slit-array mask from 0 to 45 deg.

We used two lasers to characterize the spectral resolution of the reported platform. Figure 3 shows the captured intensity line trace of the two lasers. The full width at half maximum of the intensity traces is $\sim 3\text{pixels}$, corresponding to a 7.7-nm spectral resolution. The spectral resolution in the reported platform is determined by the employed 4-deg wedge prism. To improve the spectral resolution, one can use a prism with a larger deflection angle and rotate the slit-array mask to increase the spectral measurement range. The total number of slits in one imaging field of view, however, would be lower in this case and the acquisition time for WSI would be inevitably longer. It has been shown that a higher spectral resolution does not necessarily lead to a better result for distinguishing multiplexed labeling.²⁰ The chosen 7.7-nm spectral resolution in our setup is a good compromise between the spectral resolution and the acquisition time for WSI.

Fig. 3

The measured spectral resolution using two lasers. The measured full width at half maximum is $\sim 3\text{pixel}$, corresponding to a 7.7-nm spectral resolution.

To acquire the hyperspectral images using the reported platform, we need to scan the sample along the $x$-direction, similar to the pushbroom configuration in conventional hyperspectral imaging settings. In our experiments, we synchronized the motorized stage with the camera in the acquisition process. In the captured images, the separation between adjacent slits is 80 pixels. Therefore, we scan the sample 80 times to acquire images at one field of view. By reassembling different columns from these 80 images, we can obtain 80 images corresponding to different wavelengths. The acquisition time for the 80 images is 1.07 s, corresponding to a data throughput of $0.375\text{gigapixels}/\mathrm{s}$. The throughput of the reported platform is limited by the data transfer link of the camera instead of the optomechanical hardware. For each captured image, we select the lines corresponding to the specific wavelengths, as shown in Fig. 2(b). For different captured images, the selected lines correspond to a different spatial region as the sample is moving along the $x$-axis. As such, the selected lines from the 80 images can be reassembled to form an image at a specific wavelength. In Fig. 4, we compare the resolution performance of the reported approach with that of the regular widefield microscopy. In this experiment, we use a $20\times$, 0.75 NA objective lens for both settings and a United States Air Force resolution target to quantify the performance. In the slit-array approach, we recover the images at two different wavelengths in Figs. 4(b) and 4(c), respectively. We can see that both the regular approach and the reported slit-array approach can resolve up to group 10, element 6 of the resolution target, corresponding to a $0.225\text{-}\mu \mathrm{m}$ half pitch linewidth. The image contrast is high in both the long (632 nm) and short (470 nm) wavelengths in Figs. 4(b) and 4(c), respectively.

Fig. 4

Resolution performance quantification using a resolution target. (a) Image captured by the regular widefield microscopy. (b) and (c) Images captured by the reported slit-array approach. Images at (b) 632 nm and (c) 470 nm are shown for comparison.

To acquire the hyperspectral data cube, we typically ignore six images at the edge of the passband due to the filter cutoff effect and spectral overlapping from the adjacent bands. The final output is 74 5-megapixel images at different wavelengths of one field of view, as shown in Fig. 5(a). We note that removing six images means removing information at the edge of the bandpass filter. There are no six-pixel gaps in the spatial domain. Compared with the 7.7-nm spectral resolution, the sampling step in the spectral domain is 2.7 nm. On the other hand, the Nyquist sampling rate is 3.8 nm ($7.7\mathrm{nm}/2$). The oversampling factor is, thus, 1.4 in our setup ($3.8/2.7\mathrm{nm}$). We slightly oversampled the data to avoid the loss of spectral information. Figure 5(b) shows nine images of a hematoxylin- and eosin (H&E)-stained pathology slide at different wavelengths (Video 2). Figure 5(c) shows the combined color image using images at 460, 535, and 635 nm. Figure 5(d) shows the measured spectra at positions “A” and “B” in Fig. 5(c). In Video 3, we show the hyperspectral images of a Pap smear sample prepared with the ThinPrep staining protocol.²¹ We note that, in Videos 2 and 3, we can see periodic strip artifacts when the image contrast is low. These artifacts are due to the motion accuracy of the motorized stage. In the current implementation, we assume the motorized stage moves at a constant speed when capturing images. Therefore, the selected lines from the 80 images can be reassembled to form an image at a specific wavelength. If the motion is not strictly constant, there will be some slight mismatch between the adjacent 80-pixel strips. There are three solutions to address this problem. First, we can perform Fourier-domain filtering as we know the periodicity is 80 pixels for the strips. Second, we can use a digital mirror device (DMD) to project the moving slit-array pattern. In this case, there is no mechanical motion of the sample during image acquisition. Third, use a more-accurate motorized stage.

Fig. 5

(a) The captured hyperspectral data cube (Videos 2 and 3). (b) The recovered images of an H&E pathology slide at nine different wavelengths. (c) The combined color image using three wavelengths at 460, 535, and 635 nm. (d) The measured spectra of positions “A” and “B” at (c) (Video 2, MOV, 3437 KB [URL: https://doi.org/10.1117/1.JBO.23.6.066503.2] and Video 3, MP4, 791 KB [URL: https://doi.org/10.1117/1.JBO.23.6.066503.3]).

To cover a whole-slide sample with an area of $15\mathrm{mm}\times 15\mathrm{mm}$, we need to acquire $47\times 40$ tiles of the sample. For each row (47 tiles), we first use the two LEDs to acquire the focus map of the sample of that row.^18,19 Based on this focus map, we acquire the 47 tiles and each tile contains 74 images at different wavelengths. This process is then repeated for other rows. This focus map surveying process is different from the conventional process, where the focus map of the entire sample is acquired at the beginning.²² In our experiment, we observe a thermal drift of the stage due to the relatively long acquisition time of the entire hyperspectral data cube. Therefore, we acquire the focus maps of individual rows instead of the entire sample before collecting the hyperspectral data of the same row. The added time for focus map surveying is $\sim 20\mathrm{s}$, which is negligible compared with the acquisition time of the terapixel dataset.

Figure 6(a) shows the color whole-slide image by combining the recovered images at 460-, 535-, and 635-nm-wavelengths. Figure 6(b) shows the zoom-in views of the whole-slide image of the pathology slide. Figure 6(c) shows the measured spectra of positions “A” and “B” in Figs. 6(b2) and 6(b3), respectively. In this experiment, the acquisition time for $\sim 1$ terapixel whole-slide hyperspectral dataset is $\sim 45\min$. This dataset was streamed to a solid-state drive through the PCI-express interface in our experiment. The use of slit-array facilitates parallel measurements of the spectral information, significantly shortening the acquisition time for WSI.

Fig. 6

(a) The whole-slide image of a pathology slide, (b) different zoom-in views, and (c) the measured spectra of positions “A” and “B.”

3.

Slit-Array Projection Scheme

Placing the slit-array mask on the image plane is a simple solution for transmission-based measurements (i.e., absorption measurements). However, most of the photons from the sample are blocked by the mask and cannot reach the image sensor. If we use this scheme for fluorescence imaging, we need to use excitation light to illuminate the entire sample while only detecting a small portion of the fluorescence emission. Therefore, this scheme is not suitable for fluorescence imaging due to the photobleaching or phototoxicity concern. To address this issue, we place the slit-array mask at the epi-illumination arm for sample illumination, as shown in Fig. 7(a). In this case, the light from the slit-array mask is projected onto the sample and the wedge prism at the detection path disperses the fluorescence-emission photons. As such, all fluorescence photons can be detected by the image sensor and the induced phototoxicity to the sample is minimized. This is different from conventional filter-based implementations that only detect photons at a certain spectral band, whereas discarding all other valuable photons.

Fig. 7

(a) The slit-array projection scheme for hyperspectral fluorescence imaging, (b) two recovered images at two wavelengths, (c) the combined color image based on the recovered spectral information, and (d) the measured spectra (green and orange solid line) of two different types of fluorescence microspheres. The black dash lines represent the ground-truth spectra measured by a regular spectrometer (FluoroMax Plus, HORIBA Scientific).

To calibrate the system, we used a standard mercury lamp as the light source and a mirror as the object to measure the spectrum of the mercury lamp. The measured spectrum was then calibrated to match the ground-truth spectrum measured by a regular spectrometer (FluoroMax Plus, HORIBA Scientific).

To test our platform for fluorescence imaging, we coated a coverslip with green and orange fluorescence microspheres (yellow fluorescent microspheres and orange-yellow fluorescent-coated glass microspheres fluorescence microspheres, Cospheric). Similar to the slit-array-modulated detection scheme, we scanned the sample to the direction perpendicular to the slit array to acquire 80 images of the sample. The exposure time was set to 200 ms for each image to obtain an adequate signal to noise ratio. As such, it takes $\sim 16\mathrm{s}$ to acquire a hyperspectral data cube of one field of view of the microscope system. The reason for the long acquisition time of our setup is partially due to the relatively low excitation flux of the employed mercury lamp [coupled to a liquid light guide in Fig. 7(a)]. It is possible to use a laser to replace the mercury lamp to increase the illumination flux and shorten the acquisition time. A cylindrical lens array can also be used to better couple light through the slit-array mask. Figures 7(b1) and 7(b2) show the two recovered images at two different wavelengths. Figure 7(c) shows the combined color image based on the recovered spectral information. The green and orange lines in Fig. 7(d) show the measured spectra of two different types of fluorescence microspheres. The black dash lines in Fig. 7(d) show the spectra measured by a regular spectrometer (FluoroMax Plus, HORIBA Scientific). We can see that they are in a good agreement with each other.

4.

Conclusion

We report the development of a high-throughput hyperspectral WSI system based on slit-array dispersion. We demonstrate a slit-array detection scheme for absorption-based measurements and a slit-array projection scheme for fluorescence-based measurements. Compared with the conventional pushbroom configuration, the use of slit-array mask facilitates parallel measurements of the spectral information and shortens the acquisition time for WSI. The spectral resolution and spectral range in the reported system can be adjusted by simply changing the orientation of the slit-array mask. The capability of adjusting spectral resolution and spectral range in the reported system provides a flexible solution to multilabeling schemes. This feature, to the best of our knowledge, has not been reported before. The slit-array projection scheme enables the detection of all fluorescence emission from the sample and the induced phototoxicity is minimized. This is different from conventional filter-based implementations, where many valuable photons are discarded in the acquisition process.

The development of the reported system is timely as well. WSI systems have been approved for the primary diagnostic use by the US FDA in 2017. The performance of cost-effective complementary metal-oxide semiconductor camera has been substantially improved in the past few years. High-speed USB 3.0/3.1 datalink and terabyte-scale solid-state drives have also entered the mainstream market and become more affordable recently. We envision that all these advancements will greatly facilitate the development of next-generation terapixel hyperspectral WSI systems.

There are a few future directions for further developing the reported systems. First, we can use the slit-array mask for both illumination and detection. In this case, we can perform hyperspectral confocal imaging. Second, the scanning process in the current platform relies on the motorized microscope stage. We can use a DMD to replace the slit-array mask. The scanning process can be digitally performed using the DMD instead of the motorized stage. We can also use the DMD-controlled mask for both illumination and detection. Third, we can improve the light delivering efficiency by placing a cylindrical lens array between the light source and the slit-array mask. Fourth, we can employ other image processing methods for the slit-array projection scheme in Fig. 7. For example, we can recover multilayer information from the captured data²³ or recover information beyond the frequency limit of the employed objective lens.^24,25