Significance: High-speed 3D imaging methods have been playing crucial roles in many biological discoveries.
Aim: We present a hybrid light-field imaging system and image processing algorithm that can visualize high-speed biological events.
Approach: The hybrid light-field imaging system uses the selective plane optical illumination, which simultaneously records a high-resolution 2D image and a low-resolution 4D light-field image. The high-resolution 4D light-field image is obtained by applying the hybrid algorithm derived from the deconvolution and phase retrieval methods.
Results: High-resolution 3D imaging at a speed of 100-s volumes per second over an imaging field of 250 × 250 × 80 μm3 in the x, y, and z axis, respectively, is achieved with a 2.5 times enhancement in lateral resolution over the entire imaging field compared with standard light-field systems. In comparison to the deconvolution algorithm, the hybrid algorithm addresses the artifact issue at the focal plane and reduces the computation time by a factor of 4.
Conclusions: The new hybrid light-field imaging method realizes high-resolution and ultrafast 3D imaging with a compact setup and simple algorithm, which may help discover important applications in biophotonics to visualize high-speed biological events.
Visualization of 3D high-speed biological events, e.g., signaling of neural circuits, has been the driving force of many emerging 3D imaging techniques, such as random-access microscopy1,2 and light-sheet microscopy (LSM).3,4 The former approach selectively scans the laser beam in the regions of interest to monitor biological events at speeds up to 10-s kHz,1,2 and the latter approach exploits a light sheet to parallelly illuminate and record 2D images at frame rates up to 100-s Hz;3,4 accordingly, clear 3D images can be obtained through the application of custom-designed phase masks3 or synchronization with an axial scanner,4 achieving a volume image rate of .
Light-field microscopy (LFM)5 is a high-speed 3D imaging technique that captures a volume image via a single exposure from a camera, where the imaging speed is limited by cameras. Briefly, a 3D image in LFM can be restored from a recorded “light-field image” that simultaneously contains the positions of objects, i.e., 2D image, and angles of illumination, i.e., another 2D image, which is termed a “4D” image. Comparing with other fast fluorescent imaging methods, LFM achieves the highest volume imaging rate at the expense of low spatial and angular resolution,5,6 which prevents the broad adoption of LFM in biological studies. To address the issue, both computational and optical methods have been developed. For example, deconvolution algorithms have been applied to LFM7 for imaging neuronal activities at high speed.8–10 Although simple and effective, this method is prone to generating artifacts near the native image plane (NIP) and requires powerful computers for processing dense 3D images. In addition to the deconvolution algorithm, the phase space retrieval method11 has been developed to computationally recover a high-resolution light-field image. Comparing with the deconvolution method, this method shows decreasing fidelity away from the NIP. In terms of optical methods, insertion of phase masks,12 optimization of optical distances,13 adoption of an orthogonal detection arm,14 and selective plane illumination15 have been demonstrated to improve the spatial and angular resolution of LFM. However, the increased complexity in optical design often prevents the easy application of computational methods, e.g., deconvolution algorithms.
In this work, we present a new hybrid LFM via selective optical illumination, which reconstructs a high-resolution 4D light-field image from a simultaneously recorded 2D image and low-resolution 4D light-field image. Briefly, multi-layer 2D images within the depth of the focus (DOF) are generated via deconvolution. Next, the multi-layer images are used as constraints for the iterated Fourier transform of the interpolated 4D light-field image to reconstruct a high-resolution 4D light-field image. Accordingly, the hybrid LFM achieves an imaging speed of 100-s volumes per second (vps) with a 3D field of view of in the , and axes, respectively. The measured axial and lateral resolutions, i.e., full width at half maximum (FWHM), are 9.36 and , respectively.
Optical Configuration of the Hybrid LFM
Figure 1 shows the optical configuration of the hybrid LFM. The light source is a 488-nm continuous-wave laser (MLD, Cobolt). The laser first passes through the beam expander and neutral density filter and enters the polygonal mirror scanner (DT-72-290-025, Lincoln Laser), which scans the laser beam horizontally to form a light sheet. Next, the light sheet is relayed to a galvanometric scanner (6220HB, Cambridge Technology) via a 4-f system, i.e., L1 and L2, and scanned vertically over a range of . Then, the laser is coupled to the illumination objective (Plan , Nikon), which images the scanning light sheet to the focal region to form a uniformly illuminated imaging volume of . For detection, the emissions are first collected by the detection objective (DO) (APO LWD , Nikon) and coupled to two imaging arms via a tube lens (L5). The emissions are equally separated by a beam splitter (50:50, Chroma) to a scientific complementary metal-oxide-semiconductor (sCMOS) camera, i.e., sCMOS 1 (Zyla 5.5, Andor), which generates high-resolution 2D images; and a light-field camera, which consists of a microlens array (MLA, APO-Q-P150-F3.5, OKO Tech), macrolens (180-mm F3.5, Canon), and sCMOS 2 (C13440, Hamamatsu). Specifically, the MLA is located at the focal plane of L5, and the macrolens maps the MLA focal plane to sCMOS 2. (It is worthy to note that the two sCMOS cameras have the same pixel sizes of .) Lastly, the collected 2D and light-field images are combined to form high-resolution light-field images via a custom-developed program.
Figure 2 shows the flowchart of the hybrid algorithm. () and () represents low-resolution light-field images from sCMOS 2 and high-resolution 2D images from sCMOS 1, respectively, where () and () represent the position and angular coordinates; and is an interpolation constant. and are simultaneously obtained by scanning the light sheet vertically over a range of via galvo within a single exposure time of sCMOS cameras, i.e., 10 ms. To begin, is resized to by a factor of in the spatial domain via the bicubic interpolation, followed by a 4D fast Fourier transform (FFT) to obtain , i.e., 4D Fourier information. It is worthy to note that and represent the -dimension FFT and inverse FFT (IFFT), respectively. Next, multidepth high-resolution images () () are calculated from via deconvolution, where the image at the focal plane () is replaced by the high-resolution image . It is worthy to note that based on Fourier slicing theorem FFT of represents slices along different directions in the Fourier domain of the 4D light-field data, which is used as constraints for slicing in the 4D Fourier domain . Lastly, a 4D IFFT is performed to reconstruct the 4D Fourier signals in the spatial domain. By setting the negative and complex numbers to zero, high-resolution light field images, i.e., (), are obtained. Table 1 presents the pseudocodes of the hybrid algorithm.
Pseudocode of hybrid LFM algorithm.
|1: Obtain and from sCMOS 1 and sCMOS 2, simultaneously|
|2: Calculate by resizing with a magnification of|
|3: Obtain the 4D Fourier domain of by 4D FFT|
|4: Calculate by deconvolution from|
|5: // the image at the focal plane is replaced by|
|6: Obtain the 2D Fourier domain of by 2D FFT|
|7: // is used as constraints in the Fourier slicing theorem|
|8: For each discrete image depth , do|
|9: Calculate the depth parameter ; (F = MLA focal length)|
|10: Update the 4D Fourier space by slicing at by:|
|13: Obtain 4D high resolution image from by 4D IFFT|
This section presents the experimental results of the proposed hybrid LFM. First, we discuss the resolution enhancement over the standard LFM. For standard LFM, the resolution can be estimated by the size of a microlens () and the magnification () of the DO, i.e., resolution at the focal plane (). It is worthy to note that the lateral resolution decreases when the imaging plane moves away from the focal plane. For the hybrid algorithm, a large value can directly improve the lateral resolution of the reconstructed images, especially at the focal plane. As continues to increase, the resolution enhancement effect gradually diminishes; in the meantime, the required computational power increases at a rate of . To select the optimal value of , simulated experiments have been devised and conducted in MATLAB based on wave optics and scalar Debye theory.7,16 In the simulation, we calculate the fluorescence imaging results from sCMOS 1 and sCMOS 2, where the emissions are from an infinitely small spot at the focal plane. Next, the two simulated images are imported to the hybrid algorithm. Specifically, 11 2D images from to with intervals are generated via deconvolution, where the image at the focal plane is replaced by the 2D image from sCMOS 1. The 11 high-resolution images are used as constraints in the Fourier slicing theorem to reconstruct a high-resolution 4D light field image. As shown in Fig. 3, the lateral resolution of the reconstructed images is plotted as a function of the interpolation parameter and axial positions; and as a reference, the resolution of the standard LFM and diffraction limited are also plotted together in Fig. 3. From the results, one may observe the lateral resolution has been improved over the entire scanning range and the effect becomes more prominent with increasing distances away from the focal plane, i.e., the resolution of the hybrid LFM remains relatively constant over the range in comparison with standard LFM. The resolution in the axis fluctuates as the hybrid algorithm uses high-resolution deconvoluted images at the selected depths (i.e., , , , , and ) as constraints to reconstruct the entire 3D volume based on phase retrieval. As such, better resolution is observed at these positions. From the results, we can confirm the increasing values will enhance the resolution with a diminishing effect for larger than 5. Considering the tradeoffs between resolution and computational cost, we select in the hybrid algorithm for imaging experiments in the later sections; this is close to the resolution enhancement limit of the deconvolution algorithm for standard LFM.7
In terms of axial resolution, the hybrid LFM should have comparable resolution with standard LFM. To estimate the axial resolution, we first apply the Sparrow criterion,17 where the Sparrow limit can be described as the closest distance on the microlens imaging plane that two focal points can be resolved ():
Given , , and , from Eqs. (1) and (2), we can find and . From a previous study,5 the axial resolution () and maximum axial imaging range () of LFM can be calculated via Eqs. (3) and (4), respectively,
To verify the predicted resolution and range, we performed imaging experiments on fluorescent beads (F8819, Thermo Fisher Scientific) at different depths to characterize the lateral and axial point spread functions (PSF) throughout the entire imaging volume. The recorded raw images are processed by the hybrid LFM algorithm. To begin, we first identify the optimal value of by setting to different values and recording the corresponding images at , i.e., an out-of-focus position. The results are shown in Figs. 4(a)–4(c), where is set to 1, 3, and 5, respectively. From the results, one may observe that the lateral resolution is improved substantially when is equal or . When , the enhancement becomes visually indistinguishable, which is consistent with our prediction, as shown in Fig. 3.
We record the lateral and axial PSF images over the entire work volume, followed by image reconstruction () via the hybrid algorithm. Figures 5(a) and 5(b) show the reconstructed lateral and axial PSF, i.e., 5.25 and , respectively, of the hybrid LFM at , i.e., worst-case scenario at the far end of the imaging field. In comparison, Figs. 5(c) and 5(d) show the lateral and axial PSF, i.e., 9.07 and , respectively, of the standard LFM at . Figure 5(e) plots the reconstructed lateral and axial PSF over the entire volume for both the hybrid and standard LFM. From the results, we can conclude the hybrid FLM has effectively improved the lateral resolution by a factor of over the entire imaging field. The lateral and axial PSF measured at the focal plane () are 1.85 and , respectively. We also find the hybrid LFM achieves better axial resolution than the standard LFM in the vicinity of the focal region, i.e., .
In this section, we perform 3D imaging experiments on biological specimens, i.e., fern spores (B17124 R8, Walter Products Inc.), to verify the performance of our hybrid LFM. Figures 6(a)–6(e) show reconstructed images from the hybrid LFM at five different depths, i.e., , , 0, 5, and , with a field of view of . (It is worthy to note that corresponds to the DO focal plane). In comparison, Figs. 6(f)–6(j) show reconstructed images from the deconvolution algorithm; and Figs. 6(k)–6(o) show images obtained from a standard LFM at the same imaging field. From the results, one may find the hybrid system yields images of better resolution and contrast throughout the imaging field in comparison to the standard LFM. Comparing with the deconvolution method, we find that the hybrid algorithm shows improved resolution around the focal plane, i.e., , without any artifact; and comparable resolution at regions away from the focal plane (). In terms of computation time, the hybrid algorithm can improve the speed by a factor of 3 to 4 due to the use of sparse deconvolution images, e.g., 10 deconvoluted images in our hybrid imaging process in Fig. 6. For example, when processing a light-field image with a size of , the deconvolution algorithm uses 56 min to reconstruct a 3D image stack from to with a step of ; in contrast, the hybrid algorithm uses 16 min to reconstruct a 3D image stack over the same imaging field. These calculations are performed on a workstation equipped with Intel(R) Core (TM) i9-7920X CPU at 2.90 GHz, 64 G RAM, and Nvidia Quadro P4000 graphic cards. To compare quantitatively, Fig. 6(p) plots the cross-sectional intensity profiles along the red, blue, and green dashed lines in Figs. 6(e), 6(j), and 6(o), respectively, where the intensity curves are normalized over each image, i.e., Figs. 6(e), 6(j), and 6(o), for fair comparisons. From the results, the calculated signal-to-background ratio (SBR) in Figs. 6(e), 6(j), and 6(o) are 1.55, 1.66, and 1.23 with a mean background signal of 0.32, 0.30, and 0.43, respectively. The imaging results affirm the performance improvement of the hybrid LFM over standard LFM and the deconvolution algorithm (in terms of speed and the resolution at the NIP). It is worthwhile to note that the imaging resolution of the hybrid LFM can be further increased using more than one light-sheet images, where each additional light-sheet image can improve the resolution over a local volume at the expense of increased system complexity and cost.
In conclusion, we have presented the design and characterization of a new hybrid LFM that generates high-resolution 3D images at a speed of 100-s vps over an imaging field of in the , and axis, respectively. The collected raw 2D and 4D light-field images are processed by the hybrid algorithm, developed based on the Fourier slicing theorem; to balance the resolution and computational power, an optimal interpolation constant () is selected experimentally. Analytical models have been developed to guide the system design and predict the performance, i.e., imaging resolution and range. In the experiments, we first characterize the 3D PSF of both the hybrid and standard LFM. The results confirm the lateral resolution has been improved for 2.5 times over the entire imaging field comparing with standard LFM. Next, 3D imaging experiments on fern spore samples have been performed, where the hybrid LFM demonstrates substantially improved resolution and SBR over the standard LFM. In comparison with the deconvolution algorithm, the hybrid LFM has addressed the artifact issue at the focal plane and reduced the computation time by a factor of 4. The improved resolution and speed present great opportunities for scientists to adopt LFM for in vivo biological studies.
The authors have no relevant financial interests in this article and no potential conflicts of interest to disclose.
This research was supported by HKSAR Research Grants Council, General Research Fund (Grant No. 14206517); Innovation and Technology Commission, Innovation Technology Fund (Grant No. ITS/427/17).
Qiang Geng received his BE degree in mechanical engineering from Tsinghua University, Beijing, China, in 2015. He received his PhD from the Department of Mechanical and Automation Engineering, Chinese University of Hong Kong in 2019. His research interests include ultrafast laser applications and nonlinear microscopy.
Zhiqiang Fu received his BE degree in mechatronics engineering from Zhejiang University, Zhejiang, China, in 2018. He is currently a PhD student in the Department of Mechanical and Automation Engineering, the Chinese University of Hong Kong. His research interests include ultrafast laser applications, nanofabrication, and 3D microscopy.
Shih-Chi Chen received his BS degree in mechanical engineering from the National TsingHua University, Taiwan, in 1999. He received his SM and PhD degrees in mechanical engineering from the Massachusetts Institute of Technology, Cambridge, in 2003 and 2007, respectively. Following his graduate work, he entered a post-doctoral fellowship in the Wellman Center for Photomedicine, Harvard Medical School, where his research focused on biomedical optics and endomicroscopy. From 2009 to 2011, he was a senior scientist at Nano Terra, Inc., a startup company founded by Professor George Whitesides at Harvard University, to develop precision instruments for novel nanofabrication processes. Joining since 2011, he is presently a professor in the Department of Mechanical and Automation Engineering at the Chinese University of Hong Kong. His current research interests include ultrafast laser applications, biomedical optics, precision engineering, and nanomanufacturing.