Translator Disclaimer
12 October 2020 High-resolution 3D light-field imaging
Author Affiliations +
Abstract

Significance: High-speed 3D imaging methods have been playing crucial roles in many biological discoveries.

Aim: We present a hybrid light-field imaging system and image processing algorithm that can visualize high-speed biological events.

Approach: The hybrid light-field imaging system uses the selective plane optical illumination, which simultaneously records a high-resolution 2D image and a low-resolution 4D light-field image. The high-resolution 4D light-field image is obtained by applying the hybrid algorithm derived from the deconvolution and phase retrieval methods.

Results: High-resolution 3D imaging at a speed of 100-s volumes per second over an imaging field of 250  ×  250  ×  80  μm3 in the x, y, and z axis, respectively, is achieved with a 2.5 times enhancement in lateral resolution over the entire imaging field compared with standard light-field systems. In comparison to the deconvolution algorithm, the hybrid algorithm addresses the artifact issue at the focal plane and reduces the computation time by a factor of 4.

Conclusions: The new hybrid light-field imaging method realizes high-resolution and ultrafast 3D imaging with a compact setup and simple algorithm, which may help discover important applications in biophotonics to visualize high-speed biological events.

1.

Introduction

Visualization of 3D high-speed biological events, e.g., signaling of neural circuits, has been the driving force of many emerging 3D imaging techniques, such as random-access microscopy1,2 and light-sheet microscopy (LSM).3,4 The former approach selectively scans the laser beam in the regions of interest to monitor biological events at speeds up to 10-s kHz,1,2 and the latter approach exploits a light sheet to parallelly illuminate and record 2D images at frame rates up to 100-s Hz;3,4 accordingly, clear 3D images can be obtained through the application of custom-designed phase masks3 or synchronization with an axial scanner,4 achieving a volume image rate of 30  Hz.

Light-field microscopy (LFM)5 is a high-speed 3D imaging technique that captures a volume image via a single exposure from a camera, where the imaging speed is limited by cameras. Briefly, a 3D image in LFM can be restored from a recorded “light-field image” that simultaneously contains the positions of objects, i.e., 2D image, and angles of illumination, i.e., another 2D image, which is termed a “4D” image. Comparing with other fast fluorescent imaging methods, LFM achieves the highest volume imaging rate at the expense of low spatial and angular resolution,5,6 which prevents the broad adoption of LFM in biological studies. To address the issue, both computational and optical methods have been developed. For example, deconvolution algorithms have been applied to LFM7 for imaging neuronal activities at high speed.810 Although simple and effective, this method is prone to generating artifacts near the native image plane (NIP) and requires powerful computers for processing dense 3D images. In addition to the deconvolution algorithm, the phase space retrieval method11 has been developed to computationally recover a high-resolution light-field image. Comparing with the deconvolution method, this method shows decreasing fidelity away from the NIP. In terms of optical methods, insertion of phase masks,12 optimization of optical distances,13 adoption of an orthogonal detection arm,14 and selective plane illumination15 have been demonstrated to improve the spatial and angular resolution of LFM. However, the increased complexity in optical design often prevents the easy application of computational methods, e.g., deconvolution algorithms.

In this work, we present a new hybrid LFM via selective optical illumination, which reconstructs a high-resolution 4D light-field image from a simultaneously recorded 2D image and low-resolution 4D light-field image. Briefly, multi-layer 2D images within the depth of the focus (DOF) are generated via deconvolution. Next, the multi-layer images are used as constraints for the iterated Fourier transform of the interpolated 4D light-field image to reconstruct a high-resolution 4D light-field image. Accordingly, the hybrid LFM achieves an imaging speed of 100-s volumes per second (vps) with a 3D field of view of 250×250×80  μm3 in the x,y, and z axes, respectively. The measured axial and lateral resolutions, i.e., full width at half maximum (FWHM), are 9.36 and 1.85  μm, respectively.

2.

Experiments

2.1.

Optical Configuration of the Hybrid LFM

Figure 1 shows the optical configuration of the hybrid LFM. The light source is a 488-nm continuous-wave laser (MLD, Cobolt). The laser first passes through the beam expander and neutral density filter and enters the polygonal mirror scanner (DT-72-290-025, Lincoln Laser), which scans the laser beam horizontally to form a light sheet. Next, the light sheet is relayed to a galvanometric scanner (6220HB, Cambridge Technology) via a 4-f system, i.e., L1 and L2, and scanned vertically over a range of ±40  μm. Then, the laser is coupled to the illumination objective (Plan 10×/0.30  W, Nikon), which images the scanning light sheet to the focal region to form a uniformly illuminated imaging volume of 250×250×80  μm3. For detection, the emissions are first collected by the detection objective (DO) (APO LWD 40×/0.80  W, Nikon) and coupled to two imaging arms via a tube lens (L5). The emissions are equally separated by a beam splitter (50:50, Chroma) to a scientific complementary metal-oxide-semiconductor (sCMOS) camera, i.e., sCMOS 1 (Zyla 5.5, Andor), which generates high-resolution 2D images; and a light-field camera, which consists of a microlens array (MLA, APO-Q-P150-F3.5, OKO Tech), macrolens (180-mm F3.5, Canon), and sCMOS 2 (C13440, Hamamatsu). Specifically, the MLA is located at the focal plane of L5, and the macrolens maps the MLA focal plane to sCMOS 2. (It is worthy to note that the two sCMOS cameras have the same pixel sizes of 6.5  μm.) Lastly, the collected 2D and light-field images are combined to form high-resolution light-field images via a custom-developed program.

Fig. 1

Optical configuration of the hybrid LFM.

JBO_25_10_106502_f001.png

Figure 2 shows the flowchart of the hybrid algorithm. ILF (Nx*Ny*Nu*Nv) and IRe (mNx*mNy) represents low-resolution light-field images from sCMOS 2 and high-resolution 2D images from sCMOS 1, respectively, where (x,y) and (u,v) represent the position and angular coordinates; and m is an interpolation constant. IRe and ILF are simultaneously obtained by scanning the light sheet vertically over a range of ±40  μm via galvo within a single exposure time of sCMOS cameras, i.e., 10 ms. To begin,ILF is resized to ILF_High by a factor of m in the spatial domain via the bicubic interpolation, followed by a 4D fast Fourier transform (FFT) to obtain G, i.e., 4D Fourier information. It is worthy to note that Fn and Fn represent the n-dimension FFT and inverse FFT (IFFT), respectively. Next, multidepth high-resolution images IRe_High (n*mNx*mNy) (n=number of layers) are calculated from ILF via deconvolution, where the image at the focal plane (z=0) is replaced by the high-resolution image IRe. It is worthy to note that based on Fourier slicing theorem FFT of IRe_High represents slices along different directions in the Fourier domain of the 4D light-field data, which is used as constraints for slicing in the 4D Fourier domain G. Lastly, a 4D IFFT is performed to reconstruct the 4D Fourier signals G in the spatial domain. By setting the negative and complex numbers to zero, high-resolution light field images, i.e., ILF_new (mNx*mNy*Nu*Nv), are obtained. Table 1 presents the pseudocodes of the hybrid algorithm.

Fig. 2

Flowchart of the hybrid algorithm.

JBO_25_10_106502_f002.png

Table 1

Pseudocode of hybrid LFM algorithm.

1: Obtain IRe and ILF from sCMOS 1 and sCMOS 2, simultaneously
2: Calculate ILF_High by resizing ILF with a magnification of m
3: Obtain the 4D Fourier domain G of ILF_High by 4D FFT
4: Calculate IRe_High by deconvolution from ILF
5:  // the image at the focal plane is replaced by IRe
6: Obtain the 2D Fourier domain Gf of IRe_High by 2D FFT
7:  // Gf is used as constraints in the Fourier slicing theorem
8: For each discrete image depth f, do
9: Calculate the depth parameter α=f/F; (F = MLA focal length)
10: Update the 4D Fourier space by slicing at f by:
11:
G[x,y,xmNx(α1),ymNy(α1)]=Gf(x,y);
12: end
13: Obtain 4D high resolution image ILF_new from G by 4D IFFT

2.2.

Experimental Results

This section presents the experimental results of the proposed hybrid LFM. First, we discuss the resolution enhancement over the standard LFM. For standard LFM, the resolution can be estimated by the size of a microlens (dpitch) and the magnification (M) of the DO, i.e., resolution Rlateral=dpitch/M=150  μm/40=3.75  μm at the focal plane (z=0). It is worthy to note that the lateral resolution decreases when the imaging plane moves away from the focal plane. For the hybrid algorithm, a large m value can directly improve the lateral resolution of the reconstructed images, especially at the focal plane. As m continues to increase, the resolution enhancement effect gradually diminishes; in the meantime, the required computational power increases at a rate of m2. To select the optimal value of m, simulated experiments have been devised and conducted in MATLAB based on wave optics and scalar Debye theory.7,16 In the simulation, we calculate the fluorescence imaging results from sCMOS 1 and sCMOS 2, where the emissions are from an infinitely small spot at the focal plane. Next, the two simulated images are imported to the hybrid algorithm. Specifically, 11 2D images from z=40 to 40  μm with 8  μm intervals are generated via deconvolution, where the image at the focal plane is replaced by the 2D image from sCMOS 1. The 11 high-resolution images IRe_High are used as constraints in the Fourier slicing theorem to reconstruct a high-resolution 4D light field image. As shown in Fig. 3, the lateral resolution of the reconstructed images is plotted as a function of the interpolation parameter m and axial positions; and as a reference, the resolution of the standard LFM and diffraction limited are also plotted together in Fig. 3. From the results, one may observe the lateral resolution has been improved over the entire scanning range and the effect becomes more prominent with increasing distances away from the focal plane, i.e., the resolution of the hybrid LFM remains relatively constant over the range in comparison with standard LFM. The resolution in the z axis fluctuates as the hybrid algorithm uses high-resolution deconvoluted images at the selected depths (i.e., z=±40, ±32, ±24, ±16, and ±8  μm) as constraints to reconstruct the entire 3D volume based on phase retrieval. As such, better resolution is observed at these positions. From the results, we can confirm the increasing m values will enhance the resolution with a diminishing effect for m larger than 5. Considering the tradeoffs between resolution and computational cost, we select m=3 in the hybrid algorithm for imaging experiments in the later sections; this is close to the resolution enhancement limit of the deconvolution algorithm for standard LFM.7

Fig. 3

Lateral resolution of the hybrid LFM under various m values along the z axis.

JBO_25_10_106502_f003.png

In terms of axial resolution, the hybrid LFM should have comparable resolution with standard LFM. To estimate the axial resolution, we first apply the Sparrow criterion,17 where the Sparrow limit can be described as the closest distance on the microlens imaging plane that two focal points can be resolved (Robj):

Eq. (1)

Robj=0.47λNAM,
where λ is the wavelength and NA and M are the numerical aperture and magnification of the DO, respectively. The total resolvable spots (N) per microlens can be determined by the size of individual microlenses (dpitch), as described in Eq. (2)

Eq. (2)

N=dpitchRobj.

Given dpitch=150  μm, M=40, and NA=0.8, from Eqs. (1) and (2), we can find Robj=12.5  μm and N=12. From a previous study,5 the axial resolution (Raxial) and maximum axial imaging range (zrange) of LFM can be calculated via Eqs. (3) and (4), respectively,

Eq. (3)

Raxial(2+N)λn2NA2,

Eq. (4)

zrange(2+N2)λn2NA2,
where N represents the resolvable spots in a microlens, and n is the refractive index of immersion media (n=1.33 for water). By substituting our system parameters into Eqs. (3) and (4), the axial resolution (Raxial) and imaging range (zrange) of the hybrid LFM are found to be 7.74 and 80.24  μm, respectively. It is worthy to note that within the working volume, i.e., 40 to 40  μm, the axial resolution decreases as the imaging plane moves away from the focus.

To verify the predicted resolution and range, we performed imaging experiments on 1-μm fluorescent beads (F8819, Thermo Fisher Scientific) at different depths to characterize the lateral and axial point spread functions (PSF) throughout the entire imaging volume. The recorded raw images are processed by the hybrid LFM algorithm. To begin, we first identify the optimal value of m by setting m to different values and recording the corresponding images at z=32  μm, i.e., an out-of-focus position. The results are shown in Figs. 4(a)4(c), where m is set to 1, 3, and 5, respectively. From the results, one may observe that the lateral resolution is improved substantially when m is equal or >3. When m=5, the enhancement becomes visually indistinguishable, which is consistent with our prediction, as shown in Fig. 3.

Fig. 4

Reconstructed lateral PSF at z=32  μm for different m values.

JBO_25_10_106502_f004.png

We record the lateral and axial PSF images over the entire work volume, followed by image reconstruction (m=3) via the hybrid algorithm. Figures 5(a) and 5(b) show the reconstructed lateral and axial PSF, i.e., 5.25 and 15.69  μm, respectively, of the hybrid LFM at z=32  μm, i.e., worst-case scenario at the far end of the imaging field. In comparison, Figs. 5(c) and 5(d) show the lateral and axial PSF, i.e., 9.07 and 17.49  μm, respectively, of the standard LFM at z=32  μm. Figure 5(e) plots the reconstructed lateral and axial PSF over the entire volume for both the hybrid and standard LFM. From the results, we can conclude the hybrid FLM has effectively improved the lateral resolution by a factor of 2.5 over the entire imaging field. The lateral and axial PSF measured at the focal plane (z=0) are 1.85 and 9.36  μm, respectively. We also find the hybrid LFM achieves better axial resolution than the standard LFM in the vicinity of the focal region, i.e., z=±15  μm.

Fig. 5

3D PSF of the hybrid and standard LFM.

JBO_25_10_106502_f005.png

In this section, we perform 3D imaging experiments on biological specimens, i.e., fern spores (B17124 R8, Walter Products Inc.), to verify the performance of our hybrid LFM. Figures 6(a)6(e) show reconstructed images from the hybrid LFM at five different depths, i.e., 30, 5, 0, 5, and 30  μm, with a field of view of 250×250  μm2. (It is worthy to note that z=0  μm corresponds to the DO focal plane). In comparison, Figs. 6(f)6(j) show reconstructed images from the deconvolution algorithm; and Figs. 6(k)6(o) show images obtained from a standard LFM at the same imaging field. From the results, one may find the hybrid system yields images of better resolution and contrast throughout the imaging field in comparison to the standard LFM. Comparing with the deconvolution method, we find that the hybrid algorithm shows improved resolution around the focal plane, i.e., z=±5  μm, without any artifact; and comparable resolution at regions away from the focal plane (z=±30  μm). In terms of computation time, the hybrid algorithm can improve the speed by a factor of 3 to 4 due to the use of sparse deconvolution images, e.g., 10 deconvoluted images in our hybrid imaging process in Fig. 6. For example, when processing a light-field image with a size of 1892×1892  pixels, the deconvolution algorithm uses 56 min to reconstruct a 3D image stack from 40 to 40  μm with a step of 2  μm; in contrast, the hybrid algorithm uses 16 min to reconstruct a 3D image stack over the same imaging field. These calculations are performed on a workstation equipped with Intel(R) Core (TM) i9-7920X CPU at 2.90 GHz, 64 G RAM, and Nvidia Quadro P4000 graphic cards. To compare quantitatively, Fig. 6(p) plots the cross-sectional intensity profiles along the red, blue, and green dashed lines in Figs. 6(e), 6(j), and 6(o), respectively, where the intensity curves are normalized over each image, i.e., Figs. 6(e), 6(j), and 6(o), for fair comparisons. From the results, the calculated signal-to-background ratio (SBR) in Figs. 6(e), 6(j), and 6(o) are 1.55, 1.66, and 1.23 with a mean background signal of 0.32, 0.30, and 0.43, respectively. The imaging results affirm the performance improvement of the hybrid LFM over standard LFM and the deconvolution algorithm (in terms of speed and the resolution at the NIP). It is worthwhile to note that the imaging resolution of the hybrid LFM can be further increased using more than one light-sheet images, where each additional light-sheet image can improve the resolution over a local volume at the expense of increased system complexity and cost.

Fig. 6

3D image of fern spores reconstructed via the hybrid algorithm, standard LFM, and deconvolution algorithm. (a)–(e) Reconstructed cross-sectional images from the hybrid algorithm at z=30, 5, 0, 5, and 30  μm, respectively; (f)–(j) reconstructed images from the deconvolution algorithm at z=30, 5, 0, 5, and 30  μm, respectively; (k)–(o) reconstructed images from the standard LFM at z=30, 5, 0, 5, and 30  μm, respectively; and (p) normalized intensity profiles along the red, blue, and green dashed lines in (e), (j), and (o), respectively. Scale bar=50  μm. (See Videos 1 and 2 for demonstration.) (Video 1 [URL: https://doi.org/10.1117/1.JBO.25.10.106502.1] MP4, 303 KB and Video 2 [URL: https://doi.org/10.1117/1.JBO.25.10.106502.2] MP4, 176 KB).

JBO_25_10_106502_f006.png

3.

Conclusion

In conclusion, we have presented the design and characterization of a new hybrid LFM that generates high-resolution 3D images at a speed of 100-s vps over an imaging field of 250×250×80  μm3 in the x,y, and z axis, respectively. The collected raw 2D and 4D light-field images are processed by the hybrid algorithm, developed based on the Fourier slicing theorem; to balance the resolution and computational power, an optimal interpolation constant (m) is selected experimentally. Analytical models have been developed to guide the system design and predict the performance, i.e., imaging resolution and range. In the experiments, we first characterize the 3D PSF of both the hybrid and standard LFM. The results confirm the lateral resolution has been improved for 2.5 times over the entire imaging field comparing with standard LFM. Next, 3D imaging experiments on fern spore samples have been performed, where the hybrid LFM demonstrates substantially improved resolution and SBR over the standard LFM. In comparison with the deconvolution algorithm, the hybrid LFM has addressed the artifact issue at the focal plane and reduced the computation time by a factor of 4. The improved resolution and speed present great opportunities for scientists to adopt LFM for in vivo biological studies.

Disclosures

The authors have no relevant financial interests in this article and no potential conflicts of interest to disclose.

Acknowledgments

This research was supported by HKSAR Research Grants Council, General Research Fund (Grant No. 14206517); Innovation and Technology Commission, Innovation Technology Fund (Grant No. ITS/427/17).

References

1. 

K. Nadella et al., “Random-access scanning microscopy for 3D imaging in awake behaving animals,” Nat. Methods, 13 1001 –1004 (2016). https://doi.org/10.1038/nmeth.4033 1548-7091 Google Scholar

2. 

Q. Geng et al., “Digital micromirror device-based two-photon microscopy for three-dimensional and random-access imaging,” Optica, 4 (6), 674 –677 (2017). https://doi.org/10.1364/OPTICA.4.000674 Google Scholar

3. 

Y. Chang et al., “Synchronization-free light sheet microscopy based on a 2D phase mask,” Optica, 4 (9), 1030 –1033 (2017). https://doi.org/10.1364/OPTICA.4.001030 Google Scholar

4. 

M. B. Ahrens et al., “Whole-brain functional imaging at cellular resolution using light-sheet microscopy,” Nat. Methods, 10 413 –420 (2013). https://doi.org/10.1038/nmeth.2434 1548-7091 Google Scholar

5. 

M. Levoy et al., “Light field microscopy,” ACM Trans. Graphics, 25 924 –934 (2006). https://doi.org/10.1145/1141911.1141976 ATGRDF 0730-0301 Google Scholar

6. 

R. Ng et al., “Light field photography with a hand-held plenoptic camera,” 1 –11 (2005). Google Scholar

7. 

M. Broxton et al., “Wave optics theory and 3D deconvolution for the light field microscope,” Opt. Express, 21 (21), 25418 –25439 (2013). https://doi.org/10.1364/OE.21.025418 OPEXFF 1094-4087 Google Scholar

8. 

R. Prevedel et al., “Simultaneous whole-animal 3D imaging of neuronal activity using light-field microscopy,” Nat. Methods, 11 727 –730 (2014). https://doi.org/10.1038/nmeth.2964 1548-7091 Google Scholar

9. 

T. Nöbauer et al., “Video rate volumetric Ca2+ imaging across cortex using seeded iterative demixing (SID) microscopy,” Nat. Methods, 14 811 –818 (2017). https://doi.org/10.1038/nmeth.4341 1548-7091 Google Scholar

10. 

T. Skocek et al., “High-speed volumetric imaging of neuronal activity in freely moving rodents,” Nat. Methods, 15 429 –432 (2018). https://doi.org/10.1038/s41592-018-0008-0 1548-7091 Google Scholar

11. 

C. Lu et al., “High-resolution light-field imaging via phase space retrieval,” Appl. Opt., 58 (5), A142 –A146 (2019). https://doi.org/10.1364/AO.58.00A142 APOPAI 0003-6935 Google Scholar

12. 

N. Cohen et al., “Enhancing the performance of the light field microscope using wavefront coding,” Opt. Express, 22 (20), 24817 –24839 (2014). https://doi.org/10.1364/OE.22.024817 OPEXFF 1094-4087 Google Scholar

13. 

H. Li et al., “Fast, volumetric live-cell imaging using high-resolution light-field microscopy,” Biomed. Opt. Express, 10 (1), 29 –49 (2019). https://doi.org/10.1364/BOE.10.000029 BOEICL 2156-7085 Google Scholar

14. 

N. Wagner et al., “Instantaneous isotropic volumetric imaging of fast biological processes,” Nat. Methods, 16 497 –500 (2019). https://doi.org/10.1038/s41592-019-0393-z 1548-7091 Google Scholar

15. 

T. Truong et al., “High-contrast, synchronous volumetric imaging with selective volume illumination microscopy,” Commun. Biol., 3 74 (2020). https://doi.org/10.1038/s42003-020-0787-6 Google Scholar

16. 

M. Gu, Advanced Optical Imaging Theory, Springer, Berlin (2000). Google Scholar

17. 

S. Inoué and K. Spring, Video Microscopy, Springer, Berlin (1997). Google Scholar

Biography

Qiang Geng received his BE degree in mechanical engineering from Tsinghua University, Beijing, China, in 2015. He received his PhD from the Department of Mechanical and Automation Engineering, Chinese University of Hong Kong in 2019. His research interests include ultrafast laser applications and nonlinear microscopy.

Zhiqiang Fu received his BE degree in mechatronics engineering from Zhejiang University, Zhejiang, China, in 2018. He is currently a PhD student in the Department of Mechanical and Automation Engineering, the Chinese University of Hong Kong. His research interests include ultrafast laser applications, nanofabrication, and 3D microscopy.

Shih-Chi Chen received his BS degree in mechanical engineering from the National TsingHua University, Taiwan, in 1999. He received his SM and PhD degrees in mechanical engineering from the Massachusetts Institute of Technology, Cambridge, in 2003 and 2007, respectively. Following his graduate work, he entered a post-doctoral fellowship in the Wellman Center for Photomedicine, Harvard Medical School, where his research focused on biomedical optics and endomicroscopy. From 2009 to 2011, he was a senior scientist at Nano Terra, Inc., a startup company founded by Professor George Whitesides at Harvard University, to develop precision instruments for novel nanofabrication processes. Joining since 2011, he is presently a professor in the Department of Mechanical and Automation Engineering at the Chinese University of Hong Kong. His current research interests include ultrafast laser applications, biomedical optics, precision engineering, and nanomanufacturing.

© The Authors. Published by SPIE under a Creative Commons Attribution 4.0 Unported License. Distribution or reproduction of this work in whole or in part requires full attribution of the original publication, including its DOI.
Qiang Geng, Zhiqiang Fu, and Shih-Chi Chen "High-resolution 3D light-field imaging," Journal of Biomedical Optics 25(10), 106502 (12 October 2020). https://doi.org/10.1117/1.JBO.25.10.106502
Received: 9 June 2020; Accepted: 24 September 2020; Published: 12 October 2020
JOURNAL ARTICLE
9 PAGES


SHARE
Advertisement
Advertisement
Back to Top