In the past decades, fluorescence microscopy has evolved toward superresolution imaging.22.214.171.124.6.–7 Structured illumination microscopy (SIM),89.10.–11 as one of the most promising superresolution techniques, possesses the merits of a high framing rate and low excitation intensity, compared to the stimulated emission depletion microscopy (),12 the reversible saturable optical fluorescence transition microscopy (),13 and the saturated SIM ().14 SIM in its linear form can increase the imaging resolution by a factor of two beyond the traditional wide-field microscopy when illuminating through the imaging objective lens. In this approach, a periodic sinusoidal fringe illumination with a spatial frequency close to the diffraction limit is adopted to shift the unresolved high-frequency features of the sample to the optical transfer function (OTF) domain of the objective lens. In addition, speckle-based SIM,15,16 instant SIM,11,17 and other forms of SIM18 are presently gaining increasing interest. These techniques further enrich the theory and address parts of the limitations of traditional SIM. Nevertheless, the maximum resolution extension of SIM is only twofold over the conventional microscopy. Further improvement of resolution can be realized by utilizing the nonlinear response of fluorescence molecules, such as saturated excitation (SE)14,19 or a photoswitchable (PS)20,21 mechanism. In practice, the SE-based nonlinear SIM requires high excitation intensity to satisfy the saturation condition that may harm the biological activity of living samples. For the PS-based nonlinear SIM, a relatively low intensity is permitted to activate the PS effect, but the photostability requirement for fluorescent dyes dramatically increases. Recently, the PS fluorescent protein was used in the PS-based nonlinear SIM technique developed by Betzig et al. to achieve 45- to 62-nm resolution.21 As the nonlinear SIM demands high excitation intensity or special fluorescent dyes with much more raw images required than the linear SIM, they also used a 1.7-NA objective to achieve 84-nm resolution under the linear total internal reflection fluorescence (TIRF)-SIM mode. Due to the advantages of low excitation power and high imaging speed, the linear SIM is still an ideal choice for dynamic imaging of living cells.
The generation and rapid phase control of high-quality fringe patterns is the key requirement in hardware for the SIM technique. This can be done by employing spatial light modulators (SLM), such as liquid crystal on silicon SLM, to generate adjustable orientation fringes by interference of two beams diffracted from the phase gratings addressed on the SLM.22 However, for maximal interference contrast, additional polarization control to maintain s-polarization for different fringe orientations is technically demanded, which often makes the optical configuration more complex. In addition, the high coherence of a laser beam inevitably produces speckle noises, which will seriously degrade the image quality. Although dynamic averaging of the speckle noise using a rotating diffuser or random waggling of the homogenizers such as a multimode fibers phase modulator will produce a smooth image, it will also limit the image acquisition speed and increase the complexity of the system. To address these limitations, we previously developed a micromirror device (DMD)-based light-emitting diode (LED)-illumination SIM system.23,24 In that approach, a binary fringe pattern loaded on DMD is demagnified and projected onto the specimen. The higher orders of spatial frequencies of the binary fringe are naturally blocked off due to the low-pass filtering effect of the objective lens, leading to a sinusoidal fringe illumination in the sample plane. The DMD projection and LED-illumination system have a few advantages, including no polarization control, high throughput, ease of multiwavelength switching, and free of speckle-noise. However, it should be noticed that the modulation depth of the sinusoidal fringe illumination projected on the specimen is determined by the OTF of the objective, which makes it decrease with the increase of the fringe frequency and the imaging depth. The low modulation depth of patterns will affect the quality of the final reconstructed images. So, the image reconstruction algorithm is an important issue for high-quality superresolution image retrieval. In this process, it is crucial to precisely determine the parameters of the fringe illumination pattern, especially the initial phase estimation.25 Incorrect initial phase estimation will seriously affect the superresolution result and leads to additional artifacts. Shroff et al.26 have presented a method for estimating the pattern phase by analyzing the phase of peaks (POP) of the delta function in the spectral space of spatial frequency of the captured image. It is a commonly used algorithm in superresolution image reconstruction for the linear SIM.27,28 However, for high-frequency or low modulation depth illumination patterns, this algorithm is less reliable. Wicker et al.29,30 have proposed two alternative methods based on iterative cross-correlation and noniterative auto-correlation reconstruction (ACR) algorithms, respectively. These algorithms predominantly circumvent the limitation of the POP method in the condition of a high-frequency illumination pattern. Nevertheless, for a low modulation depth illumination pattern, they did not provide a detailed discussion. In short, although the DMD-projection-based LED-illumination SIM system successfully addresses parts of limitation arising from the liquid crystal spatial light modulator-based laser-illumination SIM system, the optimal reconstruction algorithm still remains a question especially at low modulation depth of illumination patterns to keep the fidelity of the superresolution image.
In this paper, we propose a reconstruction algorithm based on an image recombination transform (IRT) scheme to determine the initial phase accurately and simplify the process of extracting the high-order spectral components as well. The precise solution of the initial phase can be obtained without any approximate conditions even at a very low modulation depth of fringe illumination. The IRT algorithm does not contain an iterative operation, which makes the whole reconstruction process fast and automatic. The validity of the IRT algorithm is demonstrated by imaging the bovine pulmonary arterial endothelial (BPAE) cells in our built DMD-projection-based, multicolor-LED-illumination SIM system. The merit of the system incorporated with the proposed algorithm allows for fluorescence imaging at excitation intensity as low as using the LED light source, which represents a great decrease in excitation intensity compared to existing SIM systems (e.g., 30 to ).21 The IRT algorithm can also be applied for the laser interference-based SIM as a substitution of the POP algorithm.
Considering a specimen characterized by a spatial distribution of fluorophores density , it is illuminated by a cosine fringe intensity pattern with the form31 Therefore, the out-of-focus projection in the detected plane can be simplified as a nonmodulation item . Then, the final detected image can be rewritten as
The frequency distribution of and is the unresolvable high-frequency features of the sample and shift to the support of the OTF, . Generally, in the reconstruction procedure of SIM, three raw images are taken with the fringe phase shifted by between adjacent images. Combined with the deconvolution operation,32,33 the unresolved high-frequency features can be solved by Eq. (6) and only the frequency component contains the out-of-focus information
Shroff et al.26 have proposed the POP algorithm for estimating the initial phase, in which they analyze the value of the Fourier image at the pattern peak (i.e., ) with the form
The POP algorithm works well when all the above assumptions are fulfilled. For example, in the TIRF-SIM where the fluorescence excitation is generated in a thin volume with a depth typically below 200 nm, this excitation scheme has a very weak background and the POP algorithm can provide good precision in estimating the initial phase and separating the components well. However, in some cases, such as an existing strong background, or having a weak modulation depth or high-frequency illumination pattern, the POP algorithm cannot give an appropriate result.29,30
To address the limitation of the POP algorithm, here, we employ the phase shift between two adjacent images by rather than the commonly used value of . Using Eq. (4), we can easily obtain the expression of the three captured images
The Fourier transform of the recombined complex image is presented in the form
Compared to the POP algorithm described by Eq. (9), the proposed IRT algorithm can provide a precise solution of the fringe initial phase without the assumptions of weak background and high modulation depth. In addition, the IRT algorithm can directly solve the unresolved in-focus high-frequency components of and by Eqs. (17) and (18), avoiding having to solve the matrix equation of Eq. (6).
To verify the validity and feasibility of the IRT method, we first make a numerical simulation. In the simulation, the fringe illumination pattern is projected onto the sample through an oil-immersed objective (, , ). The fluorescence signal (emission wavelength at 461 nm) is collected by a digital camera (, pixel size ). The virtual object and its Fourier transform spectrum are shown in Figs. 1(a) and 1(b), respectively.
According to the Abbe’s diffraction limit equation, the minimum period of the projection fringe is 155 nm at with objective of . The initial phase estimation error under the condition of low modulation depth ranging from 0.001 to 0.12 with the fringe period of 193 and 156 nm are calculated by the POP, ACR, and IRT algorithms, respectively, as shown in Fig. 2. In theory, the phase error of the IRT algorithm should be zero in the absence of noise. The phase errors of the POP and ACR algorithms increase rapidly when the modulation depth is below 0.04 and 0.02, respectively, even without noise contribution, while the IRT algorithm always retains a relatively high precision over the whole range of the modulation depth. Considering Poisson noise is the predominant type of noise in low intensity images, we first add the Poisson noise in simulation [Figs. 2(a) and 2(c)] for comparing the performance of different algorithms. In this case, the maximal phase errors of the POP, ACR, and IRT algorithms read 7.9 deg, 7.3 deg, and 0.2 deg, respectively, at the period of 193 nm, corresponding to 80% of the maximum frequency supported by the OTF. When the fringe period decreases to 156 nm, approximating to the diffraction limit of the simulated system, the maximal phase errors of the POP algorithm increases up to 61 deg at the modulation depth of 0.001. On the other hand, the major contribution of electronic noise is mostly the Gaussian distribution. Thus, the comparison for the phase errors with Gaussian noise is also simulated in Fig. 2. The variances of the noise distribution are set to 0.001 [Figs. 2(b) and 2(d)]. Note that the sample spectrum signal is so weak at the OTF edge that the Gaussian noise becomes a major contribution. Therefore, the error curves fluctuate strenuously with the fringe period decreasing in the presence of Gaussian noise. In that condition, the IRT algorithm still can provide a relatively acceptable result, especially at the fringe period of 193 nm.
In short, for raw images with high modulation depth, all the algorithms are able to give a good performance in reducing the phase error. But in the condition of low modulation depth, only the IRT algorithm can estimate the initial phase with the best precision. However, similar to the other algorithms, if the noise is too strong, the IRT algorithm may not guarantee an acceptable precision of the initial phase.
To make a superresolution image reconstruction of SIM, three orientations of illumination fringes at 0 deg, 60 deg, and 120 deg are performed at the fringe period of 156 nm. The reconstructed images of the simulated object are shown in Fig. 3 with different algorithms at different modulation depth of fringe for comparison. Figure 3(a) is the wide-field image and its central region marked by a dashed box is magnified. Figures 3(b)–3(d) are the corresponding magnified superresolution images reconstructed by the IRT algorithm, Figs. 3(e)–3(g) and 3(h)–3(j) are corresponding to the ACR algorithm and the POP algorithm. The Fourier spectra of each reconstructed image are displayed at the top-right corners of the magnified images. It can be seen that all of the algorithms can successfully extend the support of the OTF in the frequency domain and enhance the spatial resolution of the image. However, with the decrease of the modulation depth, it is more and more difficult for the POP algorithm to separate the components well, and thus some residual errors come into being because of the estimation error of the initial phase. As previously speculated, owing to the incorrect spectrum expansion, a series of wave-like artifacts appear in the reconstructed images. This is particularly evident in Fig. 3(h). On the other hand, although the ACR algorithm is an effective method for phase estimation, the reconstruction process is time-consuming compared to the POP and IRT algorithms. Meanwhile, when the modulation depth is less than 0.01, a slight wave-like artifact also appears in the reconstructed image. In contrast, the IRT algorithm always retains a good performance even under low modulation depth, as seen in Fig. 3(b). When the modulation depth increases to more than 0.1, all of the algorithms give the same results.
Experiments and Results
To further prove the applicability of the IRT algorithm, we used our developed DMD-projection-based LED-illumination SIM system23,24 to carry out the following experiment. The scheme of the SIM system is shown in Fig. 4. A four-wavelength high-power LEDs assembly (LED4D251, Thorlabs Inc.) with switchable wavelengths of 405, 470, 565, and 625 nm is employed as the illumination source for multiwavelength excitation. The LED light enters the TIR-Prism and is reflected to the DMD chip (DLP7000UV, Texas Instruments Inc.), where the fringe pattern is loaded. The light modulated by the DMD passes through a demagnifying optical system, consisting of a collimating lens, and a objective (Apo TIRF, NA1.49, Nikon Inc., Japan), to produce a sinusoidal fringe illumination projected onto the specimen. A quad-band filter set (including a 390/482/563/640 nm quad-band bandpass excitation filter, a 446/523/600/677 nm quad-band bandpass emission filter, and a R405/488/561/635 nm quad-edge dichroic beamsplitter, Semrock Inc.) are used to separate the excitation lights and the emission fluorescence signals. A sCMOS camera with a maximum full-frame rate of 100 fps (Orca Flash4.0, , 16 bits gray depth, Hamamatsu Inc., Japan) is used to capture the two-dimensional image. The phase shifts of the fringe are controlled by the DMD chip without mechanical movement. If the fringe period is designed to occupy pixels on the DMD chip, the minimum value of the phase shift will be . Thus, to achieve a phase shift, just setting four pixels per period on the DMD chip, the illumination fringe phases corresponding to , , and , will be obtained, respectively. In order to achieve a high imaging speed and a simplified pattern design, we apply only two perpendicular illumination fringe orientations in the experiment. Nevertheless, it is possible to generate multiorientation illuminations with specifically designed illumination patterns.
The photobleaching and phototoxicity effects have constantly been obstacles to long-term observation of living cells in fluorescence microscopy. The high excitation intensity will accelerate the photobleaching of fluorescent molecules and be harmful to specimens.34 The excitation intensities for the existing SIM systems (e.g., 30 to ) are usually much stronger than that of the wide-field or light-sheet microscopies. In this experiment, we could image at intensities as low as , which is close to the life evolved condition ().35 This makes the system more suitable for bioimaging of low excitation intensities and long-term observation of specimens.
To demonstrate the feasibility of the IRT algorithm for the projection-based SIM, we employ the trichrome-stained BPAE cells (F36924, Thermo Fisher Scientific Inc.) as the specimen, which are labeled with MitoTracker Red CMXRos for Mitochondria, Alexa Fluor 488 phalloidin for F-actin, and DAPI for nuclei. The cells are sequentially illuminated using the excitation wavelengths of 405, 470, and 565 nm to achieve multicolor images. In this experiment, the modulation depth of the fringe is measured to be 0.046. Figure 5(a) shows a high-fidelity superresolution image of the F-actin obtained by the IRT algorithm. By applying the POP algorithm to process the same raw data, some unwanted residues appear in the Fourier spectrum as indicated by the pale-green circles in Fig. 5(g), resulting in some artifacts in the reconstructed image of Fig. 5(c). Compared to the POP algorithm, the ACR algorithm obtains an improvement in artifacts suppression and the residual fringes are hardly seen in its reconstructed image [Fig. 5(d)]. But there still exist a few extra components in the Fourier spectrum [Fig. 5(h)], leading to the reduction of image quality. In terms of computation time, the POP, ACR, and IRT algorithms take 2.97, 9.65, and 3.02 s, respectively, to calculate one superresolution image from six raw images () using a personal computer (Intel Core i7-4790K 4 GHz processor and 32 GB RAM) with the MATLAB software (R2013a) operating under Windows 7 (SP1) x64. Similar experimental results of superimposed tricolor images of BPAE cells are shown in Fig. 6. Here, the mitochondria are rendered in red, the F-actins are rendered in green, and the nuclei are rendered in blue. By the POP algorithm, the F-actin and mitochondria exhibit some extra stripes in the reconstructed images [Figs. 6(b) and 6(f)]. These artifacts may confuse the scientific evaluation of biological morphology. However, in the IRT superresolution images [Figs. 6(d) and 6(h)], such residual stripes are not observed and the image qualities are better than the ACR images shown in Figs. 6(c) and 6(g). This once again proves that the IRT algorithm is a robust and high-accuracy reconstruction method for superresolution imaging.
Although the IRT algorithm is immune to the modulation depth of the illumination fringe, it should be noticed that the noise in the captured structured illumination images is another challenge to the accurate determination of the initial phase. In fact, when the image noise is too high, Eq. (15) cannot be strictly tenable, because the noise contribution must be considered. As a result, the solution of the initial phase of Eq. (16) will contain a phase error. So, obtaining a high signal-to-noise ratio (SNR) of structured illumination images is crucial for increasing the accuracy of initial phase determination. In our system, we use a high sensitivity sCMOS camera with 16 bits gray depth to capture high SNR images, which guarantees obtaining high-fidelity superresolution images. Based on the potential advantages of the DMD-projection-based LED-illumination SIM system, incorporated with the proposed IRT superresolution algorithm, future work will be extended to fast multicolor live-cell imaging and object tracking.
We have proposed an IRT superresolution image reconstruction algorithm for SIM. Compared to the traditional POP algorithm and the ACR algorithm, the IRT algorithm can obtain the initial phase of fringe with high precision, and extract the high-order spectral components without matrix operation. Numerical simulations show that the IRT algorithm always keeps a higher precision of initial phase estimation than the POP algorithm. The immunity of the IRT algorithm against the modulation depth of fringe patterns and the high precision of initial phase determination enable reconstructing an artifact-free superresolution image as long as the SNR of the captured images is reasonably high. We have proved the feasibility and reliability of the IRT algorithm by employing it in our built DMD-projection-based, multicolor-LED-illumination SIM system to image the BPAE cells at low excitation intensity. The experimental results demonstrate that the IRT method can effectively avoid artifacts to produce high-fidelity superresolution images. The IRT algorithm is applicable in either the projection-based SIM or the interference-based SIM.
This research was supported by the Natural Science Foundation of China (NSFC) under Grant Nos. 61522511, 11404389, 81427802, and 61275193.
Xing Zhou is a PhD candidate at Xi’an Institute of Optics and Precision Mechanics, Chinese Academy of Sciences, and a jointly educated PhD candidate at School of Science, Xi'an Jiaotong University. He received his BS degree in physics at Nankai University in 2012. His research focuses on the super-resolution optical microcopy and optical sectioning imaging, especially the structured illumination microscopy and light sheet microscopy.
Ming Lei is a professor at Xi’an Institute of Optics and Precision Mechanics, Chinese Academy of Sciences. He received his PhD degree at Xi’an Institute of Optics and Precision Mechanics, Chinese Academy of Sciences in 2007, and the BE degree at the School of Physics and Optoelectronic Engineering, Xidian University, in 2000. He has published more than 80 papers and holds 9 Chinese patents. His current research focuses on super-resolution microscopy and optical trapping technologies.
Dan Dan is an assistant professor and a PhD candidate at Xi’an Institute of Optics and Precision Mechanics, Chinese Academy of Sciences. He received his MS degree in physical electronics at the School of Physics and Optoelectronic Engineering, Xidian University, in 2010. His current research focuses on structured illumination microscopy for super-resolution and optical sectioning.
Baoli Yao is a professor at Xi’an Institute of Optics and Precision Mechanics, Chinese Academy of Sciences. He received his PhD degree at Xi'an Institute of Optics and Precision Mechanics, Chinese Academy of Sciences, in 1997. His research areas include super-resolution microscopy, digital holographic microscopy, optical micromanipulation, optical data storage, and special optical beams. He has published more than 180 papers in peer-reviewed journals and three book chapters, and owned sixteen authorized Chinese invention patents.
Yanlong Yang is a senior engineer and a PhD candidate at Xi’an Institute of Optics and Precision Mechanics, Chinese Academy of Sciences. He received his MS degree in applied physics at Xi'an Jiaotong University in 2001. His research is focused on the development of two-photon excitation laser scanning fluorescence stereomicroscopy.
Jia Qian is a PhD candidate at Xi’an Institute of Optics and Precision Mechanics, Chinese Academy of Sciences. He received his BS degree at the School of Physics and Optoelectronic Engineering, Xidian University, in 2012. His research topic is the full-color structured illumination optical sectioning microscopy with large field of view and high resolution.
Guangde Chen is a professor at School of Science, Xi’an Jiaotong University. He received his MS degree in optics in the Department of Physics, Xi’an Jiaotong University, in 1989. His research focuses on investigating the mechanisms of optical transitions and lasing in nitride materials to understand the fundamental optical transitions and dynamic processes and to study the physics components of optoelectronic devices.
Piero R. Bianco is an associate professor at the Department of Microbiology and Immunology, University at Buffalo, United States. He received his BS degree in biology at Abilene Christian University in 1987, and the PhD degree in biochemistry at University of Texas Medical School at Houston in 1993. He pursued postdoctoral work in physical biochemistry and biophysics, UC Davis. His research activity is focused on the genetic recombination and DNA repair and single molecule studies.