Our structured illumination microscopy (SIM) is based on a spatial light modulator (SLM) instead of an illumination mask, which does not need to be attached to a linear stage. This SIM can easily design the period of the one-dimensional grid related to the optical sectioning strength and can rapidly acquire three-dimensional data. The optimization of SIM with an SLM is proposed. Previous studies primarily varied magnification with a high numerical aperture objective to optimize the axial response. It is feasible to obtain the maximum optical sectioning strength by designing a grid pattern that has an appropriately high spatial frequency and to uniformly cover the entire frequency spectrum of the sample by rotating a grid pattern. We have successfully optimized SIM with such a grid and covered the frequency spectrum by rotating a grid pattern in multiple orientations.
In the field of electronics and advanced materials, there is a growing demand for information on microstructures or surface morphology because of recent advances in the technology of fine particles. Recently, the structured illumination microscopy (SIM) has been developed for the three-dimensional (3-D) measurements of samples with optical sectioning, which decreases blurring and increases the axial resolution.1 The SIM performance can be further improved by increasing the axial response2,3 and covering the entire frequency spectrum of a sample.4 In previous studies, the axial response was increased by varying the magnification with a high numerical aperture objective between a grid plane and a sample plane, or the spatial frequency was changed in a grid-patterned mask.2,3 This approach has a disadvantage in that it consumes cost and time because the optical system configuration must be modified or a high spatial grid-patterned mask suitable for the imaging system must be manufactured.
In this paper, we suggest an optimization of SIM by designing an appropriately high spatial frequency of the grid pattern in a spatial light modulator (SLM) to increase the axial response and rotating the grid pattern to uniformly cover the entire frequency spectrum of the sample without modifying the optical system. To investigate the correlation of the spatial frequency with the optical sectioning strength, we measured the axial response by changing the spatial frequency of the grid pattern in an SLM.3 From the result, the grid pattern with an appropriately high spatial frequency was selected among various spatial frequencies of the grid patterns for the maximum optical sectioning strength. In addition, we rotated the grid pattern in multiple orientations to effectively extend the optical transfer function (OTF) in an SLM, thereby uniformly covering the entire frequency spectrum of the sample. Moreover, an SLM is expected to be broadly used for the application of SIM because it can easily design the spatial period, and the 3-D data acquisition speed is fast unlike a grid-patterned mask.5–9
The remainder of this paper is organized as follows. Section 2 theoretically analyzes the effective OTF shifted by the period of the grid pattern with a single spatial frequency. In addition, we discuss the extended frequency spectrum of the sample after assembling the OTF with multiple orientations. Section 3 describes the experimental setup and the results. Section 4 concludes the paper.
Structured Illumination Microscope System
A Kohler illumination-based optical system was configured to operate SIM. The warm white LED was projected onto the SLM containing the grid pattern. Light reflected from the SLM was reflected by a beam splitter and transmitted through the band-pass filter to improve the grid pattern contrast.3,11 The filtered light was reflected into the objective by a beam splitter to form the grid pattern image of the SLM onto the sample. Optical sectioning was performed by the phase shift of a sinusoidal wave in the grid pattern, and the phase shift was electrically implemented in the SIM based on the SLM. The intensity of the samples is modulated by the square-law detection algorithm:2,12
While the samples loaded on the linear stage were axially scanned with the phase shift, the axial intensity was obtained by optical sectioning performed by the square law detection. The focal axial position was determined by the peak axial intensity in each pixel of the camera. The 3-D height map can be obtained by calculating the focal axial position in the Cartesian coordinate system with respect to all the pixels of the camera; this provides the height information of the sample. The performance of the SIM can be improved by increasing the axial response. Hence, as the full width at half maximum (FWHM) of the axial response is narrower, the SIM provides more precise height information.2 It is important to take the appropriate spatial frequency because higher spatial frequency results in reducing the contrast of the grid pattern,10 thereby weakening the optical sectioning strength. Therefore, the grid pattern with an optimal period with the normalized spatial frequency of unity must be formed, which improves the axial response.
Analysis of the Frequency Spectrum
For an incoherent illumination system, the detected image intensity is the result of the convolution of the light intensity () from the sample and the point spread function (PSF) () of the imaging system:
When the grid pattern is projected uniformly onto the sample, as illustrated in Fig. 1(a):Fig. 1(b):
The first term is the detected image of the frequency spectrum cutoff by low-pass filtering and has no high-frequency spectra, and the second and third terms are the detected images of the high-frequency spectra, which represent small structures in the sample. These high-frequency spectra are effectively shifted by a single spatial frequency in the grid pattern and are passed through the cutoff frequency of the imaging system, and they overlap the low spatial frequency spectra near the DC frequency, permitting small structures in the sample to be observed, as illustrated in Fig. 1(c). These overlapped spectra can be extracted by optical sectioning, as shown in Fig. 1(d). After optical sectioning, the cutoff frequency of the reconstructed frequency spectrum is extended by the spatial frequency (). Therefore, the cutoff frequency is higher than that of a conventional optical microscope, and the high-frequency spectra of the sample can be observed. In particular, optimal sectioning occurs when the spatial frequency () is half of the cutoff frequency () of the OTF.2,13
To cover the entire frequency spectrum of the sample, the OTF must be effectively extended in multiple orientations by rotating the grid pattern. A description of the raw image, with a frequency spectrum of the sample by the OTF extended to the entire area, can be written as follows:
The rotation shows different areas in the frequency spectrum of the sample that can provide more accurate height information of the sample and reduce the background noise. However, high-frequency spectra overlap with the low-frequency spectra, which degrades the raw image quality before optical sectioning. Therefore, these components must be extracted and reconstructed for observing small structures in the sample. To separate them, optical sectioning is performed by the square law detection from three phase-shifted images, which correspond to the relative spatial phases of After optical sections in multiple orientations, the reconstructed frequency spectrum of the sample extended to other frequency spectrum regions, as depicted in Fig. 1(e). Four optical sections corresponding to four different orientations of the grid pattern are required at any one longitudinal position to measure the entire frequency spectrum of the sample. After optical sectioning, the components in which the high-frequency spectrum overlaps the low-frequency spectrum are extracted, and the low-frequency components are eliminated. As a result, we can obtain the sectioned image with the entire frequency spectrum of the sample. It is expected to provide more accurate height information and reduce background noise.
An optical imaging system was constructed to operate the SIM, as shown in Fig. 2. The warm white LED (Thorlabs, MWWHLP1) was used as the light source; the grid pattern loaded on the SLM was projected onto the sample through the objective lens (Mitutoyo, 20X Plan Apo Infinity Corrected Long WD Objective); and the final image with the grid pattern was formed on the camera (The Imaging Source, DMK 33GX174e). The image of the sample was clearly formed on the camera when the grid pattern was matched on the sample. The phase shift was implemented by electrically shifting a sinusoidal wave pattern by one third of the spatial period of the pattern in the SLM, which has a spatial period of and a pixel pitch of along with the objective lens used in the experiment. Optical sectioning was performed by square-law detection using three phase-shifted images with zero spatial phase. As the sample was axially scanned, we took optically sectioned images for all axial positions and extracted the height information of the sample at the focused axial position. Finally, a 3-D height map was reconstructed based on these sectioned images.
Designing the Grid Pattern in the SLM
As the spacing of the grid pattern becomes narrower, the normalized spatial frequency becomes close to unity. Hence, the optical sectioning strength becomes stronger than that of the low spatial frequency of the grid pattern designed for the SLM. In the experiment, we designed grid patterns with different spatial frequencies () that were 100 line pairs/mm, 64 line pairs/mm, and 50 line pairs/mm, and the normalized spatial frequencies were 1.16, 0.75, and 0.58, respectively, by calculating , where and denote the magnification of the imaging system and the wavelength, respectively, is the spatial frequency of the grid pattern and NA is the numerical aperture. Raw images in grid patterns with different spatial frequencies were formed, as shown in Fig. 3. Of course, all the methods presented in this paper are based on the invariant PSF. In a noninvariant PSF, the axial extent of the PSF increases as the focal position becomes deeper along the -axis, which causes aberration.14,15 However, the objective lens of our system was well calibrated, and aberrations did not occur because each relay lens was well placed at the focal distance. Figure 3 shows that there was no distortion or blurring at the edge of the grid pattern, which indicates that aberration did not occur. The axial responses between each system were measured under the condition that the system was stable.
Axial responses were measured for the grid patterns with different spatial frequencies, as shown in Fig. 4, to investigate the correlation of the spatial frequency with the optical sectioning strength. We introduce the FWHM to facilitate the comparison of these axis responses. The FWHM refers to actual defocus at an axis response of 0.5. In general, normalized spatial frequency and normalized defocus are inversely proportional, so the higher the normalized spatial frequency, the lower the normalized defocus. As the normalized defocus is related to the actual defocus through , we can estimate the actual defocus by obtaining the normalized defocus according to the normalized spatial frequency, and comparing actual defocuses can evaluate each axis response.2 Therefore, it can be confirmed from the aforementioned equation that the actual defocus changes by the normalized spatial frequency, which affects the axial response. The FWHMs of axial responses at each normalized spatial frequency were measured to facilitate comparison of axial responses. In practice, we measured 15.8, 18.8, and for the FWHMs of axial responses, which correspond to grid patterns with spatial periods of 10, 15.63, and , respectively. The maximum optical sectioning strength was shown at the grid pattern with the spatial period of in which the normalized spatial frequency is close to unity. As the FWHM narrowed, there was an improvement in the axial response, and the FWHM was minimized when the normalized spatial frequency was approaching unity, as shown in Fig. 4. Consequently, the FWHM continuously became narrower as the normalized spatial frequency approached unity, leading to improved axial response. For the grid pattern with the narrowest FWHM of the axial response, SIM provides high precision when extracting the height information of the sample to be observed.
Reconstructing the Sectioning Image
We performed SIM experiments with a coin as the sample to verify the effects of different axial responses on SIM performance. Figure 5 shows the sectioning image, which is reconstructed by a square-law detection algorithm from the phase shift images. First, we discuss the sectioned image in the out-of-focus region. In this region, the sectioned image, which is obtained by Eq. (1), should have zero intensity for all image pixels, provided that the object is ideally out-of-focus. However, as shown in Figs. 5(b) and 5(c), the sectioned images of the object are constructed even in the out-of-focus area. This means that the image is measured as if there are samples even in the out-of-focus image, causing an error in the 3-D height map information. In addition, the shorter period of the grid pattern of the SLM leads to the smaller intensity of the sectioned image relative to the larger period of the grid pattern in out-of-focus. This is because the depth-of-field becomes smaller as the period becomes smaller. It is also consistent with the narrower FWHM discussed in Sec. 3.2. This means that the imaging system has a high dynamic range in a short spatial period of the grid pattern and can observe the sample well in the focal plane. Consequently, it is expected that the smaller period of the grid pattern provides more accurate height map information. Second, we discuss the sectioned image in the focused area. In this area, as shown in Figs. 5(d)–5(f), the shorter period results in significantly clearer surface information in the sectioned image. As the period of the grid was sufficiently shortened, the sectioning strength was improved, which is described in Sec. 2. Consequently, it is expected that the sufficient small period of the grid pattern provides more accurate height map information (Fig. 6).
We also rotated the grid pattern in multiple orientations and merged each sectioned image to obtain the sectioned image of the fully extended OTF in all areas. This showed that the OTF was shifted in the or direction when optical sectioning was performed in only one orientation, whereas the OTF was effectively expanded in the and directions when optical sectioning was performed in multiple orientations. It was demonstrated that the fully extended frequency spectra of the sample were observed in the raw image with the rotated grid pattern. From the results, the axial resolution was increased in the grid pattern with the normalized spatial frequency of unity compared with other spatial frequencies (Fig. 7).
Reconstructing the Height Map
The 3-D height maps were reconstructed for the plane mirror between each system. The axial scanning was performed with step and total scan range was . A better optical sectioning occurred in the grid pattern with the spatial period of , where the normalized spatial frequency is close to unity. It was confirmed that the height map reconstructed by the grid pattern with the spatial period of had more accurate height information compared with height maps reconstructed by grid patterns with spatial periods of 15.625 and .
To examine the roughness of the plane mirror, standard deviations were calculated for the reconstructed 3-D height maps. The standard deviation values were 0.25, 0.37, and , which correspond to grid patterns with spatial periods of 10, 15.625, and , respectively. This showed that the height map reconstructed by the grid pattern with the normalized spatial frequency of unity yielded a smaller standard deviation compared with height maps reconstructed by grid patterns with different spatial frequencies. These results suggest that SIM can provide a high precision measurement for grid patterns with normalized spatial frequency of unity (Fig. 8).
We also measured the height of the coin for a region of interest (ROI) of size in the 3-D height map. To obtain the objective data, we set the value measured with the confocal microscope (PRECITEC, CHRocodile C) as the reference value. Each optical sectioning was evaluated by comparing the reference value to the value measured by grid patterns with different spatial frequencies. The axial scanning was performed with step and a scan range of . The height was computed by subtracting the average height of the two faces of the ROI. The height of the coin in the ROI was measured as 29.44, 30.96, and , which correspond to grid patterns with spatial periods of 10, 15.625, and , respectively. The height of the same ROI was also measured as using the confocal microscope. The SIM using the grid pattern with an appropriately high spatial frequency provided high accuracy, considering that the error in the measurement of the confocal microscope is 5%.
For optical sectioning performed in multiple orientations, the noise was dramatically reduced in the height map, as shown in Fig. 9. The grid pattern was rotated by 45 deg. At least three rotations (four grid orientations) allowed for the measurement of the entire frequency spectrum of the sample. The frequency spectrum of the sample was measured by an effectively extended OTF. The more the area occupied by the OTF, the more uniformly the frequency spectrum of the sample can be measured. Therefore, if the rotation angle of the grid pattern is reduced and rotated several times, more frequency spectrum areas of the sample can be uniformly measured. Of course, measuring multiple rotations increases the integration time of optical sectioning images, which is disadvantageous for the measurement speed. The number of grid orientations is equal to 180 divided by the angular step size. Considering the measurement speed, we adopted an angular step size of 45 deg. In other words, the grid pattern was rotated three times, and the number of grid orientations was 4. This provided adequate measurement speed and uniform frequency spectrum measurement. It is shown in Fig. 9(b) that noise was reduced in the 3-D height map.
Therefore, we successfully optimized the SIM by designing a proper grid pattern and performing optical sectioning in multiple orientations in the SLM. The accuracy obtained from the measurement is comparable to that of a confocal microscope.
We demonstrated that the optimization of SIM was achieved by designing an appropriately high spatial frequency and rotating a grid pattern using an SLM. To examine the correlation of the spatial frequency with the optical sectioning strength, the axial response was measured between each system. It was confirmed that the optimal sectioning occurred at the grid pattern in the SLM having a normalized spatial frequency close to unity. Furthermore, we could uniformly measure the high-frequency spectra of the sample with the effectively extended OTF by rotating the grid pattern. The frequency spectrum area to be measured in the sample was determined by the angular step size and the grid orientations, and the optimum angular step size was determined considering the measurement speed. Consequently, it was confirmed that noise in the 3-D height map was remarkably reduced. In our optical system, we successfully optimized SIM using an SLM without the modification of the optical system and changing a grid-patterned mask.
This research was supported by the Ministry of Culture, Sports and Tourism (MCST) and the Korea Creative Content Agency (KOCCA) in the Culture Technology (CT) Research & Development Program 2017 (R2017060005, Development of AR Platform based on Hologram).
Jeong-Heon Han received his BS degree from the Department of Electrical Materials Engineering, Kwangwoon University, Seoul, Korea, in 2012. He received his MS degree from the Department of Electrical Engineering and Computer Science, Gwangju Institute of Science and Technology (GIST), Gwangju, Korea, in 2015. Currently, he is a PhD course student at Korea University, Seoul, Korea. His research interests include optical imaging systems and machine vision.
Nak-Won Yoo received his BS degree from the Department of Semiconductor Systems Engineering, Sungkyunkwan University, Suwon, Korea, in 2012. He received his PhD from the Department of Electrical Engineering and Computer Science, Seoul National University(SNU), Seoul, Korea, in 2018. Currently, he works at Korea Institute of Science and Technology (KIST). His research interests include circuit design and image sensor development.
Ji-Hoon Kang received his BS degree from the Department of Electronic Engineering, Inha University, Incheon, Korea, in 2014, and the PhD in the Department of Electrical Engineering of Korea University, in 2019. Now, he works at Korea Institute of Science and Technology (KIST). His research interests include optical scanning holography (OSH) and image sensing.
Byeong-Kwon Ju received his MS degree from the Department of Electronic Engineering, University of Seoul, Seoul, Korea, in 1988, and the PhD degree in Semiconductor Engineering from Korea University, Seoul, in 1995. He is currently an associate professor with Korea University, where his main interests are in flexible electronics (OLED and OTFT), field emission device, MEMS (Bio and RF), and carbon nanotube-based nano systems.
Min-Chul Park received his PhD at Tokyo University in 2000. Currently, he is a principal research scientist at Korea Institute of Science and Technology and visiting scientist at Massachusetts Institute of Technology. His research interests include 3-D imaging systems, especially making use of III-V compound semiconductors to visualize invisible light, and signal processing with artificial intelligence. He developed several structured light measurement and autostereoscopic display systems, and a 4K+4K resolution smart AR glasses in 2018. Now, he is a senior member of SPIE.