1.

Introduction

The charge-coupled device (CCD) has revolutionized imaging in the digital era.¹ In microscopy and photography, a lens is typically employed to translate the object of an image to be captured by the CCD or the CMOS sensor. Hence, the size of the CCD pixel has become both a bottleneck and a benchmark parameter for the resolution of digital imaging. The modulated transfer function is an efficient way to evaluate the performance of microscopy or photography as the contrast is plotted in the frequency domain. An interesting approach for super-resolution is that, in structured illumination microscopy (SIM),^2–5 the structured pattern is employed to expand the frequency domain, which can improve the spatial resolution by $\sim 2\times$.^6–8 Similarly, in Fourier ptychography microscopy, the sample is illuminated with multiple angles for a synthetic aperture, which is an effective expansion in the frequency domain for an improvement in spatial resolution.^9,10

Lens-free microscopy (LFM) is a novel imaging technique that breaks the space–bandwidth product of conventional lens-based microscopic techniques by avoiding the application of the lens; instead its resolution is directly coupled with the pixel size.^11–13 LFM has several distinct advantages, such as single-shot 3D imaging, cost-effectiveness, compactness, and no space–bandwidth product limitation.¹⁴ Applications based on LFM have been realized, such as field-portable microscope,¹⁵ point-of-care testing,¹⁶ telemedicine,^17,18 and air quality monitoring.¹⁹ However, the spatial resolution of LFM is still a major bottleneck that prohibits its wide application as it is limited by the pixel size of the 2D detector.¹² On the one hand, several variations of LFM have been demonstrated to overcome the resolution barrier imposed by the pixel size, such as lateral shift-based pixel super-resolution,²⁰ multiangle illumination,^21,22 multiheight,²³ wavelength scanning,²⁴ fiber-optic taper,²⁵ and synthetic aperture,²⁶ as well as multiangle illumination achieved with a tunable wavelength semiconductor laser source and a volume phase grating.²⁷ On the other hand, no lateral resolution substantially higher than the size of a pixel with a single imaging exposure has been reported.

In this paper, we utilize the anisotropic resolution of a 2D detector to present a method that can enhance the spatial resolution by $1.4\times$. Previously, it is widely accepted that the pixel size is the smallest unit for digital imaging. Here, by exploiting the nonsymmetry of the frequency domain, we propose frequency-domain diagonal extension (FDDE) microscopy in which the resolution can be improved to a $0.7\times$ pixel size, even in one snapshot, through sampling in the diagonal direction. The principle is straightforward: the diagonal direction samples more densely, thus containing more frequency information. Translating the spatial image to the frequency domain through Fourier transform also clearly reveals this fact.

We first demonstrate the principle of FDDE with LFM,^12,15,28 as in LFM the resolution is mainly limited by the pixel size. Then, through a combination of multiple frequency components from different angles, a super-resolution image with isotropic resolution can be achieved, similar to the frequency-domain image processing procedure of SIM.^4,8 The resolution enhancement of FDDE is demonstrated experimentally with a mouse skin specimen and a clinical blood smear sample. We extended FDDE to conventional photography using an ISO 12233 resolution target.

2.

Theory

The principle of FDDE microscopy is very straightforward (Fig. 1), that is, the high-frequency component in the diagonal direction is utilized to enhance the resolution. In FDDE microscopy, the image is typically recorded with a 2D sensor with grid-like pixels. The optical transfer function (OTF) of the imaging system and the frequency domain of digital imaging are represented as follows:

Fig. 1

An illustration of the FDDE microscopy. (a) Illustration of the sampling interval of an image sensor with a rectangular pixel in the horizontal and diagonal directions. (b) The frequency domain of an image sensor and the OTF of an imaging system in undersampled digital imaging. The green-dash rectangle is the frequency domain of the microscopic image. The yellow-dot circle is the OTF of an imaging system. The green-line rectangle is the frequency domain of undersampled digital imaging. D and H denote the diagonal and horizontal directions, respectively. The spatial domain image of (b) is shown in the upper-right corner. (c) The optical setup of FDDE LFM. (d) Illustration of the frequency stitching algorithm of FDDE.

To elucidate the anisotropic frequency domain of undersampling digital imaging, LFM is employed as its resolution is exclusively dominated by the pixel size. As presented in Fig. 1(c), the LFM system consists of a light source, a sample stage, and an image sensor. In this work, the sample stage is mounted on a rotation stage. The raw images are obtained with different angles between the sample and the image sensor. In the case of holographic inline LFM, the theoretical resolution and the OTF are mainly determined by the wavelength and the coherence of the illumination light and the distance between the sample and the image sensor, and the technique has an isotropic OTF, which also has a circular boundary. Because the pixel size ($2.2\mu \mathrm{m}$) is much larger than the theoretical resolution ($\sim 1\mu \mathrm{m}$) in LFM, the raw image is a considerably undersampled image (the resolution is limited by pixel size). For a pixel size of $\mathrm{\Delta }$, the frequency limit is equal to $1/(2\times \mathrm{\Delta })$ in the frequency domain, and the OTF is constrained by a smaller rectangle window, as presented in Fig. 1(b). Therefore, the resulting frequency domain has a rectangular shape. In the frequency domain, the maximum frequency limit along the diagonal direction is $\sim 1.4$-fold that along the horizontal/vertical directions. This implies that the spatial resolution should be higher in the diagonal direction than in the horizontal and vertical directions.

We propose a straightforward method to achieve extended resolution isotropically [Fig. 1(d)]. As higher frequency is attributed to higher resolution in the Fourier domain, it can be utilized to extend the resolution. First, we obtain a series of images at different detection directions. The images are rotated to the same orientation and registered with each other.³⁰ Because of the rotation of the grid 2D detector, the corresponding four corners will contain higher frequencies than the four borders. Then, the high-frequency information in different directions is extracted from the frequency domain of the images. Finally, the high-frequency information is stitched together and then converted back into the spatial domain to obtain a super-resolved image.

The frequency domain $F_{\mathrm{FDDE}}$ of FDDE generated with raw images of $n$ frames is expressed as follows:

To realize the FDDE algorithm, a blank matrix of the FDDE Fourier domain is created, then each pixel of this matrix is filled with the amplitude and the phase value at the same location from the frequency domain of the raw images. Because the raw images are acquired in different directions, the boundary of the frequency domain of each raw image is different in each direction. Thus, the matrix is filled with the complex value from the biggest boundary of the frequency domain of the raw images, as depicted in Eq. (2). Each pixel in the frequency domain has an angle with respect to the center point. By knowing the angle of each raw image in the experiment, we have the region of the angle for the largest boundary of the frequency domain, given as the red-line marked area in Fig. 1(b). Then we know which raw image should be chosen for each pixel in the frequency domain of FDDE. In this way, we fill every pixel in the frequency domain of FDDE. As a result, the frequency domain in the corresponding angle region from each of the raw images is selected, and then, they are stitched together, as depicted in Fig. 1(d). Then, FDDE is obtained with inverted fast Fourier transformation on the stitched frequency domain.

3.

Results

To illustrate the anisotropic resolution of the conventional LFM, a resolution target is imaged. In Fig. 2, the resolution-measuring results with two different orientations of the image sensor are presented. The experiments are carried out with a CMOS chip, which has a pixel size of $2.2\mu \mathrm{m}$. In Fig. 2(a), the resolution target is placed horizontally with respect to the pixel array of the CMOS chip. To elucidate the anisotropic resolution in the diagonal direction, in Fig. 2(b), the resolution board is placed 45 deg with respect to the pixel array of the CMOS chip. Figures 2(a) and 2(b) were reconstructed with the angular spectrum method in Ref. 31 and interpolated with a Fourier zero-padding algorithm by four times. In Figs. 2(a) and 2(b), the lines of group 8, element 1 can be resolved in both images. Moreover, element 3 of group 8 can be resolved in Fig. 2(b), but they are irresolvable in Fig. 2(a). Therefore, the half-pitch resolutions for Figs. 2(e) and 2(f) are 1.94 and $1.55\mu \mathrm{m}$, respectively. The resolution enhancement in the diagonal direction is $\sim 1.3\times$. Comparing the Fourier domain of the results, the larger boundary in the diagonal direction in Fig. 2(d) is beneficial to enhancing the resolution in Fig. 2(b). To further illustrate that the enhanced resolution results from the higher frequency components located in the diagonal direction of the Fourier domain, we applied the boundary of the effective Fourier domain of Fig. 2(c) as a rectangular filter on the frequency domain of the 45 deg reconstructed image in Fig. 2(d). After the application of the filter, the lines of $1.55\mu \mathrm{m}$ are no longer resolvable as the red dotted line in Fig. 2(f), and the red dotted line profile looks the same as that in Fig. 2(e). Note that the image quality could be further enhanced by utilizing a twin-image eliminating algorithm.^32,33

Fig. 2

LFM imaging with different directions of the image sensor. (a) and (b) The reconstructed hologram images of LFM in the horizontal and diagonal directions, respectively. (c) and (d) The frequency domain of the reconstructed images in (a) and (b), respectively. The dotted rectangle marked in (d) represents the effective frequency boundaries in (c). (e) and (f) The line profiles (element 3 of group 8) marked in (a) and (b), respectively. The red dotted line profile in (f) is the same location in the image that filtered the high frequency out of the yellow rectangle, as illustrated by the inset in (d). The yellow arrows indicate the direction of the sample.

Then, we test the performance of FDDE LFM with a hematoxylin-eosin (H&E) stained mouse skin sample, as presented in Fig. 3. Biological samples often contain rich structures, and high frequencies are located in different directions. For example, the stratum corneum on the surface of the skin forms layered structures that can be much smaller than the pixel size of the image sensor ($2.2\mu \mathrm{m}$), and the distance between the corneum structures could be as small as several micrometers. These kinds of details are not observable with a single holographic image. To extract such fine details, first, three rotational holographic raw images are acquired with directions of 0, $\pi /6$, and $\pi /3$ between the sample and the image sensor. Then, the frequency domains of these three images were synthesized through FDDE, as shown in Figs. 3(c4) and 3(d4).

Fig. 3

Demonstration of FDDE imaging with a mouse skin sample. (a) The FDDE LFM image of the mouse skin sample. (b) An enlarged view of the region marked in (a). (c) LFM images. (c1), (c2), and (c3) are the same area as (c4) in the three-phase images with different orientations. The arrows in the upper-right corner correspond to the direction of the sample in the experiment. The three arrows indicate the FDDE image. In addition, (c2) and (c3) and (d2) and (d3) are rotated back to the same direction as in (c1) and (d1), respectively, for a comparison. The line profile in (c4) is marked between the arrows. The inset in (c4) is imaged with a $10\times$ bright-field microscope, presented as the ground truth. (d) The frequency domains of the three-phase images and the FDDE image. The yellow rectangle is the boundary of the lensfree microscope. The red line area in (d1)–(d3) is combined into (d4) based on the principle of FDDE.

Fig. 4

Analysis of FDDE imaging with a blood smear specimen. (a) The FDDE image with the blood smear specimen. (b) An enlarged view of the region marked in (a). (c1)–(c3) The conventional LFM of different angles, and (c4) is the LFM with FDDE. The thick arrows in the upper-right corner correspond to the direction of the sample in the experiment in (c1)–(c3). The three arrows indicate the combined FDDE image in (c4). (d) The bright-field image of the same area, presented as the ground truth. (e) The line profile from (c1)–(c4) marked in (c4).

Benefiting from the 1:1 magnification of LFM, the field-of-view (FOV) of our FDDE system can reach $\sim 8{\mathrm{mm}}^2$ ($2.8\mathrm{m}\mathrm{m}\times 2.8\mathrm{m}\mathrm{m}$). As shown in Fig. 3(c), there is a small raindrop-like structure (indicated with the arrow) in the stratum corneum, as shown by the bright-field microscopy image presented in the inset of Fig. 3(c4). However, in each image of the three phases, Fig. 3(c), this structure is not resolvable. In Fig. 3(c3), this structure is resolved as a filled line, but it is resolved as a hollow structure in the other two directions [Figs. 3(c1) and 3(c2)]. This raindrop-like structure can only be resolved in the FDDE image.

This result clearly demonstrates the performance of FDDE microscopy with the histological sample. As presented in the line profile, the distance between the stratum corneum structures is $\sim 3.0\mu \mathrm{m}$, which corresponds to $\sim 1.4\times$ the pixel size of the image sensor. According to the Nyquist–Shannon sampling theorem, to resolve a two-line structure distance of $\sim 3.0\mu \mathrm{m}$, the half-pitch resolution should be at least $1.5\mu \mathrm{m}$. This corresponds to a resolution enhancement of $\sim 1.4\times$, which is consistent with the principle of FDDE. Such a small structure is not resolvable based on the conventional principle of LFM.

Then, the FDDE system is tested with a conventional blood smear sample, as shown in Fig. 4. Because the inner diameter of blood cells is $\sim 3$ to $4\mu \mathrm{m}$, it is difficult for conventional LFM with a pixel size of $2.2\mu \mathrm{m}$ to resolve the structure of blood cells. As shown in Figs. 4(c1)–4(c3), a few blood cells can be resolved as a ring structure, and most of the blood cells are observed as dark spots due to insufficient resolution. Some of the blood cells are resolved as a rectangular shape in the conventional single-frame lens-free image, which may be attributed to the rectangular shape of the Fourier domain of LFM. In the FDDE image in Fig. 4(c4), the shape of the red blood cell becomes more circular. Additionally, more blood cells can be resolved as ring structures in the FDDE image. The marked cell is completely irresolvable in any conventional LFM image, and it is observed as a ring structure with FDDE microscopy, which is similar to that in the bright-field image. As presented in Fig. 4(e), the inner diameter of the marked blood cell is $3.3\mu \mathrm{m}$, and this result proves that the half-pitch resolution is $\sim 1.6\mu \mathrm{m}$, which is a resolution enhancement of $\sim 1.4\times$. Also, the quality of the blood smear image with FDDE is much better than that of conventional LFM due to a homogeneous high resolution.

Moreover, the principle of FDDE could be extended to lens-based systems, such as photography. In the experiment, an ISO 12233 resolution target is imaged with a camera lens (SV-10035V, VS Technology) coupled with a sCMOS camera (Andor Zyla 4.2). Figures 5(a) and 5(b) present the raw images interpolated four times with a Fourier zero-padding method and with the intensity of the images inverted. The line profiles marked in Figs. 5(a) and 5(b) are presented in Figs. 5(c) and 5(d). In the case of line 1 and 1′, both images can resolve the period of 8 pixels. When the line profile gets closer to the center of the target, the period of lines should decrease. The period is decreased to 6 pixels, as shown in line 2′. However, the distance between dips in line 2 is read as 9 pixels, which is even larger than line 1. This artifact is caused by the lack of a high-frequency component in the Fourier domain. Hence, the resolution in the horizontal and the diagonal direction is 8 and 6 pixels, respectively, which corresponds to 2 and 1.5 pixels in the raw images, respectively. These results are approximately the theoretical limit in the horizontal and diagonal directions with a pixel size determined resolution. The result reflects $\sim 1.3\times$ higher resolution for the diagonal direction than that of the horizontal direction, which is consistent with the principle of FDDE (see also Supplemental Fig. S3). Therefore, FDDE can be applied to enhance the resolution of lens-based photography as well, when the resolution is limited by the pixel size.

Fig. 5

Lens-based photography with different orientations of ISO 12233 resolution target. (a) and (b) The interpolated images captured with conventional lens and sCMOS chip. (c) and (d) The line profiles marked in (a) and (b), respectively.

4.

Conclusion

Compared with other super-resolution schemes,^27,28 this proposed method is more convenient. For example, in subpixel super-resolution schemes, a nanometer-precision motorized translation stage or a tunable laser source is required, which can substantially increase the cost and complexity of the imaging system. In the FDDE method, the accuracy of the rotating angle is not strictly required, and this operation can be easily satisfied with a manual or a motorized rotation stage. This method does not require a computationally laborious subpixel super-resolution algorithm, which is highly sensitive to noise. Another merit is that the resolution enhancement can be further improved by a smaller pixel size. In this case, an upcoming consumer-level CMOS chip with a smaller pixel size and low cost will further enhance the performance and applications of the FDDE in a straightforward manner. For instance, FDDE LFM could reach an isotropic resolution of $0.56\mu \mathrm{m}$ by utilizing a CMOS chip with a pixel size of $0.8\mu \mathrm{m}$ (IMX586, SONY, cost approximately twenty U.S. dollars). At the same time, the overall cost, complexity of the FDDE system and the algorithm, portability, and robustness are far better than those of conventional optical microscopy. By contrast, in the subpixel sampling super-resolution scheme with a motorized $x-y$ translational stage, the stage should have subpixel accuracy, which brings significant additional cost. Therefore, the FDDE method is conducive to achieving an ultra-compact and low-cost red blood cell examination instrument, which is especially important for rural and underdeveloped areas. In conventional microscopy or imaging, this method can still work if the pixel size of the detector is larger than Nyquist–Shannon sampling of the optical resolution or $>\lambda /4\mathrm{NA}$. FDDE is particularly useful for scenarios in which the pixel size restricts the system resolution. For example, the widely used electron multiplying CCD (EMCCD) features a pixel size of $\sim 16\mu \mathrm{m}$ for better sensitivity, much larger than that of a conventional CCD or sCMOS detector. With a large pixel size, either the resolution or the FOV is compromised. Here, with FDDE, one can maintain the sensitivity but gain a resolution enhancement of $1.4\times$. This is particularly useful when the resolution is limited by the pixel size, such as LFM, EMCCD-based imaging, and spectroscopy.

In summary, we proposed a novel rotational approach to achieving a higher resolution for digital imaging. The enhanced resolution results from the higher-resolution component located in the diagonal directions, which is straightforward but has not been demonstrated before, to the best of our knowledge. The resolution enhancement is attained by exploiting the higher frequency of the diffraction pattern located in the diagonal direction, in which the frequency border is effectively extended diagonally. We demonstrate FDDE first with LFM and then extend it to lens-based photography. By combining the frequency domains from the images acquired with different angles, the resulting FDDE image has an isotropic resolution enhancement of $\sim 1.4\times$. The half-pitch resolution reaches $\sim 1.5\mu \mathrm{m}$ when imaging a mouse skin sample and a blood smear sample using a CMOS detector with a pixel size of $2.2\mu \mathrm{m}$. Then, we extended this approach to lens-based photography, yielding the same resolution enhancement. Here, the FDDE image only uses three raw images with different recording directions, but more images with different directions could be acquired for better image quality. This method can also be applied to a wide range of imaging or detection areas in which the limiting factor is the pixel size of the detector due to undersampling, such as microscopy, telescope imaging, and spectroscopy.