Frequency-domain diagonal extension imaging

Abstract. The pixel size of a charge-coupled device (CCD) camera plays a major role in the image resolution, and the square pixels are attributed to the physical anisotropy of the sampling frequency. We synthesize the high sampling frequency directions from multiple frames acquired with different angles to enhance the resolution by 1.4  ×   over conventional CCD orthogonal sampling. To directly demonstrate the improvement of frequency-domain diagonal extension (FDDE) microscopy, lens-free microscopy is used, as its resolution is dominantly determined by the pixel size. We demonstrate the resolution enhancement with a mouse skin histological specimen and a clinical blood smear sample. Further, FDDE is extended to lens-based photography with an ISO 12233 resolution target. This method paves a new way for enhancing the image resolution for a variety of imaging techniques in which the resolution is primarily limited by the sampling pixel size, for example, microscopy, photography, and spectroscopy.


Introduction
The charge-coupled device (CCD) has revolutionized imaging in the digital era. 1 In microscopy and photography, a lens is typically employed to translate the object of an image to be captured by the CCD or the CMOS sensor. Hence, the size of the CCD pixel has become both a bottleneck and a benchmark parameter for the resolution of digital imaging. The modulated transfer function is an efficient way to evaluate the performance of microscopy or photography as the contrast is plotted in the frequency domain. An interesting approach for super-resolution is that, in structured illumination microscopy (SIM), 2-5 the structured pattern is employed to expand the frequency domain, which can improve the spatial resolution by ∼2×. [6][7][8] Similarly, in Fourier ptychography microscopy, the sample is illuminated with multiple angles for a synthetic aperture, which is an effective expansion in the frequency domain for an improvement in spatial resolution. 9,10 Lens-free microscopy (LFM) is a novel imaging technique that breaks the space-bandwidth product of conventional lensbased microscopic techniques by avoiding the application of the lens; instead its resolution is directly coupled with the pixel size. [11][12][13] LFM has several distinct advantages, such as singleshot 3D imaging, cost-effectiveness, compactness, and no spacebandwidth product limitation. 14 Applications based on LFM have been realized, such as field-portable microscope, 15 point-of-care testing, 16 telemedicine, 17,18 and air quality monitoring. 19 However, the spatial resolution of LFM is still a major bottleneck that prohibits its wide application as it is limited by the pixel size of the 2D detector. 12 On the one hand, several variations of LFM have been demonstrated to overcome the resolution barrier imposed by the pixel size, such as lateral shift-based pixel superresolution, 20 multiangle illumination, 21,22 multiheight, 23 wavelength scanning, 24 fiber-optic taper, 25 and synthetic aperture, 26 as well as multiangle illumination achieved with a tunable wavelength semiconductor laser source and a volume phase grating. 27 On the other hand, no lateral resolution substantially higher than the size of a pixel with a single imaging exposure has been reported.
In this paper, we utilize the anisotropic resolution of a 2D detector to present a method that can enhance the spatial resolution by 1.4×. Previously, it is widely accepted that the pixel size is the smallest unit for digital imaging. Here, by exploiting the nonsymmetry of the frequency domain, we propose frequency-domain diagonal extension (FDDE) microscopy in which the resolution can be improved to a 0.7× pixel size, even in one snapshot, through sampling in the diagonal direction. The principle is straightforward: the diagonal direction samples more densely, thus containing more frequency information. Translating the spatial image to the frequency domain through Fourier transform also clearly reveals this fact.
We first demonstrate the principle of FDDE with LFM, 12,15,28 as in LFM the resolution is mainly limited by the pixel size. Then, through a combination of multiple frequency components from different angles, a super-resolution image with isotropic resolution can be achieved, similar to the frequency-domain image processing procedure of SIM. 4,8 The resolution enhancement of FDDE is demonstrated experimentally with a mouse skin specimen and a clinical blood smear sample. We extended FDDE to conventional photography using an ISO 12233 resolution target.

Theory
The principle of FDDE microscopy is very straightforward (Fig. 1), that is, the high-frequency component in the diagonal direction is utilized to enhance the resolution. In FDDE microscopy, the image is typically recorded with a 2D sensor with grid-like pixels. The optical transfer function (OTF) of the imaging system and the frequency domain of digital imaging are represented as follows: (1) where f x and f y are the coordinates in the frequency domain, r OTF is the boundary of OTF, NA denotes the numerical aperture of the optical system, λ is the wavelength, M is the magnification of the imaging system, and Δ is the pixel size of the detector. As shown in Fig. 1(b), the boundary of the diffractionlimited OTF and the frequency domain of digital imaging is circular and rectangular, respectively. In the case of the undersampling condition, the frequency domain of digital imaging is smaller than the theoretical OTF, due to the nature of the pixel arrangement in Fig. 1(a). As shown in Fig. 1(b), the diagonal boundary of digital imaging, 2Δ , is 1.4 times as large as the horizontal/vertical direction in Eq. (1). A larger diagonal boundary means a bigger bandwidth, and this can be employed to collect higher frequency components for better resolution. This is because the sampling frequency in diagonal direction is actually much higher-the distance is only 0.7 pixels [ Fig. 1(a)] in the diagonal direction. 29 To elucidate the anisotropic frequency domain of undersampling digital imaging, LFM is employed as its resolution is exclusively dominated by the pixel size. As presented in Fig. 1(c), the LFM system consists of a light source, a sample stage, and an image sensor. In this work, the sample stage is mounted on a rotation stage. The raw images are obtained with different angles between the sample and the image sensor. In the case of holographic inline LFM, the theoretical resolution and the OTF are mainly determined by the wavelength and the coherence of the illumination light and the distance between the sample and the image sensor, and the technique has an isotropic OTF, which also has a circular boundary. Because the pixel size (2.2 μm) is much larger than the theoretical resolution (∼1 μm) in LFM, the raw image is a considerably undersampled image (the resolution is limited by pixel size). For a pixel size of Δ, the frequency limit is equal to 1∕ð2 × ΔÞ in the frequency domain, and the OTF is constrained by a smaller rectangle window, as presented in Fig. 1(b). Therefore, the resulting frequency domain has a rectangular shape. In the frequency domain, the maximum frequency limit along the diagonal direction is ∼1.4-fold that along the horizontal/vertical directions. This implies that the spatial resolution should be higher in the diagonal direction than in the horizontal and vertical directions.
We propose a straightforward method to achieve extended resolution isotropically [ Fig. 1(d)]. As higher frequency is attributed to higher resolution in the Fourier domain, it can be utilized to extend the resolution. First, we obtain a series of images at different detection directions. The images are rotated to the same orientation and registered with each other. 30 Because of the rotation of the grid 2D detector, the corresponding four corners will contain higher frequencies than the four borders. Then, the high-frequency information in different directions is extracted from the frequency domain of the images. Finally, the highfrequency information is stitched together and then converted back into the spatial domain to obtain a super-resolved image.
The frequency domain F FDDE of FDDE generated with raw images of n frames is expressed as follows: ; θ ∈ ½0,2πÞ; (2) where F i denotes the Fourier domain of the i'th raw image and θ is the azimuth angle with respect to the X axis in the Fourier domain. Because of the limitation of the matrix size, the out-ofrange frequency domain components are close to 0. However, the rotated frequency domain can introduce additional information into the frequency domain; one can simply update the frequency-domain region with the larger components. Then, by combining the extended components in the frequency domain, the resulting image contains a homogeneously super-resolution with a 1.4-fold improvement over the conventional imaging.
To realize the FDDE algorithm, a blank matrix of the FDDE Fourier domain is created, then each pixel of this matrix is filled with the amplitude and the phase value at the same location from the frequency domain of the raw images. Because the raw images are acquired in different directions, the boundary of the frequency domain of each raw image is different in each direction. Thus, the matrix is filled with the complex value from the biggest boundary of the frequency domain of the raw images, as depicted in Eq. (2). Each pixel in the frequency domain has an angle with respect to the center point. By knowing the angle of each raw image in the experiment, we have the region of the angle for the largest boundary of the frequency domain, given as the red-line marked area in Fig. 1(b). Then we know which raw image should be chosen for each pixel in the frequency domain of FDDE. In this way, we fill every pixel in the frequency domain of FDDE. As a result, the frequency domain in the corresponding angle region from each of the raw images is selected, and then, they are stitched together, as depicted in Fig. 1(d). Then, FDDE is obtained with inverted fast Fourier transformation on the stitched frequency domain.

Results
To illustrate the anisotropic resolution of the conventional LFM, a resolution target is imaged. In Fig. 2, the resolution-measuring results with two different orientations of the image sensor are presented. The experiments are carried out with a CMOS chip, which has a pixel size of 2.2 μm. In Fig. 2(a), the resolution target is placed horizontally with respect to the pixel array of the CMOS chip. To elucidate the anisotropic resolution in the diagonal direction, in Fig. 2(b), the resolution board is placed 45 deg with respect to the pixel array of the CMOS chip. Figures 2(a) and 2(b) were reconstructed with the angular spectrum method in Ref. 31 and interpolated with a Fourier zeropadding algorithm by four times. In Figs. 2(a) and 2(b), the lines of group 8, element 1 can be resolved in both images. Moreover, element 3 of group 8 can be resolved in Fig. 2(b), but they are irresolvable in Fig. 2(a). Therefore, the half-pitch resolutions for Figs. 2(e) and 2(f) are 1.94 and 1.55 μm, respectively. The resolution enhancement in the diagonal direction is ∼1.3×. Comparing the Fourier domain of the results, the larger boundary in the diagonal direction in Fig. 2(d) is beneficial to enhancing the resolution in Fig. 2(b). To further illustrate that the enhanced resolution results from the higher frequency components located in the diagonal direction of the Fourier domain, we applied the boundary of the effective Fourier domain of Fig. 2(c) as a rectangular filter on the frequency domain of the 45 deg reconstructed image in Fig. 2(d). After the application of the filter, the lines of 1.55 μm are no longer resolvable as the red dotted line in Fig. 2(f), and the red dotted line profile looks the same as that in Fig. 2(e). Note that the image quality could be further enhanced by utilizing a twin-image eliminating algorithm. 32,33 Then, we test the performance of FDDE LFM with a hematoxylin-eosin (H&E) stained mouse skin sample, as presented in Fig. 3. Biological samples often contain rich structures, and high frequencies are located in different directions. For example, the stratum corneum on the surface of the skin forms layered structures that can be much smaller than the pixel size of the image sensor (2.2 μm), and the distance between the corneum structures could be as small as several micrometers. These kinds of details are not observable with a single holographic image. To extract such fine details, first, three rotational holographic raw images are acquired with directions of 0, π∕6, and π∕3 between the sample and the image sensor. Then, the frequency domains of these three images were synthesized through FDDE, as shown in Figs. 3(c4) and 3(d4).
Benefiting from the 1:1 magnification of LFM, the fieldof-view (FOV) of our FDDE system can reach ∼8 mm 2 (2.8 mm × 2.8 mm). As shown in Fig. 3(c), there is a small raindrop-like structure (indicated with the arrow) in the stratum corneum, as shown by the bright-field microscopy image presented in the inset of Fig. 3(c4). However, in each image of the three phases, Fig. 3(c), this structure is not resolvable. In Fig. 3(c3), this structure is resolved as a filled line, but it is resolved as a hollow structure in the other two directions This result clearly demonstrates the performance of FDDE microscopy with the histological sample. As presented in the line profile, the distance between the stratum corneum structures is ∼3.0 μm, which corresponds to ∼1.4× the pixel size of the image sensor. According to the Nyquist-Shannon sampling theorem, to resolve a two-line structure distance of ∼3.0 μm, the half-pitch resolution should be at least 1.5 μm. This corresponds to a resolution enhancement of ∼1.4×, which is consistent with the principle of FDDE. Such a small structure is not resolvable based on the conventional principle of LFM. Then, the FDDE system is tested with a conventional blood smear sample, as shown in Fig. 4. Because the inner diameter of blood cells is ∼3 to 4 μm, it is difficult for conventional LFM with a pixel size of 2.2 μm to resolve the structure of blood cells. As shown in Figs. 4(c1)-4(c3), a few blood cells can be resolved as a ring structure, and most of the blood cells are observed as dark spots due to insufficient resolution. Some of the blood cells are resolved as a rectangular shape in the conventional singleframe lens-free image, which may be attributed to the rectangular shape of the Fourier domain of LFM. In the FDDE image in Fig. 4(c4), the shape of the red blood cell becomes more circular. Additionally, more blood cells can be resolved as ring structures in the FDDE image. The marked cell is completely irresolvable in any conventional LFM image, and it is observed as a ring structure with FDDE microscopy, which is similar to that in the bright-field image. As presented in Fig. 4(e), the inner diameter of the marked blood cell is 3.3 μm, and this result proves that the half-pitch resolution is ∼1.6 μm, which is a resolution enhancement of ∼1.4×. Also, the quality of the blood smear image with FDDE is much better than that of conventional LFM due to a homogeneous high resolution.
Moreover, the principle of FDDE could be extended to lens-based systems, such as photography. In the experiment, an ISO 12233 resolution target is imaged with a camera lens In the case of line 1 and 1′, both images can resolve the period of 8 pixels. When the line profile gets closer to the center of the target, the period of lines should decrease. The period is decreased to 6 pixels, as shown in line 2′. However, the distance between dips in line 2 is read as 9 pixels, which is even larger than line 1. This artifact is caused by the lack of a highfrequency component in the Fourier domain. Hence, the resolution in the horizontal and the diagonal direction is 8 and 6 pixels, respectively, which corresponds to 2 and 1.5 pixels in the raw images, respectively. These results are approximately the theoretical limit in the horizontal and diagonal directions with a pixel size determined resolution. The result reflects ∼1.3× higher resolution for the diagonal direction than that of the horizontal direction, which is consistent with the principle of FDDE (see also Supplemental Fig. S3). Therefore, FDDE can be applied to enhance the resolution of lens-based photography as well, when the resolution is limited by the pixel size.

Conclusion
Compared with other super-resolution schemes, 27,28 this proposed method is more convenient. For example, in subpixel super-resolution schemes, a nanometer-precision motorized translation stage or a tunable laser source is required, which can substantially increase the cost and complexity of the imaging system. In the FDDE method, the accuracy of the rotating angle is not strictly required, and this operation can be easily satisfied with a manual or a motorized rotation stage. This method does not require a computationally laborious subpixel super-resolution algorithm, which is highly sensitive to noise. Another merit is that the resolution enhancement can be further improved by a smaller pixel size. In this case, an upcoming consumer-level CMOS chip with a smaller pixel size and low cost will further enhance the performance and applications of the FDDE in a straightforward manner. For instance, FDDE LFM could reach an isotropic resolution of 0.56 μm by utilizing a CMOS chip with a pixel size of 0.8 μm (IMX586, SONY, cost approximately twenty U.S. dollars). At the same time, the overall cost, complexity of the FDDE system and the algorithm, portability, and robustness are far better than those of conventional optical microscopy. By contrast, in the subpixel sampling super-resolution scheme with a motorized x − y translational stage, the stage should have subpixel accuracy, which brings significant additional cost. Therefore, the FDDE method is conducive to achieving an ultra-compact and low-cost red blood cell examination instrument, which is especially important for rural and underdeveloped areas. In conventional microscopy or imaging, this method can still work if the pixel size of the detector is larger than Nyquist-Shannon sampling of the optical resolution or > λ∕4 NA. FDDE is particularly useful for scenarios in which the pixel size restricts the system resolution. For example, the widely used electron multiplying CCD (EMCCD) features a pixel size of ∼16 μm for better sensitivity, much larger than that of a conventional CCD or sCMOS detector. With a large pixel size, either the resolution or the FOV is compromised. Here, with FDDE, one can maintain the sensitivity but gain a resolution enhancement of 1.4×. This is particularly useful when the resolution is limited by the pixel size, such as LFM, EMCCDbased imaging, and spectroscopy. In summary, we proposed a novel rotational approach to achieving a higher resolution for digital imaging. The enhanced resolution results from the higher-resolution component located in the diagonal directions, which is straightforward but has not been demonstrated before, to the best of our knowledge. The resolution enhancement is attained by exploiting the higher frequency of the diffraction pattern located in the diagonal direction, in which the frequency border is effectively extended diagonally. We demonstrate FDDE first with LFM and then extend it to lens-based photography. By combining the frequency domains from the images acquired with different angles, the resulting FDDE image has an isotropic resolution enhancement of ∼1.4×. The half-pitch resolution reaches ∼1.5 μm when imaging a mouse skin sample and a blood smear sample using a CMOS detector with a pixel size of 2.2 μm. Then, we extended this approach to lens-based photography, yielding the same resolution enhancement. Here, the FDDE image only uses three raw images with different recording directions, but more images with different directions could be acquired for better image quality. This method can also be applied to a wide range of imaging or detection areas in which the limiting factor is the pixel size of the detector due to undersampling, such as microscopy, telescope imaging, and spectroscopy. He is now pursuing his PhD in biomicrofluidic engineering at University of Technology Sydney. His current research interests include microfluidicbased 3D models culture, organ-on-chip, optofluidic imaging, and optical fiber sensing technology.
Xinzhu Xu received his BS degree in optoelectronic information science and engineering from Northwestern Polytechnical University in 2018, and is now a PhD candidate in biomedical engineering at Peking University, College of Engineering. He is currently interested in the field of imagescanning microscopy, stimulated emission depletion microscopy, minimal photon flux microscopy, and the related optical modulation technology in super-resolution microscopy, which can be utilized in biological research as well as other interdisciplinary studies.
Dayong Jin is a distinguished professor at the University of Technology Sydney since 2017 and a chair professor at Southern University of Science and Technology since 2019. He obtained his PhD from Macquarie University in 2007. His research has been in the physical, engineering, and interdisciplinary sciences. He is a technology developer with expertise covering optics, luminescent materials, sensing, automation devices, microscopy imaging, and analytical chemistry to enable rapid detection of cells and molecules and engineering of sensors and photonics devices.
Zhen Liu received his BS degree in physics and MSc degree in optics in 2010 and 2013, respectively. After years of industrial experience in signal processing and telecommunication, he is now pursuing his PhD in University of Southampton. His research interesting is focusing on nonlinear optics-based biosensing and related signal processing.
Kebin Shi works in the field of optical imaging and spectroscopy for biophotonics and precision metrology. His recent research interests include super-resolution imaging, nonlinear holography, and femto-second frequency comb metrology. His research publication includes over 80 technical papers and 6 patents.
Fan Bai received BSc degree in physics from Peking University in 2003 and DPhil in biophysics from University of Oxford in 2008. After three years postdoctoral training at University of Oxford and Osaka University, he returned to China in 2011 to lead his own research team. The Bai Lab studies frontiers of biomedical research by combining cutting-edge singlemolecule fluorescence imaging and single-cell sequencing, aiming at (1) study circulating tumor cells to gain deeper insights into the molecular mechanism of cancer metastasis and explore its clinical value for noninvasive cancer diagnosis, prognosis, and therapy evaluation; (2) study the genomic features of Chinese cancers; and (3) study bacterial motility, virulence, and mechanisms of antibiotic and bacterial drug resistance and tolerance.
Shu Wang is a professor and a surgeon in Peking University People's Hospital, China. One of her interests is in translational cancer research on molecular and image diagnosis. She has done a series study on fluorescence-guided sentinel lymph node biopsy and intraoperative cancer diagnosis by high-resolution optical coherence tomography.
Peng Xi is an associate professor at the College of Engineering, Peking University, China. His research interest is on the development of optical super-resolution microscopy techniques. He has published over 60 scientific journal papers in peer-reviewed journals, including Nature, Nature Methods, etc., and delivered over 30 keynote/invited talks at international conferences hosted by SPIE and OSA. He is a senior member of OSA. He has been awarded the Beijing Distinguished Young Scholar in 2018. He is on the editorial board of five SCI-indexed journals: Light: Science and Applications, Advanced Photonics, Microscopy Research and Techniques, Micron, and Scientific Reports.