The pixel size of a charge-coupled device (CCD) camera plays a major role in the image resolution, and the square pixels are attributed to the physical anisotropy of the sampling frequency. We synthesize the high sampling frequency directions from multiple frames acquired with different angles to enhance the resolution by 1.4 × over conventional CCD orthogonal sampling. To directly demonstrate the improvement of frequency-domain diagonal extension (FDDE) microscopy, lens-free microscopy is used, as its resolution is dominantly determined by the pixel size. We demonstrate the resolution enhancement with a mouse skin histological specimen and a clinical blood smear sample. Further, FDDE is extended to lens-based photography with an ISO 12233 resolution target. This method paves a new way for enhancing the image resolution for a variety of imaging techniques in which the resolution is primarily limited by the sampling pixel size, for example, microscopy, photography, and spectroscopy.
The charge-coupled device (CCD) has revolutionized imaging in the digital era.1 In microscopy and photography, a lens is typically employed to translate the object of an image to be captured by the CCD or the CMOS sensor. Hence, the size of the CCD pixel has become both a bottleneck and a benchmark parameter for the resolution of digital imaging. The modulated transfer function is an efficient way to evaluate the performance of microscopy or photography as the contrast is plotted in the frequency domain. An interesting approach for super-resolution is that, in structured illumination microscopy (SIM),2–5 the structured pattern is employed to expand the frequency domain, which can improve the spatial resolution by .6–8 Similarly, in Fourier ptychography microscopy, the sample is illuminated with multiple angles for a synthetic aperture, which is an effective expansion in the frequency domain for an improvement in spatial resolution.9,10
Lens-free microscopy (LFM) is a novel imaging technique that breaks the space–bandwidth product of conventional lens-based microscopic techniques by avoiding the application of the lens; instead its resolution is directly coupled with the pixel size.11–13 LFM has several distinct advantages, such as single-shot 3D imaging, cost-effectiveness, compactness, and no space–bandwidth product limitation.14 Applications based on LFM have been realized, such as field-portable microscope,15 point-of-care testing,16 telemedicine,17,18 and air quality monitoring.19 However, the spatial resolution of LFM is still a major bottleneck that prohibits its wide application as it is limited by the pixel size of the 2D detector.12 On the one hand, several variations of LFM have been demonstrated to overcome the resolution barrier imposed by the pixel size, such as lateral shift-based pixel super-resolution,20 multiangle illumination,21,22 multiheight,23 wavelength scanning,24 fiber-optic taper,25 and synthetic aperture,26 as well as multiangle illumination achieved with a tunable wavelength semiconductor laser source and a volume phase grating.27 On the other hand, no lateral resolution substantially higher than the size of a pixel with a single imaging exposure has been reported.
In this paper, we utilize the anisotropic resolution of a 2D detector to present a method that can enhance the spatial resolution by . Previously, it is widely accepted that the pixel size is the smallest unit for digital imaging. Here, by exploiting the nonsymmetry of the frequency domain, we propose frequency-domain diagonal extension (FDDE) microscopy in which the resolution can be improved to a pixel size, even in one snapshot, through sampling in the diagonal direction. The principle is straightforward: the diagonal direction samples more densely, thus containing more frequency information. Translating the spatial image to the frequency domain through Fourier transform also clearly reveals this fact.
We first demonstrate the principle of FDDE with LFM,12,15,28 as in LFM the resolution is mainly limited by the pixel size. Then, through a combination of multiple frequency components from different angles, a super-resolution image with isotropic resolution can be achieved, similar to the frequency-domain image processing procedure of SIM.4,8 The resolution enhancement of FDDE is demonstrated experimentally with a mouse skin specimen and a clinical blood smear sample. We extended FDDE to conventional photography using an ISO 12233 resolution target.
The principle of FDDE microscopy is very straightforward (Fig. 1), that is, the high-frequency component in the diagonal direction is utilized to enhance the resolution. In FDDE microscopy, the image is typically recorded with a 2D sensor with grid-like pixels. The optical transfer function (OTF) of the imaging system and the frequency domain of digital imaging are represented as follows:Fig. 1(b), the boundary of the diffraction-limited OTF and the frequency domain of digital imaging is circular and rectangular, respectively. In the case of the undersampling condition, the frequency domain of digital imaging is smaller than the theoretical OTF, due to the nature of the pixel arrangement in Fig. 1(a). As shown in Fig. 1(b), the diagonal boundary of digital imaging, , is 1.4 times as large as the horizontal/vertical direction in Eq. (1). A larger diagonal boundary means a bigger bandwidth, and this can be employed to collect higher frequency components for better resolution. This is because the sampling frequency in diagonal direction is actually much higher—the distance is only 0.7 pixels [Fig. 1(a)] in the diagonal direction.29
To elucidate the anisotropic frequency domain of undersampling digital imaging, LFM is employed as its resolution is exclusively dominated by the pixel size. As presented in Fig. 1(c), the LFM system consists of a light source, a sample stage, and an image sensor. In this work, the sample stage is mounted on a rotation stage. The raw images are obtained with different angles between the sample and the image sensor. In the case of holographic inline LFM, the theoretical resolution and the OTF are mainly determined by the wavelength and the coherence of the illumination light and the distance between the sample and the image sensor, and the technique has an isotropic OTF, which also has a circular boundary. Because the pixel size () is much larger than the theoretical resolution () in LFM, the raw image is a considerably undersampled image (the resolution is limited by pixel size). For a pixel size of , the frequency limit is equal to in the frequency domain, and the OTF is constrained by a smaller rectangle window, as presented in Fig. 1(b). Therefore, the resulting frequency domain has a rectangular shape. In the frequency domain, the maximum frequency limit along the diagonal direction is -fold that along the horizontal/vertical directions. This implies that the spatial resolution should be higher in the diagonal direction than in the horizontal and vertical directions.
We propose a straightforward method to achieve extended resolution isotropically [Fig. 1(d)]. As higher frequency is attributed to higher resolution in the Fourier domain, it can be utilized to extend the resolution. First, we obtain a series of images at different detection directions. The images are rotated to the same orientation and registered with each other.30 Because of the rotation of the grid 2D detector, the corresponding four corners will contain higher frequencies than the four borders. Then, the high-frequency information in different directions is extracted from the frequency domain of the images. Finally, the high-frequency information is stitched together and then converted back into the spatial domain to obtain a super-resolved image.
The frequency domain of FDDE generated with raw images of frames is expressed as follows:
To realize the FDDE algorithm, a blank matrix of the FDDE Fourier domain is created, then each pixel of this matrix is filled with the amplitude and the phase value at the same location from the frequency domain of the raw images. Because the raw images are acquired in different directions, the boundary of the frequency domain of each raw image is different in each direction. Thus, the matrix is filled with the complex value from the biggest boundary of the frequency domain of the raw images, as depicted in Eq. (2). Each pixel in the frequency domain has an angle with respect to the center point. By knowing the angle of each raw image in the experiment, we have the region of the angle for the largest boundary of the frequency domain, given as the red-line marked area in Fig. 1(b). Then we know which raw image should be chosen for each pixel in the frequency domain of FDDE. In this way, we fill every pixel in the frequency domain of FDDE. As a result, the frequency domain in the corresponding angle region from each of the raw images is selected, and then, they are stitched together, as depicted in Fig. 1(d). Then, FDDE is obtained with inverted fast Fourier transformation on the stitched frequency domain.
To illustrate the anisotropic resolution of the conventional LFM, a resolution target is imaged. In Fig. 2, the resolution-measuring results with two different orientations of the image sensor are presented. The experiments are carried out with a CMOS chip, which has a pixel size of . In Fig. 2(a), the resolution target is placed horizontally with respect to the pixel array of the CMOS chip. To elucidate the anisotropic resolution in the diagonal direction, in Fig. 2(b), the resolution board is placed 45 deg with respect to the pixel array of the CMOS chip. Figures 2(a) and 2(b) were reconstructed with the angular spectrum method in Ref. 31 and interpolated with a Fourier zero-padding algorithm by four times. In Figs. 2(a) and 2(b), the lines of group 8, element 1 can be resolved in both images. Moreover, element 3 of group 8 can be resolved in Fig. 2(b), but they are irresolvable in Fig. 2(a). Therefore, the half-pitch resolutions for Figs. 2(e) and 2(f) are 1.94 and , respectively. The resolution enhancement in the diagonal direction is . Comparing the Fourier domain of the results, the larger boundary in the diagonal direction in Fig. 2(d) is beneficial to enhancing the resolution in Fig. 2(b). To further illustrate that the enhanced resolution results from the higher frequency components located in the diagonal direction of the Fourier domain, we applied the boundary of the effective Fourier domain of Fig. 2(c) as a rectangular filter on the frequency domain of the 45 deg reconstructed image in Fig. 2(d). After the application of the filter, the lines of are no longer resolvable as the red dotted line in Fig. 2(f), and the red dotted line profile looks the same as that in Fig. 2(e). Note that the image quality could be further enhanced by utilizing a twin-image eliminating algorithm.32,33
Then, we test the performance of FDDE LFM with a hematoxylin-eosin (H&E) stained mouse skin sample, as presented in Fig. 3. Biological samples often contain rich structures, and high frequencies are located in different directions. For example, the stratum corneum on the surface of the skin forms layered structures that can be much smaller than the pixel size of the image sensor (), and the distance between the corneum structures could be as small as several micrometers. These kinds of details are not observable with a single holographic image. To extract such fine details, first, three rotational holographic raw images are acquired with directions of 0, , and between the sample and the image sensor. Then, the frequency domains of these three images were synthesized through FDDE, as shown in Figs. 3(c4) and 3(d4).
Benefiting from the 1:1 magnification of LFM, the field-of-view (FOV) of our FDDE system can reach (). As shown in Fig. 3(c), there is a small raindrop-like structure (indicated with the arrow) in the stratum corneum, as shown by the bright-field microscopy image presented in the inset of Fig. 3(c4). However, in each image of the three phases, Fig. 3(c), this structure is not resolvable. In Fig. 3(c3), this structure is resolved as a filled line, but it is resolved as a hollow structure in the other two directions [Figs. 3(c1) and 3(c2)]. This raindrop-like structure can only be resolved in the FDDE image.
This result clearly demonstrates the performance of FDDE microscopy with the histological sample. As presented in the line profile, the distance between the stratum corneum structures is , which corresponds to the pixel size of the image sensor. According to the Nyquist–Shannon sampling theorem, to resolve a two-line structure distance of , the half-pitch resolution should be at least . This corresponds to a resolution enhancement of , which is consistent with the principle of FDDE. Such a small structure is not resolvable based on the conventional principle of LFM.
Then, the FDDE system is tested with a conventional blood smear sample, as shown in Fig. 4. Because the inner diameter of blood cells is to , it is difficult for conventional LFM with a pixel size of to resolve the structure of blood cells. As shown in Figs. 4(c1)–4(c3), a few blood cells can be resolved as a ring structure, and most of the blood cells are observed as dark spots due to insufficient resolution. Some of the blood cells are resolved as a rectangular shape in the conventional single-frame lens-free image, which may be attributed to the rectangular shape of the Fourier domain of LFM. In the FDDE image in Fig. 4(c4), the shape of the red blood cell becomes more circular. Additionally, more blood cells can be resolved as ring structures in the FDDE image. The marked cell is completely irresolvable in any conventional LFM image, and it is observed as a ring structure with FDDE microscopy, which is similar to that in the bright-field image. As presented in Fig. 4(e), the inner diameter of the marked blood cell is , and this result proves that the half-pitch resolution is , which is a resolution enhancement of . Also, the quality of the blood smear image with FDDE is much better than that of conventional LFM due to a homogeneous high resolution.
Moreover, the principle of FDDE could be extended to lens-based systems, such as photography. In the experiment, an ISO 12233 resolution target is imaged with a camera lens (SV-10035V, VS Technology) coupled with a sCMOS camera (Andor Zyla 4.2). Figures 5(a) and 5(b) present the raw images interpolated four times with a Fourier zero-padding method and with the intensity of the images inverted. The line profiles marked in Figs. 5(a) and 5(b) are presented in Figs. 5(c) and 5(d). In the case of line 1 and 1′, both images can resolve the period of 8 pixels. When the line profile gets closer to the center of the target, the period of lines should decrease. The period is decreased to 6 pixels, as shown in line 2′. However, the distance between dips in line 2 is read as 9 pixels, which is even larger than line 1. This artifact is caused by the lack of a high-frequency component in the Fourier domain. Hence, the resolution in the horizontal and the diagonal direction is 8 and 6 pixels, respectively, which corresponds to 2 and 1.5 pixels in the raw images, respectively. These results are approximately the theoretical limit in the horizontal and diagonal directions with a pixel size determined resolution. The result reflects higher resolution for the diagonal direction than that of the horizontal direction, which is consistent with the principle of FDDE (see also Supplemental Fig. S3). Therefore, FDDE can be applied to enhance the resolution of lens-based photography as well, when the resolution is limited by the pixel size.
Compared with other super-resolution schemes,27,28 this proposed method is more convenient. For example, in subpixel super-resolution schemes, a nanometer-precision motorized translation stage or a tunable laser source is required, which can substantially increase the cost and complexity of the imaging system. In the FDDE method, the accuracy of the rotating angle is not strictly required, and this operation can be easily satisfied with a manual or a motorized rotation stage. This method does not require a computationally laborious subpixel super-resolution algorithm, which is highly sensitive to noise. Another merit is that the resolution enhancement can be further improved by a smaller pixel size. In this case, an upcoming consumer-level CMOS chip with a smaller pixel size and low cost will further enhance the performance and applications of the FDDE in a straightforward manner. For instance, FDDE LFM could reach an isotropic resolution of by utilizing a CMOS chip with a pixel size of (IMX586, SONY, cost approximately twenty U.S. dollars). At the same time, the overall cost, complexity of the FDDE system and the algorithm, portability, and robustness are far better than those of conventional optical microscopy. By contrast, in the subpixel sampling super-resolution scheme with a motorized translational stage, the stage should have subpixel accuracy, which brings significant additional cost. Therefore, the FDDE method is conducive to achieving an ultra-compact and low-cost red blood cell examination instrument, which is especially important for rural and underdeveloped areas. In conventional microscopy or imaging, this method can still work if the pixel size of the detector is larger than Nyquist–Shannon sampling of the optical resolution or . FDDE is particularly useful for scenarios in which the pixel size restricts the system resolution. For example, the widely used electron multiplying CCD (EMCCD) features a pixel size of for better sensitivity, much larger than that of a conventional CCD or sCMOS detector. With a large pixel size, either the resolution or the FOV is compromised. Here, with FDDE, one can maintain the sensitivity but gain a resolution enhancement of . This is particularly useful when the resolution is limited by the pixel size, such as LFM, EMCCD-based imaging, and spectroscopy.
In summary, we proposed a novel rotational approach to achieving a higher resolution for digital imaging. The enhanced resolution results from the higher-resolution component located in the diagonal directions, which is straightforward but has not been demonstrated before, to the best of our knowledge. The resolution enhancement is attained by exploiting the higher frequency of the diffraction pattern located in the diagonal direction, in which the frequency border is effectively extended diagonally. We demonstrate FDDE first with LFM and then extend it to lens-based photography. By combining the frequency domains from the images acquired with different angles, the resulting FDDE image has an isotropic resolution enhancement of . The half-pitch resolution reaches when imaging a mouse skin sample and a blood smear sample using a CMOS detector with a pixel size of . Then, we extended this approach to lens-based photography, yielding the same resolution enhancement. Here, the FDDE image only uses three raw images with different recording directions, but more images with different directions could be acquired for better image quality. This method can also be applied to a wide range of imaging or detection areas in which the limiting factor is the pixel size of the detector due to undersampling, such as microscopy, telescope imaging, and spectroscopy.
This work was supported by the National Natural Science Foundation of China (NSFC) (Grant Nos. 31971376, 61705252, 61729501, 91750203, and 51720105015), the Beijing Natural Science Foundation (Grant No. JQ18019), the Natural Science Foundation of Jiangsu Province (Grant No. BK20170388), Australia-China Joint Research Centre for Point-of-Care Testing (Grant Nos. ACSRF65827, SQ2017YFGH001190), and Science and Technology Innovation Commission of Shenzhen (Grant No. KQTD20170810110913065). The authors declare that there are no conflicts of interest regarding the publication of this article.
Shan Jiang received his BS degree in electronic science and technology from Heilongjiang University in 2008 and his MS degree and PhD in physical electronics from Harbin Institute of Technology in 2010 and 2015, respectively. He is currently a postdoctoral researcher at Peking University. His current research interests include super-resolution microscopy and holography.
Meiling Guan received her BS degree in biomedical engineering from Beijing University of Technology, Beijing, China, in 2018. She is pursuing her PhD in biomedical engineering at Peking University, Beijing, China. Her current interests include image restoration and reconstruction algorithms for image scanning and super-resolution microscopy, phase retrieval and compressed sensing reconstruction of lens-free inline holography, and super-resolution dipole orientation mapping via polarization demodulation.
Jiamin Wu received his BS degree and PhD from the Department of Automation, Tsinghua University, Beijing, China, in 2014 and 2019, respectively. He is currently a postdoctoral fellow within the Institute for Brain and Cognitive Sciences at Tsinghua University. His current research interests include computational microscopy and optical computing, with a particular emphasis on developing computation-based optical setups for observing large-scale biological dynamics in vivo.
Guocheng Fang received his BS and MS degrees in electric science engineering from North University of China, in 2014 and 2017, respectively. He is now pursuing his PhD in biomicrofluidic engineering at University of Technology Sydney. His current research interests include microfluidic-based 3D models culture, organ-on-chip, optofluidic imaging, and optical fiber sensing technology.
Xinzhu Xu received his BS degree in optoelectronic information science and engineering from Northwestern Polytechnical University in 2018, and is now a PhD candidate in biomedical engineering at Peking University, College of Engineering. He is currently interested in the field of image-scanning microscopy, stimulated emission depletion microscopy, minimal photon flux microscopy, and the related optical modulation technology in super-resolution microscopy, which can be utilized in biological research as well as other interdisciplinary studies.
Dayong Jin is a distinguished professor at the University of Technology Sydney since 2017 and a chair professor at Southern University of Science and Technology since 2019. He obtained his PhD from Macquarie University in 2007. His research has been in the physical, engineering, and interdisciplinary sciences. He is a technology developer with expertise covering optics, luminescent materials, sensing, automation devices, microscopy imaging, and analytical chemistry to enable rapid detection of cells and molecules and engineering of sensors and photonics devices.
Zhen Liu received his BS degree in physics and MSc degree in optics in 2010 and 2013, respectively. After years of industrial experience in signal processing and telecommunication, he is now pursuing his PhD in University of Southampton. His research interesting is focusing on nonlinear optics-based biosensing and related signal processing.
Kebin Shi works in the field of optical imaging and spectroscopy for bio-photonics and precision metrology. His recent research interests include super-resolution imaging, nonlinear holography, and femto-second frequency comb metrology. His research publication includes over 80 technical papers and 6 patents.
Fan Bai received BSc degree in physics from Peking University in 2003 and DPhil in biophysics from University of Oxford in 2008. After three years postdoctoral training at University of Oxford and Osaka University, he returned to China in 2011 to lead his own research team. The Bai Lab studies frontiers of biomedical research by combining cutting-edge single-molecule fluorescence imaging and single-cell sequencing, aiming at (1) study circulating tumor cells to gain deeper insights into the molecular mechanism of cancer metastasis and explore its clinical value for non-invasive cancer diagnosis, prognosis, and therapy evaluation; (2) study the genomic features of Chinese cancers; and (3) study bacterial motility, virulence, and mechanisms of antibiotic and bacterial drug resistance and tolerance.
Shu Wang is a professor and a surgeon in Peking University People’s Hospital, China. One of her interests is in translational cancer research on molecular and image diagnosis. She has done a series study on fluorescence-guided sentinel lymph node biopsy and intraoperative cancer diagnosis by high-resolution optical coherence tomography.
Peng Xi is an associate professor at the College of Engineering, Peking University, China. His research interest is on the development of optical super-resolution microscopy techniques. He has published over 60 scientific journal papers in peer-reviewed journals, including Nature, Nature Methods, etc., and delivered over 30 keynote/invited talks at international conferences hosted by SPIE and OSA. He is a senior member of OSA. He has been awarded the Beijing Distinguished Young Scholar in 2018. He is on the editorial board of five SCI-indexed journals: Light: Science and Applications, Advanced Photonics, Microscopy Research and Techniques, Micron, and Scientific Reports.