Untrained neural network enhances the resolution of structured illumination microscopy under strong background and noise levels

Abstract. Structured illumination microscopy (SIM) has been widely applied in the superresolution imaging of subcellular dynamics in live cells. Higher spatial resolution is expected for the observation of finer structures. However, further increasing spatial resolution in SIM under the condition of strong background and noise levels remains challenging. Here, we report a method to achieve deep resolution enhancement of SIM by combining an untrained neural network with an alternating direction method of multipliers (ADMM) framework, i.e., ADMM-DRE-SIM. By exploiting the implicit image priors in the neural network and the Hessian prior in the ADMM framework associated with the optical transfer model of SIM, ADMM-DRE-SIM can further realize the spatial frequency extension without the requirement of training datasets. Moreover, an image degradation model containing the convolution with equivalent point spread function of SIM and additional background map is utilized to suppress the strong background while keeping the structure fidelity. Experimental results by imaging tubulins and actins show that ADMM-DRE-SIM can obtain the resolution enhancement by a factor of ∼1.6 compared to conventional SIM, evidencing the promising applications of ADMM-DRE-SIM in superresolution biomedical imaging.

internalization of clathrin-coated pits (CCPs), 8 and tubulin and kinesin dynamics. 9 However, the resolution of SIM with linear illumination mode is only 2 times over that of wide-field microscopy, which reaches about 100 nm, because the maximal frequency of structured illumination pattern is limited by the diffraction. Thus, SIM is inferior to some other superresolution techniques in spatial resolution, such as stimulated emission depletion microscopy, 10 photoactivated localization microscopy, 11 or stochastic optical reconstruction microscopy, 12 which can obtain the resolution up to about 10 nm. By now, many methods have been reported to increase the spatial resolution of SIM. For example, fluorescence saturation was utilized to induce nonlinear structured illumination for higher spatial frequency shift. 13 Similarly, photoswitching proteins were used as the fluorescent labels for nonlinear SIM with low excitation power. 14 With these methods, the nonlinear SIM with a resolution below 50 nm has been realized, but unfortunately it comes at the expense of reducing the imaging speed. In addition, a sparse deconvolution method (sparse-SIM) was proposed to further obtain the resolution enhancement of SIM while keeping the imaging speed by employing the sparsity and continuity priors of biological structures. 15 Without the modification to the hardware device or acquirement mode of SIM, sparse-SIM achieves almost twofold resolution enhancement compared with conventional SIM. However, as an iterative deconvolution method, sparse-SIM is greatly affected by the artificial selection of multiple parameters, which are used to balance the weights of various priors. Therefore, the image processing in sparse-SIM is time-consuming.
In the experimental acquisition of a fluorescent image, strong background and noise are common issues. These issues may be tolerated in normal microscopy analysis but can lead to severe artifacts in the superresolution image reconstruction, which make it difficult to distinguish the real structures and fake artifacts. The strong background mainly results from the out-offocus fluorescence and cellular autofluorescence, while the strong noise often occurs in the image acquirement by the camera, especially when the fluorescent photon number is limited by the short exposure time or low excitation power. To further improve the resolution of SIM under strong background and noise levels, here we propose a deep resolution enhancement (DRE) method by an untrained neural network associated with alternating direction method of multipliers (ADMM) framework, termed as ADMM-DRE-SIM. An untrained neural network with implicit priors is utilized as a generator for desired resolution-enhanced image, and an image degradation model containing optical transfer of SIM and additional background map provides the criterion for updating the parameters of the neural network. The untrained neural network is quite convenient to obtain the resolution enhancement of SIM because no training datasets are required (it is rather difficult to experimentally acquire the training datasets containing the SIM images and paired resolution-enhanced images). An iterative framework of ADMM embedded with Hessian prior is used to optimize the network by exploiting the intrinsic continuity of biological structures. Based on these special designs, ADMM-DRE-SIM realizes the resolution enhancement of SIM under the strong background and noise while completely preserving the advantage of SIM with high imaging speed. Compared with conventional SIM, a resolution enhancement by a factor of about 1.6 is experimentally demonstrated by imaging tubulins and actins in cells with ADMM-DRE-SIM. Moreover, ADMM-DRE-SIM shows better performance in the fidelity of biological structure compared to other deconvolution methods, which is key for image analysis, and it will provide a powerful tool for the studies of organelle dynamics.

Methods
The resolution of the image acquired in wide-field microscopy is restricted by the optical diffraction limit due to the finite aperture of the objective. To exceed the resolution limitation, a structured illumination pattern is utilized to modulate the sample image with spatial frequency shift in SIM. The recorded image in SIM can be formatted in the frequency domain as where DðkÞ is the spatial spectra of recorded image in SIM, SðkÞ is the structured illumination pattern, EðkÞ is the spatial spectra of fluorescent sample, OðkÞ is the optical transfer function (OTF) of the microscopy system, NðkÞ is the noise during the image acquirement, S is the max modulation intensity, m is the modulation factor, p is the modulation frequency vector, and ϕ is the modulation phase. The recorded images in SIM are linear combinations of the origin and frequency-shifted components. Normally, the structured illumination patterns with three-step phase shifts in three directions are required to acquire nine images in total, which are then processed to obtain a superresolution image by separating and recombining the frequency components. Since the max frequency shift in linear SIM is close to the bound of OTF, SIM can only acquire an equivalent OTF twice as large as that of wide-field microscopy. The reconstructed SIM image in the spatial domain can be seen as the convolution of fluorescence distribution and equivalent point spread function (PSF) of SIM. The further resolution enhancement from the reconstructed SIM image is similar to deconvolution microscopy, 16 which increases the contrast and resolution of a widefield image. However, the resolution enhancement from the SIM image is much more complex. The noise amplification in the image reconstruction of SIM seriously hampers deconvolution, which makes it difficult to converge. Considering the strong background and noise levels, this problem becomes tougher because the serious artifacts and distortion contaminate the actual structures. High-fidelity image reconstruction for the resolution enhancement in SIM is still limited by the amplified noise as well as the strong initial background. Under the condition of strong fluorescent background and noise levels, the reconstructed SIM image in the spatial domain can be written as where I SIM is the reconstructed SIM image; I sample is the fluorescence distribution of the sample; and PSF SIM is the equivalent PSF of SIM, which considers the cut-off frequency bound of the microscope and the frequency shift of structured illumination patterns; G is the background distribution; and N is the noise from the image acquirement and reconstruction. It should be noted that G and N correspond to the background and noise after the image reconstruction of SIM. The background occupies the low spatial frequency components. Therefore, the special frequency characteristic allows the extraction of the background to some extent. A wavelet-based background subtraction was proposed to handle this task. 17,18 However, the background subtraction in the low-photon images easily causes structural defects because some structures with the low fluorescence intensity are overwhelmed in the background and noise.
To further improve the resolution of SIM under the strong background and noise levels, we adopt a concept of deep image prior (DIP), which indicates that the structure of a generator network itself can capture lots of statistics priors of low-level image for various tasks instead of learning from training datasets. 19 The untrained neural network incorporated with various physical models has shown to be a well-established tool in solving the image reconstruction of computational imaging methods, such as coherent phase imaging, 20 digital in-line holography, 21,22 ghost imaging, 23 and SIM. 24,25 Here we achieve the further resolution enhancement of SIM by the untrained neural network combining with the background interrupted optical transfer model of SIM. The implicit priors inside the neural network provide the bridge between the available information in the SIM image and the higher frequency information corresponding to the higher resolution. Meanwhile, the optical transfer model containing the diffraction limit in SIM and the background map is employed to supervise the neural network and guarantee the image fidelity with background suppression. Furthermore, an ADMM framework embedded with Hessian prior is used to optimize the parameters of the neural network with high precision, which is different from the previous work with only an untrained neural network. 26 The flow chart of ADMM-DRE-SIM is shown in Fig. 1. The original SIM image x 0 is sent into the neural network with a U-net structure to acquire a resolution-enhanced image f θ ðx 0 Þ. An image degradation model, containing the convolution with equivalent PSF of SIM and the additional background, is then used to provide the estimation of the recorded image in SIM from the original fluorescence distribution of the sample. In addition, a pixel upsampling has been incorporated in the network, and a downsampling process is correspondingly embedded in the image degradation model, which is used to adjust the magnification of the reconstructed image. The estimation can be written as (3) where f θ ð:Þ is the neural network with the parameter θ to be optimized, and m is the estimated background map from the SIM image. The equivalent PSF of SIM is estimated from the optical parameters of the SIM system, considering the cut-off frequency bound of the microscope objective and the frequency shift of structured illumination spatial frequency patterns, which can be expressed by the following equation as where J 1 is the first-order Bessel function, λ is the fluorescence wavelength, NA is the numerical aperture of the objective lens, m f is the modulation factor used to modulate the equivalent NA after SIM reconstruction, m f ¼ ð1 þ k f ∕k c Þ, k f is the structured illumination frequency, k c is the cut-off frequency for acquisition, and r is the radial space coordinate of object surface. The additional background map is estimated from the input SIM image with wavelet transform. 17,18 A "db6" wavelet base is used, decomposing the signal into the seventh level. Once the estimation is optimized to approach the input SIM image x 0 , the resolution-enhanced image can be acquired. An ADMM framework embedded with a Hessian prior is used to iteratively optimize the parameters of the neural network. Generally, the resolution enhancement of SIM can be converted into searching for the optimal parameters of the neural network that satisfy the minimization of the following question, and is given as where R Hessian is the Hessian norm, ρ is the regularization parameter, and k:k 2 is the L 2 norm. The Hessian matrix can exploit the continuity of biological structures as the prior knowledge and provide a tool to constrain image restoration and suppress noise, which has shown good performance in the image reconstruction of SIM with structural fidelity and artifact suppression. 4 Here, the Hessian prior is used to constrain the parameter optimization of the neural network. The Hessian matrix can be expressed as where D xx , D yy , and D xy denote the discrete second-order derivatives at the pixel i of the image I along the three axes of horizontal, vertical, and diagonal directions, respectively. Based on the ADMM framework, Eq. (5) can be transferred into the following problem by introducing the auxiliary variable v and is written as þ ρR Hessian ðvÞ; subject to Df θ ðx 0 Þ ¼ v: By further introducing the auxiliary variable u, Eq. (7) can be divided the minimization into the following subproblems and is given as The subproblem in Eq. (8) is similar to the traditional DIP solution by forcing Df θ ðx 0 Þ to approach v k − u k ∕β with the neural network; here the parameter θ is optimized by a gradient descent method. 27,28 In Eqs. (9) and (10), the second-order derivatives in different directions are separately processed. By the alternate iterative optimization, a resolution-enhanced image with low noise and background removal can be obtained. It is worth mentioning that ADMM-DRE-SIM can optimize the parameters of the network in a self-supervised mode without the training datasets.

Results and Discussion
To demonstrate the resolution enhancement of SIM by ADMM-DRE-SIM, a theoretical simulation is conducted. A simulated image containing the curves with width of 32.5 nm is used as the ground truth. The wide-field image is generated by the convolution of the ground truth and the PSF of a microscope with the NA of 1.2 and fluorescence wavelength of 560 nm. Nine images are simulated by sequential structured illumination and PSF convolution based on the ground truth. A fairSIM algorithm is utilized to recover the superresolution SIM image. 29 Then, the SIM image is further processed by ADMM-DRE-SIM to obtain a resolution-enhanced image. All the image processing is performed on a server with Intel i9-10920X, 192 GB of RAM and NVIDIA GeForce RTX3090. The simulation results are shown in Fig. 2, and the corresponding spatial frequency spectra and the intensity profiles along the labeled lines are also provided. As can be seen, only the low-frequency components are preserved in the wide-field image due to the limited OTF of the microscope, which lead to low spatial resolution, and the curves along the labeled line are seriously blurred. The SIM image has a frequency bound twice that of the wide-field image, and the intensity profile of the curves can be distinguished with difficulty. However, the ADMM-DRE-SIM image has a greatly extended frequency bound, which is close to the ground truth, and the adjacent curves can be clearly distinguished.
In actual experiments, the background and noise are inevitable, and thus the recorded images with structured illumination in SIM will be contaminated. Although SIM can suppress the background to some extent, heavy noise will cause vague artifacts during SIM reconstruction. As for the resolution improvement in noisy conditions, the noise will mislead the optimization direction in iterative calculations. The heavier the noise is, the more effort the algorithm takes to cover the noise. This leads to a bias between the reconstructed result and the real structure, which finally results in serious artifacts. Besides, the more resolution enhancement is conducted, the more sensitive to noise the reconstructed result is. The SIM images involving background and various noise levels are processed to test the resolution enhancement performance by ADMM-DRE-SIM. The simulated results are shown in Fig. 3; here, a uniform background combined with Gaussian noise σ ¼ 0.1, 0.3, 0.5, and Poisson noise mixed with Gaussian noise σ ¼ 0.5 are, respectively, added in the recorded image with structured illumination in SIM. For comparison, some traditional deconvolution algorithms are also utilized to process the SIM image, including Wiener deconvolution, 30 Richardson-Lucy total variation (RLTV) deconvolution, 31 and Hessian deconvolution. 32 The SIM images are also reconstructed by fairSIM, associated with the resolution-enhanced images by various traditional algorithms and ADMM-DRE-SIM, as shown in Fig. 3(b).
Obviously, ADMM-DRE-SIM is able to improve the resolution of SIM and simultaneously suppress the background and noise. In contrast, other deconvolution algorithms bring serious artifacts with slight resolution enhancement. In addition, structural similarity values of the images processed by Wiener, RLTV, Hessian deconvolution and ADMM-DRE-SIM in the condition of Poisson noise mixed with Gaussian noise σ ¼ 0.5 are calculated to be 0.07, 0.10, 0.09, and 0.68, respectively. In a word, ADMM-DRE-SIM shows good performance in the resistance to the background and noise during the image processing of resolution enhancement, which is very useful in processing the experimental SIM images.
Furthermore, two experiments are performed by imaging the intracellular structures with a commercial SIM system (Nikon, N-SIM) to further demonstrate the resolution enhancement capability of ADMM-DRE-SIM. The exposure time is set as 200 ms with illumination power of about 22.9 mW. First, the tubulins in mouse embryonic fibroblast (MEF) cells are investigated, as shown in Fig. 4(a). Tubulin is part of the cytoskeleton of cells and has a filamentous distribution with large space. Compared to SIM, ADMM-DRE-SIM provides finer structural details of the tubulins, and the overlapped fibers can be distinguished more clearly. Meanwhile, the intensity profiles along the labeled line also intuitively show the resolution enhancement. To quantitatively determine the resolution enhancement, a Fourier ring correlation (FRC) method is utilized to characterize the spatial resolution, and the calculated results are given in Fig. 4(b). The resolution is improved from 145 nm of SIM to 82 nm by ADMM-DRE-SIM, which indicates that ADMM-DRE-SIM is an effective technique to improve the resolution of SIM. Second, the actins in NIH/3T3 cells are further investigated, as shown in Fig. 4(c). Different from the tubulins, the distribution of actins in cells is much denser, and the fiber is thinner. Thus, the images of actins show a stronger background due to the out of focus fluorescence, which bring the difficulty for further resolution enhancement. Similarly, ADMM-DRE-SIM shows clearer spatial structure of actins compared with SIM. Importantly, ADMM-DRE-SIM can greatly suppress the artifacts of the SIM image due to the calculation error of the background in the image reconstruction. According to the calculated results of FRC in Fig. 4(d), the resolution of ADMM-DRE-SIM is enhanced to 85 nm from 140 nm of SIM. Based on our experimental observations in Fig. 4, ADMM-DRE-SIM can realize the resolution enhancement by a factor of about 1.6.
To specifically exhibit the superiority of ADMM-DRE-SIM in suppressing the background and noise, the SIM images of tubulins with strong background or low signal-to-noise ratio (SNR) are separately collected and processed in experiment. As shown in Fig. 5(a), the tubulin fibers in the SIM image are seriously contaminated by the background, although SIM has the function of optical sectioning. 33 The strong background results from imperfect fluorescence staining during sample preparation. The resolution-enhanced images by conventional deconvolution algorithms of RL, RLTV, and Hessian are all affected by the background and therefore contain severe artifacts. On the contrary, ADMM-DRE-SIM can eliminate the background and keep the fiber structures with enhanced resolution because the image degradation model in ADMM-DRE-SIM has separated the background. And the sparse deconvolution has similar performance. As shown in Fig. 5(b), the SIM image has quite a low SNR due to the recorded nine images, with structured illumination being acquired at high speed with a short exposure time (0.5 ms), 4 which results in a great challenge for further resolution enhancement. In the same way, the noise is greatly amplified using conventional deconvolution algorithms to process the SIM image with low SNR, and the artifacts are very serious and almost confused with the tubulin fibers. In contrast, ADMM-DRE-SIM succeeds in suppressing the artifacts and acquiring a resolution-enhanced image with high-fidelity structures. To verify the superiority of ADMM-DRE-SIM in structural fidelity, the SIM image of tubulins with strong background 6 is processed by ADMM-DRE-SIM and modified ADMM-DRE-SIM; here the modified ADMM-DRE-SIM means that the background estimation is directly subtracted from the input SIM image, and the image degradation model has only the convolution of PSF of SIM. The SIM image in Fig. 6(a) has the strong background, and the background map can be estimated by the wavelet transform, as shown in Fig. 6(b). The background subtracted SIM image has obvious discontinuity, as shown in Fig. 6(c). As shown in Figs. 6(f) and 6(g), ADMM-DRE-SIM and the modified version both can achieve the resolution enhancement with sharper profiles compared with the SIM image. The direct subtraction of the estimated background map leads to the accidental removal of the useful structures, as shown in Fig. 6(e) and therefore it is very difficult to recover the missing structures during the image processing of resolution enhancement without the additional information, as shown in Fig. 6(f). However, ADMM-DRE-SIM addresses this issue by inserting the background estimation into the image degradation model instead of the direct subtraction, where the accidental removal of the structures can be eliminated during the iterative optimization  of the network for the resolution enhancement. The resolutionenhanced image in Fig. 6(g) by ADMM-DRE-SIM shows the higher fidelity with fewer missing structures compared with the image processed by modified ADMM-DRE-SIM with direct subtraction of background estimation in Fig. 6(f). By comparison, it is obvious that ADMM-DRE-SIM has the stronger robustness and can preserve the structural information while removing the background and noise.
For the superresolution imaging of small intracellular structures, the pixel size of the camera is also an important factor for the final imaging performance. The smaller pixel size can contribute to the higher resolution bound, while it also leads to the lower SNR, and therefore the pixel size of the camera used in superresolution imaging is usually a few micrometers. However, the pixel effect sometimes results in a discrete profile of the superresolution structures. To acquire a resolution-enhanced image of the finer structures with smooth profile, a 2× upsampling is embedded in the front of the neural network. In this way, a 2× enlarged superresolution image with more details can be acquired. The CCPs, which play an important role in transporting proteins between organelles, have ring shapes, and it is difficult to be recognized in wide-field microscopy due to its small size. To demonstrate the performance of ADMM-DRE-SIM with upsampling, the CCPs are first imaged by SIM, 6 and then processed by ADMM-DRE-SIM with and without 2× upsampling, respectively. As shown in Fig. 7(a), some CCPs show the ring shape in SIM, while some others have the ball shape because of the nonuniform size. By the image processing of ADMM-DRE-SIM, the background is suppressed and the ring shape of most CCPs can be visualized through the resolution enhancement. As can be seen in the enlarged images of the selected area in Figs. 7(b) and 7(c), ADMM-DRE-SIM can extract the ring shape from the ball shape of the SIM image, and ADMM-DRE-SIM with 2× upsampling can further provide a smoother and finer structure, which will be very helpful for the analysis of fine structures and functions.
As shown above, ADMM-DRE-SIM shows a powerful ability to transfer the resolution-limited SIM image to the resolution-enhanced image by bridging the low and high frequencies with the implicit priors from the network and Hessian continuity. Meanwhile, an inner image degradation model containing the equivalent PSF of SIM and background map offers the resistance to the strong background. It should be pointed out that the image degradation model is utilized in each iteration during the optimization to maintain the structural integrity because a simple subtraction of the background estimation may lead to the structural defects, especially under the low SNR condition. ADMM-DRE-SIM not only realizes the resolution enhancement of SIM but also retains the advantages of SIM involving high imaging speed, low photo-damage, and wide applications for various fluorescent labels. Therefore, ADMM-DRE-SIM provides a well-established tool to capture the high-speed dynamics of finer structures, which will have important applications in many biological areas, such as organelle dynamics 34 and cell metabolism. 35 However, ADMM-DRE-SIM has also some limitations. The imaging degradation model is based on the premise of uniform PSF across the whole image, which is an ideal situation and not very consistent with actual experiments; thus the bias may induce uneven resolution enhancement. This issue can be addressed by employing space-variation PSF. In addition, the SIM images in this work are reconstructed by fairSIM, which is a classic algorithm without the priors, and so the effect of resolution enhancement by ADMM-DRE-SIM is obvious due to the use of implicit priors. If the SIM images are reconstructed by TVSIM 36 or Hessian-SIM, 4 which have already exploited the priors, the resolution enhancement ability of ADMM-DRE-SIM will be slightly reduced.

Conclusions
In summary, we have developed an ADMM-DRE-SIM to further improve the resolution of SIM by an untrained neural network optimized with an ADMM framework. By incorporating an image degradation model containing equivalent PSF of SIM and additional background map, ADMM-DRE-SIM can significantly improve the resolution of SIM even under the condition of strong background and noise levels. A resolution enhancement by a factor of about 1.6 was experimentally demonstrated by the superresolution imaging of tubulins and actins in cells. Possessing the resolution enhancement ability while maintaining the intrinsic advantages of SIM, ADMM-DRE-SIM can provide a flexible and robust tool for studying the high-speed dynamics of biological fine structures, which will greatly promote the development of biomedical imaging. Furthermore, by inserting the imaging degradation model of three-dimensional (3D) SIM 37 into the framework of ADMM-DRE-SIM, 3D resolution enhancement can also be achieved. Additionally, ADMM-DRE-SIM is not limited to the resolution enhancement of SIM. The superresolution microscopy images acquired by some other techniques can also be further processed to obtain the higher resolution once the corresponding equivalent PSF is provided, for example, confocal spinning-disk, 38 two-photon, 39 and expansion microscopy. 40

Data and Code Availability
The data and code presented in this study are available from the corresponding author upon request.