Translator Disclaimer
8 November 2017 Super-resolution structured illumination in optically thick specimens without fluorescent tagging
Author Affiliations +
This research extends the work of Hoffman et al. to provide both sectioning and super-resolution using random patterns within thick specimens. Two methods of processing structured illumination in reflectance have been developed without the need for a priori knowledge of either the optical system or the modulation patterns. We explore the use of two deconvolution algorithms that assume either Gaussian or sparse priors. This paper will show that while both methods accomplish their intended objective, the sparse priors method provides superior resolution and contrast against all tested targets, providing anywhere from ∼1.6× to ∼2× resolution enhancement. The methods developed here can reasonably be implemented to work without a priori knowledge about the patterns or point spread function. Further, all experiments are run using an incoherent light source, unknown random modulation patterns, and without the use of fluorescent tagging. These additional modifications are challenging, but the generalization of these methods makes them prime candidates for clinical application, providing super-resolved noninvasive sectioning in vivo.



Advances in light sources, computational power, and molecular tags have resulted in a renewed interest in improvements in microscopy. In this work, we focus on four measures of microscope performance: (1) resolution of smaller objects even below the diffraction limit, (2) increased field of view (FOV) toward gigapixel images, (3) the ability to “section” a turbid medium, rejecting out-of-focus objects and thus providing three-dimensional images, and (4) high contrast without exogenous materials, which enables in vivo imaging. For example, in imaging of skin cancers, the clinician would like to resolve subcellular components to at least below the dermoepidermal junction (100-mm deep), with as large an FOV as possible. High-numerical aperture (NA) objectives generally have high magnification, thus limiting the FOV. Confocal reflectance microscopy1 uses a high-NA objective to provide resolution and sectioning without added contrast, but the high NA and consequent high magnification determine that increased FOV requires mosaics of images.

One relatively new and practical solution to provide quick and accurate sectioning is structured illumination microscopy (SIM).2 By projecting a high-frequency modulation pattern through the illumination path onto the specimen, isolation of discrete planes is demonstrated, eliminating clutter that would otherwise obscure useful information. Due to the high frequency of the modulation patterns, it has been found that high-resolution information beyond the diffraction limit, previously inaccessible, can be recovered through innovative processing. Thus, we can reduce the NA and increase FOV while still maintaining most of the resolution and sectioning ability of the confocal microscope. We demonstrate the use of random modulation patterns of incoherent [light-emitting diode (LED)] light for resolution enhancement and sectioning. The optimal solution to form an image with high lateral resolution requires a pattern to be at the Nyquist limit, which, after some reconstruction in the frequency space, allows for double the resolution.3 In practice, there is a trade-off between sectioning depth and the total resolution of the system based on the spatial frequency of the modulation pattern.4

Original developments of super-resolution using structured illumination required the precise movements of a sine pattern with a discrete phase and frequency. To illuminate the image fully, a 1/3 period phase shift must be applied to the sine pattern, requiring an extremely finely tuned mechanical device. Additionally, the pattern must be rotated to a least two other positions to provide resolution enhancement in the X- and Y-(lateral) directions. Recently, attempts have been made to reduce the necessity for a well-defined sine pattern. Removing this dependency reduces the overall cost and complexity of manufacturing and manipulating the pattern on a micron scale, as well as removing the need for perfect phase alignment within a turbid media.58 Dynamic speckle illumination microscopy uses a speckle illumination pattern to provide sectioning at significant depths but has been demonstrated using fluorescence only.9,10 Instead, we use a random pattern in reflectance, comprised of N different samples. This method has the benefit of not requiring exact phase shifts but also allows for in vivo imaging without any tagging.4,11 However, without discrete frequencies and phase shifts, the original method for super-resolution, developed by Gustafsson, cannot be applied to random patterns. Thus, we develop two methods of providing both sectioning at depth and resolution enhancements using a random pattern.

Two algorithms are demonstrated for producing sectioning and super-resolution with an unknown random pattern. The first method we will consider is the maximum a posteriori probability (MAP) processing technique.5 MAP processing produces a high-resolution, unsectioned image and a low-resolution, sectioned image, which are then combined to produce a high-resolution, sectioned image. Typically, when MAP is used in conjunction with SIM, a sinusoidal pattern with a known frequency and phase is required to produce sectioning. The literature does not extend the work to sectioning at depth and makes no attempt to demonstrate their functionality against a subsurface object. Additionally, the transfer function of the system must be reasonably characterized to estimate the modulation pattern at the detector accurately. To alleviate some of these constraints, we modify the algorithm to use random patterns that are not known ahead of time. To validate the techniques developed above, we test and consider the quality of sectioning and super-resolution.

Second, we consider a different technique of deblurring that relies on sparse priors.12 Due to the fact that the point spread function (PSF) will vary as a function of depth, deblurring a thick specimen can often be tedious and a source of errors as it requires multiple estimations of the PSF to correctly reconstruct the image. By sectioning the image using SIM first, we can account for the depth dependency of the PSF. When this sectioned image is deblurred, we show that the results for thick specimens are significantly better than deblurring alone. Sparsity makes the assumption that the object is comprised mostly of sharp edges, allowing us to deconvolve a generic PSF with excellent results. Throughout this paper, we will use the same nomenclature as Levin et al.,12 referring to this technique as one of sparse priors. To our knowledge, these two techniques described above have not been used for subsurface imaging in biological specimens and provides a very promising technique for extracting both axial and lateral information about a specimen, even within turbid medium.

The deblurring techniques leveraged in this paper fall under a class of algorithms known as deconvolution, where an estimated PSF is decoupled from a signal in the presence of noise.13 Each algorithm requires multiple iterations to converge on a solution, which are constrained to their respective regularization functions. To further simplify these methods, we implement all methods using an incoherent light source (reducing cost and increasing safety) and take all images without fluorescent tagging, making extension to most in vivo applications extremely practical. Mainstream techniques typically rely on fluorescent tagging, alleviating many signal-to-noise constraints but limiting the use in in vivo applications. The methods described throughout this paper are applied to images taken in reflectance only. It is understood that less information will be obtained using reflectance. However, given the limited number of available exogenous labels, a label-free technique is of great value. By using a 1951 Air Force Resolution Chart (AFT), we are able to demonstrate the resolution, and sectioning improvements are robust at the surface as well as within a turbid, semiopaque media.

In short, this paper demonstrates an MAP-based algorithm that produces both sectioning and super-resolution without capturing data about the patterns a priori. Further, we show that this MAP processing algorithm benefits from the use of a random pattern for sectioning, as three-phase sectioning does not section well within a turbid media. Second, we use a sparse priors algorithm against a sectioned image. We demonstrate that this provides superior sectioning against targets than deconvolution alone and does not require the adjustment of the PSF for variations in depth, as previous algorithms required.


Data Collection and Processing


Image Construction

In all cases, we make the assumption that the measurements are constructed in the following manner:

Eq. (1)

where In is a single image out of N total images (set to 80 for this paper), x is the unknown high-frequency image, which has been modulated by our random pattern Mn, convolved with the PSF of the microscope h, and added to random white noise n. Given this construction, the resolution is limited in large part due to the PSF, h, or its Fourier transform, H, known as the optical transfer function (OTF). Removal of h from the measurements In to retrieve x is called deblurring or deconvolution. This is a nontrivial task, which, with the correct assumptions, can be optimized for the problem of super-resolution in microscopy.14 Further, without knowledge of h a priori, any errors can be exacerbated when estimating the sharp image x. Selection of the proper regularization parameters will be critical to achieve the best results. Given the ill-posed nature of deconvolution (i.e., numerous possible solutions), regularization provides a constraint on our algorithm to ensure that any final result does not vary significantly from the original image. For all experiments, a regularization value was selected empirically to provide the best resolution without over smoothing the image or deviating drastically from the wide-field image. The exact values of these are described further in their respective sections. Beyond deblurring, structured illumination is used to isolate a particular plane of interest, as is done in confocal microscopy,15 known as optical sectioning. Optical sectioning in structured illumination is achieved by modulating an XY plane, at a discrete z-axis position (focal plane), with a high-frequency illumination pattern. Locations where the illumination pattern is not in focus, i.e., above and below the focal plane, blurring occurs. Typically, the modulated or AC portion of the signal is isolated from the unmodulated portion or DC, through the use of the differencing scheme2,3,5

Eq. (2)

IAC=(I0  degI120  deg)2+(I0  degI240  deg)2+(I120  degI240  deg)2,
where the demodulated image IAC is constructed from three separate frames, where the modulation pattern in each frame has been phase shifted by 120 deg each.


Experimental Setup

Our experimental setup is specifically designed to allow for manipulation of the pupil on both the transmit (before the specimen) and receive (after the specimen) side of the illumination path. Knowing that the pupil plane contains the Fourier transform of the image, a simple aperture can be used in each pupil to band limit the spatial frequency of the microscope.16 Constricting the aperture changes, the diffraction limit of the system. This adjustment allows for the raw, high-NA images to be compared to the processed low NA counterpart. The following sections will show how each processing technique is conducted and how well they compare to the ground truth.

In all experiments, 80 random, unknown patterns are used to thoroughly illuminate the entire specimen. The random patterns are binary in nature using the rand() function in MATLAB® with uniform distribution and adjusted to a 25% fill factor. Using a random pattern, the frequency of the pattern spans the entire bandwidth of the system and is limited only by NA=0.25. This fill factor was selected to ensure that the specimen was adequately illuminated and that we were able to achieve strong contrast at depth. This contrast will be demonstrated in Sec. 2.2 as we show objects can still be resolved even through a semiopaque media. Using a lower fill factor allows us to achieve deeper sectioning but requires additional images.17 The number of samples used, 80, was selected empirically. There is a trade-off between image contrast and processing time, which both increase as the number of images increases. Previous work has shown that in vivo sectioning is possible in as little as 30 frames.4 However, here, the higher number of samples is chosen to best demonstrate the processes developed in this paper.

A 635-nm incoherent LED with a linewidth of 17 nm was used as the light source. The patterns were projected onto the specimen using a TI Lightcrafter digital micromirror device (DMD). The images were then captured using an Allied Vision Guppy PRO F-125 CCD camera with a sampling rate of 22 Hz, providing all 80 samples in just over 3.5 s. All data were processed in MATLAB® R2013a, using an Intel Core i7-2675QM CPU at 2.20 GHz. The objective has a nominal magnification of 10× in air with an NA of 0.25. During the experimentation, by adjusting the apertures in the pupil planes, the NA of the system was significantly lowered to test for super-resolution. A layout of the microscope is shown in Fig. 1. Before constricting the apertures, the maximum resolution of the system was measured as 1.8  μm in agreement with Zemax simulations of the system. All images are compared to their wide-field equivalent that is constructed by taking the average of all images as described by

Eq. (3)

This sum represents the image that one would expect to see through the eyepiece of the microscope without any additional processing or modulation. Figure 1 provides a graphical representation of the microscope. For the structured illumination processing to work properly, the pattern projected by the DMD must be conjugated with the focal plane, as well as detection plane. The dashed lines in the image planes (labeled with an “I”) denote the location of the pattern in space. All other sections labeled with an “F” denote the pupil plane, which contains the Fourier transform of the image. In F1 and F4, an aperture is added to low-pass filter the frequency content of both pattern and specimen before it arrives at the camera.

Fig. 1

Layout of the microscope. Each image plane is labeled with an “I” and each pupil plane is labeled with an “F.” A beam splitter is located just before the objective lens to relay the light toward the CCD device. Placing an aperture in F1 and F4 gives the ability to artificially lower the NA of the system.



Sectioning Techniques

Many different sectioning techniques have been developed to provide sectioning from three phases.8,1821 However, as shown by Hoffman and DiMarzio,11 the susceptibility of phase misalignment in turbid media renders most images useless. Using a random pattern, rather than one of discrete phase changes, ensures that sectioning at depth produces nominally artifact-free images.4 To decouple the modulated and unmodulated portions of light, the following pair-wise comparison [Eq. (4)] is applied and results in uniformly sectioned images when using multiple, random, modulated patterns. This is just an extension of the typical differencing scheme from Eq. (2) but ensures that all 80 images are subtracted from one another

Eq. (4)


Additionally, we rely heavily on the use of a random modulation pattern, as this ensures that we are working at the limits of the OTF. The use of random patterns increases the amount of data required for sectioning but simplifies the hardware and characterization of the system required for proper functioning. An experiment is run to quantify the axial resolution of this sectioning technique. The random pattern is projected onto a mirror at a focal plane where the pattern is completely in focus. The location of the mirror relative to the focal plane is adjusted by increments of 0.5  μm. The results are plotted in Fig. 2. At 1.0  μm, there is a drop in amplitude across all frequencies of about 1 to 4 dB, enough to blur the pattern, but not likely to significantly impact the sectioning. However, around 1.5  μm difference from the focal plane, the amplitude drops by 5 to 15 dB throughout the spectrum. With this much loss in signal, there is no longer a detectable signal from the modulation pattern, thus, all light from these depths will be rejected by our sectioning algorithm. From this, we should expect a strong drop in signal outside of ±1.5  μm (above and below the focal plane) yielding a sectioning thickness of about 3.0  μm.

Fig. 2

Comparing the signal strength of the pattern at various distances from the focal plane. The same pattern is projected on a mirror at different relative distances. As the distance increases, the strength drops off significantly.


Another advantage of a random broadband pattern is that the maximum frequency present will always be limited by the OTF of the system. This ensures that the image will always be modulated at the Nyquist limit of the system. This can be visualized in Fig. 3. Here, we show experimentally the frequency spectrum of three patterns projected onto a mirror. Figure 3(a) contains a low-frequency discrete square pattern well within the limits of the OTF (first-order side bands are shown with white arrows). Because this is a square wave, we note the higher order harmonics make up the diagonal line in the figure. The black circle in the figure represents the approximate OTF of this system. Figure 3(b) is a high-frequency square pattern much closer to the edge of the OTF. This setup will provide much better resolution enhancement during reconstruction. As demonstrated by Gustafsson,3 a total of 2× resolution improvement can be obtained when the frequency of the pattern exists at the edge of the OTF. Should the OTF shrink, this may begin to alias, no longer providing sectioning or super-resolution. However, the random pattern in Fig. 3(c) conforms to the OTF continuing to provide both sectioning and super-resolution at any OTF size. This feature obviates the need to characterize the system before developing the projection patterns. Ultimately, there is a trade-off between resolution and depth of sectioning. For a high-resolution modulation pattern, we are able to get close to the full 2× improvement as demonstrated by Gustafsson. However, within a turbid media, contrast is rapidly lost, producing poor sectioning. In contrast, a low-frequency pattern provides much better sectioning at depth but limited resolution improvement.

Fig. 3

Comparison of three different modulation patterns, magnitude in dB. Aperture stop superimposed with a black circle. (a) Low-frequency square wave pattern projected on top of mirror target. First-order harmonics (arrows in white) is within aperture stop providing sectioning at depth but limited resolution improvement. (b) High-frequency square wave pattern projected on a mirror closer to the limits of the spatial resolution of the system, providing limited sectioning but increased resolution improvement. (c) Random modulation pattern on a mirror, spanning the entire allowable frequency space, providing both sectioning and resolution improvement at the cost of additional data samples.



Super-Resolution Techniques


Maximum a posteriori probability

The first technique that we explore is known as MAP processing. The MAP processing algorithm is outlined in Fig. 4. Previous implementations of MAP processing required the pattern to be known a priori; however, our research will show that we can still perform sectioning and super-resolution, even without prior knowledge of the patterns. Starting with 80 samples modulated with a random pattern [In(x,y)], a wide-field image, IWF, is produced using Eq. (3), and a sectioned image, Isec(x,y), is produced using Eq. (4). The wide-field image is then subtracted from each sample to produce an estimate of the random modulation pattern as follows, Mn*(x,y)=In(x,y)IWF(x,y). This operation is completed in the image domain and ideally removes the specimen component of the image (IWF) from the pattern + image, returning on an estimation of the pattern only. The results from this process can be seen in Fig. 5. The MAP processing then attempts to minimize the mean square error between the estimated high-resolution images and the observed images; In(x,y) and Mn*(x,y), finding the most probable high-resolution image required to produce the sampled array of low-resolution images. As described by Lukeš et al.,5 a gradient decent algorithm is applied to iteratively minimize the following equation:

Eq. (5)

Here, we are solving for the high-resolution image IHR from the constructed image in Eq. (1), where λΓ(x) is a regularization function, composed of first-order derivatives at each pixel. To minimize noise in the image, MAP processing applies Gaussian priors, slightly smoothing out the image. From this, a high-resolution, unsectioned image is produced. This high-resolution is first high-pass filtered and then combined with a low-passed version of the sectioned image. By combining the two in the frequency domain, a high-resolution, sectioned image is produced.21,22 This process is completed by extracting the low- and high-frequency components of their respective images and adding the two together as described by

Eq. (6)

where Isec is spectrally filtered by a Gaussian function of standard deviation ρ at frequency f, weighted by β and combined with an inversely filtered version of IHR. The two filtered images are combined in the frequency domain, which after taking the inverse Fourier transform (F1) produces the final high-resolution, sectioned image IHR-SIM.

Fig. 4

Flow chart outlining the MAP processing.


Fig. 5

For a single frame captured, which is comprised of the specimen modulated by the random pattern, above demonstrates the isolation of the pattern from the specimen (here, the target is an air force resolution chart). This is achieved by subtracting (a) the wide-field image producing an estimation of the pattern, as shown in (b). White bar represents a distance of 50  μm.


Typical implementations of MAP expect that the modulation pattern is known a priori.8 In the case of imaging in turbid media, the exact position of the pattern will not be consistent from specimen to specimen. Thus, to remove this constraint, we estimate the modulation after the data have been collected. This removes the necessity of characterizing the system ahead of time, whose positions will change within turbid media, as required in previous implementations of MAP. The MAP processing algorithm that is implemented throughout the rest of this paper is based on a modified version of the SIMToolbox, developed by the Multimedia Technology Group, Czech Technical University, Prague.22 The algorithm has been modified using the pattern estimation technique described above. On average, the MAP processing took 5 iterations to converge on a solution in a total time of 35 s. We note that all experiments going forward will use our modified version of the MAP processing algorithm, which allows for the use of blind modulation patterns. All experiments use no a priori knowledge of the pattern and are integrated into the MAP algorithm. The above equations describe the modifications applied in this paper, but we refer readers to the original paper for additional details.


Sparse priors

Next, we apply a deconvolution technique that relies on sparse priors. In sparse reconstruction, images are assumed to be mostly binary in nature and their details of interest are expected to have sharp edges. These assumptions allow for high-quality reconstruction against small point scatters and thin lines, such as fibers. Objects such as these, which are beyond the working diffraction limit, typically reconstruct nicely. The deconvolution processing is based on the algorithm developed by Levin et al.12 and implemented as described. To estimate the super-resolution image, we minimize the following:

Eq. (7)

where ρ is a heavy tailed function, where ρ(z)=|z|0.8 while iteratively solving for x, which represents an intermediary, sharp, image until convergence is achieved.12 In previous implementations, this algorithm relied on varying the PSF to account for depth. Our research shows that, with sectioning, this depth dependency can be removed, requiring only a single PSF estimation for deconvolving the image. Additionally, within turbid media, the deconvolution process is far less effective without sectioning first.

The algorithm is outlined in Fig. 6. After the 80 modulated images are collected, the sectioned image is produced. The sectioned image is then deconvolved from an estimated PSF. The PSF is iteratively refined until the algorithm converges. An assumption of sparse priors is applied in this technique. There are two reasons why sectioning first is necessary in this algorithm: first, the PSF will vary as a function of depth. By sectioning first (isolating a single depth plane), the PSF should be approximately uniform across the entire image, greatly simplifying the deconvolution procedure. Second, the use of sparse priors will constrain the edges of single point objects. In our case, the modulation pattern itself contains many point objects. By sectioning first, the pattern is nominally removed, leaving only the specimen to be processed. Similar to the MAP processing, this algorithm is run iteratively until convergence, continuously updating the PSF. For our application, we run this process about 200 times in 28 s, including sectioning, which provides a reasonable trade-off between overall convergence and minimal processing time. Adjustment of the total number of iterations or increasing the size of the PSF will have a dramatic effect on how long it takes to complete the processing. The process has been shown to work well in other microscopy techniques in various biological specimens.2325

Fig. 6

Flow chart outlining the sparse priors processing.


For this experiment, a PSF is modeled as an Airy disk26 and then the Fourier transform is taken to produce the OTF estimation

Eq. (8)

where fc is the cutoff frequency of the OTF. For this paper, all PSF values are selected empirically to get the best results. However, this technique can be improved by creating a smarter selection criterion for the PSF, producing higher quality images and faster convergence time. For these experiments, the amount of blur for any given pixel is modeled by a symmetrical 2-D function that smears any single pixel across 12 pixels in the x- and y-directions, resulting from Eq. (8). Again, the reader is referred to the original paper by Levin for details about the algorithm that go beyond our modifications.



A 1951 USAF resolution test chart (AFT) is used to evaluate and compare the two algorithms. For the experiments going forward, we will define the resolution as the distance between two black lines on the chart, regardless of the thickness of the line. Figure 7 compares the full-resolution image at full NA, 0.25 (a), to low-resolution image when the aperture in the detection path is constricted (b). At full resolution, the elements can be resolved, with the smallest lines 2.2  μm in width (group 7, element 6). The aperture is then closed making the smallest resolvable line 6.2  μm in width (group 6, element 3), corresponding to an NA of 0.07. It is found that both the MAP (c) and sparse processing (d) provide a resolution improvement. The MAP processing reveals elements with a linewidth of 4.4  μm (group 6, element 6), increasing the resolution by 1.4×. The sparse prior processing reveals elements with a linewidth of 3.9  μm (group, 7 element 1), increasing the resolution by about 1.6×. With respect to total resolution improvement, it can be seen that the sparse processing produced an image that more accurately reflected the pattern seen in the ground truth image (a). However, it is found that both techniques introduce a few artifacts in very low-frequency regions. These artifacts can be seen in the background, as well as the black box between the 7 and 6. The cause of these artifacts is likely nonuniform illumination from the random patterns that are used. These artifacts can also be seen in the wide-field image (b), where there is still some structure present in the image. Further, we see that relative to the MAP processing, the sparse deconvolution overregularizes the location of the three lines, reconstructing the lines as slightly thinner than those seen in the ground truth image. Cross sections from two elements of the AFT have been plotted in Fig. 8(a); group 6 element 5 and Fig. 8(b); group 7, element 1. Here, it is clear how well the three elements are resolved, as well as the contrast improvement provided by each of the techniques.

Fig. 7

Comparison of the AFT before and after processing, using 80 random patterns. Here, we give an “effective NA” where we calculate the approximate NA based on the smallest resolvable object. (a) The full-resolution target as a ground truth, with the aperture in the pupil fully dilated (NA=0.25, resolution 2  μm: group 7, element 6), (b) the resolution target after the aperture has been closed (NA=0.07, resolution 5.5  μm: group 6 element 3), (c) after MAP processing (effective NA0.1, resolution 3.9  μm: group 6, element 6), and (d) after sparse processing (effective NA0.13, resolution 3.1  μm: group 7, element 1). Black scale bar in bottom right corner of each image represents a distance of 50  μm.


Fig. 8

(a) Cross section of the AFT from group 6, element 5, measuring a resolution of 4.92  μm. Both processing techniques match the ground truth but are otherwise lost in the wide-field image. (b) Cross section of the AFT from group 7, element 1, measuring a resolution of 3.91  μm. Only the deconvolution with sparse priors matches the ground truth. Each cross section is from a single line of pixels from the image.


Next, microbeads, made of polystyrene, 2  mm in diameter are placed on top of a mirror, as shown in Fig. 9. The resolution of the system is reduced (NA0.13) and the wide-field image is shown in Fig. 9(a). The region boxed in red is shown in the image below [Fig. 9(d)]. The MAP processed image is shown in (b) and (e), whereas the sparse priors processed image is shown in (c) and (f). In the low-resolution image, individual beads cannot be discerned when they are adjacent to one another. The MAP processing creates contrast and sharpness around the edges. However, because the MAP processing uses a Gaussian prior as its regularization function, it has a tendency to smooth over edges making many beads appear as one large blob. Although it sharpens large areas of contiguous beads, it is incapable of segregating adjacent beads from one another. In the sparse priors technique, the processing does an excellent job of differentiating each bead. Arrows have been superimposed onto the figure, to show two regions where the resolution has undergone marked improvement. In Fig. 10, a cross section of the beads shows an almost a 2× resolution improvement from the original image to that of the sparse priors processing. We measure the full width half max (FWHM) of the low-resolution image to be 4  μm as expected. After the sparse processing, there are two well-defined beads with a FWHM of 2  μm, demonstrating the accuracy of this procedure. As for the MAP processing, it is apparent that the deconvolution process is merging the individual beads in the image making it difficult to isolate one from one another.

Fig. 9

Images taken of 2-μm microbeads on a mirror (beads are in black), showing the differences between each of the processing techniques. (a) The wide-field image where the individual beads cannot be seen. The red box zooms in on the region shown in the image below (d). (b) and (e) The MAP processing and zoomed-in region, respectively. (c) and (f) The sparse processing and the zoomed-in region, respectively.


Fig. 10

Cross section of two adjacent beads, taken from Fig. 4.3. Amplitude has been inverted for clarity. Using the sparse processing, the two beads are now distinguishable from one another. The MAP processing is incapable of isolating the two beads from one another.


Given the desire to apply these techniques to in vivo applications, a biological specimen is investigated to ensure that both the super-resolution and sectioning aspects are present in the reconstructed image. Figure 11 shows a layer of onion cells placed 30  μm above the AFT. The space in between the two targets is filled with ultrasound gel (n=1.33). Based on previous experiments on our sectioning technique, we have shown that sectioning resolution of this random pattern technique is 1.0  μm. Therefore, we expected to remove scattered light from the AFT at depth, as well as scattered light from the onion cells a few microns below the surface. In the first image (a), dark areas are seen below the onion cells due to absorption of the light from AFT beneath the cells. Additionally, contrast among each of the cells is reduced from scattered light throughout the sample. The sectioning process is verified by noting that the dark regions have been removed in both (c) and (d), as well as clarifying the interface between cells. In both of the processed images, distinct lines between the cells of each pair are seen, giving strong contrast in the image. Comparing the MAP processing in (c) to the sparse processing in (d), many of the point scattering objects, as well as the regions between the cells are even further defined. To ensure that the sectioning and super-resolution are complimentary, the deconvolution is applied directly to the wide-field image (b). Without applying the sectioning first, the scattered light from the onion cells and AFT below are not rejected. As such, the super-resolution technique actually enhances the contrast of the AFT below. It is worth noting the contrast at the edges of the onion cells. In the wide-field example, the regions among cells are almost indistinguishable. Both processing techniques increase the contrast greatly. In reviewing the sparse priors technique (d), the edges are further narrowed, producing high-contrast, high-frequency edges between each of the cells. In the deconvolution only image, some of the lines are enhanced slightly, however, they are not nearly as effective without the sectioning.

Fig. 11

Images taken of onion cells: (a) plots the wide-field image of the cells, the AFT can be seen beneath cells. (b) The wide-field image after sparse deconvolution (no sectioning), (c) contains the onion cells after using the MAP processing, and (d) contains the onion cells after the sparse processing.


To further verify the technique, the focus is adjusted to a depth below the onion cells, as demonstrated in Fig. 12. The lines are distorted as a result of an index of refraction change from the air objective to the onion cells. Use of an index matching material would likely mitigate this distortion, greatly improving the results. It can be seen that the sparse priors technique gives additional, albeit, modest resolution improvement, which is consistent with previous results. Specifically, three lines in group 8 element 6 (lower-right side) are now resolvable, which are not visible in the other three images. There are some air bubbles between the onion cells and the resolution chart, which is present in the ultrasound gel. Due to the index of refraction change, in some areas, the modulation pattern does not focus on the correct plane. As a result, there are some dark lines and spots in the image. However, on the whole, the sectioning has removed much of the scattered light from the cells above the resolution chart, providing better contrast.

Fig. 12

Images taken of the resolution chart 30  μm beneath the onions: (a) contains the wide-field image of the AFT beneath the onion cells, (b) the wide-field image after sparse deconvolution (no sectioning), (c) contains the AFT after using the MAP processing, and (d) contains the AFT after the sparse processing.


Last, we compare these methods to two other common deconvolution techniques without any sectioning. We apply a Lucy–Richardson and maximum likelihood deconvolution algorithm. Each method was implemented in MATLAB® 2013a by MathWorks, using a template PSF, similar to the one applied in the sparse deconvolution method. These results are plotted in Fig. 13. This plot looks at cross sections from the resolution bars of Fig. 12, group 6, element 6 (bottom-right section of the imaged resolution chart). The plot on the left side of Fig. 13 plots a cross section of the vertical bars. Both the sparse method and MAP processing methods greatly increase the contrast of these bars. The other two methods produce results similar to no processing at all. The right graph plots the cross section of the horizontal bars. The sparse method succeeds in producing contrast between the lines while the MAP processing improvement is modest at best. However, the other two deconvolution methods, without sectioning, again do not appear to produce quality results. This experiment agrees with the original hypothesis that applying the sectioning first allows for superior deconvolution and better overall results.

Fig. 13

Comparison of MAP and sparse methods against Lucy–Richardson and maximum likelihood method. These plots compare the resolution bars of group 6, element 6 (4.38  μm). In (a) cross section of the vertical bars is plotted. In (b) cross section of the horizontal bars is plotted.




Random patterns of incoherent light can be used in SIM to provide resolution enhancement and sectioning. These experiments demonstrate that even with limited a priori knowledge of the pattern and the optical system we can, with the right processing, achieve both optical sectioning and spatial resolution beyond the diffraction limit. Because the algorithm is robust with respect to variations in the pattern, the severe alignment constraints of many structured illumination approaches are eliminated. Given that traditional three-phase SIM and super-resolution SIM do not work at depth, we are able to show that random patterns are capable of providing both. Recognizing that only a few fluorescent probes are approved for in vivo imaging, we have chosen to test our system and algorithms with reflectance imaging, which requires no exogenous materials. In this work we achieved a transverse resolution enhancement of a factor of 1.4 using the MAP algorithm with Gaussian priors and 1.6 with sparse priors.

Considering the particular application of imaging skin, to obtain results comparable to conventional laboratory examination of hemotoxylin-and-eosin-stained biopsy specimens, we need resolution of about 1 mm to match the laboratory microscope resolution and sectioning of about 5 mm to mimic the typical microtome sections. Using this approach to SIM, we can reduce the NA and magnification of the objective to improve the FOV while recovering much of the resolution and sectioning ability.

The sparse priors approach is particularly suited to isolated point objects, such as the subcellular organelles, and provides resolution enhancement of a factor of 1.6, even without knowledge of the random patterns or the PSF of the microscope. For less sparse objects, the Gaussian priors algorithm provides resolution enhancement of a factor of 1.4. We have demonstrated a particularly useful technique for producing high-quality sectioned images at resolutions and FOVs greater than conventional microscopy, with minimal knowledge about the system a priori. This advance should prove to be extremely useful in the context of low-cost, low-maintenance microscopy in in vivo applications.


The authors declare no conflicts of interest exist.


The authors would like to thank Guoan Zheng, Kaikai Guo, and Joseph Hollmann for their input on the paper, as well as Kalina Yang for helping with additional data collection. This work was done in part under the National Science Foundation Grant No. CBET-1510281.



M. Rajadhyaksha et al., “In vivo confocal scanning laser microscopy of human skin: melanin provides strong contrast,” J. Invest. Dermatol., 104 (6), 946 –952 (1995). Google Scholar


M. A. A. Neil, R. Juškaitis and T. Wilson, “Method of obtaining optical sectioning by using structured light in a conventional microscope,” Opt. Lett., 22 (24), 1905 –1907 (1997). OPLEDP 0146-9592 Google Scholar


M. G. L. Gustafsson, “Surpassing the lateral resolution limit by a factor of two using structured illumination microscopy,” J. Microsc., 198 (2), 82 –87 (2000). JMICAR 0022-2720 Google Scholar


Z. R. Hoffman and C. A. DiMarzio, “Structured illumination microscopy using random intensity incoherent reflectance,” J. Biomed. Opt., 18 (6), 061216 (2013). JBOPFO 1083-3668 Google Scholar


T. Lukeš et al., “Three-dimensional super-resolution structured illumination microscopy with maximum a posteriori probability image estimation,” Opt. Express, 22 (1624), 29805 –29817 (2014). OPEXFF 1094-4087 Google Scholar


E. Mudry et al., “Structured illumination microscopy using unknown speckle patterns,” Nat. Photonics, 6 (5), 312 –315 (2012). NPAHBY 1749-4885 Google Scholar


S. Dong et al., “High-resolution fluorescence imaging via pattern-illuminated Fourier ptychography,” Opt. Express, 22 (17), 20856 –20870 (2014). OPEXFF 1094-4087 Google Scholar


P. Křížek, I. Raška and G. M. Hagen, “Flexible structured illumination microscope with a programmable illumination array,” Opt. Express, 20 (22), 24585 –24599 (2012). OPEXFF 1094-4087 Google Scholar


C. Ventalon and J. Mertz, “Quasi-confocal fluorescence sectioning with dynamic speckle illumination,” Opt. Lett., 30 (24), 3350 –3352 (2005). OPLEDP 0146-9592 Google Scholar


C. Ventalon and J. Mertz, “Dynamic speckle illumination microscopy with translated versus randomized speckle patterns,” Opt. Express, 14 (16), 7198 –7209 (2006). OPEXFF 1094-4087 Google Scholar


Z. R. Hoffman and C. A. DiMarzio, “Single-image structured illumination using Hilbert transform demodulation,” J. Biomed. Opt., 22 (5), 056011 (2017). JBOPFO 1083-3668 Google Scholar


A. Levin et al., “Image and depth from a conventional camera with a coded aperture,” ACM Trans. Graphics, 26 (3), 70 (2007). Google Scholar


A. Levin et al., “Understanding and evaluating blind deconvolution algorithms,” in IEEE Conf. on Computer Vision and Pattern Recognition (CVPR’09), 1964 –1971 (2009). Google Scholar


R. C. Gonzalez, R. E. Woods and S. L. Eddins, Digital Image Processing Using MATLAB, Pearson Education, India (2004). Google Scholar


M. Minsky, “Microscopy apparatus,” United States Patent 3,013,467 (1961).


C. A. DiMarzio, Optics for Engineers, CRC Press, Boca Raton, Florida (2011). Google Scholar


O. Mandula et al., “Line scan-structured illumination microscopy super-resolution imaging in thick fluorescent samples,” Opt. Express, 20 (22), 24167 –24174 (2012). OPEXFF 1094-4087 Google Scholar


R. Heintzmann and P. A. Benedetti, “High-resolution image reconstruction in fluorescence microscopy with patterned excitation,” Appl. Opt., 45 (20), 5037 –5045 (2006). APOPAI 0003-6935 Google Scholar


T. Lukeš et al., “Comparison of image reconstruction methods for structured illumination microscopy,” Proc. SPIE, 9129 91293J (2014). PSISDG 0277-786X Google Scholar


K. O’Holleran and M. Shaw, “Optimized approaches for optical sectioning and resolution enhancement in 2D structured illumination microscopy,” Biomed. Opt. Express, 5 (8), 2580 –2590 (2014). BOEICL 2156-7085 Google Scholar


D. Karadaglić and T. Wilson, “Image formation in structured illumination wide-field fluorescence microscopy,” Micron, 39 (7), 808 –818 (2008). MICNB2 0047-7206 Google Scholar


P. Křížek et al., “Simtoolbox: a MATLAB toolbox for structured illumination fluorescence microscopy,” Bioinformatics, 32 (2), 318 –320 (2015). BOINFP 1367-4803 Google Scholar


S. D. Babacan et al., “Cell imaging beyond the diffraction limit using sparse deconvolution spatial light interference microscopy,” Biomed. Opt. Express, 2 (7), 1815 –1827 (2011). BOEICL 2156-7085 Google Scholar


M. Mir et al., “Visualizing Escherichia coli sub-cellular structure using sparse deconvolution spatial light interference tomography,” PLoS One, 7 (6), e39816 (2012). POLNCL 1932-6203 Google Scholar


T. Kim et al., “White-light diffraction tomography of unlabelled live cells,” Nat. Photonics, 8 (3), 256 –263 (2014). NPAHBY 1749-4885 Google Scholar


J. W. Goodman, Fourier Optics, 3 Roberts and Company Publishers, Greenwood Village, Colorado (1968). Google Scholar


Zachary R. Hoffman is a PhD candidate at Northeastern University where he works in Professor Charles DiMarzio’s lab. His research focuses on developing and improving methods of structured illumination microscopy for resolving subsurface information in-vivo. He is also working as an engineer at Draper Laboratory, developing lightbased inertial sensors.

Charles A. Di Marzio holds degrees from the University of Maine, WPI, and Northeastern University. After 14 years at Raytheon Company in coherent laser radar, he joined Northeastern University. He is an associate professor of electrical and computer engineering,mechanical and industrial engineering, and bioengineering in Northeastern University. He is the author of the textbook Optics for Engineers. His interests include confocal microscopy, structured illumination, and the interaction of light and sound.

© 2017 Society of Photo-Optical Instrumentation Engineers (SPIE) 1083-3668/2017/$25.00 © 2017 SPIE
Zachary R. Hoffman and Charles A. DiMarzio "Super-resolution structured illumination in optically thick specimens without fluorescent tagging," Journal of Biomedical Optics 22(11), 116003 (8 November 2017).
Received: 24 July 2017; Accepted: 23 October 2017; Published: 8 November 2017


CHORUS Article. This article was made freely available starting 08 November 2018

Back to Top