Considerable progress has been made in the area of optical microscopes and their applications during the past two decades.18.104.22.168.6.7.–8 Optical tools with numerous variations and techniques have become a major part of research and development in the biorelated fields. Usage of through-focus (TF) optical imaging is steadily gaining momentum, particularly in biological applications.910.11.12.13.14.15.–16 TF imaging is sometimes informally and interchangeably referred to using terms such as volumetric, out-of-focus, blurred, defocused, extended-focused, extended-depth-of-field, axial scanning, and three-dimensional (3-D) imaging.
Several uses of the TF optical image applications such as biological imaging, microscopy, adaptive optics, material processing, optical data storage, and optical inspection have been reported.17 In this paragraph, we highlight some specific applications and methods that use TF images. TF scanning optical microscopy (TSOM) makes use of a set of defocus optical images for 3-D shape metrology of target sizes ranging from sub-10 nm to over 100 μm, including nanoparticles,18,19 with subnanometer resolution.1822.214.171.124.126.96.36.199.188.8.131.52.184.108.40.206.35.36.–37 The ability to analyze optical illumination was also reported20 using the TSOM method. Much interest is given to two-photon (or multiphoton) microscopy that makes use of TF imaging to acquire 3-D volumetric data of biological samples, including brain tissue and bone calcium.11,14,15 High-speed TF imaging is used to track single-molecules in 3-D and observe their behavior during cell division,38 and it has also been used to image entire embryos.39 By simultaneously imaging different focal planes within the sample, it was possible to track the 3-D dynamics in live cells at high temporal and spatial resolution.12,13 3-D position, alignment, and orientation of submicroscopic features were made possible by TF polarization imaging in label-free as well as fluorescently labeled specimens.40 Cellular network dynamics such as spatiotemporal activity patterns in neuronal and astrocytic networks was demonstrated using TF imaging in 3-D.14 3-D automated nanoparticle tracking was demonstrated using TF images.41
Significance of TF Image Collection
Three important developments have increased the prevalence of TF imaging:
I. TF data conventionally include out-of-focus or blurred images. These blurred images were once considered to be of either inferior quality or not useful and were therefore mostly discarded, as in confocal microscopy.42 However, with the development of technology, improved optical simulations, and new insights, it was found that the out-of-focus TF images do contain useful information regarding the target being imaged. New applications and uses are being found for blurred optical images that were previously considered unusable.20,29,32,34,43220.127.116.11.–48 The major challenge was to extract high quality, usable information from the blurred TF optical images. TSOM1818.104.22.168.22.214.171.124.126.96.36.199.188.8.131.52.35.36.–37 is an example of a method where useful 3-D shape information can be extracted from a set of TF, blurred, optical images. TSOM enables 3-D shape metrology with subnanometer measurement resolution using a conventional optical microscope.24,25 TSOM has demonstrated the capability of extracting 3-D shape information of targets ranging in size from sub-10 nm to , with measurement resolution comparable with scanning electron microscopy or atomic force microscopy.
II. The second development is the advent of new applications that require collection of TF or volumetric data. Examples include high-speed 3-D tracking of nanoparticles,38,41 observation of high-speed cell division38 and cellular network dynamics in 3-D,14 and 3-D volumetric data of biological samples such as brain tissue and bone calcium.11,14,15
III. The third development is that advances in technology, instrumentation, simulations, and computation have enabled collection of TF images with sufficiently high quality and at sufficiently high speed to allow TF information to be readily utilized for many applications. For example, the development of objective lens scanning using piezoscanners has made it possible to readily adopt any conventional optical microscope into a high-speed TF image collection tool with acquisition times as small as 200 ms.41 Optical tool developments have also enabled the simultaneous collection of TF images, with no scanning parts.3839.–40
It is now firmly established that TF optical images are beneficial for many applications. A survey of prior work reveals that many optical methods for TF image collection have been developed. It would be impossible to describe or even mention all of them in this review. However, most of the methods that have been published are variations of a few unique or basic methods. We here review the TF methods that represent the foundation of most published techniques.
Through-Focus Image Collection Methods
In this section, we describe most of the basic TF methods that have been published. These methods are depicted using simplified schematics and brief summaries. Note that in the following figures, the portions of the diagrams enclosed using dashed boxes indicate components or aspects of the technique that are unique to that particular method. Note also that some figures (e.g., Fig. 1) show multiple interrelated techniques.
Scanning the Sample Stage Along the Focus Axis
Sample stage scanning, shown schematically in Fig. 1(a), is the classic and the most widely used method to obtain TF images. It is simple and straightforward and nearly every optical microscope has some provision to adjust the sample along the focus axis: these methods range from manual focusing on basic instruments to more sophisticated motorized or piezoscans on high-end modern tools. The scanning range can be up to several tens of millimeters, and positional accuracy can be better than 10 nm. However, TF scan speeds are typically regarded as relatively slow. Depending on the focus scan range, the exposure time, and the number of focal steps required, the total scan time could extend up to several minutes. The focus accuracy of this TF technique can be among the best, with high reproducibility and low distortion. However, optical mechanical instability32 and illumination aberrations49,50 have also been reported for such systems.
TF optical images collected by the stage-scanning method can be stacked at their respective focus positions to create a 3-D space filled with the optical intensities. Plotting the optical intensities in a vertical cross-section through this 3-D space results in a TSOM image [Fig. 1(e)]. Even though TSOM uses conventional optical images (i.e., not a resolution enhancement method), it provides subnanometer 3-D shape measurement resolution24 as it uses additional information present in the out-of-focus optical images. TSOM images are sensitive to changes in the (i) 3-D shape of a target,24,29 (ii) position of a target in a 3-D space,41 (iii) optical properties of a target, and (iv) illumination.20 A differential TSOM image (pixel-by-pixel difference between two well-aligned TSOM images) highlights all these differences with much higher signal-to-noise ratio compared to a conventional best-focus, top-down optical image.18,19 TSOM is a strong candidate to analyze the position, shape, and optical properties of soft nanoparticles (e.g., hydrogel nanoparticles) in their native liquid environment, in addition to hard nanoparticles.41
Scanning the Objective Lens Along the Focus Axis
Another method of collecting TF images is by scanning the objective lens along the focal axis [Fig. 1(b)], generally using piezomotors.14,41,51 A relatively high-speed TF image collection time of 200 ms has been reported41 using this method, which makes it suitable, e.g., for nanoparticle tracking in 3-D space. It is relatively easy to convert a conventional microscope into a TF image collecting tool by replacing its objective lens base with one of the several commercially available objective-scanning piezomotors. In this high-speed scanning mode, the image quality could be degraded. However, this approach has the advantage of not disturbing the specimen because the sample stage remains stationary during imaging. Mechanically, this method is more stable than the stage-scanning method.
Scanning the Image Plane Along the Focus Axis
Through focus image collection can also be accomplished by scanning the image plane or camera,12,32 as shown in Fig. 1(c). The same effect can also be achieved by scanning a replica of the image plane.52 This method is reported to have the advantage of avoiding the spherical aberration common to other optical refocusing systems; it also allows for fast TF scans, extending the working distance, and keeping the specimen on stage stationary and undisturbed.52
Multifocal Plane Microscopy
Instead of scanning the image plane, one alternative is to split the imaging beam into several fixed beams that are refocused simultaneously onto multiple cameras positioned at different focal distances,12,13,32 as shown in Fig. 1(d). This is also called image plane sharing microscopy.32 It has the main advantage of avoiding scanning altogether (of stage, objective or imaging plane). In this configuration, the TF images are collected simultaneously that enables high-speed TF image collection. However, since the imaging beam is split into several beams, the available optical intensity at each camera is reduced, which may lower the signal-to-noise ratio of the optical data, unless the collection time is increased. The number of focal planes available is also limited to the number of times the imaging beam is split. Since there are no moving (scanning) parts, this is one of the most mechanically stable configurations.
Wavelength Scanning Method
For microscope objectives with chromatic aberration, differing wavelengths will have different focus positions. Normally microscope objectives are designed to minimize chromatic aberration. However, chromatic aberration can also be used to obtain TF images.32,34,53 In this method, both the chromatic aberrated objective and the sample stage are kept at fixed positions. Instead, the wavelength of illumination is varied (or wavelength-scanned) so each wavelength focuses at a different focus position on the sample, enabling TF image collection (Fig. 2). A variation of this approach, which utilizes the chromatic dependence of diffraction, has been developed using tricolored LED illumination and a spatial light modulator (SLM).53 Since the position of sample stage is not scanned, the wavelength scanning approach generally has superior mechanical stability compared to conventional stage-scanning optical microscopes. It also allows high-throughput TF image collection due to the ability to quickly and precisely tune the wavelength34 (faster than scanning a stage).
Flexible-Membrane Liquid Lens
While there are many variations in design and implementation, the general concept of the approach shown in Fig. 3(a) is that the fluid pressure inside two flexible membranes is varied to make the membranes inflate or deflate. This process alters the radius of curvature of the membranes and thus changes the effective focal length of the lens, allowing TF image collection.54184.108.40.206.59.60.61.–62 The membrane curvature can alternatively be varied by different mechanisms such as changing the aperture diameter thus squeezing the membranes.63 These lenses are generally fast, but are prone to have optical aberrations.56
This method, shown in Fig. 3(b), is also known as a variable focus liquid lens. These lenses utilize refraction at a liquid–liquid boundary interface. Focal adjustment is achieved by using a variable voltage to tune the curvature at the boundary interface of two immiscible liquids. A spherical surface is formed at the boundary of a polar and an apolar liquids. The curvature of the interface can be controlled by adjusting the relative wettability through electro-wetting.15 This creates a lens with variable focal length to enable TF image collection.15,55,63,64 Liquid lenses are relatively fast and very durable and exhibit a high degree of phase shift, i.e., focal length change. These lenses have a long functional life and low sensitivity to mechanical stress, and they are also advantageous for autofocus applications due to their relatively low power consumption. Liquid-tunable lenses have relatively low cost and are commercially available. However, the optical performance of liquid-tunable lenses suffers from significant spherical as well as higher-order, gravity-induced aberrations.56
As shown in Fig. 4, adaptive optical elements such as a tip/tilt or deformable mirror can be inserted into the optical path of a microscope to enable TF image collection.32,65,66 Common examples of adoptive optics elements are deformable mirror devices or liquid crystal SLMs. The adoptive optics is used to alter the phase of incident wave fronts to displace the focal spot. Tip/tilt applied to the adoptive optics shifts the focus laterally within the focal plane, while defocus translates along the optic axis.65 In the implementation with a tip/tilt mirror, a basic closed-loop quad cell is used to control the mirror. A high-order deformable mirror with a Shack–Hartmann sensor is used in the deformable mirror implementation. This approach can also correct high-order residual aberrations as well as performing the TF scanning without -axis movement. A key advantage of these methods is high-speed TF image collection (). Adaptive optics technology and hardware are readily available and are most commonly used for biological applications.16
Multifocus microscopy (MFM) allows rapid and simultaneous acquisition of TF images using a single exposure (Fig. 5). Unlike a conventional microscope, there are no scanning parts.3839.–40 This is achieved by placing a multifocus grating (MFG) at the Fourier plane followed by a chromatic correction grating (CCG) and prism [Fig. 5(a)]. The combination of these elements divides the primary image into several TF images and simultaneously projects them onto the plane of the CCD camera. Unlike a conventional microscope, the CCD camera in MFM is divided into several squares, each one of which is used to collect one of the focal plane images [Fig. 5(b)]. To date, MFM has been used primarily for biology applications, and this technology is not yet widely available.
Aperture-Scanning Fourier Ptychography
In this method, an aperture is placed at the Fourier plane as shown in Fig. 5(c), and it is scanned in a raster manner while simultaneously collecting intensity images of the object. The acquired images are then synthesized in the frequency domain to recover the complex hologram of extended objects. This can then be digitally propagated into different planes along the optical axis to extract TF images of the object.67 An alternative method is to scan the camera instead of an aperture.67 Because of the requirement to scan certain parts, this approach could be slower.
In confocal microscopy, out-of-focus image information is selectively discarded by placing a pin hole in front of the camera (Fig. 6). This allows collection of image-slices that contain primarily in-focus image information. By scanning the sample along the focal axis, multiple such image-slices through the sample are collected, and this enables a 3-D reconstruction of the sample surface. Numerous alternative implementations and variations on this basic method of data acquisition have been published. This is one of the most widely used TF imaging methods, particularly in biology.
Light Sheet Microscopy
In light sheet microscopy (LSM), the sample (usually transparent or semitransparent) is illuminated from the side using a thin sheet of light. Typically, the light source is rigidly coupled to the objective and aligned with its focal plane (Fig. 7).6869.–70 By scanning the sample stage along the focal axis, volumetric images of the specimen can be obtained. In a strict sense, this method is not a TF method because the imaging plane is always in focus. However, it can extract 3-D volumetric image information similarly to a confocal microscope. Unlike a confocal microscope, however, the out-of-focus image information is not discarded, it is simply not illuminated. Numerous variations of this basic technique have been published.69,71 LSM is mainly used to study biological tissues. Because of the opacity of such tissues, 3-D imaging is typically limited to a depth of 500 to .72,73 However, in some studies, LSM appears to outperform confocal microscopy.10
Light Field Microscopy
In this microscopy, a microlens array is inserted into the optical train of a conventional microscope between the main lens and sensor plane, and this enables the capture of light-fields of specimens in a single image (Fig. 8).74,75 TF images can then be extracted through application of 3-D deconvolution to this single image. LFM also has the capability to extract different perspective views from the single image. In an LFM, spatial resolution is determined by the number of microlenses. A disadvantage of LFM is the trade-off between spatial resolution and angular resolution.74
Phase Retrieval Techniques
Phase retrieval techniques make use of the relationship between phase and propagation direction in an optical microscope. This method initially requires a few TF images that are inverted to recover phase and amplitude quantitatively. By making use of the retrieved phase and the amplitude, the entire set of TF images can then be calculated.7677.–78 A variation of this method, which is sometimes called quantitative phase imaging, has found many useful applications in biology.79
Digital Holography Microscopy
Holographic microscopy (including digital holography) is a coherent imaging system, and its advantage lies in the instantaneous and quantitative acquisition of both amplitude and the phase information from the reconstruction of the wavefront.8081.82.83.84.85.–86 Digital holography microscopy can numerically extract TF images from a single experimentally recorded hologram without the need to move the sample (Fig. 9). The digital holographic approach has been successfully implemented to speed up TSOM image acquisition.31 Since no mechanical scanning is involved, and a complete set of TF images can be extracted from a single holographic image, it is a high-speed TF imaging method.
Acousto-optic lens scanning17 can achieve a pure focal scan at a very high speed (). This is accomplished using two adjacent counterpropagating acoustic waves with a synchronized frequency chirp so the transverse scans subtract to cancel each other, whereas the focal scans add. Acousto-optic modulators and acousto-optic scanners are used along with a laser beam, and there is a 180-deg phase shift between the two frequency-modulating signals. The light is focused on a CCD camera that is mounted on a translation stage and used to measure the focal distance. Temporal and spatial focusing87 achieves high TF speeds with no moving parts. Full-field optical coherence microscopy produces volumetric imaging with all in-focus images similar to confocal microscopy.88,89 It is based on the spatial coherence gate principle and generates in parallel complete two-dimensional TF type images without scanning.
Many variations of the preceding general TF methods have also been published. Some are a combination of two (or more) TF methods with some other methods. For example, a combination of high speed sinusoidal vibration of the microscope objective along with “smart” movements of galvanometric scanners (to repeatedly scan the laser focus along a closed 3-D trajectory) enables high-speed acquisition of TF images for two-photon microscopy that permits fast fluorescence measurements.14
Optical images such as the TSOM images [Fig. 1(e)] can be constructed using any one of the TF method presented here. However, for the TF methods such as confocal and light-sheet microscopy, all the images are in-focus creating a different type of TSOM image.
TF Images Categorization
Optical Data Type
TF optical images can be broadly divided into three types depending on the type of optical data
I. In the first type, the set of TF images usually contains a best focus image, along with many out-of-focus (or blurred) images on the either sides of the best focus image. A typical example for this type of TF data collection is the conventional stage-scanning method.37
II. In the second type, nearly all the TF images are either in-focus or contain only in-focus image information enabling one to visualize or reconstruct the 3-D volume of the sample. Confocal microscopy42,90 is a typical example for this kind of TF data collection method. Some TF methods are able to extract both the first and the second types of TF optical data such as the digital holography82 method.
III. The third type, which is often referred to as either super-resolution microscopy4 or nanoscopy,1 is a highly localized imaging method with measurement resolution down to tens of nanometers. In this stochastic approach, only a small subset of molecules is switched on in a given 3-D volume at any particular moment in time using fluorescence principle while the majority remains in a nonfluorescent “dark” or “off” state.91 Super-resolved images are reconstructed from the positions of thousands to millions of single molecules that have been recorded in thousands of camera frames.92 Several thorough review articles have been published on this subject,12.3.4.–5 and hence no further discussion of this type of TF imaging is included in this review.
Depending on the presence or absence of scanning, TF image collection methods can be divided into two groups.
I. In the first group, the TF image collection involves scanning (or continuous variation) of some parts of the optical microscope. Typical examples include objective scanning and light-sheet microscopy. Even though the phase retrieval technique does not require scanning, it still requires at least two images at different focal planes.
II. In the second group of methods, no scanning is involved to acquire TF images. MFM and digital holography microscopy are typical examples of this group.
Depending on the image extraction method, two groups of TF approaches can be formed:
I. Some TF methods enable direct image acquisition, such as stage scanning, liquid-tunable lens, etc.
II. Some TF methods require indirect extraction (or computational extraction) of TF images. Phase retrieval and digital holographic microscopy are typical examples for this group.
A complete categorization of all the TF methods is presented in Table 1.
Complete categorization of the TF scanning methods.
|Section no.||TF method||TF image data includes||Scanning||Scanning method||Image extraction||Best speedsa|
|3.1||Stage scanning||In-focus and out-of-focus||Yes||Stage||Direct||A few hundred ms|
|3.2||Objective scanning||In-focus and out-of-focus||Yes||Objective||Direct||A few hundred ms|
|3.3||Image plane scanning||In-focus and out-of-focus||Yes||Camera||Direct||A few hundred ms|
|3.4||Multifocal plane microscopy||In-focus and out-of-focus||No||NA||Direct||High-speed/instantaneousb|
|3.5||Wavelength scanning||In-focus and out-of-focus||Yes||Wavelength||Direct||A few hundred ms|
|3.6||Flexible-membrane liquid lens||In-focus and out-of-focus||Yes||Focus||Direct||A few ms|
|3.7||Liquid-tunable lens||In-focus and out-of-focus||Yes||Focus||Direct||A few ms|
|3.8||Adaptive optics||In-focus and out-of-focus||Yes||Focus||Direct||A few ms|
|3.9||MFM||In-focus and out-of-focus||No||NA||Direct||High-speed/instantaneousb|
|3.10||Aperture-scanning Fourier ptychography||In-focus and out-of-focus||Yesc||NA||Computational||Tens of secondsd|
|3.11||Confocal microscopy||In-focus only||Yes||Stage||Direct||A few hundred ms|
|3.12||LSM||In-focus only||Yes||Stage||Direct||A few hundred ms|
|3.13||LFM||In-focus and out-of-focus||No||NA||Computational||High-speed/instantaneousb,d|
|3.14||Phase retrieval techniques||In-focus and out-of-focus||Yes/noe||Anye||Computational||High-speed/instantaneousb,d|
|3.15||Digital holography microscopy||In-focus and out-of-focus||No||NA||Computational||High-speed/instantaneousb,d|
NOTE: NA, not applicable.
Approximate values to collect a set of TF images.
Either only one image or several images simultaneously are collected.
Scanning of an aperture.
Plus computational time.
Either scanning or nonscanning methods can be used to obtain a few images at different focal positions.
Speed and Quality of TF Image Collection
One of the main challenges in modern optical microscopy is fast and sensitive acquisition of TF data.11,38 In some applications, such as in single molecule tracking methods,38,41 it is essential to have a high-speed TF image collection. In contrast, for certain reference metrology applications, high-speed collection may not be necessary, but high precision is of paramount necessity.24,25 In the following sections, we discuss relative speeds and quality of TF image collection methods.
The speed of TF image collection and image quality will usually trend in opposite directions. In this review, the image quality mostly refers to signal-to-noise ratio. Typically, higher collection speeds reduce image quality and vice versa. In the scanning-based image collection methods, TF image collection time can vary considerably depending on the microscope conditions such as exposure time, the number of TF images needed, illumination source intensity, sample type, and method of scanning. All such factors must be optimized to achieve a suitable, low-noise image in the acceptable condition. Increasing the exposure time and the number of TF images will usually increase collection time, while increasing illumination source intensity decreases it due to lesser required exposure time. A sample that returns a large scattered intensity is beneficial in reducing the exposure time and hence TF image collection time. TF collection times of the scanning methods span from a relatively slow (in the order of minutes) to a relatively high-speed (in the order of less than a millisecond).
Scanning can either be performed continuously or using repeated, short, step-scans for the scanning-based TF image collection methods. In continuous-scanning approaches, the scanning part and the camera are synchronized that enables relatively high-speed TF image collections. For example, in the objective scanning method (Sec. 3.2), it is possible to continuously scan the objective back-and-forth and simultaneously record images at a relatively high-speed (200 ms41). The focus step size depends on the time gap between each exposure and the scanning speed of the objective. Even though the adaptive optics method (Sec. 3.8) is a type of continuous-scanning method, it achieves a very high-speed collection (32). Acousto-optic lens (),17 liquid lenses (),11,93 and adaptive optics ()66 can also achieve a high-speed TF image collection.
Step-scan methods are relatively slow as scanning must be stopped briefly for image collection at every step, and this increases the overall TF image collection time. For example, in the stage-scanning method (Sec. 3.1), each time the stage is axially translated at a given step size, it is stopped for image collection, repeating the process until all the TF images are collected.
Usually, nonscanning methods are relatively fast. For example, in MFM, TF images are collected simultaneously in one exposure (or in one image). This results in TF image collection as fast as any single image collection using a conventional microscope. Multifocal plane microscopy also can collect TF images simultaneously at a high-speed. However, the number of images is restricted to the number of times imaging beam is split. High-speed TF imaging can also be achieved using DHM. A summary of the best speeds possible to collect a set of TF images for different TF methods is presented in Table 1.
Not all TF imaging methods produce the same quality images. In fact, a wide range of image qualities can be expected depending on the TF method and the experimental conditions. It is the opinion of the author that the best quality images can be collected relatively easily using step-scan type of TF methods (e.g., stage-scanning method), but this is at the expense of collection speed. The presence of optical tool mechanical instability and other mechanical vibrations can deteriorate the image quality. In reality, scanning-based methods are usually prone to image deterioration of this kind.34 Consequently, if careful attention is not given to these factors, the continuous-scanning methods will usually result in lower quality images. Liquid-tunable lenses and flexible membrane liquid lenses suffer from significant aberrations56 which reduce image quality. Image quality in MFM can suffer because of splitting of the primary beam into several images.
As presented above, every TF method has certain advantages and disadvantages. The selection of a TF method depends on several factors, including type of data needed, image quality, applicability, speed of acquisition, cost, mechanical stability, accuracy, noise, simplicity, ease of use, and availability of the technology. Several aspects of optical microscopes have been identified to minimize degradation of optical images (e.g., laser stability, flat-field correction, camera performance, optical aberrations, noise, spectral reproducibility, lateral resolution, lens cleanliness, lens characteristics, temporal variability of signal and noise, absolute intensity calibration, and correcting field-dependent aberrations90,94220.127.116.11.99.100.101.–102). However, to the best of our knowledge, there is no published method to test the fidelity of a set of TF images. Two aspects are uniquely associated with TF type of data collection: (i) focus step size and (ii) sample/stage vibration (or lateral movement) during mechanical scanning. Either imprecise focus step size (including in the extracted TF images) or lateral displacement of sample is highly likely to deteriorate TF optical data. A combination of (i) and (ii) could also be presented. For example, scanning methods could have mechanical instabilities and vibration issues34 resulting in increased overall noise. If there is a mismatch between the sample stage scan axis with the optical axis along the focus direction, TF optical images appear to shift laterally32 creating a large error. Similar lateral image shift can also be observed if the aperture diaphragm is not correctly aligned with the optical axis.20 For these reasons, it is important to evaluate fidelity of the TF data. We are actively working to develop a method to test TF data fidelity.
We have presented a review of over 15 distinct TF imaging methods from the literature. There is a wide choice to select from depending on the need. As with any optical images, the set of TF images collected using any method or under any conditions needs to be tested for fidelity. We hope to present such a test in the near future.