9 May 2014 Simultaneous imaging of cellular morphology and multiple biomarkers using an acousto-optic tunable filter–based bright field microscope
Author Affiliations +
An acousto-optic tunable filter (AOTF)-based multispectral imaging microscope system allows the combination of cellular morphology and multiple biomarker stainings on a single microscope slide. We describe advances in AOTF technology that have greatly improved spectral purity, field uniformity, and image quality. A multispectral imaging bright field microscope using these advances demonstrates pathology results that have great potential for clinical use.


Introduction and Motivation

High-throughput screening of multiply stained clinical pathology samples is currently a difficult and laborious task requiring expert review. In examining a patient slide, pathologists rely primarily on examination of cellular morphology and, in the case of histology, tissue architecture. While this approach has been the backbone of pathology for many decades, the discovery of numerous molecular biomarkers of disease has lent increasing importance to the role of biomarkers in detecting abnormalities and determining proper treatment.1 However, individual biomarkers generally do not have sufficient sensitivity and specificity to identify disease, and there has been growing recognition that the use of multiple biomarkers is highly desirable.

The possibility of further combining these multiple biomarkers with traditional morphology is of particular interest to pathologists. Often, adjunct tests for biomarkers are done by methods in which individual cells in the sample are not visualized. Alternatively, imaging biomarker tests are done, but without the ability to simultaneously correlate this information with morphology on a cell-by-cell basis. Optimally, however, the pathologist would like to see the multiple biomarkers of particular interest and the morphology of interest simultaneously throughout the slide. Biomarkers could be used both to locate potential abnormal regions on a slide, as well as to clarify interpretation of these regions. Although in some situations, it may be possible to rely solely on the presence of certain biomarker combinations to identify disease, it is likely that in many cases morphology will remain the gold standard for accurate interpretation, with biomarkers providing critical additional information in regions having ambiguous morphology.

However, multiply stained slides can be difficult to interpret by eye. Multispectral imaging systems are designed to provide an image in which every pixel contains detailed spectral information. If the pixels are imagined as arranged in a matrix in the xy-plane, then the spectral information can be displayed along the z-axis giving rise to a three-dimensional data structure termed an “image cube.” By analyzing this image cube, individual stains on a slide can be cleanly separated. We have developed a multispectral imaging system for traditionally stained clinical pathology samples to which additional stains labeling multiple specific biomarkers have been added.


Multispectral Microscopy for Pathology


Staining Samples for Clinical Use

Cellular morphology and tissue architecture are conventionally identified using transmission stains; common examples include hematoxylin, hematoxylin and eosin, Pap stain, Masson’s trichrome, and so on. In contrast, biomarkers can be labeled with either fluorescence probes or transmission stains. When using multiple biomarkers, fluorescence labeling has the advantage that fluorescence labels can be chosen to have minimal spectral overlap, enabling the different labels to be clearly distinguished by eye or with a properly chosen set of interference filters. Fluorescence labeling can be done with either conventional organic fluorphores or quantum dots, and has the added advantage that fluorescence intensity increases linearly with biomarker expression, allowing quantitation.

If one wishes to view both biomarkers and morphology on a single slide, however, the use of fluorescence biomarker labeling is problematic. First, hematoxylin (by itself or in combination with other stains) is one of the most common stains used for morphology. In its conventional formulation, however, hematoxylin contains an aluminum mordant which quenches fluorescence (both from organic fluorophore and from quantum dots). Hence, it is difficult to obtain adequate signal from fluorescently labeled biomarkers in the presence of many types of morphological stainings. Second, commonly used transmission stains, such as hematoxylin and eosin, absorb and fluoresce strongly throughout the visible wavelength region. As the amount of these transmission stains varies throughout the sample, this results in a spatially varying decrease in the biomarker fluorophore excitation and emission efficiency (due to the stain absorbance) as well as a spatially varying background fluorescence (due to the stain’s own fluorescence). While the use of near-infrared (NIR) fluorophores can potentially reduce this second problem somewhat, it is difficult to find multiple high-efficiency NIR fluorophores, and in any case the quenching problem still remains.

Biomarker multiplexing with morphology staining, therefore, is more effectively accomplished using transmission biomarker staining. There are many transmission stains available for labeling biomarkers that are stable and interact minimally with other stains. Transmission stains typically have broad absorbance profiles, however, and when combined with morphology staining, the results can no longer be easily interpreted by eye. In such situations, multispectral imaging and analysis are needed to separate the morphology staining components from those of the various biomarkers. This technique has been widely used, even when the stains in question are highly overlapping both spectrally and spatially.


Multispectral Imaging of Multiply Labeled Clinical Samples

Multispectral imaging results in a spectral image cube, consisting of a set of images each at a different wavelength. The final acquired data arise from the combined effect of various independent, often co-localized, spectrally varying components present in the sample imaged. While various approaches to spectral imaging analysis exist, the most straightforward one is linear spectral unmixing (LSU), which assumes that the acquired spectra at each pixel of the imaged region may be expressed as a linear combination of the contribution from each of these independent elements, each having its own distinct “reference spectrum.” The technique of LSU allows the final image to be decomposed into separate “rule images,” one for each of the reference spectra, with the pixel-by-pixel intensity of each rule image corresponding to the calculated amount of that spectral component present at said pixel in the image.

This procedure requires unambiguous determination of the reference spectra of the individual stains and a minimization of nonlinearities between them. Reference spectra can be most easily obtained by preparing separate test samples using each of the individual stains to be used. Spectra derived from these samples will work effectively in the combined multistained sample, provided care is taken to use moderate staining intensities to reduce absorbance-dependent spectral nonlinearities in the reference spectra, to choose stains that have minimal reactivity with each other, and to avoid stains whose primary interaction with light is via scattering rather than absorbance.10

In principle, the number of wavelengths required to be imaged should equal the number of independent reference spectra in the sample. It turns out, however, that many more than this is often needed for the highest-quality results. The actual number required depends on the number of stains used, the degree to which they spectrally overlap, and the complexity of the sample being imaged.

Figure 1 illustrates this important point. On the far left, (a) is a color image of Pap-stained cervical cells co-stained for the biomarker p16 with an additional stain, Deep Space Black. The Pap stain is actually a combination of four separate stains—hematoxylin, eosin, fast green, and OG6—each of which stains different parts of different types of cells, for a total of five stains in this sample. The unmixed hematoxylin rule image (the main nuclear stain, of primary importance for morphologic evaluation) for various number of unmixing wavelengths, N, is shown in the four images to the right (b), with a graph of relative image quality shown in (c), and the spectra used for unmixing in (d). The average of the local contrast factor11 was used to quantify the relative image quality.

Fig. 1

Illustration of the effect of number of wavelengths used for unmixing on rule image quality. (a) Image as it appears under the microscope; (b) unmixed hematoxylin rule images for different numbers of unmixing wavelengths; (c) relative image quality for the rule images obtained by quantifying local image contrast; and (d) spectra used for the unmixing.


For each N, the wavelengths used for the unmixing were chosen to provide the best result possible for this choice of wavelength number; these choices are shown in Table 1. From the images in Fig. 1(b) and the graph shown in Fig. 1(c), it may be seen that analysis results using six wavelengths are far from adequate, and approximately 30 wavelengths are required for best results.

Table 1

Wavelength sets used for the unmixing of Fig. 1.

NWavelength Choices
6490, 525, 565, 610, 630, 750
15490, 510–570, by 10, 590, 615, 635, 650, 665, 700, 750
30480–520 by 10, 520–575 by 5, 575–615 by 10, 615–655 by 5, 700, 750
60450–750 by 5

A clinical sample often requires many hundreds of fields of view in order to cover an entire slide under a microscope, even at low magnification. For a multistained sample such as that shown in Fig. 1, this requires thousands of wavelength changes per slide. In high-throughput clinical applications, it is essential that the slide scan time be reduced to a minimum. For multispectral high-throughput scanning, we therefore require that wavelength switching be accomplished fast enough so as not to slow down the native frame rate of the camera. Since, in this situation, there is no intrinsic relative movement between sample and detector, and only specific wavelengths need to be acquired, band sequential spectral imaging technologies are preferred over pushbroom imagers. For a modest number of wavelengths (10, e.g.,), this task can be accomplished using a properly designed interference filter wheel, once the optimum wavelengths have been decided upon. When the number of wavelengths required becomes much greater than this, this approach becomes less practical. Liquid crystal tunable filters (LCTFs) have the required flexibility, but their 100-ms switching times make them incompatible with the speed requirement. AOTFs have approximately 100-μs switching time and spectral flexibility, which together with their superior imaging quality, make them an ideal choice for this application.


Acousto-Optic Tunable Filter–Based Bright Field Microscopy


Acousto-Optic Tunable Filter Design

An AOTF is an optical component that uses the strong interaction of light and sound waves within a suitable material to selectively filter out a single color from a white-light image. The sound is generated by a megaHertz frequency electronic signal driving a piezoelectric transducer bonded to an acousto-optic (AO) interaction crystal medium. The sound waves periodically modulate the crystal optical index to form a moving volume diffraction grating. By varying the acoustic wave frequency, a specific wavelength of light is selectively diffracted and angularly separated from the remainder of the input light. The AOTF is, therefore, a completely solid-state device in which the driving electronic signal can be varied rapidly to create a programmable sequence of color filters, enabling very-high-speed multispectral imaging.12

The material most frequently used for visible range AOTFs is crystalline tellurium dioxide, TeO2, a material that is transparent from approximately 380 nm to 4.4 μm. Gooch and Housego grow crystals from raw powder TeO2 in boules of up to 90 mm in diameter and 80 mm in length, weighing approximately 3 kg. Such large boules can be used to produce two 25-mm-aperture AOTF devices.

The boule is orientated using x-rays and then cut to the orientation required. Figure 2(a) shows the next step in producing the AOTF, wherein an acoustic transducer of lithium niobate is bonded to the Cr/Au groundplane conductor previously applied to the device and the top Cr/Au electrode is applied. The highly anisotropic acoustic properties of TeO2 are very useful for high-efficiency diffraction of light, but also result in the acoustic beam produced by the transducer “walking off” at approximately 45 deg from the normal to the transducer, requiring a significantly longer crystal to obtain the desired interaction length. For a conventional AOTF, it is desirable to use a traveling acoustic wave, but this wave must be absorbed once it has crossed the optical aperture of the device. In order to produce a traveling acoustic wave in the aperture and to avoid any appreciable acoustic reflections, an acoustic absorber is constructed on the bottom surface, opposite the transducer (not shown). Figure 2(b) shows the fully assembled AOTF device. The broadband matching circuit board can be seen to the left; in this device, the acoustic beam travels from left to right.

Fig. 2

(a) 25-mm-aperture crystal for AOTF hyperspectral imaging system with bonded transducer and (b) fully assembled AOTF module.


An AOTF is a device operating in the Bragg regime. The conditions for this are discussed in books on AOs,13,14 but broadly speaking this means that the transducer is many acoustic wavelengths long and the sound wave acts as a simple grating. Most AOTFs use a single uniform transducer for which the interaction region may be modeled using a rectangular or “top hat” function. For low diffraction efficiency, we can directly Fourier transform (FT) this and take the modulus squared of the result to give a wavelength response in the form of a “sinc squared” function. This assumption agrees fairly well with what is found in practice; the spectral sidelobe height is approximately 13.5dB at low diffraction efficiency, and rises to approximately 9.5dB at high diffraction efficiency. Figure 3 shows the experimental results using a 15-mm-aperture AOTF designed for use in the visible with a “textbook” rectangular uniform transducer.

Fig. 3

Representative transmission spectrum of AOTF with a rectangular transducer.



Optimized Acousto-Optic Tunable Filter Electrodes


Solid Gaussian electrode

Rectangular uniform transducers produce large spectral sidelobes, as shown in Fig. 3. Lithographically patterning the gold electrode connection to the transducer is a simple method to realize more sophisticated shapes and improve performance by reducing sidelobes, a technique known as apodization. Since the FT of a Gaussian curve is itself a Gaussian, a reasonable approach for designing a transducer to produce a spectrum with no sidelobes would be a Gaussian-type pattern. One approach to this is shown in Fig. 4(a), in which the white area represents the metal of the top electrode and the arrow shows the direction of optical propagation. The Gaussian is truncated at 10% at either end and is reflected through the horizontal axis; the total length in this and the other electrodes discussed below is 22 mm. The total out-of-band light from this transducer is shown in Fig. 4(b), and is not appreciably better than that of the rectangular transducer shown in Fig. 3.

Fig. 4

(a) Top-electrode pattern of a solid Gaussian apodized transducer showing direction of acoustic propagation and symmetry axis and (b) corresponding AOTF transmission spectrum.


In designing a structured electrode, its effect on the spatial uniformity of the spectrally filtered image also needs to be considered. Unlike a rectangular electrode, light traversing a crystal with a solid Gaussian electrode near its edges [the top and bottom of Fig. 4(a)] will have a much shorter AO interaction length than light traversing near the center. This difference will have two effects. First, the spectral passband of the AOTF will tend to increase for the off-axis intermediate image points corresponding to off-axis object points, leading to a position-dependent bandpass function. In particular, the spectrum of a beam of light traversing the region of the crystal corresponding to the acoustic pattern arising from the top of a solid Gaussian electrode is broadened and red shifted by several nanometers relative to the spectrum of a beam traversing through the region of the crystal corresponding to that arising from the center of the electrode (data not shown). Such a variation is highly nonoptimum for many applications. Second, diffraction efficiencies will fall away from the center symmetry axis, making the image dimmer as one moves off-axis.

This variation of diffraction efficiency across the field is of great practical importance. When this effect is coupled with those due to inhomegeneities of the acoustic near-field, significant modulations of image brightness appear in the final image plane, easily measurable when viewing an otherwise uniform object. For small variations, this effect can be corrected by flat fielding, if the resulting patterns do not vary rapidly with acoustic wavelength (and therefore selected color band), but significant modulation and beam intensity reduction near the image edges clearly need to be avoided.

Figure 5 illustrates the image nonuniformity that results from a solid Gaussian top electrode. A uniformly illuminated white card was imaged through a solid Gaussian-type AOTF, as shown in Fig. 5(a), with the intensity profile along the direction of acoustic propagation shown in Fig. 5(b) [vertical direction in Fig. 5(a)]. It can be seen that there is a sharp fall-off at the edges, together with a large “spike” at the center, arising from the horizontal centerline symmetry of the solid Gaussian pattern.

Fig. 5

Image nonuniformity in a solid Gaussian electrode AOTF. (a) Image of a uniformly illuminated white card and (b) intensity profile along the vertical acoustic propagation direction.



Side-dithered pseudorandom electrode

To improve performance, we have developed a patented15 scheme in which a pseudorandom contiguous pattern of metal pixels is defined using photolithography. Our method of apodization offers a way to simultaneously apodize the device and to keep, on average, translational invariance. Figure 6(a) shows an example of a pseudorandom apodized transducer top electrode; the white regions represent the electrode material left after the photolithographic process. The detail inset illustrates that the gold top electrode is one contiguous piece of metal; if the metal pixels in each column were summed, the result would be a Gaussian histogram, truncated at 10%, although other windowing functions are of course possible. This electrode has a 10-μm-feature (pixel) size, and has 1100 rows and 2200 columns, making the border a rectangle of 22×11mm.

Fig. 6

(a) Pseudorandom apodized top electrode (white) with detail shown in inset and (b) top electrode of a “sideways-dithered” pseudorandom transducer.


To further improve performance, we divided the top-electrode pattern into a series of rectangles and “shuffled” them by random amounts in the lateral direction, while preserving electrical connectivity. The idea is a simple one: it is not possible to make an acoustic transducer with any apodization scheme that generates a textbook homogeneous plane wave near to the transducer surface at high Fresnel numbers, at least not by any means accessible to the current state of the art of the AO industry. However, this is not actually necessary—all that is needed is a transducer that produces a homogeneous integrated AO effect,16 i.e., one that produces an effect that “averages out” along any given ray passing through the AO device. Figure 6(b) shows an example of such a “sideways-dithered” top electrode using approximately 10% randomization in the transverse direction.

Experimental responses of AOTF transmission/wavelength for this electrode are shown in Fig. 7. Out-of-band light is greatly reduced using this approach relative to the solid Gaussian, with sidelobes dropping to approximately 25 dB. On- and off-axis spectra are virtually identical.

Fig. 7

Representative transmission spectrum of AOTF using a side-dithered pseudorandom apodized transducer.


Both the dithered and nondithered versions of the pseudorandom apodized electrode improve image nonuniformity relative to the solid Gaussian, as shown in Figs. 8 and 9. Figures 8(a) and 9(a) show white card images taken through the nondithered and dithered devices, with intensity profiles of these images taken in the vertical direction shown in Figs. 8(b) and 9(b). Both the overall image uniformity and peak modulation height are significantly reduced in the dithered device. These residual variations are easily and robustly corrected using standard flat-fielding techniques.

Fig. 8

Image nonuniformity in a nondithered pseudorandom apodized electrode. (a) Image of a uniformly illuminated white card and (b) intensity profile along the vertical acoustic propagation direction.


Fig. 9

Image nonuniformity using a sideways-dithered apodized electrode. (a) Image of a uniformly illuminated white card and (b) intensity profile along the vertical acoustic propagation direction.



Optimized Optical Relay

An AOTF’s response depends on the wavelength and direction of the light and the wavelength and direction of the acoustic waves. The combination of all these effects can lead to a significant degradation in the final resolvable image. In particular, the finite size of the acoustic transducer coupling with the range of rays impinging on the AOTF from a complex sample can result in an image surrounded by a weak “halo” or blur. This effect can be significant in an afocal imaging configuration in which rays from any point on the object plane are collimated as they traverse the AOTF.

A solution that minimizes this problem, as pointed out by Suhre et al.,17 is to modify the imaging system so that an intermediate image of the object is formed inside the AOTF itself. Thus, a point on the object will be relayed to a point-like image in the AOTF, and rays all diverging from this point-like image can now be brought to a common well-defined focus. The small extra amount of angular deviation present in the rays corresponding to the extremes of the spectral bandpass will also diverge away from this same point, which can now be cleanly imaged onto the charge-coupled device (CCD).

Figure 10 shows this arrangement in a schematic form. Rays from two object points, A and B, are shown. As the front aperture stop is placed at a distance f1 in front of the lens L2, the system is telecentric in the object space; as one moves on the object plane from points A to B, the chief ray of the converging pencil of rays entering the AOTF remains horizontal and translates upward. An intermediate image is formed inside the AOTF. The zero-order light is not shown, but the back aperture stop is placed so as to optimally reject it. The “dog-leg” in the optical path at the AOTF is necessary because the diffracted beam exits the crystal at an angle to the incident beam. In the assembled instrument, a C-mount connector is included at the entrance port for mounting to a microscope and an Allied Vision Technologies (Burnaby, British Columbia, Canada) GC1380H CCD camera (Sony ICS285 chip with 1360×1024pixels and a 30-fps frame rate) is placed at the image plane.

Fig. 10

Schematic of the confocal telecentric optical system used in our hyper-spectral imager.



Imaging Performance


Image quality

The resulting imager has very high image quality. Figure 11 shows the point spread functions (PSFs) obtained at the green and NIR ends of the tuning range. The PSF is obtained by viewing a 1-μm pinhole using a 10× objective. The imager design shown above has an overall demagnification of 2 for optimizing light throughput; a 1-μm pinhole will therefore produce an ideal image (1μm×10)/2=5μm in diameter, slightly less than the 6.45-μm pixel size of the camera used.

Fig. 11

PSF at 550 nm (a) and 800 nm (b). Airy disk diameters are 7 μm at 550 nm and 10 μm at 800 nm.


The modulation transfer function (MTF) specifies the contrast reduction of a periodic sine pattern after passing through the lens system as a function of its periodicity and orientation. Formally, the optical transfer function is the absolute value of the FT of the PSF of the imaging system. Our MTF test is based on imaging a slanted edge and processing the image with appropriate software using a standard MTF test target.

A representative set of MTFs for the instrument is shown in Fig. 12. The vertical visibility scale is from 0% to 100%, the horizontal scale is in cycles/mm. The wavelength of the passband is set, in turn, to 460, 650, and 780 nm so as to span the AOTF tuning range. It can be seen that midband visibility of >50% is maintained at up to approximately 46cycles/mm.

Fig. 12

Measured MTFs for the hyperspectral imager at three different wavelengths.



Scene shift

If a collimated beam of white light is incident on the AOTF at normal incidence to the input face, a diffracted beam of filtered light would be produced with an internal angular separation and would exit from the back face in a direction determined by refraction at that surface. Since the index of light (and, therefore, the refraction angle) depends on wavelength, this would result in the undesirable effect of causing an angular variation versus wavelength of the filtered output, with the resulting image stack not properly aligned. In order to compensate for this chromatic dispersion effect, it is conventional to angle the back face by the average separation angle. Cutting the back surface of the crystal by just a few degrees relative to the front face provides excellent compensation over a 1-octave optical tuning range. To verify this effect for the current system, we imaged a grid pattern across the tuning range of the device in order to quantify maximum color shift and found a maximum color shift of less than two pixels, close to the resolvable PSF limit. As a result, successively acquired individual wavelength images have minimal registration artifact and can be effectively combined into an image cube for subsequent pixel-by-pixel analysis.


Optical throughput

Another important parameter is the optical throughput of the instrument. The theoretical maximum is 50% because the AOTF filters only one polarization of light. Figure 13 displays throughput results for unpolarized light for the AOTF alone in black, indicating transmission above 35% (70% of the theoretical maximum) throughout much of the tuning range. The dotted curve indicates the transmission of the entire imaging system, including the additional optics shown in Fig. 10.

Fig. 13

Optical throughput for the AOTF alone (black) and the full instrument (dotted).



Performance Summary

These device improvements result in an AOTF spectral imaging instrument with submillisecond wavelength switching throughout the visible range, out-of-band rejection approximately 25 dB down from peak with no sidelobes, and superior intensity uniformity across the image. The device displays excellent image quality, resulting in >50% visibility at up to 46cycles/mm, with minimal scene shift, and unpolarized optical throughput approximately 30% through most of the tuning range.


Application of Acousto-Optic Tunable Filter–Based Multispectral Imaging to Cervical Cancer Screening


Cervical Cancer Screening

Screening for cervical cancer is conventionally done by taking a sample of cervical cells and staining them with Pap stain to look for morphologically abnormal cells. Despite its long history, the Pap test is far from perfect, as interobserver variability in interpreting the cervical cell morphology can result in significant missed or ambiguous diagnoses.18,19 Much effort has been invested in investigating biomarkers that can reliably determine the risk of cervical cancer to help reduce subjectivity of interpretation and possibly also mitigate the need for highly trained personnel to review the large volume of slides.20,21 To date, the most widely accepted of these is the human papillomavirus (HPV), of which there are 13 high-risk strains whose presence has been associated with the development of cervical cancer. HPV infection is frequently present even without disease, and hence other markers have also been heavily studied, among them p16, Ki67, MCM2, and TOP2A.

While these markers have shown promise for improving cervical cancer detection, it is clear that biomarker testing alone is an insufficient substitute for detailed morphologic assessment. Nevertheless, the current expectation is that with the intense pressure to expand coverage, reduce cost, and improve the efficiency of cervical cancer screening, prescreening with stand-alone nonimaging biomarker assays will soon gain acceptance. Since these assays will likely have high sensitivity but relatively low specificity, the need for a highly specific “reflex test” will become highly important. We believe that the combination of additional complementary biomarkers with conventional Pap-stain morphologic assessment will provide the most compelling approach to this problem. Including these biomarkers can help locate potentially abnormal cells by finding those cells that express one or more of these biomarkers, and then help with the interpretation of ambiguous morphology by providing simultaneous imaging of the biomarker expression and the morphology of the cells in question.


Demonstration with Pap and Simultaneous Biomarker Imaging

To demonstrate this approach, we have developed an assay to stain cervical cell samples with Pap stain and biomarker immunolabeling for p16 (JC8, Santa Cruz Biotechnology, Dallas, TX), using Deep Space Black stain (Biocare Medical, Concord, CA). Spectral imaging was performed using our multispectral AOTF instrument mounted on a Nikon 90i upright microscope, using conventional tungsten lamp illumination, and a Nikon 40× Plan Apo dry objective with NA=0.95 (Nikon Instruments, Melville, NY); a photo of the entire system is shown in Fig. 14.

Fig. 14

Photo of the complete spectral imaging system used for the cervical cancer screening imaging.


Images were acquired from 460 to 730 nm in 10 nm steps with an average bandwidth of 2.5 nm. Reference spectra for each of the four components of the Pap stain as well as Deep Space Black were determined by staining samples with each of the stains individually; these spectra are shown in Fig. 1(d). Using these spectra, LSU was performed on the acquired image set, producing rule images for each of the five stains. The four Pap component rule images were individually recolorized to match the actual color of each stain component (although this is not necessary, it is usually the most useful for a pathologist), and then recombined to produce the Pap image of the sample with the biomarker removed for morphologic examination, as shown in Fig. 15(a). The biomarker rule image can then be separately false colored and used as a removable overlay, as shown in Fig. 15(b). In this figure, the abnormal cell is characterized by its large dark nuclei and large nuclear-cytoplasmic ratio; the biomarker overlay indicates positivity for p16.

Fig. 15

Pap-stained cervical cells with p16 biomarker labeling: (a) spectrally unmixed and recolorized Pap-only image and (b) including false-colored p16 overlay.


Figure 16 shows Pap stained cervical cells together with biomarker labeling using a cocktail of 13 high-risk HPV DNA strains, labeled with in situ hybridization (ISH) probes and permanent red stain (all from Dako, Carpenteria, CA). Figure 16(a) also shows the AOTF-acquired, LSU-analyzed recolorized Pap-only image, similar to the sample as it appears under the microscope. Figure 16(b) includes the false-color HPV overlay (red) at the location of the abnormal cell. Note that this sample was prepared in a different laboratory, hence the difference in Pap staining color. Although we chose to recolorize each to match its actual staining color, we could equally have chosen to recolorize differently.

Fig. 16

Pap-stained cervical cells with HPV ISH labeling: (a) spectrally unmixed and recolorized Pap-only image and (b) including false-colored HPV overlay.


Figure 17 shows an example of dual biomarker staining, in which Pap staining is used together with both HPV ISH labeling (using permanent red stain) and p16 immunolabeling (using Vector Blue stain, Vector Labs, Burlingame, CA) on a section of cervical tissue from a patient with cervical cancer. Figure 17(a) shows the fully stained tissue section as it appears under the microscope, Fig. 17(b) shows the recolorized Pap-only image, Fig. 17(c) shows the HPV overlay in red, and Fig. 17(d) shows the p16 overlay in green. This figure demonstrates that this approach can work with both cytology and histology samples.

Fig. 17

Pap-stained cervical tissue with HPV and p16 labeling. (a) Image as seen through microscope, (b) Pap-only image; (c) Pap with HPV overlay, and (d) Pap with p16 overlay.


A second example of dual biomarker staining is shown in Fig. 18. This sample was prepared using the CINTec Plus protocol from Roche Molecular Systems (Pleasanton, CA) in which cervical cells are immunolabeled for p16 using DAB stain and Ki67 using fast red stain in addition to a hematoxylin counterstain. While this assay is designed for determining the presence or absence of both biomarkers by direct visualization, there are instances in which the darkness of the staining prevents clear determination of the presence of both biomarkers, as illustrated by the cells shown in Fig. 18(a). After spectral imaging, analysis, and recolorization, the presence of both p16 and Ki67 in one of the cells is now apparent [Figs. 18(c) and 18(d)]. In addition, by looking at the hematoxylin rule image [Fig. 18(b)], the underlying morphology of the cells can now be clearly elucidated.

Fig. 18

CINTec plus stained cervical cells: hematoxylin counterstain with Ki67 and p16 immunostaining. (a) Image as seen through microscope, (b) hematoxylin only image, (c) hematoxylin with Ki67 overlay, and (d) hematoxylin with p16 overlay.



Optimizing Spectral Imaging Analysis

When using spectral imaging to determine biomarker expression in a clinical context, it is of the utmost importance that false positives be reduced as much as possible. False-positive biomarker signal can result either from actual nonspecific staining of normal cells in the sample (background staining), or from artifacts resulting from the unmixing process itself. Contributions from the former can be reduced to acceptable levels by adjusting the staining protocol, so that background staining is minimized while insuring that the real biomarker staining intensity is significantly above this background even in weakly expressing cells. The biomarker rule image can then be intensity thresholded to remove any background staining detected without compromising detection of the true biomarker staining.

An example of how this is done is shown in Fig. 19. An abnormal cervical cell sample was labeled for p16 using Deep Space Black stain. The values of the biomarker stain intensity in abnormal and normal cells were measured and the intensity histogram compiled, as shown in the figure. These results indicate that background nonspecific staining in normal cells can be largely eliminated by using an intensity threshold of 0.1, whereas the value of the specific positive biomarker staining is largely above this value, suggesting that in this case a threshold value of 0.1 would work effectively.

Fig. 19

Histogram of p16 staining intensity in normal (black) and abnormal (dotted) cells in a cervical cell sample. Above an intensity of 0.1, artifactual normal cell staining is negligible.


This thresholding is not completely effective in reducing false positives due to mathematical unmixing artifacts. We have found that these artifacts can be reduced significantly, however, by using the LSU error image. Once rule images have been obtained from the LSU process, one may obtain the LSU-calculated spectrum at each pixel by multiplying the reference spectra associated with each rule image by the value of the rule image at the pixel in question and summing the results from all the rule images. The LSU error image may then be computed by calculating the square root of the sum of the squares of the difference between the LSU-calculated spectrum and the actual measured spectrum at each pixel location. Nonzero errors at a particular pixel location may arise from measurement noise, the use of incorrect reference spectra, the failure to include a reference spectrum that is actually present, or a breakdown of the linearity assumption of the LSU mathematics. Unlike the noise contribution to the LSU error image, which generally appears as a “salt and pepper” pattern, the other three contributions listed can lead to consistently high error values at specific areas in the image.

The LSU error image has been used extensively in the past for various applications, including identifying regions in which reference spectra were identified incorrectly or omitted entirely,22 determining the reliability of previously identified spectral components,23 and, most notably, determining optimum reference spectra when they are not known a priori.24,25

In the present work, we present a novel application of the LSU error image to multiply stained clinical samples in which the reference spectra are known reliably and completely. In particular, we have observed that localized high LSU error locations are not uncommon. These most often coincide with structural anomalies in the cell, such as dense regions due to unusual cytoplasmic folds, cell clumping, or atypical nuclear or cytoplasmic membrane edges. These effects often, but not always, lead to a greater than average uptake of stain. It would seem that these anomalies lead to an unusually high degree of scattering, resulting in a wavelength-dependent variation of the acquired signal not included in the standard LSU methodology—thereby resulting in a higher than average LSU error at that location.

When performing LSU on a sample using a known reference spectra set, it is well known that the LSU algorithm may result in a nonzero value for each of the reference spectra being used, even when that particular spectrum is not present at the pixel in question. Typically, such nonzero “unmixing crossover” values are quite small in magnitude, and simple thresholding is indeed adequate to eliminate this potential source of false-positive results. In structurally anomalous regions where error values are particularly high, however, the values associated with such unmixing crossover can be considerably elevated. Heuristically, this may be understood to result from the LSU algorithm attempting to use all reference spectra at its disposal to match the extra scattering term at these high-error pixels. Hence, if a single threshold value is used to distinguish between true and false biomarker staining throughout the entire sample—for example, the value 0.1 sufficient to eliminate the majority of background staining in Fig. 19—false-positive biomarker values will result in many high-error value locations on the slide, resulting in a situation untenable for clinical applications.

This effect is illustrated in Fig. 20. For this figure, cervical cells were stained with Pap stain only and unmixed with the five spectra listed above, so that the biomarker unmixing values shown are due solely to unmixing artifact. Figure 20(a) shows the results of compiling these artifactual values for low-error and high-error regions, where these regions are chosen by selecting an error threshold value that eliminates nearly all the random noise–related values in the error image, such that low-error regions correspond to pixels in the error image below this threshold, and high-error regions correspond to pixels above this threshold. Note that whereas in this example, a biomarker intensity threshold of 0.1 would effectively remove most of the unmixing artifacts in the low-error regions, it would not be sufficient to remove the artifacts in the high-error regions; these regions would require a threshold of approximately 0.4. We can now replot the histogram shown in Fig. 19, segmenting the abnormal cell intensity values of Deep Space Black according to whether they were found in low-error or high-error regions, as shown in Fig. 20(b). It can be seen that threshold values of 0.1 and 0.4 (shown as dashed vertical lines) would effectively remove most of the unmixing artifacts as well as the background staining, while retain the vast majority of the true-positive biomarker staining in both the low-error and high-error regions.

Fig. 20

(a) Histogram of artifactual biomarker unmixing values obtained from a slide without any biomarker staining; note that artifact intensities are much greater in high-error regions; (b) Histogram of true biomarker staining values from Fig. 19, with abnormal cells now segregated into low-error and high-error regions.


In regions where artifactual biomarker unmixing values occur, there is a corollary degradation of the rule images of the reference components that are actually present. Interestingly, we have observed that once identified as artifactual, these values can be arithmetically added back into the rule images of the reference components known to be present at that location, and through this process the degradation can be completely corrected.

An example of this is shown in Fig. 21 where the hematoxylin rule image of a cervical cell stained with Pap stain alone has been unmixed with just the four reference spectra corresponding to the primary components of the Pap stain [Fig. 21(a)], and then with two additional reference spectra (Deep Space Black, permanent red) corresponding to biomarkers that are not actually present in the cell [Fig. 21(b)]. The degradation due to the inclusion of the two additional spectra is evident in the degradation of the cytoplasmic detail in Fig. 21(b). In Fig. 21(c), the artifactual rule images for the additional reference spectra are added to the degraded rule image of Fig. 21(b), resulting in a recovered hematoxylin rule image, Fig. 21(d), which can be seen to compare favorably with the original hematoxylin rule image of Fig. 21(a).

Fig. 21

Hematoxylin rule images from a Pap-only stained sample. (a) Unmixing results using Pap component spectra only, (b) including two additional biomarker spectra, (c) correction of degradation by addition of artifactual biomarker signals, and (d) recovered image.



Using Spectral Imaging to Highlight Morphology

When examining the morphology of cervical cells using the Pap stain, it is a common occurrence (even when using modern liquid Pap preparations) to find clumps of darkly stained cells in which nuclei are difficult to differentiate by eye. When confronted with such a situation, the pathologist uses a number of techniques to help with the visualization, including changing the intensity of the microscope illumination source, adjusting the objective focus, and changing the focus of the condenser. While these techniques are often quite helpful, the process can be time consuming, and there are many instances in which cell clumps need to be skipped over (so-called “skipocytes”).

Using the techniques described, individual recolorized images are produced for each of the Pap component stains. These are ordinarily all combined to create the full recolorized Pap image as has been described, but it is straightforward to adjust the relative intensity or remove altogether one or more of these component stains in order to better visualize the underlying morphology. Two examples of this are shown in Fig. 22, in which a full Pap image is shown of two different fields of view before [Figs. 22(a) and 22(c)] and after [Figs. 22(b) and 22(d)] removal of three of the four Pap component stains using the user interface shown in Fig. 22(e). The remaining hematoxylin stain shown in the images at the right is the primary nuclear stain of the Pap combination. By “turning off” the other stains, a clear view of the nuclei in these cell clumps can be obtained, enabling the pathologist to confirm that either there is a potential cause for concern, as in Fig. 22(b), or there is little cause for concern, as in Fig. 22(d).

Fig. 22

Using spectral unmixing to elucidate morphology. Pap-stained cervical cells are shown in (a) and (c) which contain densely stained clumps. Individual components of the Pap stain can be removed, leaving, e.g., just the hematoxylin staining presents as shown in (b) and (d). Control of relative intensity of the components is accomplished using the graphical user interface shown in (e).




The combination of morphology staining and multiple biomarker staining on a single slide has great potential for improving the practice of pathology. In order to actualize this approach, however, it is necessary to use chromogenic transmission biomarker staining rather than fluorescence biomarker labeling, which in turn requires the use of multispectral imaging in order to extract out the individual stains from the fully stained slide. In many instances, this can require upward of 10 wavelengths acquired at each location to produce the highest-quality results, necessitating high speed wavelength switching as well as excellent image quality. AOTFs are an excellent choice for this application. When used in conjunction with appropriate artifact-reduction algorithms, this approach can be appropriate for clinical use, a potential application of which could be to cervical cancer screening. The recolorized images produced can also be adjusted to highlight morphologic features that might not otherwise be visible, and can be used in conjunction with the cell-by-cell biomarker images to produce false-colored overlays. This combination can provide the pathologist with significant additional information to aid in more accurate interpretation.


The AOTFs used in this work were designed, produced, and tested by members in the Orlando, Cleveland, Palo Alto, New Jersey, and Ilminster, UK, branches of Gooch and Housego. The authors would specifically like to acknowledge the indispensable assistance of Matt Whittaker, Matt Francis, Dieter Jundt, and Hai Huang. The assay development was performed in collaboration with Margaret Jones, OxFabs Laboratory, Oxford University Hospital, Oxford, UK, and Dr. Katia Manova, Dr. Afsar Barlas, Dr. Dmitry Yarilin, and Dr. Mesruh Turkekul of the Molecular Cytology Core Facility at Memorial Sloan-Kettering Cancer Center, Manhattan, NY. Cervical cell samples were provided by the BioBank, Oxford University Hospital, Oxford, UK, and Dr. Robert Swedarsky, Pennsylvania Cytology Services, Monroeville, Pennsylvania.


1. M. Kalia, “Personalized oncology: recent advances and future challenges,” Metab. 62(Suppl 1), S11–S4 (2013).21BHAV0026-0495 http://dx.doi.org/10.1016/j.metabol.2012.08.016 Google Scholar

2. C. M. van der Loos, “Multiple immunoenzyme staining: methods and visualizations for the observation with spectral imaging,” J. Histochem. Cytochem. 56(4), 313–328 (2007).JHCYAS0022-1554 http://dx.doi.org/10.1369/jhc.2007.950170 Google Scholar

3. C. M. van der Loos, “Chromogens in multiple immunohistochemical staining used for visual assessment and spectral imaging: the colorful future,” J. Histotechnol. 33(1), 31–40 (2010).JOHIDN0147-8885 http://dx.doi.org/10.1179/his.2010.33.1.31 Google Scholar

4. R. M. LevensonJ. R. Mansfield, “Multispectral imaging in biology and medicine: slices of life,” Cytometry, Part A 69(8), 748–758 (2006).1552-4922 http://dx.doi.org/10.1002/(ISSN)1552-4930 Google Scholar

5. W. J. CukierskiQ. XinD. J. Foran, “Moving beyond color: the case for multispectral imaging in brightfield pathology,” in Proc. IEEE Int Symp Biomed Imaging: from Nano to Macro, Vol. 5193251, pp. 1111–1114, IEEE, Piscataway, NJ (2009). Google Scholar

6. R. M. LevensonA. FornariM. Loda, “Multspectral imaging and pathology: seeing and doing more,” Expert Opin. Med. Diagn. 2(9), 1067–1081 (2008).EOMDA91753-0059 http://dx.doi.org/10.1517/edg.2008.2.issue-9 Google Scholar

7. J. R. MansfieldC. HoytR. M. Levenson, “Visualization of microscopy-based spectral imaging data from multi-label tissue sections,” Curr. Protoc. Mol. Biol. 84, 14.19.1–14.19.15 (2008). http://dx.doi.org/10.1002/0471142727.mb1419s84 Google Scholar

8. M. V. Macvilleet al., “Spectral imaging of multi-color chromogenic dyes in pathologic specimens,” Anal. Cell. Pathol. 22(3), 133–142 (2001).ACPAER0921-8912 Google Scholar

9. R. M. LevensonJ. BeechemG. McNamara, “Spectral imaging in preclinical research and clinical pathology,” Anal. Cell. Pathol. (Amst), 35(5), 339–361 (2012).ACPAER0921-8912 http://dx.doi.org/10.3233/ACP-2012-0062 Google Scholar

10. R. M. Levenson, “Multispectral imaging, image analysis, and pathology,” in Proc. 26th Southern Biomedical Engineering Conference SBEC 2010, Springer Press, Berlin, Germany, pp. 564–567 (2010). Google Scholar

11. K. Matkovicet al., “Global contrast factor – a new approach to image contrast,” in Proc. the First Eurographics conference on Computational Aesthetics in Graphics, Visualization and Imaging, pp. 159–167, Eurographics Association, Geneva, Switzerland (2005). Google Scholar

12. E. S. WachmanD. L. Farkas, “AOTF Microscope for imaging with increased speed and spectral versatility,” Biophys. J. 73(3), 1215–1222 (1997).BIOJAU0006-3495 http://dx.doi.org/10.1016/S0006-3495(97)78154-2 Google Scholar

13. J. XuR. Stroud, Acousto-Optic Devices, Principles, Design and Applications, Wiley Interscience, New Jersey (1992). Google Scholar

14. A. YarivP. Yeh, Optical Waves in Crystals, Wiley Classics Library, New Jersey (2003). Google Scholar

15. C. PannellE. S. Wachman, “Acousto-optic devices having highly apodized acoustic fields and methods of forming the same,” U.S. Patent No. 7,283,290 B1 (2007). Google Scholar

16. W. D. CookE. CavanaghH. D. Dardy, “A numerical procedure for calculating the integrated acousto-optic effect,” IEEE Trans. Sonics. Ultrason. 27(4), 202–207 (1980).IESUAU0018-9537 http://dx.doi.org/10.1109/T-SU.1980.31173 Google Scholar

17. D. R. SuhreL. J. DenesN. Gupta, “Telecentric-confocal optics for aberration correction of acousto-optic tunable filters,” Appl. Opt. 43(6), 1255–1260 (2004).APOPAI0003-6935 http://dx.doi.org/10.1364/AO.43.001255 Google Scholar

18. D. K. Guptaet al., “Interobserver and intraobserver variability in the cytologic diagnosis of normal and abnormal metaplastic squamous cells in pap smears,” Acta Cytol. 45(5), 697–703 (2001).ACYTAN0001-5547 http://dx.doi.org/10.1159/000328290 Google Scholar

19. M. H. StolerM. Schiffman, “Interobserver reproducibility of cervical cytologic and histologic interpretations,” J. Am. Med. Assoc. 285(11), 1500–1505 (2001).JAMAAP0098-7484 http://dx.doi.org/10.1001/jama.285.11.1500 Google Scholar

20. A. P. Pintoet al., “Immunomarkers in gyneclogic cytology: the search the the ideal ‘biomolecular papanicolaou test,” Acta Cytol. 56(2), 109–121 (2012).ACYTAN0001-5547 http://dx.doi.org/10.1159/000335065 Google Scholar

21. C. A. Brownet al., “Role of protein biomarkers in the detection of high-grade disease in cervical cancer screening programs,” J. Oncol. 2012, 289315 (2012).JOONA21687-8469 http://dx.doi.org/10.1155/2012/289315 Google Scholar

22. M. SlaterR.B. Herrick, “Hyperspectral imaging method and apparatus,” U.S. Patent No. 6,008,492 (1999). Google Scholar

23. S. DetermanP.J. Miller, “Spectral unmixing for visualization of samples,” U.S. Patent No. 8,462,981 (2013). Google Scholar

24. R.A. Neheret al., “Blind source separation techniques for the decomposition of multiply labeled fluorescence images,” Biophys. J. 96(9), 3791–3800 (2009).BIOJAU0006-3495 http://dx.doi.org/10.1016/j.bpj.2008.10.068 Google Scholar

25. J.H. GruningerA. J. RatkowskiM. L. Hoke, “The sequential maximum angle convex cone (SMACC) endmember model,” Proc. SPIE 5425, 1–14 (2004). http://dx.doi.org/10.1117/12.543794 Google Scholar


Elliot S. Wachman is vice-president of research and development at Gooch and Housego, Lakewood, New Jersey. He received his BA degree in physics from Swarthmore College in 1983 and his MS and PhD degrees in applied physics from Cornell University, New York, in 1986 and 1991, respectively. He was a member of the research faculty at Carnegie-Mellon University for approximately 10 years before joining Gooch and Housego. His current research interests focus on applications of spectral imaging to biomedicine.

Stanley J. Geyer has served as a faculty member in the pathology departments at the schools of medicine of the University of Pittsburgh, the University of Washington, Georgetown University, Temple University, and the University of Utah. He has published over 50 papers in peer-reviewed journals and serves as a reviewer for journals devoted to cytopathology. He currently practices pathology at the Indiana Regional Medical Center.

Joel M. Recht received his PhD degree in mathematics from Temple University, Philadelphia, and served as an assistant professor of mathematics at the Camden campus of Rutgers University. Following his academic career, he served in private industry as a software specialist for embedded systems, and for the past 23 years as a designer and developer of image processing-based algorithms. He has been awarded 6 patents, all of which are related to image processing apparatuses and methods.

Jon Ward received his BS degree in physics from London University in 1977 and joined Standard Telephone and Cables (STC) working in telecoms. He then moved to the aerospace sector working for the Westland Group specializing in the field of optical fiber sensors. In 1991, he joined Gooch and Housego (G&H), initially working on fiber/acousto-optic components and later concentrating on specialist acousto-optic devices. More recently, he has led the G&H UK effort on acousto-optic R&D.

Bill Zhang is a senior optical engineer at Gooch and Housego in Orlando, Florida. He has more than 20 years of experience in imaging and nonimaging optics, electro-optical and photonics systems, and optical MEMS. Since 2006, he has been designing and developing optics for grating spectrometers, FT spectrometers, programmable light sources, optical isolators, optical fiber couplers and collimators, OCT systems, and AOTF-based hyperspctral imaging systems.

Murray Reed received his MSc degree in physics from Auckland University in 1982 and a PhD degree in applied physics from Stanford University, California, in 1990. He was employed at Coherent as a laser engineer for many years, leading the development of a wide range of diode pumped and ultrafast laser products, and was appointed as vice-president of engineering. After being General Manager of New Focus for Newport, he joined Gooch and Housego as chief technology officer (CTO) in 2012.

Chris Pannell received his BSc degree in theoretical physics and a PhD degree in the field of laser fluid flow measurement. He worked as a postdoc at the University of Kent, UK; then took up the post of lecturer at the Optoelectronics Research Center, UK. He became a professor in 2000 and ran a research group at the University of Kent before taking up the position of chief scientist at Gooch and Housego, based in Orlando, Florida.

© The Authors. Published by SPIE under a Creative Commons Attribution 3.0 Unported License. Distribution or reproduction of this work in whole or in part requires full attribution of the original publication, including its DOI.
Elliot S. Wachman, Elliot S. Wachman, Stanley J. Geyer, Stanley J. Geyer, Joel M. Recht, Joel M. Recht, Jon D. Ward, Jon D. Ward, Bill G. Zhang, Bill G. Zhang, Murray K. Reed, Murray K. Reed, Christopher N. Pannell, Christopher N. Pannell, } "Simultaneous imaging of cellular morphology and multiple biomarkers using an acousto-optic tunable filter–based bright field microscope," Journal of Biomedical Optics 19(5), 056006 (9 May 2014). https://doi.org/10.1117/1.JBO.19.5.056006 . Submission:


High efficiency LWIR AOTF
Proceedings of SPIE (August 26 2008)
Monolithic bulk shear wave acousto-optic tunable filter
Proceedings of SPIE (February 16 2003)
AOTF-based system for image cytometry
Proceedings of SPIE (March 22 2005)

Back to Top