1.

Introduction

Hyperspectral remote sensing (HRS), also known as imaging spectroscopy or hyperspectral imaging (HSI), is rapidly emerging as a distinct and key discipline in the science of Earth and planetary observation. Special charge-coupled device (CCD) cameras combined with gratings, prisms, or interferometers enable solar-reflected energy (350 to 2500 nm) and emitted (7500 to 13,500 nm) energy to be separated into hundreds of narrow, contiguous bands that capture the interactions of photons with the atomic and molecular structure of a target material. We focus here on visible/near-infrared to shortwave infrared (VNIR/SWIR; $\sim 400$ to 2500 nm) HSI. Based on analytical chemistry and spectroscopic analysis principles utilized in the laboratory since the nineteenth century, HRS systems generate multiple images based on narrow bandwidths, creating a coregistered “stack” of images, each with a different band-center wavelength. When calibrated (spectrally and radiometrically) and processed to remove sensor and atmospheric artifacts, each pixel in the image is expressed as a vector through the stack that represents a spectrum essentially identical to that achieved under laboratory conditions.¹ Figure 1 graphically depicts the concept of a hyperspectral image stack comprised of hundreds of bands of imagery, collected at slightly different wavelengths across the solar reflected spectrum from 390 to 2500 nm. The data structure is commonly known as a data cube or image cube.

Fig. 1

Hyperspectral images are collections of hundreds of bands of coregistered imagery. Each band collects reflected energy in a slightly different wavelength and results in what is called an image cube. The image shown here, of Moffett Field, California, was collected on August 20, 1992, from a height of 20,000 m.²

HSI data permit the identification of materials based on matching against a library (or database) of spectral signatures. Equally important, HSI provides broad area, synoptic scale coverage with overhead imagery for mapping the composition of large areas of the Earth’s surface with traditional spectroscopic, statistical signal processing, chemometric, and machine-learning algorithms and tools. And in addition to material identification, many biophysical, environmental, and geological/sedimentological processes such as photosynthesis, toxic spill evolution, and coastal erosion can be analyzed and quantified via imaging spectroscopy.

While most hyperspectral reflectance measurements are single-point samples acquired in the field or laboratory or airborne or space-borne imagery, there have been applications, mostly in medical research, where hyperspectral analysis is directed inward at the microscopic level.^3–7 Imaging spectroscopy at the microscopic level offers the same material identification and characterization advantages as overhead, imaging spectroscopy. However, most hyperspectral microscopy applications have been limited to the VNIR spectral range and have focused on very specific applications. There exists a need in the remote sensing community to characterize materials with high spatial resolution imaging spectroscopy, and to do this over the full solar-reflected (350 to 2500 nm) spectral range. Here, we describe the development of a hyperspectral microscope based on commercial off the shelf (COTS) camera and lighting systems and a tunable laser that enables a full-range hyperspectral scan. Example applications are included.

2.

Background

Contemporary microscopy, in a variety of forms, has received increased attention from the scientific community. The 2014 Nobel Prize in Chemistry was awarded to E. Betzig, S. Hell, and W.E. Moerner for their parallel development of super-resolved fluorescence microscopy.⁸ The resolution of conventional light microscopes was thought to be optically limited to the full-width at half-maximum (FWHM) of the point spread function, as demonstrated by Abbe.⁹ The optical limitation was generally reached around 250 nm. Betzig, Hell, and Moerner independently demonstrated, using fluorescence, that the nonlinear response of fluorophores to excitation could be exploited to enhance resolution. Using lasers and the characteristics of specific fluorophores, they demonstrated that the strict diffraction limit could be functionally surpassed and, although not strictly hyperspectral, this research demonstrates the resurgent scientific interest in spectral microscopy for a number of biological applications.

Over the past 20 years, hyperspectral imaging microscopes (HIM) have been successfully developed and utilized by a number of researchers mainly for biomedical applications. Many of the current uses of HIMs utilize fluorescence but many others involve nonfluorescent reflectance spectra also. A variety of methods have been used to generate fine resolution spectral data utilizing, among a variety of methods, acouto-optical tunable filters (AOTF) for dispersing (separating) light into narrow bandwidths. This has moved imaging spectroscopy into a new dimension where hyperspectral imagers are smaller and much more cost effective than the early instruments that emerged circa 1990 and have also paved the way for microscopy applications. Table 1 lists some selected early applications of hyperspectral microscopy.

Table 1

Selected hyperspectral microscopy applications from the literature.

Early applications of hyperspectral microscopy include Treado et al.¹⁰ who developed a hyperspectral microscope with an AOTF and a CCD array and observed lipid vesicle suspensions in water and human epithelial cells. Frank et al.³ used a diode laser spectrometer to demonstrate that NIR/Raman spectroscopy could reveal significant biochemical information from biopsy samples.

Several fluorescence applications of hyperspectral microscopy in the VIS and VNIR ranges have been demonstrated by, e.g., Schultz et al.⁶ and Huebschman et al.¹¹ Sinclair et al.¹² constructed and demonstrated the advantages of a hyperspectral microarray scanner that exploits fluorescence for the separation of the emission due to green fluorescent protein in yeast cells. Timlin et al.¹³ coupled the advantages of high spectral resolution sampling with multivariate statistical analysis to understand the interference of various organic fluorophores among cellular proteins.

Gehm et al.¹⁴ described an aperture-coding approach to HSI microscopy that significantly improved throughput and delivered 1-nm spectral resolution and a $5.4\text{-}\mu \mathrm{m}$ spatial resolution over a portion of the VIS range of 550 to 665 nm. This approach was important in increasing photon efficiency which is often a problem in this type of microscopy. In a unique application for the analysis of possible biological warfare agents, Anderson et al.¹⁵ utilized a hyperspectral camera attached to a epifluorescent microscope to differentiate live, viable versus dead/nonviable bacterial endospores. Thirty-two images from 400 to 720 nm, coupled with a statistical test for spectral signature divergence, showed that the differences between viable and nonviable endospores to be statistically significant.

In a cancer research breakthrough, Horton et al.⁴ utilized microfluidics and HIM to demonstrate the ability to capture and analyze immune-phenotype circulating tumor cells which are characteristic of early stage carcinomas and signify the presence of cancer in a patient. This application demonstrates the tremendous potential for HSI/HIM technology to aid disease diagnosis, treatment selection, and response monitoring.

Bouillard et al.¹⁶ demonstrated scanning near-field optical microscopy (SNOM) with a fiber-tip-based scanning microscope that enabled the simultaneous acquisition of near-field images in a broad spectral range (400 to 1000 nm), thus recovering near-field spectroscopic information. This permitted the spectral characterization of plasmonic nanostructures. Park et al.¹⁷ developed an HIM that utilized an AOTF and very sensitive lifetime fluorescence characterization techniques to identify foodborne pathogenic bacteria. Bertani et al.¹⁸ demonstrated a near-full range solar reflected confocal microscope, fed by a supercontinuum laser source that could provide accurate measurements out to the limit of solar-reflected energy at about 2500 nm. Demonstrated usefulness included engineering applications as well as the characterization of melanoma cells.

3.

Emerging Applications

Hyperspectral microscopy has recently enjoyed continued development in a number of new fields and applications. In terms of instrumentation, microchip technologies, and computer miniaturization, such as the raspberry pi, have generally reduced the size and shape of hyperspectral instruments, especially in the VNIR. The introduction of laser technology has also advanced HIM and have improved infrared detection capabilities, especially in aqueous environments.¹⁹ VNIR hyperspectral cameras have been reduced to two to three pounds and can capture images without scanning or movement of the camera or the target.²⁰ Gao et al.²¹ described a snapshot image mapping spectrometer with a high enough sampling density to be used in microscopy applications. There is an amazing variety of new HSI sensors that have been developed for microscopy and unmanned systems application and there has even been work on developing a comparative methodology for evaluating and comparing the performance characteristics of different hyperspectral microscopy systems.²²

HIM was essentially borne of medical applications and unique systems have been continually under development. Siddiqi et al.²³ developed a hyperspectral microscopic test to determine normal, precancerous, and cancerous cells. Li²⁴ showed that different medical conditions could be identified through hyperspectral retinal scans. Dicker²⁵ showed that cancerous skin cells and melanoma could be identified through HSI. Uhr et al.²⁶ showed that HIM could precisely identify and quantify 10 molecular markers in individual cancer cells in a single pass. Akbari et al.²⁷ showed the effectiveness of HIM in the detection of prostate cancer. Excellent reviews of hyperspectral technology in the medical field can be found in Refs. 28 and 24. Although not strictly microscopic, prostate cancer detection has been demonstrated by Akbari et al.,²⁷ using HIM data and least squares support vector machines that were developed and evaluated for classifying hyperspectral data in order to enhance the detection of cancer tissue.

Hyperspectral microscopy has found many relevant applications in the detection of plant diseases and as a quality control mechanism in factory food production. Eady et al.²⁹ developed a technique for detection of Salmonella with hyperspectral microscopy. Fusarium infections in wheat have been detected using HIM.³⁰ Several researchers have shown a direct connection between plant hyperspectral reflectance imaging characteristics and specific plant pathogens.^31–34 Kuska et al. showed that there is potential for automated analysis of plant diseases using HIM data.³² Dale et al.³⁵ outlined HIM techniques for food quality analysis production quality control.

Identification and analysis of materials at the nanomaterial scale is another emerging application of hyperspectral microscopy. Roth et al.³⁶ showed that HSI with state-of-the-art optics and computer software could enable the rapid identification of materials at the micro- and nanoscales. Roth, Pena et al., and Grabinski et al.^37–39 showed that HM could be effective at identifying nanomaterials in biomedical research.

Patskovsky et al.⁴⁰ designed a hyperspectral microscopy systems for the identification of plasmonic nanoparticle of Au Ag and Au/Ag alloys in human cancer cell preparations and for optical biosensing. Mortimer et al.⁴¹ showed the capability of HIM for examining cellular uptake of different metal-based nanoparicles, including nanosized metals (silver and gold, both citrate stabilized), metal oxides (copper oxide and titanium dioxide), and CdSe/ZnS core/shell quantum dots at subtoxic concentrations. Badireddy et al.⁴² showed how HIM could be applied water quality and especially the detection of engineered nanoparticles.

Hyperspectral microscopy has been successfully applied to the detection of explosives by demonstrating the detection of absorption spectra of individual explosives particles and demonstrate subnanogram detection limits.⁴³

In planetary science, HIM is being planned for analysis of soils on Mars.^44,45 The European Space Agency’s MicrOmega instrument is planned to characterize in situ the composition of samples at their grain size scale efficiently through the utilization remote sensing hyperspectral imagers.⁴⁶ In addition to Martian soils analysis, Korablev et al.⁴⁷ demonstrated the utilization of a robotic arm with a hyperspectral microscope that could be used to analyze lunar soils also.

4.

Instrument Description

The functional HSI microscope utilizes two independent image sources. The VNIR data cubes were acquired by a Resonon Pika II imaging spectrometer with a Xenoplan 1.4/23-0902 objective lens and is shown in Fig. 2. The Pika II is mounted nadir-looking at a mechanical translation table on which the sample to be imaged is placed. The Pika II is a pushbroom sensor with a slit aperture, thus the need for a translation table to move the sample to facilitate hyperspectral image cube formation. The height of the sensor above the table is user selectable and is dependent upon spatial resolution requirements. Significantly higher resolution has been achieved with a higher magnificantion lens, not described in this work. Though capable of acquiring 240 bands from 400 to 900 nm, the sensor is generally configured to acquire 80 bands by binning spectrally by three resulting in a sampling interval of $\sim 6.25\mathrm{nm}$ and high signal-to-noise ratio spectra.

Fig. 2

Photograph of the VNIR section of the HSI microscope. The Resonon Pika II is shown with the Xenoplan 1.4/23-0902 objective lens.

Four quartz–tungsten–halogen lamps are used for illumination roughly approximating a hemispherical-directional illumination/viewing geometry. Sensor and translation table operation, data acquisition, and data calibration are achieved by the instrument manufacturer’s software. Conversion to reflectance requires a measurement of dark frame data (i.e., acquiring a cube with the lens cap on) and a measurement of a polytetrafluoroethylene (PTFE) reference plaque (large enough to entirely fill the field of view). Then, for each HSI cube measured, the sensor’s software first subtracts the dark data and then uses the PTFE data (also dark subtracted) to ratio the spectral measurements to give relative reflectance.^48,49

The SWIR data cubes were collected by a custom-built system based on monochromatic laser illumination and an imager and used to acquire spectral imagery in the 900- to 2500-nm spectral range. The laser used is a Continuum Surelite III-10: Q-switched Nd:YAG, 10 Hz, 850 mJ at 1064 nm and a Panther^® EX PLUS: OPO; tunable from 410 to 2550 nm with a linewidth $<6.0{\mathrm{cm}}^{-1}$. The laser source is projected free space to a diffuser oriented at an angle of 45 deg with respect to the illumination source and an equal and opposite angle with respect to the sample of the same plane. The imager used in this study is a Xenics Xeva 2.5-320. It is an HgCdTe (MCT)-based imager, 14 bit radiometric resolution, with a spectral response ranging from 900 to 2500 nm. The detector array is $320\times 256$ with a pixel pitch of $30\mu \mathrm{m}$. The lens used was a 25-mm focal length, $f/1.4$, optimized for use in the NIR/SWIR. Other lenses may be used depending upon magnification requirements; and here, too, the sensor may be moved up and down to vary the spatial resolution. A LabVIEW program was used for operation and data acquisition. At each laser pulse, a signal from the laser is used to trigger the camera. The program controls the laser stepping sequentially through the wavelengths, triggers the camera, and collects a set of images at each wavelength.

A sample is placed on the sample stage and the lens is focused at a wavelength near the midpoint of the intended spectral range ($\sim 1700\mathrm{nm}$) (see Fig. 3). An eight step gray-scale target was included in the field of view. Additional processing is required due to a nonlinearity in the laser-imager system; more will be mentioned about this in a later section. The data acquisition program scans the sample collecting 20 images at each wavelength and then saves a mean of the 20 images. The program advances the wavelength in 5-nm increments (finer sampling is possible) and repeats the image collection until the full NIR/SWIR spectral range is scanned. Once the sample is scanned, a reference reflectance plaque is scanned. This provides a means to correct for spatial nonuniformities. A dark frame is also acquired. The collection of images is compiled into a data cube and processed for corrections in nonlinearity of the imager’s radiometirc response, spatial uniformity, and conversion to reflectance.

For general applications, the instrument resolution is $\sim 0.17\mu \mathrm{m}$ and the full data cube is collected in 600 s.

Fig. 3

The tunable laser source (insert, upper right) of the HSI microscope designed to collect spectroscopic image data from 900 to 2500 nm and complete the full solar-reflected range of the HSI microscope. The NIR/SWIR imager and lens are on the left; the VNIR camera and lens [not currently used routinely (see text)] are on the right. This configuration is housed in a dark black plastic case (not shown) for laser light protection.

5.

Applications

We review next several studies previously published that have utilized HSI microscopy and the instruments described above. All have in common a demonstration of the unique perspective that imaging brings to laboratory spectroscopy and how such a measurement capability is directly relevant to studies of Earth remote sensing from traditional air- and space-borne sensors. Indeed, we maintain that spectroscopy for ground-truth spectral measurements in support of Earth remote sensing should, whenever possible, be acquired by imaging spectrometers (and at a range of magnifications—to include microscopic) rather than point spectrometers alone.

5.1.

Microscene for Quantifying Hyperspectral Uncertainty

While HSI can effectively detect and identify a target material embedded in background clutter, determining how well this can be achieved as a function of various measurement conditions or parameters can be a challenge.⁵⁰ Almost all HSI sensors that are currently in use are calibrated and characterized, producing information to assess measurement uncertainty. However, these sensor uncertainties alone do not provide an obvious indication of the ability to detect or classify substances of interest in a scene. Beyond sensor performance, the target material, collection method and geometry, and choice of processing and analysis algorithms can all be significant factors affecting the final results. Kerekes and Baum provided a template for evaluating these factors using an end-to-end analysis model. Their model may be used as a cursory, system design-level tool to estimate the impact of those factors that might be significant contributors to the overall uncertainty. This model can be used as a comprehensive means to quantify and understand which factors might be significant contributors to the overall uncertainty.⁵⁰

Ideally, a test scene used in an end-to-end analysis will have the same constituents (target, background, and confuser materials) of interest for a specific application. The materials of interest (more specifically the spectral signatures) are of particular importance, since the features of a spectral signature may or may not be highly dependent on all of the other factors in an end-to-end analysis.

An alternative method (or complementary, if used as input data for a numerical model) is the use of a laboratory, benchtop hyperspectral imager in which the sensor collects a hyperspectral data cube by observing a Petri dish filled with materials of interest. Here, we describe a relatively simple, low cost method of generating test data that can be used for system trade studies and for algorithm development and testing. The method is facilitated by the increasing number of compact HSI sensors that are now commercially available, enabling benchtop measurements of a physically simulated scene in the convenience—and control—of a laboratory setting. This setup can be considered as a means to produce a proxy data set and an alternative to a sensor collecting imagery over a terrestrial area. The composition of the scene may be engineered to contain the materials of interest by physically placing representative or actual classes of targets, backgrounds, and possible confusers.

Adding materials to the microscene Petri dish may be considered analogous to applying paint from a palette to a canvas where the different colored paints represent different distributions of points in $n$-dimensional ($n$-D) hyperspace. In order to validate that the data set is adequately populated with points in $n$-D space comparable to data acquired with airborne sensors, data dimensionality estimates such as those derived from a principal components analysis and box counting (yielding a fractal dimension) can be employed as quantitative metrics.

We assert that a microscene data set—and the microscene concept—is a proxy for an actual air- or spaceborne remotely sensed multi- or hyperspectral data cube. We further assert that a microscene is a suitable alternative data set generated by numerically modeled scenes. These assertions may be substantiated both qualitatively and quantitatively.

Here we provide as an illustration the use of a microscene for a system trade study: the impact of wavelength error on classification. A microscene and its reference spectra (in a microarray) are measured and subsequently used in a straightforward scene classification analysis. The scene is then reanalyzed but with an error in center wavelength position to assess the impact on classification results. A wavelength shift represents one of many possible errors that might be explored with microscene data. Classification differences are quantified using an error matrix to determine the changes in classified and unclassified pixels, and errors of omission and commission. This allows for the establishment of factors that might affect the final data product in a way that is application specific by direct measurement of materials of interest.

Using the Pika II system described in the previous section, hyperspectral images were collected of a 96-well sample tray, painted flat black, and filled with selected materials to form a “microarray” as shown in Fig. 4. The main purpose of the microarray was to independently collect reference spectra of the same substances. Fourteen materials were selected and each was placed in a well of the microarray. One well was filled with powdered PTFE for use as a white reference and another left empty as a dark reference. Other materials selected to provide a range of spectral signatures including some that are very unique such as neodymium (III) oxide (${\mathrm{Nd}}_2{\mathrm{O}}_3$), aquamarine blue pigment, and sulfur, along with some that are very similar such as paprika, chili powder, and cumin. These substances were chosen to yield a data cube with a complicated, spectral mix (in terms of their distribution in hyperspace).

Fig. 4

A normal color composite using spectral bands 641, 549, and 458 nm for red, green, and blue, of the 96-well sample tray (microarray) filled with the materials used in the microscene. The black is an empty cell in the sample tray.

The microscene was arranged with a base layer of impure quartz sand, $\sim 1\text{-}\mathrm{cm}$ deep, with the other substances randomly deposited in piles on top of the sand. The microscene is shown in Fig. 5; Table 2 lists the substances and their three-letter code.

Fig. 5

A normal color composite of the hyperspectral microscene using spectral bands 641, 549, and 458 nm for red, green, and blue. The scene is masked to eliminate the edges of the Petri dish and the translation table platform. The key listing each of the substances is provided in Table 2.

Table 2

The substances emplaced in the microscene and their three letter abbreviations.

Substance	Abbreviation	Substance	Abbreviation
1. Parsley	veg	8. Sulfur	sul
2. Paprika	pap	9. Aquamarine blue	blu
3. Chili powder	chl	10. Quartz sand	snd
4. Cumin	cmn	11. Asphalt	asp
5. Turmeric	tmr	12. Neodymium oxide	ndo
6. Iron	irn	13. Copper	cop
7. Clay	cly	14. Coffee	cof

5.2.

Scene Classification

The classified microscene is shown in Fig. 6(a) in which false colors are applied to each classified substance. Not all of the material patches are solidly defined such as sulfur; some are nearly solidly filled, others have misclassified pixels (i.e., paprika, chili powder, and cumin), and some not classified, shown as black. The classification of the microscene was repeated with an induced wavelength error of 7 nm and is shown in Fig. 6(b) using the same color scheme. Significant differences can be observed for this extreme example, including misclassifications and substances that are unclassified (errors of omission and commission).

Fig. 6

The classified microscenes are shown with false colors representing each class. (a) The original microscene and (b) the wavelength error classified microscene.

5.4.

Analyzing Nonlinear Spectral Mixing

It is well known that intimately mixed materials frequently exhibit nonlinear spectral mixing. Granular materials, such as soils, are often intimate mixtures of numerous different inorganic (minerals) and organic (humic) substances.⁵¹ And since soils are often significant constituents of spectral remote sensing scenes, intimate mixing may safely be assumed to be a common phenomenon. Thus, the impact of nonlinear spectral mixing on algorithm results must be well understood if we are to achieve a major goal of HSI: the areal mapping and quantification of materials that comprise remotely sensed scenes.

It is also very common, as part of the data analysis process, to apply linear spectral unmixing (LSU) to imaging spectrometer data. However, an LSU inversion applied to an intimate mixture exhibiting nonlinear spectral mixing will yield subpixel material abundance estimates that do not equal the true values of the mixture’s constituents. An example of this is provided by Keshava and Mustard.⁵² To address this, several methods of spectral unmixing of nonlinear spectral mixtures have been proposed. Examples include Hapke theory, the extended endmember matrix method, kernel-based methods, and Gaussian processes. Reviews of nonlinear spectral mixture analysis are given in Refs. 5253.–54.

Using the hyperspectral microscope described in Sec. 2, three binary mixtures (and the two endmembers) of glass beads were constructed and emplaced in the wells of a 96-well sample plate: 0%/100%, 25%, 75%, 50%/50%, 80%/20%, and 100%/0% of didymium/soda-lime (percentages by volume). We describe here the results of analyses of the HSI microscopy data (in reflectance) of the intimate mixtures. Analysis methods applied to the reflectance data are LSU; LSU applied to reflectance converted to single-scattering albedo (SSA) using Hapke theory; and two kernel-based methods (those of Broadwater and Banerjee⁵⁵ and Chen et al.⁵⁶).

Some results are shown in Figs. 7Fig. 8Fig. 9–10. The five regions of interest (ROI) mean spectra of the didymium and soda-lime glass beads (endmembers and mixtures) are shown in Fig. 8. The ROIs from which these mean spectra were derived are shown in Fig. 7. Note that the didymium spectrum has many features in the VNIR region of the spectrum in stark contrast to that of the soda-lime (Fig. 8).

Fig. 7

(a) 35-mm digital single-lens reflex (DSLR) camera photograph of the 96-well plate containing the glass beads, and (b) an HSI microscope normal color composite image (2% linear stretch of the bands used in the RGB image). All percentages are by volume of glass bead type indicated.

Fig. 8

Mean spectra from the ROI shown in Fig. 9. Plot color corresponds to ROI color in Fig. 9. Fractions are by volume; “DiDy” = didymium; “Lime” = soda-lime.

Fig. 9

A graph of the results for the FCLS, GKLS at different levels of nonlinearity at measured by the parameter $\gamma$, and SSA methods is displayed. Higher values of $\gamma$ correspond to greater degree on nonlinearity.⁴¹

Fig. 10

The observed and estimated spectra at location $(x,y)=(324,49)$ within the 50/50% mixed region using the FCLS, GKLS, and SSA methods. The $y$-axis of (a) and (b) is in reflectance units; the $y$-axis of (c) is in albedo units.⁴¹

The testing of the LSU models yielded variable results. Figure 10 shows one of the best results as demonstrated by the generalized kernel fully constrained least squares (GKLS) model developed by authors of Ref. 55. Other models, such as the SSA, sum-to-one, and fully constrained least squares show slightly higher error rates. But our main purpose here is simply to demonstrate the utility of hyperspectral microscopy to model and evaluate various complex processing methods that might be used in airborne or spaceborne hyperspectral data. A more complete discuss of the these results can be found in Ref. 51.

5.5.

Mineralogy and Petrology

A commercially obtained cut-and-polished slab of coarse-grained pink granite was imaged with the tunable laser HSI microscope. The rock is from the Fujian Province of China, a large region of granitic magmatism.⁵⁷ A 35-mm photograph of the slab is shown in Fig. 11 along with a grayscale image from the HSI cube at 2010 nm. An eight-step gray calibration target is imaged alongside the slab and is required for converting the image data to reflectance. The slab is square with a side length of $\sim 7.6\mathrm{cm}$ ($\sim 3$ in.); the region imaged by the HSI microscope is indicated by the red rectangle.

Fig. 11

Granite images. (a) A 35-mm true-color DSLR photograph of the slab. (b) The HSI microscope captures the area indicated by the red box. A grayscale image from the HSI cube at 2010 nm. An eight-step gray calibration target is captured in the scene. The image is displayed with an arbitrarily chosen linear stretch (to maximize image quality).

The microscope measures biconical reflectance—the realization of a bidirectional reflectance measurement geometry.⁵⁷ The cube is 320 samples by 256 lines by 306 bands from 906 to 2500 nm. The granite is covered by 170 samples by 256 lines.

In addition to an image cube of the granite slab, a dark current cube and a cube of a 12% gray reflectance standard (and which completely fills the field of view of the microscope) are acquired. The 12% gray reflectance standard is utilized for correction to reflectance. The absolute reflectance (biconical—as the laboratory realization of bidirectional) of the 12% gray panel is determined by independent measurement.

The need for the eight-step gray calibration target is to correct for nonlinearity in the response of the Xenics Xeva SWIR camera due to the short (5 ns) high intensity laser pulse. When not outright saturated, the imager response is nonlinear with a reduced dynamic range. This effect necessitated the use of a 12% reflectance standard versus the more typical 99% reflectance reference panel. It is worth noting that the HSI microscope seems to counter common experience. Very good, high signal-to-noise ratio spectra are generated by materials with lower reflectance; noise content increases with reflectance (because of the higher signal strength entering and overwhelming the sensor).

The straightforward calculation of reflectance by a ratio of the subject cube to the reflectance standard cube (both dark current subtracted) followed by multiplication by the reflectance standard is not sufficient to generate reflectance from the laser HSI microscope cubes (though it is standard practice for typical spectral signature measurements with point and close-range laboratory imaging spectrometers). An empirical line method-like procedure is required but since the camera response is nonlinear, a custom program was created to apply an “empirical cubic method.” Cubic (i.e., third-order polynomial) correction equations are generated from ROI on the eight-step gray calibration panels combined with their laboratory-measured truth spectra (biconical reflectance with a spectral evolution spectrometer). Retrieval of reflectance from a third-order polynomial equation is not as direct as from a linear equation; the bisection method root-finding algorithm was implemented to achieve this.

In practice, there are additional processing steps applied to generate the data shown in Fig. 11. Following measurement of the three cubes mentioned above, the precise procedure is as follows: (1) apply a spectral calibration correction to the data. This is merely assigning each band its proper wavelength based on independent spectral calibration measurements of the tunable laser using a point spectrometer measurement of laser light in an integrating sphere. (2) Calculate reflectance using the usual spectroscopy method discussed above; i.e., obtain the ratio of the subject cube to the reflectance standard cube (both dark current subtracted); postmultiply by the spectrum of the standard. (3) Apply a mask to eliminate the bad/dead pixels that commonly occur with MCT detectors. (4) Create a spectral subset cube that eliminates bad/noisy bands. Bands with not-a-number values disrupt the empirical cubic method reflectance retrieval. (5) Apply a $3\times 3\text{-}\text{pixel}$ median filter to each image. This is largely a cosmetic step to mitigate the presence of the masked bad pixels (which would show as a “peppering” of dark pixels). (6) Apply the empirical cubic method correction using ROI on the eight-step calibration panels and their truth reflectance spectra.

The granite slab sample has medium grained phaneritic (granitic) texture as is evident in the photograph in Fig. 11. It is comprised of alkali feldspar (pink and buff; largely subhedral), plagioclase feldspar (white; subhedral to euhedral), quartz (clear, gray; anhedral), and biotite (black; subhedral to euhedral) as the predominant Fe/Mg-rich mafic mineral. Though a petrographic analysis of a thin-section has not, thus far, been performed, zoning and alteration are apparent in hand sample and confirmed via inspection with a hand lens. Other textural features are evident as well such as a glomeroporph of plagioclase laths. The plagioclase feldspars most likely have albite and Carlsbad twinning and contain patches altered to serricite and/or sausserite; the alkali feldspars likely show perthitic texture and tartan twinning. The biotite grains are undoubtedly pleochroic. Although these textural features are not observable at the hand sample scale (with the exception of suspected patches of alteration), they are essentially ubiquitous in granitic rocks and can be assumed to be present and will contribute to spectral signature diversity (next section). The widespread anhedral quartz grains are translucent to transparent; in many instances, minerals beneath are discernible through them. It is not known if the quartz grains show undulatory extinction (common in granitic rocks); the presence of myrmekitic and granophyric textures (also common in granitic rocks) is unknown. These features, too, if present, will contribute to spectral signature diversity.

5.5.1.

Spectral signatures

The relatively straightforward mineralogical makeup of granites, in general, and of this sample in particular (i.e., alkali and plagioclase feldspar, quartz, and biotite) notwithstanding, there is an astonishing diversity of spectral signatures. And because spectral signature quality is very good, differences in the spectral signatures exceed the noise and are discernible. Several single-pixel spectra are shown in Fig. 12. Three-by-three pixel ROI mean spectra around the locations of the spectra in Fig. 12 are given in Fig. 13. The nine pixel average spectra are somewhat smoother, as expected. Contrast Figs. 12 and 13 with Fig. 14: a plot of the slab mean spectrum compared to a point spectrometer measurement of the slab. A single point spectrum of a granite is not indicative of the true range of spectral signatures actually present in the rock. Though not shown, the spectra of Fig. 14 match quite closely the spectra of microcline (for the laboratory point spectrometer spectrum), an alkali feldspar, and albite (for the scene mean spectrum), a plagioclase feldspar, in the USGS mineral spectral library.⁵⁸ Both feldspars (i.e., microcline and albite), although perhaps good proxies for granitic rocks, do not portray the wide range of spectral signatures as they occur in the rock nor yield information on the mafic (and, in the present sample, hydroxide-bearing) mineral content. Additional spectral signatures are shown in Fig. 15.

Fig. 12

Single pixel spectra randonly selected from the granite slab HSI cube.

Fig. 13

Three-by-three pixel ROI mean spectra around the locations of the single-pixel spectra in Fig. 12. The nine-pixel ROI mean spectra are smoother, as expected, than their counterparts shown in Fig. 12.

Fig. 14

A plot of the slab mean spectrum (black; a mean of all spectra in the laser HSI microscope cube) and a point spectrometer measurement of the slab (blue).

Fig. 15

Several additional spectral signatures in the HSI data. The spectra are vertically offset for clarity of presentation.

5.5.2.

Mapping the minerals

It is challenging to generate a classification map due to the variability in spectral signatures. A spectrum from a plagioclase grain (or an ROI mean spectrum from a grain), for example, does not adequately capture all of the variability plagioclase presents in the rock. Numerous attempts to generate a classification image using spectral angle mapping (SAM) and LSU were made. A representative result is shown in Fig. 17. A spectral angle mapper was applied with several spectra such as those shown in the plots above. The results from three spectra are used to build a false color composite (Fig. 16). The blue regions correspond to plagioclase feldspar (the white minerals in the normal color digital image), the reddish/purplish regions are quartz, the tan/buff/orange regions are alkali feldspar, and the white regions are biotite.

Fig. 16

Representative classification results. (a) A 35-mm normal-color DSLR photo. (b) A false color composite based on spectral angle mapper results. The red circles correspond to the same locations on the slab; the corresponding minerals are also indicated.

Fig. 17

(a) A normal color composite image of the granite slab acquired with the PikaII VNIR HSI sensor. (b) A grayscale image also from the PikaII and showing four ROI—one for each of the major mineral phases in granite: red, quartz; green, plagioclase feldspar; yellow, alkali feldspar; and blue, biotite/other Fe-Mg phases. ROI mean spectra are used in LSU.

Although not shown here, several individual SAM and LSU output planes highlight the zoning and other compositional variability shown by most of the crystals in the rock. Indeed, spectral signature variability is likely complicated by mixing effects between grains at their boundaries and the transmission of signature information through transparent and translucent grains (introducing nonlinear spectral mixing effects that could mimic or appear to be zoning).

5.5.3.

VNIR data of granite

The granite sample can also be classified with the VNIR portion of the spectrum. LSU is used to map each of the major mineral phases in granite: quartz, plagioclase feldspar, alkali feldspar, and biotite/other Fe–Mg phases. Figure 17 shows a normal color composite image.

The mean spectra of the ROI contain very little in terms of distinct or remarkable spectra of the major rock-forming minerals (Fig. 18). Nonetheless, the differences evident here are effective for a spectral unmixing analysis (Fig. 19).

Fig. 18

ROI mean spectra for the ROIs shown in Fig. 17.

Fig. 19

(a) The same normal color composite image of the granite slab shown in Fig. 18. (b) A false-color composite of the spectral unmixing fraction planes.

The seemingly unremarkable spectra of the granite ROIs are shown in Fig. 18. However, using LSU, the major mineral phases are successfully identified and mapped. Red is quartz, green is plagioclase feldspar, and blue is alkali feldspar. Three of the four endmember fraction planes are utilized here to display the image shown. The four endmembers are effective at separating and highlighting the major two-feldspar plus quartz mineralogy of a granite.

5.5.4.

Cyanobacteria and harmful algal blooms

Harmful algal blooms (HAB) are a growing water quality problem worldwide. Generally composed of strains of various types of cyanobacteria, HABs often contain toxin-producing varieties that pose a threat to human and aquatic life, in addition to being a nuisance to navigation and esthetics.⁵⁹ Hyperspectral microscopy affords a unique analysis of this phenomenon and may hold the key to separating toxic from nontoxic strains of bacteria, which could be critically important in setting remediation priorities. Figure 20 shows an example of hyperspectral microscope images of Microcystis and Aphanizomenon, two genera of cyanobacteria, at three different scales of magnification. An LSU algorithm applied to the spectral derivative of the images shows promise in spectrally separating and identifying each genera. One genus (Microcystis) often produces a toxin while the other does not. Being able to use remote sensing to separate these two species could have important consequences for remediation efforts (Fig. 21).

Fig. 20

Images of the cyanobacteria. (a) Normal-color composites of Microcystis and Aphanizomenon. (b) The corresponding false-color composites of LSU results, where endmembers (red: Microcystis, green: Aphanizomenon, blue: background) were identified based on the first derivative of the spectrum at each pixel. Scale bars from left to right are 100, 50, and $20\mu \mathrm{m}$.

Fig. 21

First derivative spectra of the two genera of cyanobacteria showing the inherent separability of the two species.

6.

Conclusion

There are distinct advantages to characterizing sample using HIM. A key advantage is better spectral characterization of target materials. A fundamental part of the imaging spectroscopy process is matching imagery-derived spectra with laboratory spectra for a given material. But this can be complicated by the variability in the composition of the material as well as the variability encountered in real-life mixtures and background interferences.

A second advantage relates to the ability to separate constituents of a composite sample. As shown in the granite example above, the major mineral phases are identified and mapped based on ROI mean spectra and are done using in LSU.

A final important characteristic of hyperspectral microscopy, especially as it related to overhead remote sensing, is the addition of a new scale in identifying materials and processes. Scale is a fundamental concept in remote sensing and geography in general. In the strictly quantitative sense, scale is a mathematical relationship between an actual ground distance and the representative fractional distance of a geographic object such as an image or map. But the term also has other important meanings, not the least of which is related to ecological processes and understandings. Scale is intimately related to concepts of pattern and process and in that sense is deliberately void of strict definitions and is even sometimes tied to perceptual perspectives.⁶⁰ However, even in remote sensing and geography, events such as changes in biological distribution, as might be expected from climate change, are intimately dependent on scale and processes important at one scale may not be as predictive at another scale.⁶¹

This leads us to the major point that, even with the current maturing of hyperspectral technology, spectroscopic analysis and understanding of material and processes at the Petri dish level is not fully understood and research in this area can only add to our understanding of multiple scales of geographic phenomenon. This continues a trend that is emerging with unmanned aerial systems (UAS) development which is bringing hyperspectral analysis down to the submeter level. Soon remote sensing science will be able to apply imaging spectroscopic analysis from the Petri dish (microscope) to the submeter (UAS) to the meter (aircraft) to the decameter (satellite) scales and understanding the connection between pattern and processes at those scales will be an important challenge to remote sensing science.

We have demonstrated the utility of hyperspectral imaging microscopy (HIM) as an investigational tool for remote sensing. More specifically we provided examples over the full solar-reflected range using both COTS systems and a custom laser-based system. The examples are intended to demonstrate the ability to address common remote sensing changes including material detection and classification, nonlinear mixture analysis, and species level discrimination, at the sub-cm spatial scale. These demonstrations are not intended to be interpreted as fully implemented best practices, as significant refinement is still necessary. Rather, it is intended to stimulate interest and adoption of HIM as a means to evaluate materials in order to make better use of their inherent spectral signatures.