Live-subject microscopies, including microendoscopy and other related technologies, offer promise for basic biology research as well as the optical biopsy of disease in the clinic. However, cellular resolution generally comes with the trade-off of a microscopic field-of-view. Microimage mosaicking enables stitching many small scenes together to aid visualization, quantitative interpretation, and mapping of microscale features, for example, to guide surgical intervention. The development of hyperspectral and multispectral systems for biomedical applications provides motivation for adapting mosaicking algorithms to process a number of simultaneous spectral channels. We present an algorithm that mosaics multichannel video by correlating channels of consecutive frames as a basis for efficiently calculating image alignments. We characterize the noise tolerance of the algorithm by using simulated video with known ground-truth alignments to quantify mosaicking accuracy and speed, showing that multiplexed molecular imaging enhances mosaic accuracy by leveraging observations of distinct molecular constituents to inform frame alignment. A simple mathematical model is introduced to characterize the noise suppression provided by a given group of spectral channels, thus predicting the performance of selected subsets of data channels in order to balance mosaic computation accuracy and speed. The characteristic noise tolerance of a given number of channels is shown to improve through selection of an optimal subset of channels that maximizes this model. We also demonstrate that the multichannel algorithm produces higher quality mosaics than the analogous single-channel methods in an empirical test case. To compensate for the increased data rate of hyperspectral video compared to single-channel systems, we employ parallel processing via GPUs to alleviate computational bottlenecks and to achieve real-time mosaicking even for video-rate multichannel systems anticipated in the future. This implementation paves the way for real-time multichannel mosaicking to accompany next-generation hyperspectral and multispectral video microscopy.
Hyperspectral (HS) imaging systems have become important tools in an array of fields due, in part, to the superior molecular recognition capabilities provided by high-resolution spectral information. Provided the user has a library of spectral fingerprints representing the individual molecular contents, one may decompose each HS pixel into a sum of its constituent species using a linear least-squares fitting routine with a non-negativity constraint, (i.e., spectral unmixing). This method, while robust, presents a significant computational bottleneck that precludes real-time HS image analysis. In this work, we use GPUs to accelerate the fast non-negative least squares (FNNLS) algorithm and present unmixing analysis results using images acquired from 4 commercial HS imaging systems. In all cases, we demonstrate video-rate speeds (> 15 fps) using one and two NVIDIA GTX 1080Ti GPUs, representing an average data throughput of 2.5 GB/s and 5.0 GB/s, respectively. This implementation enables online HS feature recognition and is easily integrated into computer-based and mobile platforms with current NVIDIA GPU technology. The method is also applied to a hyperspectral fluorescence imaging system to show online 5-color optical biopsy (5 protein biomarkers) in a mouse model of ovarian cancer to monitor responses to PDT.
Molecular imaging using fluorescence microendoscopy offers promise for the optical biopsy of cancer to inform precision medicine. However, microscopic resolution generally comes with the trade-off of a tiny field of view and tunnel vision. Micro-image mosaicking offers the capability of stitching together larger scenes of the tissue to aid visualization and interpretation. The development of hyperspectral microendoscopes provides motivation for adapting mosaicking algorithms to process a plurality of simultaneous channels. We present an algorithm that mosaics hyperspectral microendoscopic video by correlating channels of consecutive frames as a basis for calculating image alignments. A typical raster path to produce suitable data for mosaicking images the same location several times redundantly in different frames, making this algorithm well-suited for analyzing video-rate data. To complement this data rate, we employ parallel processing via GPUs to alleviate computational bottlenecks and approach video-rate mosaicking speeds. This implementation lays the foundation for real-time multi-channel mosaicking to accompany video-rate hyperspectral microendoscopic probes.