This PDF file contains the front matter associated with SPIE Proceedings Volume 11654, including the Title Page, Copyright information, and Table of Contents.

High-frame-rate two-photon imaging is becoming increasingly important in neuroscience for recording fast voltage and neurotransmitter signals from many sources at a time. I will summarize developments in this field and my lab’s recent efforts to build a DMD-based random-access two-photon microscope capable of recording thousands of sources at kilohertz framerates, without the need for computational image reconstruction.

The brain is one of the most important organs in our body, but it is functionally the least understood one. It is composed of millions of neurons, whose interconnection, i.e. connectome, determines its function. Although the interaction of neurons in vitro has been well studied in the past century, no existing tool can capture whole-brain emergent properties at single neuron or even synapse resolution. To understand functional connectome, an imaging system that can cover a whole brain in vivo with spatial resolution of micrometers (neuron) to nanometers (synapse) as well as temporal resolution in sub-seconds (calcium) to milliseconds (action potential) is highly desirable. In this invited talk, we introduce our recent efforts to improve optical microscopy in terms of speed, depth, and spatial resolution, toward the goal of understanding the brain of Drosophila, which offers a small brain with sophisticated functions and genetic control capabilities.

Uncovering the structure and function of cortical vascular network down to the level of capillaries can provide useful insights on brain physiology and pathology. Ideally, the probing method should allow concurrent observation of vascular morphology and hemodynamics, with sufficient spatiotemporal resolution to resolve individual capillaries and track blood cell motion in the scattering mammalian brain. By employing an all-optical scanning two-photon fluorescence microscope, we realized kilohertz full-frame recording of cortical blood vessels beyond 700 µm deep in the mouse brain and measured blood flow speeds up to 15 mm/s

Nondegenerate two-photon absorption (NTA) is a method for detecting infrared (IR) radiation with wide bandgap photodetectors. We use NTA to detect mid IR (MIR) light on a Si-based detector with the aid of a near-IR (NIR) pump beam. This enables the use of mature, robust Si technologies to detect IR light at room temperature. We show that NTA facilitates MIR spectroscopy in both a single pixel photodiode and a high-resolution CCD, allowing for chemically selective MIR imaging. We demonstrate the utility of NTA by including MIR images and videos of moving targets.

Optical-resolution photoacoustic microscopy (OR-PAM) has become a popular tool in small-animal studies. However, previous OR-PAM techniques variously lacked a high imaging speed, a high spatial resolution, and/or a large field of view. Here we report a high-speed OR-PAM system using an innovative water-immersible polygon-mirror scanner, which has achieved a cross-sectional frame rate of as high as 2400 Hz over a 12-mm scanning range. Using this polygon-scanner-based OR-PAM system, we have performed various studies on mouse models with stroke and cardiac arrests. We expect that the new OR-PAM system will become a powerful tool for imaging hemodynamics and neuronal functions.

Photoacoustic imaging is a promising biomedical imaging technology that can provide the biological information of animals and humans in vivo. Taking advantages of both optics and ultrasound simultaneously, the photoacoustic imaging has rich optical contrast and high ultrasonic resolution in deep tissues. Especially, the optical-resolution photoacoustic microscopy (OR-PAM), the major implementation of the photoacoustic imaging, achieves sharp spatial resolution with focused optical beams. However, developed OR-PAM systems are disadvantaged by limited temporal and/or spatial resolutions by hardware parts. Here, we introduce a high speed super-resolution localization OR-PAM that overcomes the limited resolutions. First, to improve a temporal resolution, we equipped a galvanometer scanner, which has been used in many optical microscopies. Previously, due to the vulnerability of the scanner to water, the PAM system using the scanner scanned only light, not ultrasound. However, our system overcame the weakness by immersing only a scanning part except to a galvanometer part, leading to scanning light and ultrasound simultaneously. Steering both laser beams and ultrasound waves with a scanner’s mirror immersed in the water, our system achieves a wide scanning range and a high signal-to-noise ratio as well as the B-mode imaging speed of 500 Hz. Furthermore, we acquired a super-resolved microvascular image in vivo using an agent-free localization imaging technology based on the fast scanning speed. These results show that our super-resolution high-speed OR-PAM can be used in a variety of fields, including neurology, oncology, and pathology.

Light-sheet microscopes with an extended depth of field (EDOF) offer a simple but powerful route toward fast volumetric imaging. However, methods for EDOF typically result in a loss of signal-to-noise ratio. Here, we propose a parallelization strategy as a simple solution. By illuminating multiple acoustically generated light sheets at different axial positions within the EDOF, and following an encoding sequence, information from several in-focus planes can be simultaneously retrieved. After applying a decoding algorithm, volumetric images are reconstructed with enhanced signal and level of detail. Our strategy paves the way for exploiting the full speed capabilities of EDOF light-sheet systems.

Multiphoton microscopy enables sub-micron resolution, label-free structural and functional imaging of living tissues with contrast from multiple modalities, including second harmonic generation and two-photon excited fluorescence. We developed a fast, large area multiphoton exoscope (FLAME) portable system with enhanced label-free molecular contrast for macroscopic imaging of human skin with microscopic resolution. It combines optical and mechanical scanning mechanisms with deep learning image restoration to produce 3D sub-cellular resolution images that encompass sub-mm2 to cm2 scale areas of tissue within minutes. We demonstrate the performance and utility of the instrument by fast ex vivo and in vivo imaging of human skin.

Dual-comb coherent anti-Stokes Raman scattering (DC-CARS) spectroscopy is an effective tool for high-speed acquisition of vibrational spectra in the fingerprint region. DC-CARS spectroscopy also provides high spectral resolution by virtue of its ability to cover a large time delay between the pump and probe pulses, from two optical frequency combs with slightly different, fixed pulse repetition rates. However, less than 1% of the incident pulse energy is used to acquire the CARS signal because the repetition interval of the laser pulses (<1 ns) is much longer than the coherence lifetime of molecular vibrations (~3 ps). This results in a low spectral acquisition rate and a low signal-to-noise ratio. Here, we introduce a novel method for DC-CARS spectroscopy with a nearly 100% energy efficiency using a “quasi”-dual-comb laser. Specifically, one of the repetition rates of the two lasers is rapidly modulated by controlling the pumping intensity of a Ti:sapphire laser, so that the group delay between two pulses is shorter than the coherence lifetime of molecular vibrations, while the group delay is monitored with two-color interferometry to calibrate the time-domain CARS signal. With this method, we realized a spectral acquisition rate of 100,000 spectra/s, which is 10x higher than conventional DC-CARS spectroscopy. Due to its ~100% energy efficiency, sensitivity at this record high acquisition rate is even higher than conventional DC-CARS spectroscopy operating at a slower acquisition rate of 10,000 spectra/s. Our method holds promise for diverse applications in fields ranging from materials science to life science, such as high-throughput screening, flow cytometry, and live-cell imaging.

Light-sheet illumination enables major increase in multiphoton imaging speed for in vivo studies. However, photoperturbation in multiphoton light-sheet microscopy remains poorly investigated. We show here that the heart beat rate of zebrafish embryos is a sensitive probe of linear and nonlinear photoperturbations. By analyzing its behavior with respect to laser power, pulse frequency and wavelength, we derive guidelines to balance signal and photoperturbation. We then demonstrate one order-of-magnitude signal enhancement over previous implementations by optimizing the laser pulse frequency. These results open new opportunities for fast in vivo imaging.

Here we achieved record-high >500 volumes/second two-photon imaging by improving lateral and axial scanning speed via 32-channel multifocal excitation/detection, and a tunable acoustic gradient-index lens, respectively. We developed a deconvolution process to reduce scattering-induced crosstalk in multifocal detection scheme, thus enabling whole brain imaging of Drosophila with millisecond and micrometer spatiotemporal resolution. Potential applications toward brain science include studying millisecond dynamics in a neuronal network, and resolving 3D microfluidics in blood vessels.

Understanding and repairing complex biological systems, such as the brain, requires new technologies that enable such systems to be observed and controlled with great precision, across extended spatial and temporal scales. We are discovering new molecular principles that are leading to such technologies. For example, we recently discovered that it was possible to physically magnify biological specimens manyfold, in an even way, by embedding them in dense swellable polymers, mechanically homogenizing the specimens, and then adding water to isotropically swell the specimens. In this method, which we call expansion microscopy (ExM), we enable scalable, inexpensive diffraction-limited microscopes to do large-volume nanoscopy, in a multiplexed fashion – important, for example, for brain mapping. As another example, we discovered that microbial opsins, genetically expressed in neurons, could enable their electrical activities to be precisely driven or silenced in response to millisecond timescale pulses of light. These tools, called optogenetic tools, are enabling causal assessment of the contribution of defined neurons to behaviors and pathologies in a wide variety of basic science settings. Finally, we have developed new methods of directed evolution, and discovered mutant forms of optogenetic tools that enable precision fluorescent imaging of the high-speed voltage of neurons in the living brain. We share all these tools freely, and aim to integrate the use of these tools so as to lead to comprehensive understandings of neural circuits.

Fluorescence Lifetime Imaging (FLIM) is an essential tool in Life Sciences, but up to now users had to chose between high timing precision or fast data acquisition when using Time-Correlated Single Photon Counting (TCSPC) electronics. Our approach, named rapidFLIMHiRes, allows recording several FLIM images per second with a temporal resolution of 10 ps. The method combines advances in fast scanning, hybrid photomultiplier detectors, TCSPC modules, and correction algorithms to reduce decay curve distortions. Thus fast processes can be observed with the high optical and temporal resolution achievable in confocal microscopy at a rate of several frames per second.

The conventional ultrafast optical imaging methods in the ultraviolet (UV) spectral range are based on pump-probe techniques, which cannot record non-repeatable and difficult-to-produce transient dynamics. Compressed ultrafast photography (CUP), as a single-shot ultrafast optical imaging technique, can capture an entire transient event with a single exposure. However, CUP has been experimentally demonstrated only in visible and near-infrared spectral ranges. Moreover, the requirement to tilt a digital mirror device (DMD) in the system and the limitation of controllable parameters in the reconstruction algorithm also hinder CUP’s performance. To overcome these limitations, we extended CUP to the UV spectrum by integrating a patterned palladium photocathode into a streak camera. This design also nullifies the previous restrictions in DMD-based spatial encoding, improves the system’s compactness, and offers good spectral adaptability. Meanwhile, by replacing the conventional TwIST algorithm with a plug-and-play alternating direction method of multipliers algorithm, the reconstruction process is split into three secondary optimization problems to precisely update the separated variables in different steps, which considerably enhances CUP’s reconstruction quality. The system exhibits a sequence depth of up to 1500 frames with a size of 1750×500 pixels at an imaging speed of 0.5 trillion frames per second. The system’s ability of ultrafast imaging was investigated by recording the process of UV pulses travel through various transmissive targets with a single exposure. We envision that our system will open up many new possibilities in imaging transient UV phenomena.

Single-shot real-time ultra-high-speed imaging is of significance in capturing transient phenomena. Existing techniques fall short in possessing satisfying specifications in the imaging speed, sequence depth, and pixel count. To overcome these limitations, we have developed compressed optical-streaking ultra-high-speed photography (COSUP) that records a scene (x, y, t) by applying the operations of spatial encoding, temporal shearing, and spatiotemporal integrating. The COSUP system possesses an imaging speed of 1.5 million frames per second (fps), a sequence depth of 500 frames, and a pixel count of 0.5 megapixels per frame. COSUP is demonstrated by imaging single laser pulses illuminating through transmissive targets and by tracking a fast-moving object. We envision COSUP to be applied in widespread applications in biomedicine and materials science.

Structured illumination microscope enables high temporal resolution wide field-of-view super-resolution imaging but typically provides only two-fold resolution improvement over the diffraction limit. We present speckle metamaterial-assisted illumination nanoscopy (MAIN) which brings the resolution down to 40 nm and beyond. A hyperbolic metamaterial is implemented as a substrate to generate subwavelength illumination patterns in the near field of the metamaterial. Fluorescent objects located on the metamaterial are illuminated by such high spatial-frequency near field illuminations and are reconstructed by a Blind-SIM algorithm. Speckle-MAIN represents a new route for super-resolution, which may lead to important applications in bio-imaging and surface characterization.

The modern fluorescence microscope would ideally offer high spatial resolution in all dimensions, high speed and minimal photodamage to specimens. We developed a triple-view line confocal system that improves spatial resolution in all three dimensions (to ~270 nm x 250 nm x 335 nm or ~185 nm x 170 nm x 245 nm with structured illumination) in scattering samples tens of microns thick. To speed up acquisition and reduce photobleaching, deep learning is applied to denoise and enhance resolution when only using data acquired from one view.

We report a large-scale light-scattering single-cell characterization enabled by a high-throughput quantitative phase imaging platform (multi-ATOM) (10,000 cells/sec). By virtue of its subcellular resolution, multi-ATOM accesses the light-scattering information from individual cells via Fourier Transform light scattering (FTLS) analysis. Specifically, we applied FTLS analysis on multi-ATOM images to explore the statistical characteristics of single-cell fractal dimension (FD). We demonstrated that FD can be harnessed as an effective label-free phenotype that is indicative of cell types and states. FD can identify the heterogeneity among leukemia/lung cancer cell types and trace the different phases in cell cycle progression.

Attaining three-dimensional data at high throughput is a grand challenge in microscopy. I will discuss two recent contributions to 3D microscopy utilizing point-spread-function (PSF) engineering: 1.a new method that extends the capabilities of flow-based imaging to 3D co-localization microscopy, and 2. efficient and fast localization of dense molecules in volumetric single-molecule-localization microscopy (SMLM) using deep learning, termed DeepSTORM3D. Together, these methods enable the study of sub-cellular biological processes at unprecedented timescales and throughputs.

In COVID-19 therapy with artificial lungs such as extracorporeal membrane oxygenation (ECMO) machines, platelets in the extracorporeal circulation are often activated by their contact with the artificial materials, leading to the formation of blood clots followed by serious complications such as stroke and heart attack. However, anticoagulation and antithrombotic management is challenging due to the lack of testing tools to evaluate the circulation. Here we demonstrate real-time monitoring of thrombogenesis in the circulation of an ECMO-equipped goat with an intelligent platelet aggregate characterizer (iPAC), which is based on imaging flow cytometry and deep-learning-based analysis of numerous platelet aggregates in blood.

Using a high-throughput imaging flow cytometer (10,000 cells/sec) multi-ATOM, we established a hierarchical biophysical phenotyping approach for label-free single-cell analysis. We demonstrate that the label-free multi-ATOM contrasts can be derived into a set of spatially hierarchical biophysical features that reflect optical density and dry mass density distributions in local and global scales. This phenotypic profile enables us to delineate subtle cellular response of molecularly targeted drug even at an early time point after the drug administration (6 hours). Based on fluorescence image analysis, we further interpreted how these biophysical phenotypes correlate with specific intracellular organelles alteration upon drug treatment.

We present a 1 megapixel single-photon avalanche diode (SPAD) camera featuring 3.8 ns time gating and 24 kfps frame rate for 1-bit images, fabricated in 180 nm CMOS image sensor technology. The SPAD sensor was used to capture 2D and 3D scenes over 2 m with depth resolution of 5.4 mm and precision better than 7.8 mm (rms). We demonstrate extended dynamic range in dual exposure operation mode and show spatially overlapped multi-object detection in single-photon time-gated time-of-flight experiments. We further demonstrate applications of the megapixel SPAD camera for fluorescence lifetime imaging microscopy (FLIM) and light-in-flight imaging.

Gold nanoparticles (AuNPs) have been widely used as optical probes to observe motions of biomolecules such as lipids and proteins in biological systems. They efficiently scatter light without photobleaching, and provide high contrast in optical images. Motions of single biomolecules, labeled with AuNPs, have been investigated by tracking center positions of AuNPs in optical image. Nanoscale stepping motions of motor proteins and fast diffusional motions of lipids in membranes have been observed. To understand the working mechanism of tiny and complex biological molecules in detail, further improvement of the localization precision, and imaging capability for multiple biomolecules will be important. We developed an annular illumination total internal reflection dark-field microscope with axicon lens to achieve localization precision at angstrom level and temporal resolution at microsecond order. Localization precisions at 0.3 nm was achieved with 40 nm AuNPs at temporal resolution of 100 µs. We used this system for the observation of bio-molecular motors, such as kinesins and dyneins. We also developed a multicolor single particle tracking system using silver and silver-gold alloy nanoparticles (AgNPs and AgAuNPs) together with AuNPs. We constructed a total internal reflection multicolor dark-field microscope with multiple lasers that match the plasmon resonance wavelength of AgNPs, AgAuNPs, and AuNPs, respectively. A spectrophotometer was used in imaging optics, to project scattering images at each wavelength on the different portions of two-dimensional high-speed CMOS camera. Motions of multiple phospholipids in supported lipid membranes, and multiple kinesins were simultaneously observed at 100 µs time resolution.

Light field microscopy is a powerful tool for fast volumetric image acquisition in biology which requires a computationally demanding and artefact-prone image reconstruction process. I will present a novel framework consisting of a hybrid light-field light-sheet microscope and deep learning-based volume reconstruction, where single light-sheet acquisitions continuously serve as training data and validation for the convolutional neural network reconstructing the LFM volume. Our framework produces video-rate reconstructions; their fidelity can be verified on demand and the network can be fine-tuned as necessary.

We report a robust method based on generative deep learning to reconstruct quantitative phase image (QPI). By employing multiplexed asymmetric-detection time-stretch optical microscopy (multi-ATOM), we simultaneously captured multiple intensity image contrasts of the same cell in microfluidic flow, revealing different phase-gradient orientations at high throughput (10,000 cells/sec). Using conditional generative adversarial networks (cGAN), we performed a systematic analysis of how different orientations of the phase-gradient contrasts and their combinations influence the QPI prediction performance, which overall general achieves a high similarity (structural similarity index > 0.91) and low error rate (mean squared error < 0.01).

We present a novel approach to remove the unwanted non-resonant background from Broadband Coherent Anti-Stokes Raman Scattering (B-CARS) spectra, based on deep learning. The unsupervised model is built as a convolutional neural network with seven hidden layers. After training on synthetic data, our model was able to process experimental B-CARS spectra and correctly retrieve all the relevant vibrational peaks. The retrieval time is 100 microseconds per spectrum, faster than the time required to record it. We expect that this model will significantly simplify and speed-up the analysis of B-CARS spectra, allowing real-time retrieval of the vibrational features.

We present a new deep compressed imaging modality by scanning a learned illumination pattern on the sample and detecting the signal with a single-pixel detector. This new imaging modality allows a compressed sampling of the object, and thus a high imaging speed. The object is reconstructed through a deep neural network inspired by compressed sensing algorithm. We optimize the illumination pattern and the image reconstruction network by training an end-to-end auto-encoder framework. Comparing with the conventional single-pixel camera and point-scanning imaging system, we accomplish a high-speed imaging with a reduced light dosage, while preserving a high imaging quality.

Standard immunofluorescence (IF) staining is labor-intensive, time-consuming and suffers from inflexibility and poor multiplicity. To overcome these limitations, we proposed a deep learning (DL) approach for virtual IF staining with high multiplicity and specificity from label-free reflectance microscopy. Our results show that DL-enabled label-free IF microscopy can predict characteristic subcellular features during different cell cycles and reveal cellular phenotypes with high accuracy.

Optical diffraction tomography (ODT) has demonstrated its potential for revealing subcellular structures and quantitative compositions in living cells without chemical staining. Recently, we developed a deep-learning based algorithm to reconstruct the 3D refractive index (RI) maps of cells using a single raw interferogram, measured from an angle-multiplexed ODT system. Using this system, we demonstrated a high throughput 3D image cytometry method, in which a microfluidic chip for controlling cell flow is integrated in the ODT system. By flowing the cells in the chip and minimizing the camera exposure time, we can achieve 3D imaging of over 6,000 cells per second.

Amino acids and peptides are basic components of proteins and have vital importance in various biological functions and diseases. In this research, we have attempted to detect and distinguish 20 kinds of amino acids and 39 kinds of peptides without any labeling. By using Raman microscopy, more than two thousand Raman spectra were obtained within five minutes from each analyte, at femtomolecular levels. Furthermore, deep learning analyses of the spectra yielded accuracies greater than 96 percent in discriminating between the amino acids and the peptides.

We designed a compact, real-time LED-based endoscopic imaging system for the detection of various diseases including cancer. In gastrointestinal applications, conventional endoscopy cannot reliably differentiate tumor from normal tissue. Current hyperspectral imaging systems are too slow to be used for real-time endoscopic applications. We are investigating real-time spectral imaging for different tissue types. Our objective is to develop a catheter for real-time hyperspectral gastrointestinal endoscopy. The endoscope uses multiple wavelengths within UV, visible, and IR light spectra generated by a micro-LED array. We capture images with a monochrome micro camera, which is cost-effective and smaller than the current hyperspectral imagers. A wireless transceiver sends the captured images to a workstation for further processing, such as tumor detection. The spatial resolution of the system is defined by camera resolution and the distance to the object, while the number of LEDs in the multi-wavelength light source determines the spectral resolution. To investigate the properties and the limitations of our high-speed spectral imaging approach, we designed a prototype system. We conducted two experiments to measure the optimal forward voltages and lighting duration of the LEDs. These factors affect the maximum feasible imaging rate and resolution. The lighting duration of each LED can be shorter than 10 ms while producing an image with a high signal-to-noise ratio and no illumination interference. These results support the idea of using a high-speed camera and an LED-array for real-time hyperspectral endoscopic imaging.

This paper, originally published on March 5, 2021 was retracted from the SPIE Digital Library on November 10, 2021, by the publisher and in agreement with the authors, upon verification that a substantial portion of the paper was copied from the following work without attribution or permission: A.V. Polishchuk, “Development of a Raman gas analyzer for isotopologues of carbon-containing compounds with ultraspectral resolution,” Doctoral Thesis, ITMO University, November 18, 2020.