This PDF file contains the front matter associated with SPIE Proceedings Volume 11299, including the Title Page, Copyright information, Table of Contents, Author and Conference Committee lists

In this talk, I give a brief overview of the evolution of metasurfaces. By appropriately adjusting the size, orientation, geometry and arrangement of subwavelength building-blocks (SBB) of metasurfaces, one can control/change the basic properties of incident light at will. The tremendous freedom of choice and orientations for SBBs open up new physics and lead to unprecedented functionalities such as dispersion engineering and polarization control capability. Owing to these unique features, metasurfaces have a potential to complement or even replace their conventional diffractive/refractive counterparts. Also, metasurfaces straightforward fabrication makes them an attractive platform for emerging technologies ranging from AR/MR/VR to LiDAR.

Compared to conventional, rotationally-symmetric optics, free-form optics provide additional degrees of freedom that enable special geometrical and performance features. For instance, in AR and VR, some approaches based on free-forms have shown their ability to reduce the display to pupil distance (total track length, or TTL), to increase the Field of view (FOV) or to improve the apparent resolution of the VR scene. This work will discuss different forms to treat design problems with free-forms, ranging from the direct calculation of point-clouds using the so-called SMS method (followed by a NURBS representation of results) to the combination of direct calculation + optimization. In the latter approach, different types of polynomial descriptions of the free-form surfaces will be considered, with special emphasis on the importance of using a basis that is orthogonal, either with a sag metric or a gradient one.

In this paper, we will review the technical challenges, present the development of occlusion-capable OST-HMD prototypes, and demonstrate their mutual occlusion capability and performance in bright environments.

A compact near-eye visor system that can guide light from a display to the eye could transform mixed reality technology. Unfortunately, existing implementations of such a visor either suffer from small field of view or chromatic aberrations. See-through quality and vergence-accommodation conflict further make the overall performance of the visors unsuitable for a seamless mixed reality experience. Metasurfaces are an emerging class of nanophotonic elements, which can dramatically reduce the size of optical elements while adding new functionalities. I will present our research on metasurface freeform optics, and explain how such metasurface optics can potentially enable the ultimate miniaturization of the near-eye visor without sacrificing any performance.

Neural networks in which the interconnections between the nodes are randomly assigned are promising for the realization of neuromorphic devices in which the resource requirements for training are lower than for a fully deterministic system. Reservoir computing is a class of recurrent network for which the input and internal weights are random and fixed over time, and only the output weights are trained via a linear regression. In this work, we review the recent work on photonic reservoirs and describe our recent results on the implementation of a single node system based on multi-mode optical interconnect technology developed for high channel density and low power data transfer applications. We discuss the potential advantages of this approach for the realization of a photonic cluster of reservoirs.

Reservoir computing (RC) is a class of recurrent neural network that expands the dimensionality of a time-domain signal by mapping it into a higher-dimension space to capture and predict features of complex, non-linear temporal dynamics. Hardware level implementation of RC requires a reservoir with a large number of fixed nodes and the ability to activate and read the output weights of the neurons. As training is performed at a single output layer using simple linear regression techniques, RC is significantly simpler than other recurrent neural networks and thus provides a potentially faster learning framework with low training cost. Here, we report on an optical implementation of a reservoir computer using speckle generated in a multimode fiber (MMF). Neurons are activated by driving pixels of a spatial light modulator (SLM) with time domain waveforms and the output of the SLM is imaged onto the MMF. The MMF output is imaged onto a camera whose image is digitally processed and fed back into the fiber through the SLM. We demonstrate recovery of Mackey- Glass waveforms and classification of multi-frequency sinusoids using the speckle-based optical reservoir computer. As all the components used in the experiment can be readily mapped into an integrated photonic circuit our result demonstrates a framework for building a scalable, chip-scale, optical reservoir computer.

Reservoir computing (RC) has reinvigorated neuromorphic computing activities in photonics. RC radically reduces the required complexity for a hardware implementation in photonics as compared to earlier efforts in the nineties. Currently, multiple photonic RC systems show great promise for providing a practical yet powerful hardware substrate for neuromorphic computing. Among those, delay-based systems offer through a time-multiplexing technique a simple technological route to implement photonic neuromorphic computation. We will review the state of the art on delay-based RC and discuss our advances in substrates implemented as passive coherent fibre-ring cavities and semiconductor lasers with delayed optical feedback. Passive coherent reservoirs built using fiber loops have achieved record performances, but are still aided by nonlinear electro-optical transformations at the input and output. Nevertheless, when targeting all-optical reservoirs, these nonlinearities will be absent. We have found that optical nonlinearities in the fibre itself can be sufficient to enhance the task solving capabilities of a passive reservoir. Also, delay-based optical substrates for RC tend to be quite bulky employing long fiber loops or free-space optics. As a result, the processing speeds are limited in the range of kSa/s to tens of MSa/s. We have studied and developed substrates using external cavities which are far shorter than what has been realized before in experiment. Specifically, by integrating a semiconductor laser together with a 10.8 cm delay line on an active/passive InP photonic chip using the Jeppix platform, we can increase the processing speed to GSa/s.

There have been a number of rapid advances in the prediction of the dynamics of chaotic systems using a technique known as Reservoir Computing. These techniques are mostly not effective for large networks, as the complexity of the task increases quadratically both in time and memory. We report new advances in Optical Reservoir Computing using multiple light scattering to accelerate the recursive computation of the reservoir states. Different approaches to information encoding based on phase or amplitude spatial light modulations are compared. We demonstrate the scalability and the good prediction performance of our approach using the Kuramoto-Sivashinsky equation as an example of a spatiotemporally chaotic system.

Non-van Neumann compute engines such as neuromorphic electronics have shown to outperform CPUs by 3-4 orders of magnitude in terms of ‘weighted addition’, namely multiply-accumulate (MAC)-per-Joule. Here, we discuss experimental devices for a photonic neural network (NN) with an energy efficiency targeting10^18 MAC/J. We consider an electro-optic perceptron consisting of a photodetector (summation) coupled to an EO modulator (nonlinear activation function, NLAF) [George et al, Opt.Exp. 2019]. The perceptron’s efficiency is proportional to the electronic charge at the NLAF; in case of Silicon MZI modulators, this is ~10^6 charges hence the MAC/J is similar to TrueNorth. However, co-integration of emerging EO materials such as ITO into Si MZIs enables efficient modulation (e.g. VpL=0.5 V-mm [Armin et al, APL Phot. 2018]. Here we discuss latest results of a ITO-Silicon MZM with a record-low VpL=0.06 V-mm, and show noise-based NN training results of our in-house software PhotonFlow.

If electro-optic conversion of current photonic NNs could be postponed until the very end of the network, then the execution time is simply the photon time-of-flight delay. Here we discuss a first design and performance of an all-optical perceptron and feed-forward NN. Key is the dual-purpose foundry-approved heterogeneous integration of phase-change-materials resulting in a) volatile nonlinear activation function (threshold) realized with ps-short optical pulses resulting in a non-equilibrium variation of the materials permittivity, and b) thermo-optically writing a non-volatile optical multi-cell (5-bit) memory for the NN weights after being (offline) trained. Once trained, the weights only required a rare update, thus saving power. Performance wise, such an integrated all-optical NN is capable of < fJ/MAC using experimental demonstrated pump-probe [Waldecker et al, Nat. Mat. 2015] with a delay per perceptron being ~ps [Miscuglio et al. Opt.Mat.Exp. 2018] has a high cascadability.

Storing, processing, and learning from data is a central task in both industrial practice and modern science. Recent advances in modern statistical learning, particularly Deep Neural Networks (DNNs), have given record breaking performance on tasks in game playing,^{1, 2} natural language processing,³ computer vision,⁴ computational biology,^{5, 6} and many others. The rapid growth of the field has been driven by an increase in the amount of public datasets,⁷ improvements to algorithms,⁸ and a substantial growth in computing power.⁹ In order to perform well on these tasks networks have had to grow in size, learning more complicated statistical features. The training and deployment of these large neural networks has spurred the creation of many neural network accelerators to aid in the computation of these networks.^10-12

Existing general purpose computing devices such as CPUs and GPUs are limited both by thermal dissipation per unit area and yield associated with large chips.^{13, 14} The design of Application Specific Integrated circuits (ASICs) has aided in decreasing the energy consumption per workload substantially by limiting the supported operations on chip. An example of this is the first generation tensor processing unit (TPU)¹⁵ which is able to perform the inference of large convolutional neural networks in datacenter in <10ms with an idle power of 28W and an workload power of 40W. It may seen counterintuitive then that the limiting factor for the implementation of DNNs is not computation, but rather the energy and bandwidth associated with reading and writing data from memory as well as the energy cost of moving data inside of the ASIC.^{15, 16} Several emerging technologies, such as in-memory computing,¹⁷ memristive crossbar arrays¹⁸ promise increased performance, but these emerging architectures suffer from calibration issues and limited accuracy.¹⁹

Photonics as a field has had tremendous success in improving the energy efficiency of data interconnects.²⁰ This has motivated the creation of optical neural networks (ONNs) based on 3D-printed diffractive elements,²¹ spiking neural networks utilizing ring-resonators,²² reservoir computing²³ and nanophotonic circuits.²⁴ However, these architectures have several issues. 3D-printed diffractive networks and schemes requiring spatial light modulators are non-programmable, meaning that they are unable to perform the task of training. Nanophotonic circuits allow for an O(N²) array of interferometers to be programmed, providing passive matrix-vector multiplication. However, the large (≈1mm²) size of on chip electro-optic interferometers means that scaling to an array of 100x100 would require 10; 000mm² of silicon, demonstrating the limitations of scaling this architecture. To date no architecture has demonstrated high-speed (GHz) speed computation with more than N ≥ 10; 000 neurons.

Here we present an architecture that is scalable to N ≥ 10⁶ neurons. The key mechanism of this architecture is balanced homodyne detection. By scaling the architecture to such a large size we show that we can decimate energy costs per operation associated with the optical component of this architecture, reaching a bound set by shot noise on the receiving photodetectors which leads to classification error. We call this bound a standard quantum limit (SQL) which reaches 100zJ/MAC on problems such as MNIST. We also analyze the energy consumption using existing technologies and show that sub-fJ/MAC energy consumption should be possible.

This paper is organized as follows: In section 1 we will discuss the function of this architecture as a matrixmatrix processor. In section 2 we will analyze the energy consumption of the architecture. In section 3 we will discuss methods for training and extending the accelerator to a broader scope of problems, namely convolutionally neural networks (CNNs).

This Conference Presentation, Improved optical cryptosystem based on phase-shifting interferometry in a joint transform correlator was recorded at Photonics West 2020 held in San Francisco, California, United States.

The coherent Ising machine (CIM) is a network of optical parametric oscillators (OPOs) that solves for the ground state of Ising problems through OPO bifurcation dynamics. Here, we present experimental results comparing the performance of the CIM to quantum annealers (QAs) on two classes of NP-hard optimization problems: ground state calculation of the Sherrington-Kirkpatrick (SK) model and MAX-CUT. While the two machines perform comparably on sparsely-connected problems such as cubic MAX-CUT, on problems with dense connectivity, the QA shows an exponential performance penalty relative to CIMs. We attribute this to the embedding overhead required to map dense problems onto the sparse hardware architecture of the QA, a problem that can be overcome in photonic architectures such as the CIM.

Performing feature extractions in convolution neural networks for deep-learning tasks is computational expensive in electronics. Fourier optics allows convolutional filtering via dot-product multiplication in the Fourier domain similar to the distributive law in mathematics. Here we experimentally demonstrate convolutional filtering exploiting massive parallelism (10^6 channels, 8-bit at 1kHz) of digital mirror display technology, thus enabling 250 TMAC/s. An FPGA-PCIe board controls the ‘weights’ and handles the data I/O, whereas a high-speed camera detects the inverse-Fourier transformed (2nd lens) data. Gen-1 processes with a total delay (including I/O) of ~1ms, while Gen-2 at 1-10ns leveraging integrated photonics at 10GHz and changing the front-end I/O to a joint-transform-correlator (JTC). These processors are suited for image/pattern recognition, super resolution for geolocalization, or real-time processing in autonomous vehicles or military decision making.

Pulsed lidar provides a simple direct measure of range that can be scanned to produce accurate 3D point clouds for modest computational cost. A simple implementation, however, can be subject to point-cloud artifacts particularly when sensing scenes with high dynamic range.

We discuss the origins of some of these artifacts and some possible mitigations. Of particular interest is range aliasing. By creating multiple hypotheses for the range of any given return a practical system may accurately and precisely classify aliased returns..

This work describes a new method for handwriting forensic analysis using optical coherence tomography (OCT) images. The region of interest (ROI) in documents signature to be analyzed were pre-selected with a traditional optical analyzer (Docucenter 2000 – Projectina®) equipped with different wavelengths light sources and bandpass filters coupled detector. The ROI where then submitted to OCT to 3D imaging (OCP930SR, Thorlabs Inc.) with optical resolution of 6.0 microns. One original and 5 falsified signatures were produced by 5 different individuals, using a block of 5 sheets of white sulfite paper A4 size and weight of 75 g/m^2, using the same blue ballpoint pen and in the same surface. For pressure pen calibration, were developed a system computer controlled to maintain the pressure constant during writing linear traces, 15 different pressure configuration were applied to each linear trace, resulting in a relation between deformation and applied pressure. An autonomous algorithm was developed to make surface detection, across OCT tomographic images, correct the paper surface curvature, detect and measure deformation on paper due to pen pressure, generating a new image in false color from differential pressure, based in previous calibration. This new image can be considered as a “fingerprint pressure” signature. This methodology may assist the forensic expert in correctly determining the authorship of a signature.

One of the main bottlenecks of deep learning is the requirement for many training examples. In medical imaging, these examples are not always available. I will present our latest advances in the development of machine learning classification on interferometric phase microscopy (IPM) quantitative tomographic maps to obtain grading of cancer cells without staining. We first applied principle component analysis (PCA) followed by support vector machine (SVM) classifiers. To apply deep learning with small training sets, we proposed a new deep learning method, TOP-GAN, which is a hybridization between transfer learning and generative adversarial networks.

We introduce a differential measurement scheme in diffractive neural networks, where object-classes are assigned to separate opto-electronic detector pairs and the class-inference is made based on the maximum normalized differential signal. This scheme enables diffractive networks to achieve blind-testing accuracies of 98.54% and 48.51% for MNIST and CIFAR-10 datasets, respectively. These accuracies improve to 98.52% and 50.82%, when differential detection is combined with the joint-training of parallel diffractive networks, with each specializing on a separate object-class. Finally, we report independently-trained diffractive networks that project their output-light onto a common plane to achieve 98.59% and 51.44%, for the same datasets, respectively.

We demonstrate the application of deep learning for the identification of particles, directly from their backscattered light. The particles were illuminated using a single-mode fibre-coupled laser light source and the scattered light was collected by a 30-core optical fibre. The technique enabled identification of the specific species of pollen grains with an accuracy of ~97%, even in the presence of high levels of background light equivalent to daytime sunlight. In addition, the technique determined the distance between the fibre tip and the particles with an accuracy of ± 6 µm.

In this study, a smart rapid laser scanning system with 3D small object detection for disabling caterpillars has integrated. A monocular camera vision system was developed which works in tandem with rapid laser scanning system. Two caterpillar species Orgyia Postica and Porthesia Taiwana were considered and their original images were used to train YOLO for identification. Color transform from RGB to HSV was applied to Orgyia Postica, and for Porthesia Taiwana, RGB color space was maintained, to successfully detect the caterpillar’s head. The centre of the caterpillar’s head was positively approximated by using k-means clustering algorithm. These identified coordinates were then exposed to automatically controlled laser beam. Compact CW lasers of wavelength 450 nm with 1.738 W power with beam diameter 2.5 mm was used and their respective effects were studied. The entire setup was controlled using the NVIDIA Jetson TX2 embedded system. It was observed that even a precise exposure of second long laser beams on the head incapacitated the caterpillar from further ingestion of food. Therefore, this synergistic utilization of deep learning and lasers seems to be a beneficial and promising approach to effectively control the pest population, thereby preventing crop damage and improving the yield.

In coherent optical-fiber communication systems, polarization-division multiplexing is employed to double the transmission capacity. Polarization tracking based on digital signal processing (DSP) is used to cope with the polarization fluctuations of the light wave, which are caused by disturbances of the optical fibers. Usually, the polarization demultiplexing and polarization tracking are performed by using butterfly-structured finite impulse response (FIR) filters. We have proposed and investigated novel methods of polarization tracking using artificial neural networks (ANNs). An ANN can perform polarization demultiplexing because an ANN includes butterfly structures. Adaptive control of the weights of the ANN can be achieved by using decision directed least mean squares (DD-LMS) algorithm. Furthermore, the ANNs can potentially compensate waveform distortion caused by optical nonlinear effects such as self phase modulation (SPM) and cross-phase modulation (XPM). In this paper, we investigated the polarization tracking performance of the ANN under various conditions of polarization fluctuation speed by numerical simulations, comparing with that of FIR filters. Furthermore, we investigated the tracking performance depending on the number of input layer and hidden layer units of the ANN. The results show that the ANN can efficiently track the polarization fluctuation.

We compared the performance and the computational complexity of a time-domain- (TD-) artificial neural network (ANN) and a frequency-domain- (FD-) ANN used for nonlinearity compensation in optical fiber communication systems. In optical communication systems, ANNs have been used for coherent optical orthogonal frequency division multiplexing (CO-OFDM) transmission systems in the frequency domain. TD-ANN-based optical nonlinearity compensation has also been investigated in the last few years. For linear equalization to compensate for, e.g., chromatic dispersion (CD), it is known that FD-equalization outperforms TD-equalization in terms of computational complexity over a wide range of CD values. However, TD-ANNs and FD-ANNs have not been investigated in order to compare them in terms of computational complexity, to the best of our knowledge. In this paper, we investigated and compared the computational complexity of a TD-ANN and an FD-ANN which are used for optical nonlinearity compensation. We evaluated the number of complex multiplications needed for nonlinear compensation per symbol. The compensation performance was investigated using 16-ary quadrature amplitude modulation (16QAM) signal transmission over a standard single-mode fiber (SSMF) by numerical simulation. The results showed that the TD-ANN outperformed the FD-ANN in terms of computational complexity.

We investigated the problem of overfitting of artificial neural networks (ANNs) which are used for digital nonlinear equalizers in optical communication systems. ANN-based digital signal processing (DSP) can be used in the time domain or the frequency domain to compensate for the optical nonlinearity. We have reported the advantages of ANN over Volterra series transfer function (VSTF) in terms of the computational complexity. However, when pseudo-random binary sequence (PRBS) data is used to evaluate the ANN-based nonlinear equalizers, the ANNs can potentially learn the repeated PRBS patterns, resulting in overestimation of the equalization performance. In this paper, we clarify that the overfitting of the ANNs hardly occurs or requires much more neuron units than used for common nonlinear equalization, if we employ multi-level modulation signals including simple 4-ary pulse-amplitude modulation (PAM4). In our study, we compared binary signal and 4-level signal generated by PRBS data. White Gaussian noise (WGN) was added to the signals. The ANN-based nonlinear equalizers were trained using a least mean squares (LMS) algorithm, to attempt to “compensate” for the noise. The signal quality was evaluated using the error vector magnitude (EVM). When the overfitting occurs, the EVM is improved by the nonlinear equalization. The results revealed that, in the case of 4-level signal, the influence of the overfitting was suppressed in comparison with the case of the binary signal.

This paper suggests a new short wave infrared (SWIR)-imaging technique which can overcome these limitations. In addition to a two dimensional (2D) SWIR camera, the system also comprises a 2D visible light camera, an Inertial Measurement Unit (IMU), and global positioning system (GPS) to accurately determine the location of the leak using image correlation and triangulation techniques. The paper also suggests a low cost experimental setup used to assess the performance of the system to accurately quantify and localize CH4 gas leak. An Artificial Neural Network (ANN), was assessed using this setup. Series of extensive experimental tests demonstrate the capability of the system to detect, quantify, and localize CH4 gas leak for different scenarios. The corresponding results reveal that the ANN algorithm yields accurate results for gas mass leak measurement and localization using a SWIR optical filter. Uncertainties of gas mass leak flow measurement did not exceed 2.1% and 3.76% using a SWIR LED source with and without a SWIR filter respectively. This leads to state that the suggested system can be a tangible alternative for next generation leak detection systems.

This paper suggests a new concept of high GVF measurement and flow regime determination using a Terahertz-based imaging system. It explores the fact the gas phase has very low absorption of THz waves, while it yields an absorption factor that is proportional to the amount of liquid. The recent availability of low cost THz imaging systems that can generate two dimensional (2D) images at more than 100 frames/seconds make them well suitable for flow metering applications. A dedicated Support Vector Machine (SVM) was assessed using an in-house multiphase flow loop. The corresponding results reveal that the SVM yields very accurate results where a maximal error of 0.46 % for GVF in the GVF range from 80 to 100% could be achieved. In addition it could accurately determine all three type of flow regimes (i.e. annular, stratified, or slug flow). This suggests that the technique can be considered as a good candidate for next generation flow metering and imaging of multiphase flows.