This paper investigates the architectural requirements in simulating large neural networks using a highly parallel multiprocessor with distributed memory and optical interconnects. First, we model the structure of a neural network and the functional behavior of individual cells. These models are used to estimate the volume of messages that need to be exchanged among physical processors to simulate the weighted connections of the neural network. The distributed processor/memory organization is tailored to an electronic implementation for greater versatility and flexibility. Optical interconnects are used to satisfy the interprocessor communication bandwidth demands. The hybrid implementation attempts to balance the processing, memory and bandwidth demands in simulating asynchronous, value-passing models for cooperative parallel computation with self-learning capabilities.

A hardware implementation of a lightly connected artificial neural network known as the Hogg-Huberman model (1) (2) is described. The hardware is built around NCR's Geometric Arithmetic Parallel Processor (GAPP) chip. A large perfor-mance gain is shown between this implementation and a simulation done in FORTRAN on a VAX 11/780. Even though the direct processor to processor communications are limited to nearest neighbors, models which require other connections can be implemented with this hardware.

The neural net model introduced by Hogg and Huberman1/2 is examined and extended to include 2-dimensional inputs. Computer simulations of this model exhibit fault tolerant behavior in pattern recognition applications despite the local nature of the neural interconnectivity. This local interconnectivity, however, lends itself to VLSI and/or optical implementations. In this paper, previous work is reviewed and a mathematical characterization of the model is considered along with generalizations of the model to include the effects of changing the neuron computation functions. An electro-optical implementation of the 2-dimensional extension of the model will also be presented.

Logic programming is being used extensively by Artificial Intelligence researchers to solve problems including natural language processing and expert systems. These languages, of which Prolog is the most widely used, promise to revolutionize software engineering, but much greater performance is needed. Researchers have demonstrated the applicability of neural network models to the solution of certain NP-complete problems, but these methods are not obviously applicable to the execution of logic programs. This paper outlines the use of neural networks in four aspects of the logic program execution cycle, and discusses results of a simulation of three of these. Four neural network functional units are described, called the substitution agent, the clause filter, the structure processor, and the heuristics generator, respectively. Simulation results suggest that the system described may provide several orders of magnitude improvement in execution speed for large logic programs. However, practical implementation of the proposed architecture will require the application of optical computing techniques due to the large number of neurons required, and the need for massive, adaptive connectivity.

Partitioning of an object into N parts and the use of M filters with different output patterns are used to produce an NM digit symbolic encoding of the input object. The rule based system and techniques to update partitions of the object are emphasized in this paper. Three-dimensional aspect-invariant, shift-invariant and distortion-invariant pattern recognition data are considered and provided to demonstrate the usefulness of this technique for adaptive image processing.

Analog optical neural networks permit us to assemble continuous - input, continuous-output systems. These can have great advantages relative to their digital counterparts.

We consider various associative processor modifications required to allow these systems to be used for visual perception, scene analysis, and object recognition. For these applications, decisions on the class of the objects present in the input image are required and thus heteroassociative memories are necessary (rather than the autoassociative memories that have been given most attention). We analyze the performance of both associative processors and note that there is considerable difference between heteroassociative and autoassociative memories. We describe associative processors suitable for realizing functions such as: distortion invariance (using linear discriminant function memory synthesis techniques), noise and image processing performance (using autoassociative memories in cascade with with a heteroassociative processor and with a finite number of autoassociative memory iterations employed), shift invariance (achieved through the use of associative processors operating on feature space data), and the analysis of multiple objects in high noise (which is achieved using associative processing of the output from symbolic correlators). We detail and provide initial demonstrations of the use of associative processors operating on iconic, feature space and symbolic data, as well as adaptive associative processors.

A new recording technique for Hopfield-type associative/content address-able memories is proposed. The new technique is based on the finite and exponentially convergent algorithm of Ho and Kashyap[1] for the solution of a system of linear inequalities. Associative neural memories recorded with the proposed algorithm are shown to be superior to those recorded with the Hopfield's outer product and Kohonen's generalized inverse techniques. High capacity, high convergence rates to stored memories, and low convergence rates to false and oscillatory states are characteristics of this new recording algorithm. The issue of stable false and oscillatory states is raised, and it is shown that such states have a direct Boolean logic relationship with the stored memories.

This paper investigates the feasibility of constructing a Hopfield neural network model using optical techniques. The use of thin holographic elements to form the weighted interconnections is discussed as well as the desirable response of 'nonlinear processing elements. This implementation is of a bipolar vector model and processes the negative numbers involved without using a subtraction stage. Our analysis suggests that although present day technology would limit the size of the network that could be implemented, the construction of an optical machine is now possible with a connectivity exceeding any curently available electronic machine.

The performance of neural networks used for pattern recognition and classification may be improved by introducing some capacity for invariance into the network. Two measures of similarity and their relationship to the network architecture are discussed. A very efficient neural network that may be used not only as a content-addressable memory but as a general symbolic substitution network is discussed. In addition to invariance to input errors, invariance to translations and rotations are considered. This may be accomplished by modifying the network itself, or changing the interconnection scheme, or by means of some pre-processing of the input data. In some cases the preprocessing could be done by the network itself or by another network, or by optical means. The techniques discussed include the introduction of more input neurons, the preprocessing of data by means of invariant matched filters, the use of new invariant image representations and the projection of input data on stored invariant principal components. The trade-offs involved in the various proposed schemes are discussed.

Binary pattern classification that may be implemented using optical hardware and neural network algorithms is considered. Pattern classification problems that have no concise description (as in classifying handwritten characters) or no concise computation (as in NP-complete problems) are expected to be particularly amenable to this approach. For example, optical processors that efficiently classify binary patterns in accordance with their Boolean function complexity might be designed. As a candidate for such a design, an optical neural network model is discussed that is designed for binary pattern classification and that consists of an optical resonator with a dynamic multiplex-recorded reflection hologram and a phase conjugate mirror with thresholding and gain. In this model, learning or training examples of binary patterns may be recorded on the hologram such that one bit in each pattern marks the pattern class. Any input pattern, including one with an unknown class or marker bit, will be modified by a large number of parallel interactions with the reflection hologram and nonlinear mirror. After perhaps several seconds and 100 billion interactions, a steady-state pattern may develop with a marker bit that represents a minimum-Boolean-complexity classification of the input pattern. Computer simulations are presented that illustrate progress in understanding the behavior of this model and in developing a processor design that could have commanding and enduring performance advantages compared to current pattern classification techniques.

Optical processor architectures for various forms of the alternating projection neural network (APNN) are considered. Required iteration is performed by passive optical feedback using only free space and guided propagation. No electronics or slow optics (e.g. phase conjugators) are used. The processor can be taught a new training vector by viewing it only once.

The neural computing scheme of image reconstruction by the human visual system has been modeled by multi-scale zero-crossings as unique representations of bandlimited polynomial functions. The exact analytical development of such a model and its computer simulation are quite complex tasks. We propose a novel scheme for optical implementation of image reconstruction by synthesizing optical filters involving multiple orthogonal channels. Alternatively the zero crossing operator, i.e. the LOG (Laplacian of Gaussian) operator can also be implemented in a specially designed associative network. A combination of optical implementation and computer simulation of this image reconstruction model may provide exciting insight into the neural mechanisms in the human visual system as well as lead to the development of a real time hybrid signal processing system.

A quadratic neural network in which neural activity at each neuron is determined not only by every other neuron, but also by every other neuron-pair, is implemented optically. This quadratic network can be programmed through the interconnection weights and is also capable of realizing higher order nonlinear networks.

To us, and to other biological organisms, vision seems effortless. We open our eyes and we "see" the world in all its color; brightness, and movement. Yet, we havegreat difficulties when trying to endow our machines with similar abilities. In this paper we shall describe recent developments in the theory of early vision which lead from the formulation of the motion problem as an ill-posed one to its solution by minimizing certain "cost" functions. These cost or energy functions can be mapped onto simple analog and digital resistive networks. Thus, we shall see how the optical flow can be computed by injecting currents into resistive networks and recording the resulting stationary voltage distribution at each node. These networks can be implemented in cMOS VLSI circuits and represent plausible candidates for biological vision systems. This manuscript is a condensed version of ref.' .

Many articles on neural networks focus on learning, and restrict themselves to a limited class of simple neurons. The present paper is a tutorial designed, by contrast, to emphasize the "domain-specific" structure of neural networks, as well as to emphasize that technologists have much to learn from the study of neurobiological systems. The paper does not discuss optical implementations the author would appreciate comments on the feasibility of such implementations for the neural (and schema) networks described here.

The drive-reinforcement neuronal model is described as an example of a newly discovered class of real-time learning mechanisms that correlate earlier derivatives of inputs with later derivatives of outputs. The drive-reinforcement neuronal model has been demonstrated to predict a wide range of classical conditioning phenomena in animal learning. A variety of classes of connectionist and neural network models have been investigated in recent years (Hinton and Anderson, 1981; Levine, 1983; Barto, 1985; Feldman, 1985; Rumelhart and McClelland, 1986). After a brief review of these models, discussion will focus on the class of real-time models because they appear to be making the strongest contact with the experimental evidence of animal learning. Theoretical models in physics have inspired Boltzmann machines (Ackley, Hinton, and Sejnowski, 1985) and what are sometimes called Hopfield networks (Hopfield, 1982; Hopfield and Tank, 1986). These connectionist models utilize symmetric connections and adaptive equilibrium processes during which the networks settle into minimal energy states. Networks utilizing error-correction learning mechanisms go back to Rosenblatt's (1962) perception and Widrow's (1962) adaline and currently take the form of back propagation networks (Parker, 1985; Rumelhart, Hinton, and Williams, 1985, 1986). These networks require a "teacher" or "trainer" to provide error signals indicating the difference between desired and actual responses. Networks employing real-time learning mechanisms, in which the temporal association of signals is of fundamental importance, go back to Hebb (1949). Real-time learning mechanisms may require no teacher or trainer and thus may lend themselves to unsupervised learning. Such models have been extended by Klopf (1972, 1982), who introduced the notions of synaptic eligibility and generalized reinforcement. Sutton and Barto (1981) advanced this class of models by proposing that a derivative of the theoretical neuron's out-put be utilized as a reinforcement signal. Klopf (1986) has recently extended the Sutton-Barto (1981) model, yielding a learning mechanism that correlates earlier derivatives of the theoretical neuron's inputs with later derivatives of the theoretical neuron's output. Independently, Kosko (1986) has also discovered this new class of differential learning mechanisms. Kosko (1986), approaching from philosophical and mathematical directions, and Klopf (1986), approaching from the directions of neuronal modeling and animal learning research, came to the same conclusion: correlating earlier derivatives of inputs with later derivatives of outputs may constitute a fundamental improvement over a Hebbian correlation of approximately simultaneous input and output signals. Klopf's version of the learning mechanism, termed a drive-reinforcement model, has been demonstrated to predict a wide range of classical conditioning phenomena in animal learning. This will be illustrated with results of computer simulations of the drive-reinforcement neuronal model and with a videotape of a simulated network of drive-reinforcement neurons controlling a simulated robot operating in a simulated environment.

We present coupled mode equations for a holographic resonator consisting of a volume-holographic recording medium bounded by two phase-conjugating mirrors. We exhibit forms for the binary and higher order mode couplings that produce associative memory, time-sequence memory, multi-associative memory and branching behavior, and indicate how such couplings may be "programmed into" existing (or conceivable) nonlinear optical materials.

The storage capacity for neural interconnections in photorefractive crystals and the use of the dynamic nature of the photorefractive effect to train these interconnections are discussed. We describe an optical neural architecture which uses the characteristics of the photorefractive response to implement error driven learning and describe a modified perceptron algorithm which we have used to train this optical system.

Variation in the response of hardware components causes analog implementations of neural networks with soft nonlinearities to corrupt their signals with noise. In this paper we simulate the behavior of analog implementations of multiplicative and additive winner-take-all lateral-inhibitory-networks (WTA-LIN) for different levels of component variation. We observe that for an acceptable level of performance the amplitude resolution and number of the initial neuron activities are constrained by the standard deviation of the component nonuniformity.

We describe an optoelectronic resonator associative memory system which utilizes holographic interconnects. Image processing techniques are used to implement nonlinearities and feedback. We show using numerical models that both power law and sigmoidal nonlinearities improve the storage capacity. Our experimental results lead us to be optimistic that this hybrid optical/electronic approach can be extended to adaptive neural network models.

In order to produce well-focused images of moving targets using radar, distortions in the radar echos due to the target's motion should be compensated for. Techniques exist to compensate distortion due to a target's changing range but compensation for non-uniformity in the target's aspect is generally neglected. We present an iterative technique and its optical implementation for performing aspect compensation.

Electronic competiton for a fixed current resource can be used to mediate a winner-take-all operation on an array of optical beams, using a special-purpose optoelectronic integrated circuit containing modulators and detectors. Reflective competiton between an array of optical beams is proposed as the basis for an unsupervised, competitive optical learning architecture. It is suggested that self-aligning, adaptive interconnection holograms can be written in photorefractive crystals which are repeatedly exposed to the interference between phase-conjugated object waves and competitively produced reference beams. As statistically clustered patterns are input to the optical network, specific output nodes should learn to respond to the various classes of input without supervision.

A learning algorithm based on temporal difference of membrane potential of the neuron is proposed for self-organizing neural networks. It is independent of the neuron nonlinearity, so it can be applied to analog or binary neurons. Two simulations for learning of weights are presented; a single layer fully-connected network and a 3-layer network with hidden units for a distributed semantic network. The results demonstrate that this potential difference learning (PDL) can be used with neural architectures for various applications. Unlearning based on PDL for the single layer network is also discussed. Finally, an optical implementation- of PDL is proposed.