Most research in technologies for the Deaf community have focused on translation using either video or wearable devices. Sensor-augmented gloves have been reported to yield higher gesture recognition rates than camera-based systems; however, they cannot capture information expressed through head and body movement. Gloves are also intrusive and inhibit users in their pursuit of normal daily life, while cameras can raise concerns over privacy and are ineffective in the dark. In contrast, RF sensors are non-contact, non-invasive and do not reveal private information even if hacked. Although RF sensors are unable to measure facial expressions or hand shapes, which would be required for complete translation, this paper aims to exploit near real-time ASL recognition using RF sensors for the design of smart Deaf spaces. In this way, we hope to enable the Deaf community to benefit from advances in technologies that could generate tangible improvements in their quality of life. More specifically, this paper investigates near real-time implementation of machine learning and deep learning architectures for the purpose of sequential ASL signing recognition. We utilize a 60 GHz RF sensor which transmits a frequency modulation continuous wave (FMWC waveform). RF sensors can acquire a unique source of information that is inaccessible to optical or wearable devices: namely, a visual representation of the kinematic patterns of motion via the micro-Doppler signature. Micro-Doppler refers to frequency modulations that appear about the central Doppler shift, which are caused by rotational or vibrational motions that deviate from principle translational motion. In prior work, we showed that fractal complexity computed from RF data could be used to discriminate signing from daily activities and that RF data could reveal linguistic properties, such as coarticulation. We have also shown that machine learning can be used to discriminate with 99% accuracy the signing of native Deaf ASL users from that of copysigning (or imitation signing) by hearing individuals. Therefore, imitation signing data is not effective for directly training deep models. But, adversarial learning can be used to transform imitation signing to resemble native signing, or, alternatively, physics-aware generative models can be used to synthesize ASL micro-Doppler signatures for training deep neural networks. With such approaches, we have achieved over 90% recognition accuracy of 20 ASL signs. In natural environments, however, near real-time implementations of classification algorithms are required, as well as an ability to process data streams in a continuous and sequential fashion. In this work, we focus on extensions of our prior work towards this aim, and compare the efficacy of various approaches for embedding deep neural networks (DNNs) on platforms such as a Raspberry Pi or Jetson board. We examine methods for optimizing the size and computational complexity of DNNs for embedded micro-Doppler analysis, methods for network compression, and their resulting sequential ASL recognition performance.
3D scene reconstruction provides an improved representation from which features of critical objects or targets may be extracted. Both electro-optical (EO) and synthetic aperture radar (SAR) sensors have been exploited for this purpose, but each modality possesses issues resulting in different sources for reconstruction errors. Reconstruction from EO data is limited by frame rate and can be blurred by moving targets or optical distortions in the lens, which leads to errors in the 3D model. Meanwhile, SAR offers the opportunity to correct from some of these errors through its capacity for making range measurements, even under clouds or during nighttime, when EO data would not be available. Conversely, SAR imagery lacks the texture offered by optical images and is more sensitive to perspective, while moving targets can likewise result in reconstruction errors. This work aims at exploiting the strengths of both modalities to reconstruct 3D scenes from multi-sensor EO-SAR data. In particular, we consider the fusion of multi-pass Gotcha SAR data with a modeled EO-data for the particular scene. We propose a framework that fuses 2D image maps acquired from airborne EO data as well as airborne SAR, which leverages the range information of SAR and object shape information of EO imagery. From an initial 2D image of the scene, with each additional sources of sensor data (EO or SAR), a 3D reconstruction is formed that is iteratively improved. This approach allows for the potential to achieve robust and real-time 3D representations as a basis for 4D surveillance.
A critical limitation in the application of deep learning to radar signal classification is the lack of sufficient data to train very deep neural networks. The depth of a neural network is one of the more significant network parameters that affects achievable classification accuracy. One way to overcome this challenge is to generate synthetic samples for training deep neural networks (DNNs). In prior work of the authors, two methods have been developed: 1) diversified micro-Doppler signature generation via transformations of the underlying skeletal model derived from video motion capture (MOCAP) data, and 2) auxiliary conditional generative adversarial networks (ACGANs) with kinematic sifting. While diversified MOCAP has the advantage of greater accuracy in generating signatures that span to the probable target space of expected human motion for different body sizes, speeds, and individualized gait, the method cannot capture data artifacts due to sensor imperfections or clutter. In contrast, adversarial learning has been shown to be able to capture non-target related artifacts, however, the ACGANs can also generate misleading signatures that are kinematically impossible. This paper provides an in-depth performance comparison of the two methods on a through-the-wall radar data set of human activities of daily living (ADL) in the presence of clutter and sensor artifacts.
Deep neural networks have become increasingly popular in radar micro-Doppler classification; yet, a key challenge, which has limited potential gains, is the lack of large amounts of measured data that can facilitate the design of deeper networks with greater robustness and performance. Several approaches have been proposed in the literature to address this problem, such as unsupervised pre-training and transfer learning from optical imagery or synthetic RF data. This work investigates an alternative approach to training which involves exploitation of “datasets of opportunity" micro-Doppler datasets collected using other RF sensors, which may be of a different frequency, bandwidth or waveform - for the purposes of training. Specifically, this work compares in detail the cross-frequency training degradation incurred for several different training approaches and deep neural network (DNN) architectures. Results show a 70% drop in classification accuracy when the RF sensors for pre-training, fine-tuning, and testing are different, and a 15% degradation when only the pre-training data is different, but the fine-tuning and test data are from the same sensor. By using generative adversarial networks (GANs), a large amount of synthetic data is generated for pre-training. Results show that cross-frequency performance degradation is reduced by 50% when kinematically-sifted GAN-synthesized signatures are used in pre-training.
Cognitive radar is a novel concept for next-generation radar systems, which as part of the perception-action cycle to improve the measurement process based on dynamic changes in the environment. Although most work in this area to-date have focused on adaptation on the transmitted waveform, in this paper, we propose adaptive control of novel multifunctional reconfigurable antennas (MRAs) as a mechanism for action within the cognitive radar framework. Reconfigurable parasitic layer based MRAs have the capability of dynamically and simultaneously changing its electromagnetic characteristics (mode of operation), e.g. antenna beam pattern, polarization, center frequency, or a combination of thereof. Different modes of an MRA are controlled via RF switches interconnecting the pixels of the reconfigurable parasitic layer. This enhanced capability can be controlled using adaptive mode selection schemes. In particular, an array of MRAs provides more degrees of freedom, where each element of an array can be controlled to generate one of many modes depending on the environmental measured variables as a feedback mechanism. In this work, a designed and fabricated reconfigurable parasitic layer based MRA operating over 4.94-4.99 GHz band with 25 different radiation patterns, i.e., modes of operation, is utilized for cognitive direction-of-arrival (DoA) estimation and target tracking. A novel computationally efficient iterative mode selection (IMS) technique for MRA arrays is developed, where the modes are cognitively selected to minimize the DoA estimation error in target track. It is demonstrated that the proposed cognitive mode selection for MRA arrays achieves remarkably lower estimation errors compared to uniform pattern arrays without adaptive capability.
Automatic target recognition (ATR) using micro-Doppler analysis is a technique that has been a topic of great research over the past decade, with key applications to border control and security, perimeter defense, and force protection. Patterns in the movements of animals, humans, and drones can all be accomplished through classification of the target’s micro-Doppler signature. Typically, classification is based on a set of fixed, pre-defined features extracted from the signature; however, such features can perform poorly under low signal-to-noise ratio (SNR), or when the number and similarity of classes increases. This paper proposes a novel set of data-driven frequency-warped cepstral coefficients (FWCC) for classification of micro-Doppler signatures, and compares performance with that attained from the data-driven features learned in deep neural networks (DNNs). FWCC features are computed by first filtering the discrete Fourier Transform (DFT) of the input signal using a frequency-warped filter bank, and then computing the discrete cosine transform (DCT) of the logarithm. The filter bank is optimized for radar using genetic algorithms (GA) to adjust the spacing, weight, and width of individual filters. For a 11-class case of human activity recognition, it is shown that the proposed data-driven FWCC features yield similar classification accuracy to that of DNNs, and thus provides interesting insights on the benefits of learned features.
The human micro-Doppler signature is a unique signature caused by the time-varying motion of each point on the human
body, which can be used to discriminate humans from other targets exhibiting micro-Doppler, such as vehicles, tanks,
helicopters, and even other animals. Classification of targets based on micro-Doppler generally involves joint timefrequency
analysis of the radar return coupled with extraction of features that may be used to identify the target.
Although many techniques have been investigated, including artificial neural networks and support vector machines,
almost all suffer a drastic drop in classification performance as the aspect angle of human motion relative to the radar
increases. This paper focuses on the use of radar networks to obtain multi-aspect angle data and thereby ameliorate the
dependence of classification performance on aspect angle. Knowledge of human walking kinematics is exploited to
generate a fuse spectrogram that incorporates estimates of model parameters obtained from each radar in the network. It
is shown that the fused spectrogram better approximates the truly underlying motion of the target observed as compared
with spectrograms generated from individual nodes.
Radar offers unique advantages over other sensors, such as visual or seismic sensors, for human target detection.
Many situations, especially military applications, prevent the placement of video cameras or implantment seismic
sensors in the area being observed, because of security or other threats. However, radar can operate far away
from potential targets, and functions during daytime as well as nighttime, in virtually all weather conditions. In
this paper, we examine the problem of human target detection and identification using single-channel, airborne,
synthetic aperture radar (SAR). Human targets are differentiated from other detected slow-moving targets by
analyzing the spectrogram of each potential target. Human spectrograms are unique, and can be used not
just to identify targets as human, but also to determine features about the human target being observed, such
as size, gender, action, and speed. A 12-point human model, together with kinematic equations of motion
for each body part, is used to calculate the expected target return and spectrogram. A MATLAB simulation
environment is developed including ground clutter, human and non-human targets for the testing of spectrogram-based
detection and identification algorithms. Simulations show that spectrograms have some ability to detect
and identify human targets in low noise. An example gender discrimination system correctly detected 83.97%
of males and 91.11% of females. The problems and limitations of spectrogram-based methods in high clutter
environments are discussed. The SNR loss inherent to spectrogram-based methods is quantified. An alternate
detection and identification method that will be used as a basis for future work is proposed.