Generating imagery using gaming engines has become a popular method to both augment or completely replace the need for real data. This is due largely to the fact that gaming engines, such as Unity3D and Unreal, have the ability to produce novel scenes and ground-truth labels quickly and with low-cost. However, there is a disparity between rendering imagery in the digital domain and testing in the real domain on a deep learning task. This disparity/gap is commonly known as domain mismatch or domain shift, and without a solution, renders synthetic imagery impractical and ineffective for deep learning tasks. Recently, Generative Adversarial Networks (GANs) have shown success at generating novel imagery and overcoming this gap between two different distributions by performing cross-domain transfer. In this research, we explore the use of state-of-the-art GANs to perform a domain transfer between a rendered synthetic domain to a real domain. We evaluate the data generated using an image-to-image translation GAN on a classification task as well as by qualitative analysis.
Rendering synthetic imagery from gaming engine environments allows us to create data featuring any number of object orientations, conditions, and lighting variations. This capability is particularly useful in classification tasks, where there is an overwhelming lack of labeled data needed to train state-of-the-art machine learning algorithms. However, the use of synthetic data is not without limit: in the case of imagery, training a deep learning model on purely synthetic data typically yields poor results when applied to real world imagery. Previous work shows that "domain adaptation," mixing real-world and synthetic data, improves performance on a target dataset. In this paper, we train a deep neural network with synthetic imagery, including ordnance and overhead ship imagery and investigate a variety of methods to adapt our model to a dataset of real images.
Explosive Ordnance Disposal (EOD) technicians are on call to respond to a wide variety of military ordnance. As experts in conventional and unconventional ordnance, they are tasked with ensuring the secure disposal of explosive weaponry. Before EOD technicians can render ordnance safe, the ordnance must be positively identified. However, identification of unexploded ordnance (UXO) in the field is made difficult due to a massive number of ordnance classes, object occlusion, time constraints, and field conditions. Currently, EOD technicians collect photographs of unidentified ordnance and compare them to a database of archived ordnance. This task is manual and slow - the success of this identification method is largely dependent on the expert knowledge of the EOD technician. In this paper, we describe our approach to automatic ordnance recognition using deep learning. Since the domain of ordnance classification is unique, we first describe our data collection and curation efforts to account for real-world conditions, such as object occlusion, poor lighting conditions, and non-iconic poses. We apply a deep learning approach using ResNet to this problem on our collected data. While the results of these experiments are quite promising, we also discuss remaining challenges and potential solutions to deploying a real system to assist EOD technicians in their extremely challenging and dangerous role.