3.1

Learning in Complex Data Environments

The US Army’s operational environment is truly complex and often unpredictable. New AI & ML methods are needed that effectively handle datasets that have few samples, noisy data and operate in environments that contain a lot of “clutter” and “congestion” beyond just physical entities but also in terms cyberspace and social information. Not only are these data hard to collect ahead of time and still more difficult to process, but the data consists of a wide variety of data types from audio, video, text, network traffic, social media, news, wireless transmissions, and many other heterogeneous forms of information. Some of the events and information flows in this environment (e.g.megacity) may be natural and expected, while others may be deceptive and adversarial—intentionally created to thwart the Army’s progress. Contested and deceptive environments will be especially challenging when dealing with sophisticated and near-peer adversaries, spawning the need for AI and ML methods that are not only robust and resilient but can also infer, quantify, and reason about the uncertainty of various situations [12].

3.2

Resource Constrained Processing at the Point of Need

Unlike the commercial AI and ML application pipeline which largely depends upon large and stable training data as well as fairly static operational environments with access to significant computing resources, the operational environment of the military is one of of learning in complex, dynamic and under largely unpredictable circumstances. Access to powerful cloud computing facilities will likely be very limited or completely unavailable in military contexts. Power is always at a premium in the battlefield, and physical loading must be optimized to ensure that the weight and size of components can be supported by the mission. In the future battlefield consisting of many interconnected devices, this internetwork of “battlefield things” will require algorithms that run effectively in a distributed manner over a potentially large set of less powerful CPUs working with unreliable connectivity to solve complex learning problems and to provide useful, timely analytics and services to the warfighter [12,13].

3.3

Predictable and Generalizable AI

As mentioned earlier, the quantification of uncertainty is a requirement for AI & ML methods that will be employed in the future battlefield. This implies that these techniques need to be predictable so that failures are not catastrophic and performance limits are understood. Today’s commercial AI and ML methods tend to be specific and narrow, and not general. Research is needed to create methods that generalize and that can transfer learning from one model to another, even in another domain. This will require methods that have built-in quantitative models that not only produce an “optimal” decision or assessment based a set of observations, but need to have some notion of the accuracy of the recommendation or assessment and explain its reasoning process, providing justification for the course of action to the soldier. It may even be possible to have provisions for human-in-the-loop training that will allow algorithms to be programmed to meet the performance requirements of specific missions [12, 14].

To summarize, the overarching goal for Army research in AI and ML is to develop artificially intelligent agents that are both heterogeneous and distributed; and that rapidly learn, adapt, reason, and act in contested, austere, and congested environments [12].

4.1

Lack Enormous, Relevant Datasets with Labeled Examples

When examining the state of the art of AI & ML research, it is apparent that large amounts of example data are required in order to train many of the most successful algorithms [16]. In military contexts such massive datasets are rarely available. Even when there is a large amount of collected data, often it is not labeled and, therefore, not suitable for training algorithms that depend on the paradigm of supervised learning. With supervised learning, the learning algorithm must be presented with examples that each belong to a known class. The algorithm then trains on the labeled examples. Unfortunately, many algorithms such as convolutional neural networks with deep learning do not perform well when presented with sparse, incomplete datasets rendering them virtually useless for supporting military operational scenarios.

Even under the most ideal conditions, such as with self-driving cars and extremely large training sets, there have been situations where a human driver would have likely avoided an accident where the self-driving car failed. Ideally, new breakthroughs in AI and ML research could help to address this issue.

4.2

Batch-Based Processing Algorithms that Work Poorly with Sparse Data

Another important limitation of many commercial AI & ML algorithms is their dependence on “batch processing” of training datasets. With batch processing all training data is required at once in order for the learning algorithm to build a useful model that can be deployed into an operational setting. These “learned models”, rather than those handcrafted from domain expertise and first principles, have gained increasing relevance for a variety of engineering and pattern recognition tasks, from speech recognition to autonomous control of complex robotic platforms. The dominant approach to learning methods is based on neural networks, whose training requires optimization over millions of parameters and data points in advance of their use. This training procedure further requires cloud infrastructure to store massive amounts of data. When neither extensive data nor cloud services are available, learning in situ is required, for which training deep networks is intractable. The question then becomes how to design learning methods that can learn stably online while performing well after only tens to hundreds of data points, and strike at the benchmarks of deep learners as the number of observations becomes very large.

4.3

Dependence on Supervised Learning Algorithms

With supervised learning, exemplar training data is required that is completely representative of all operational scenarios that the trained agent will face. As discussed earlier, collection of this magnitude and quality of training data is generally not realistic for Army-relevant operations. Nevertheless, many of the recent successful machine learning algorithms are based on supervised learning [17]. In supervised learning systems, the learning agent is provided with the correct answer or “ground truth” label for each example input in the dataset—that is, mapping a set of inputs, X, to a set of outputs, Y. In the example provided earlier, AlexNet uses a deep convolutional neural network to classify classify 1.2 million high-resolution images into 1000 different class achieving better that previous state of the art error rates [4]. Ideally, military applications of machine learning should not depend solely on supervised methods and if possible newer techniques that use semi-supervised or unsupervised approaches should be utilized. However, the current state of the art unsupervised learning approaches are not sufficiently robust to handle militarily relevant situations. In order to operationalize AI and ML, research is needed to develop new semi-supervised and unsupervised learning algorithms that less susceptible to noise and can effectively and accurately generalize with either sparsely labeled or totally unlabeled data sets.

4.4

Inability to Operate Autonomously and Reason about the Environment

Intelligent agents need a lot of human supervision and cannot operate in the environment autonomously. Most robotics currently used in the field are teleoperated by soldiers and even if programmed to operate in an autonomous manner, most robots are unpredictable and unable to adapt to unexpected environmental conditions. Current intelligent systems cannot reason about or learn from the environment in which they operate. As conditions change, intelligent agents need to adjust to the changing conditions. For humans, this comes naturally. For example, when a person walks outside on a sunny, dry day and suddenly it begins to rain, they will sense the that the ground surface seems slippery and automatically adjust the way they walk. Likewise, on a cold winter day if icy patches appear on a sidewalk, people will deliberately change the way they take their steps, differently than they would in the rain, perhaps taking each step tentatively until they get an intuitive sense of the surface traction, automatically anticipating the possibility of slipping and falling. This innate ability is partially pre-wired and partially learned from the time the person is a small child just learning to walk. No one has to tell a human what to do in this instance, they learn through experience. Likewise, if the driver of a car senses that the road feels slippery, they will adjust their speed accordingly. An inexperienced driver may slam on the brakes the first time they encounter a slick road and need to stop, but will quickly learn that this may cause loss of control and will adjust future behavior accordingly. In fact, once a driver has gained experience, they may see glaze on the road surface and automatically change the way they drive in anticipation of potentially dangerous road conditions. Unfortunately, this ability to learn from and reason about the environment is lacking in today’s robots. In fact, most robots are tethered to humans via a wired or wireless links. If they get stuck, humans must intervene. In reality, most robots are a remote controlled extension of the human who must focus a significant amount of their attention on the task of controlling the robot. Intelligent agents have difficultly learning from a human operator either by example or by direct feedback. Instantiating truly autonomous capabilities into robots remains a challenging problem and active area of research.

4.5

Lack of Common Sense

In order for robots to operate effectively as teammates for soldiers, they will not only have to reason about the environment but they also need to have an innate “common sense,” such that a soldier can ask a robot to complete a task and robot will understand how to properly carry out the order. For example, a soldier may ask a robot to explore an area and report back if they find anything “of interest”. This task would be straightforward for humans, who will identify things such as weapons, people in the area, vehicles, buildings, and any of a number of “interesting” facts that cannot be anticipated in advance. Basic research is needed to in order for intelligent agents to be able to understand the “intent” of the commander’s orders; furthermore, the intelligent agent should be able to ask questions in order to clarify instructions that it has been given. Part of the reason this problem exists is that intelligent agents don’t have a “shared understanding” of the situation with the human commander. Research breakthroughs are required that will help an intelligent agent to share a common “frame of reference” about the environment and the operational context with the human. For example, the robot should know if the current order should be executed in a normal or urgent manner.

Often, AI and ML algorithms can be used to make decisions about or classify a situation but are not able to explain what particular attributes drove the algorithm to come to that decision. This ability to “explain” is lacking in most AI and ML methods.

Not only do intelligent agents such as robots lack common sense, they rarely have the ability to explain their decisions or even communicate in a straightforward way with their commanders. If these systems could use natural language communication and have the ability to query the commander for clarification this would provide for smoother more seamless operation in the field and even lead to the possibility of an intelligent agent to “explain” their decision process and let the human commander know how a decision was reached or the agent could seek clarifying information.

Furthermore, AI and ML methods tend to be “brittle” and experience severe failures that one would not expect from a human being. For example, there are many papers that demonstrate the apparent ease of tricking CNNs into thinking an image belongs to a completely different unrelated class. For example, in one paper an image that is obviously a “school bus” with a little bit added, but imperceptible pixel noise, is classified as an “ostrich” [18]. For a military context, this would be a catastrophic failure and a mistake that is extremely unlikely for a human to make. The fact that a few pixels in the image need to be manipulated in order to fool the CNN is troubling and represents a vulnerability that adversaries could exploit. This issue is exacerbated due to the inability of the AI/ML to explain how it came to a final classification decision thereby making it virtually impossible to understand why the image was incorrectly classified.

4.6

Summary of Gaps

Thus far we have posited 5 AI and ML gaps for military relevant operations. The rest of the paper reviews some of the basic research that is being undertaken by ISD in order to address these challenges. To review, the gaps discussed were:

5.1

Collection of Relevant Datasets (when possible)

Although the US military collects and possesses a lot of data, it is very challenging to label enough data to train AI and ML algorithms thoroughly such that systems will operate effectively in military scenarios. Perhaps the most famous example of data collection in the commercial world is ImageNet, with over 14 million images that have been labeled by humans [4, 9]. Although data collection is difficult and some question whether it is even possible to collect enough data to feed state-of-the-art algorithms such that they would perform well under militarily operational conditions, the DoD is working to do just that. In April 2017, Robert Work, former Deputy Secretary of Defense, announced the formation of Project Maven and established a an Algorithmic Warfare Cross-Functional Team (AWCFT) that is working towards fielding technology to augment and automate the processing, exploitation, and dissemination of data for tactical Unmanned Aerial System (UAS) and Mid-altitude Full-Motion Video (FMV). [20,21] In addition to efforts to automate the labeling of data, the AWCFT enlisted ARL’s ISD and other ARL researchers to advise on automatic labeling techniques and to help consolidate existing algorithm-based technology initiatives related to mission areas of the Defense Intelligence Enterprise, including initiatives that develop, employ, or field artificial intelligence, automation, machine learning, deep learning, and computer vision algorithms. The overarching objective of Project Maven is to “turn the enormous volume of data available to DOD into actionable intelligence and insights at speed.”

ARL also maintains internal efforts to collect data for research and to train AI and ML algorithms through various research programs. One such facility is the Intelligent Systems Center (ISC), which is part of ARL’s Open Campus initiative with the goal of will facilitating innovation and encouraging cross-disciplinary research with a focus on long-term basic and applied research. The Center will leverage the strength of its current research program by focusing on systems that interact with the physical world [22]. The ISC is a highly collaborative environment with cross-discipline opportunities spanning many areas touching AI and ML including traditional robotics (intelligence, perception, and mobility/manipulation), adaptive control, autonomous networking, cognitive architectures, natural language, semantics and reasoning with facilities for test, evaluation, validation and verification. The ISC has many Robotic experimental platforms (ground and air) as well as an intelligent algorithm software repository, unique military datasets and simulation tools as well as unique facilities for experimentation for ground-and air-based platforms.

In addition, ARL hosts the Automated Online Data Repository (AODR) and Sensor Information Testbed for Collaborative Research Environment (SITCORE), which are both used by ISC researchers. ARL and other government organizations, industry, and academia use AODR to store and disseminate multiple intelligence (Multi-INT) datasets collected at field exercises and demonstrations, facilitates research and advancement of analytical tools and algorithms supporting the Intelligence, Surveillance, and Reconnaissance (ISR) community. SITCORE is a virtual research development environment allowing collaboration from other locations, including DoD, industry, academia, and coalition partners. SITCORE combined with AODR provides a comprehensive end-to-end algorithm development, experimentation, demonstration, and validation platform [23].

5.2

Online, Anytime, Continuous Learning and Learning with Sparse Examples

Notwithstanding current Army efforts to collect and label large amounts of relevant data, there remain many situations where collection of adequate data required by many of the mainstream AI and ML algorithms will not be possible. For example, operations in enemy territories that are very dissimilar in nature to the archived environmental data in the Army’s repositories could be hindered if modern AI and ML methods are utilized. Creating scenarios representative of the wide variety of operational situations that could be encountered by an intelligent system on some future mission is just not possible. Some early research is underway to simulate data in order to generate large and rich training sets that are derived from existing data. Although promising, it is not yet clear whether or not this approach will provide examples that will enable algorithms to perform well under operational conditions. From ISD’s perspective, what is required are new algorithms that are capable of operating simultaneously in online learning and operating mode, receiving data one sample at a time and processing a small subset of the total observed set of samples without the need for the entire dataset to be present for training, as is the case for batch mode learning.

ISD has recently published research on a new algorithm called Parsimonious Online Learning with Kernels (POLK) that is an example of such an algorithm [24]. POLK proposes the use of nonparametric machine learning, which tends to be more accurate than parametric methods and allows for a more flexible hypothesis that can adapt to incoming data. An example of a parametric method, linear regression, is given in Equation 1.

Note that with parametric models we must make specific assumptions about the data and the parameter selection will greatly impact how well the model performs. Some examples of parametric machine learning algorithms are [25]:

Parametric machine learning methods, albeit fairly simple, offer a constrained model representation and this limits their application to complex and noisy environments. On the other hand, nonparametric methods offer several advantages in that they make no a prior assumptions about the data and, therefore, are capable of providing a more flexible model that is more accurate in complex scenarios. However, models built from nonparametric methods tend to grow in size as they are exposed to more data, requiring a large amount of storage. Because of this, most ML practitioners do not apply nonparametric methods to large online learning applications. Typical examples of nonparametric ML methods are:

With nonparametric methods, the model complexity grows with each additional data point such that the complexity is proportional to the iteration index and, therefore, does not readily lend itself to online learning; many techniques such as support vector machines rely on batch data.

POLK was specifically developed to address the complexity and memory requirements challenges of nonparametric approaches in order to operate in true streaming data scenarios, attaining state-of-the-art accuracy tradeoffs between model complexity, statistical accuracy, and stability, both for statistical inference (supervised learning) and learning-based control (reinforcement learning). This is accomplished by utilizing a stochastic gradient descent approach that uses a supervised “sparsification” scheme, which only keeps the data points that minimize the statistical loss. The POLK authors consider this to be a “greedy function subspace projection,” where, for each gradient step, data samples are either retained or discarded depending on their ability to minimize the statistical loss, resulting in a technique better suited than traditional parametric techniques for dynamic environments with limited data.

For Army-relevant scenarios, POLK and similar approaches to machine learning offer unique advantages. First, online learning is important because underlying distributions of observed data can change as events unfold. Second, in a data-impoverished environment, the ML algorithm may only observe a few examples of a phenomenon before needing to adapt to it.

POLK has a strong mathematical basis and is derived from prior online learning approaches such as Kivinen et. al’s Naïve Online Regularization Minimization Algorithm (NORMA), which uses a fixed number of data points and discards the oldest point [26]. Kivinen et. al. were the first to demonstrate how the application of classical stochastic gradient descent can lead to novel and practical algorithms for online learning using kernel methods. Subsequently, Wang et. al. introduced Budgeted Stochastic Gradient Descent (BGSD), which addresses the “curse of kernelization”, a phenomena that causes unbounded linear growth in model size and update time with data size [27]. With BGSD, any number of underlying learning algorithms can be used (such as NORMA) with a fixed number of data points, but the algorithm discards the data point with the smallest weight and “reprojects” weights onto the remaining data points. These memory reduction methods may cause learning instability due to the fact that the discarded data is not based on learning progress.

POLK provides an alternative method to discarding data that takes into account the value of the observed data and determines which data points are the most informative. This is shown pictorially in Figure 2.

Figure 2.

Visual depiction of POLK where epsilon is the size of the error neighborhood and f_t is the current data point, is the estimate of the next SGD step and f_t+1 is the POLK estimate. D_t is the kernel dictionary. The function f_t is parameterized by dictionary D_t and weight vector W_t. f_t+1 is determined from output of DKOMP on which is parameterized by [18].

The stochastic projection is defined as follows (where P is the Project function) in Equations 2 and 3:

Where:

Note that in in Equation 2, the term P_HDt+1 is the projection onto the most representative data points, while η is the learning rate and λ is the regularization parameter. Also note that [(1 – η_t) can be considered as a “forgetting factor”. η_t∇_fℓ(f_t(x_t),fy_t) is the gradient step and ℓ represents the loss function. The estimated value for

Note that without using projection D_t is merely memorizing the most recent data point x_t.

Projection is carried out by using a novel variation of Kernel Orthogonal Matching Pursuit, dubbed here as Destructive Kernel Orthogonal Matching Pursuit (DKOMP). Simply put, for each data point, DKOMP discovers the approximation error of the best function approximation if this point were removed and removes the point that incurs the least error. This process is repeated until the approximation error budget is reached as shown in Equation 7. See [18] for more details:

Fundamentally, POLK leverages greedy compression to define a projection onto most representative observations.

Koppel et. al. proved that this approach has a guaranteed bounded model order. POLK has been shown to work well with synthetic and real datasets. For example, when run with the MNIST handwritten character dataset, the following results were obtained:

The end result is optimally compressed online kernel learning algorithms that offer an alternative to deep learning and overcomes their data hunger as well as cloud reliance. Thus, these methods will be vital to the development of intelligent systems that operate in communications-denied settings without much a priori information.

5.3

Semi-Supervised and Unsupervised Learning

In addition to developing techniques for online supervised learning, ISD is making strides in developing new AI and ML methods that operate in an unsupervised manner and label data collected by agents operating in an unknown environment. Although many AI and ML methods expect virtually perfectly labeled training data, there are many challenges with creating correctly labeled datasets including expense and reliability [28]. Data labeling is expensive because it is timeconsuming and often requires experts to perform the labeling. Labeled datasets often contain mislabeling errors called “label noise”. Label noise comes from 4 main sources [28]:

Labeling ambiguity issues also derive from changes in illumination, scale, perspective, color, and background clutter [29]. To address these issues ISD researchers Wigness et. al. have developed novel methods to hierarchically cluster image data in such a way that a wide “spectrum of visual concept granularities” is effectively captured in a way that greatly increases the efficiency of labeling data and reduces label noise [29]. This work was subsequently expanded to develop a comprehensive framework intended to more effectively label datasets by focusing on reducing human labeling effort using 4 metrics—discovery, efficiency, exploitation, and accuracy—by leveraging hierarchical clustering, a novel “interestingness” metric, and an iterative group-based labeling process based upon relationships between labeled and unlabeled data [30]. This new framework dubbed “Hierarchical Cluster Guided Labeling” (HCGL) requires no a priori knowledge of the label set and, unlike other methods, requires no re-clustering for refinement of groupings, which reduces overall process latency. In HCGL, discovery is the process of learning “concepts” or categories for images. In other words, the algorithm determines what labels belong on each image by setting up a taxonomy. For example and image could be clustered into a concept for “animals” while also belonging simultaneously to “dogs”. Ultimately, this improves learning performance during classification training. Labeling efficiency refers to the extent to which the human effort for labeling data can be reduced. This is determined by accounting for both the effort and time it takes humans to assist in labeling images. The idea behind exploitation is measuring the number of images that can be labeled with a single label while accuracy is the proportion of non-erroneous labels [30].

Figure 3.

Illustration that depicts the relationships for group c in a local neighborhood of H, including its parent p and left and right children, cl and cr. [30]

HCGL utilizes several “selection heuristics,” including interestingness, exploration, and exploitation, in order to make group selections in the labeling process. Interestingness can be seen as the structural novelty within a local neighborhood of the hierarchy, while exploitation was discussed previously. Exploration is a measure that determines how similar a group is to previously labeled groups.

HGCL differs from other group-based labeling approaches in that the entire hierarchal clustering set of the training data is maintained and HCGL selects groups from this set, allowing for the encoding of many levels of “semantic granularity”. HCGL uses its novel hierarchical group selection criteria to quickly collect large amounts of labeled data, explore the feature space, and identify locations in the hierarchy that are more likely to contain groups of data that represent images from exactly one class. As compared with other group-based labeling techniques, HCGL produces higher performing visual classifiers with respect to labeling effort, which is very promising in for Army-relevant environments where time for labeling collected data is at a premium. The researchers also tested HCGL on a robot extending the algorithm so that the technique would achieve high quality visual perception requiring only a few hours of labeling effort from a human annotator [31].

While HGCL is a batch-based, semi-supervised learning algorithm, Wigness and Rogers developed a fully unsupervised, online semantic scene labeling algorithm called Unsupervised Semantic Scene Labeling (USSL) that continuously learns and adapts to incoming image data steams [32]. In this work, Wigness and Rogers address the shortcomings of batch-based processing, which prohibits online learning and discovery of novel concepts in incoming visual data streams. They also point out that even semi-supervised methods introduce processing delays as humans are still required to label some data. Unlike earlier segmentation preprocessing algorithms, USSL directly models scene semantics with regard to pixel connectivity in localized regions of the image.

Figure 4.

Comparison of segmentation output for our technique which directly models semantics without regard to locality, and hierarchical graph-based (GBH) segmentation which emphasizes segmentation coherency using strict locality rules. USSL, ARL’s approach labels all four traffic cones consistently (column two), whereas GBH assigns unique labels to each cone (column three) [32].

Note that in Figure 4, USSL is capable of identifying the 4 cones as belonging to the same class, as denoted by the purple color, whereas the hierarchical graph-based segmentation method defines each cone as belonging to different classes. USSL uses a technique called “Agglomerative Clustering,” which is a “bottom-up” approach when each object is assigned to a single cluster and then is successively merged until a desired cluster structure is obtained based on some predefined criteria [33,34]. Therefore, USSL iteratively creates and adapts a set of semantic models from streaming image data without a priori knowledge of image classes or any parameters of the data and is, therefore, capable of modeling novel images and concepts continuously. One of the key innovations in USSL is the use of overlapping sliding windows to create an ensemble of learners that are encoded in a graph structure and used to reconcile learning errors and generate label mappings and a global label set for the data stream, as depicted in Figure 5. According to the researchers, by “over-learning locally, noise introduced by unsupervised learning can be minimized, and remaining errors can be reconciled using an ensemble of local learners encoded in a graph-based structure.”

Figure 5.

Overview of the USSL algorithm. The next image from the data stream is over-segmented, and segments are agglomeratively merged with existing models from previous frames in the stream. Overlapping local models are created for a window in the stream and these local label sets are mapped and reconciled using a graph encoding to generate a global label set. [32]

5.4

Autonomous and Semi-Autonomous Operation

As discussed previously, it is common in the field of AI and ML to train learning agents by large numbers of training examples but this is not feasible for many Army scenarios. Even with recent applications of AI and ML, such as self-driving vehicles, miles of miles of data has been collected but this may not be adequate. As Knox and Stone point out, “an autonomous driving agent should not learn to drive by crashing into road barriers and endangering the lives of pedestrians.” [35] Furthermore, these researchers point out the humans have domain knowledge that can be leveraged to increase the speed and learning while avoiding the need for complex or difficult to create training examples. Knox and Stone’s framework is called “Training an Agent Manually via Evaluative Reinforcement” (TAMER) where a human trainer to interactively “shapes” an agent’s policy using positive and negative reinforcement signals [35].

ARL researchers Warnell, Waytowich, and Lawhern collaborated with one of TAMER’s creators—Stone—to extend TAMER to incorporate the use of high-dimensional state spaces and deep learning, resulting in Deep TAMER [36]. Whereas with TAMER, the agent receives signals from the non-expert human and the agent, in response, tries to estimate a hidden function that approximates the human’s feedback and adjusts its behavior to align with this estimate, Deep TAMER uses a deep neural network that is used to compute an estimate that minimizes the value of a cost function. In addition, TAMER learns once from each state-action pair, unlike Deep TAMER, which can learn multiple times for each state-action pair due to a built-in feedback replay buffer [36]. The feedback replay buffer, D increases the rate of learning by storing observed human feedback and sampling this running history at a fixed rate regardless of the rate of human feedback. That is, Stochastic Gradient Descent (SGD) updates to Ĥ occur whenever the human provides new feedback and at a fixed rate using the data sampled from D. The Deep TAMER framework is depicted in Figure 6, and the initial research specifically considered the Atari game of BOWLING, which uses a pixel-level state space.

Figure 6.

The Deep TAMER framework. A human observes an autonomous agent trying to perform a task in a highdimensional environment and provides scalar-valued feedback as a means by which to shape agent behavior. Through the interaction with the human, the agent learns the parameters of a deep neural network, ^H, that is used to predict the human’s feedback. This prediction then drives the agent’s behavior policy. [36]

In summary, Deep TAMER leverages the representational power of deep neural networks in order to learn complex tasks in just a short amount of time with a human trainer. ARL’s researchers demonstrated that Deep TAMER is able to exceed the performance of average human trainers after only 7 minutes of training, and surpasses the performance of the expert human after 15 minutes. This was accomplished on the Atari game of BOWLING a task that has proven difficult for even state-of-the-art reinforcement learning methods [36].

5.5

Intent Awareness, Reasoning and Explainability

5.5.1

Natural Language Communication between Humans and Agents

In order address the issues related to common sense, reasoning, and explainability, ARL has a research thrust that is looking at how humans can communicate with intelligent agents using natural language. The capabilities that will be derived from this research are useful in a number of ways. For example, if an intelligent agent is given an order such as “go through that door”, it would be useful if it could respond with a query such as “Do you mean the door that is closest to me?” or “Do you mean the door on the left?” This grounding of the request is important and very natural when humans communicate with each other. ARL’s “BOT Language” research program (BOTLang) aims to provide more natural ways for soldiers to interact and communicate with robots [37,38]. Current research involves using a “Wizard of Oz” approach that attempts to simulate capabilities of autonomous systems that we envision will exist in the future. This research not only informs researchers of human-system integration issues but also provides insights as to how autonomous systems should be designed.

BOTLang uses 2 wizards, the first is for robot navigation and the second is for dialog management. Experiments have the system explore and unknown environment with a human teammate, where the human gives verbal instructions to the robot from a remote location and the robot takes action based on the human requests, clarifying instructions through queries back to the human. The architecture is depicted in Figure 7.

Fig. 7.

The participant provides instructions, unaware that the robot is composed of two wizards. The Dialog Manager (DM) wizard translates spoken language commands into a constrained action list for the Robotic Navigator (RN) wizard to execute. Map updates, image data, and the robot’s dialogue responses are sent to the participant through a multimodal interface (upper right) [37,38].

The BOTLang architecture is very modular. The Robot Navigator (RN) can simply be thought of as autonomous navigation that leverages all available sensor data, while the Dialog Manager (DM) does all of the natural language processing and has the intelligence to translate verbal commands given by the participant into movement orders that will be understood by the RN. The BOTLang research has produced a corpus of data that is useful for training the AI and ML research algorithms that are being developed, and will eventually replace the RN and DM human “wizards”.

5.5.2

Common Sense Reasoning with Robustness

Often researchers will attempt to imbue intelligent agents with “common sense” by using a “knowledge base” (KB). A KB is a set of representations of objects in the world. An intelligent agent leverages a KB and an inference engine or inference algorithm such that when an agent is asked a question it can use its KB and inference engine to answer. [39]

ISD researcher Summers-Stay has noted that most common sense KBs are “brittle” and “incomplete” in that they are sensitive to how queries are posed and they are missing seemingly trivial knowledge facts [40]. In order to improve overall performance, research has been conducted that combines the deductive reasoning abilities of a KB inference engine with the ability to represent semantic similarity that is provided by “distributional semantic vector spaces”. This amounts to exploring the KB for semantically related terms to the query term. However, these systems sometimes do not result in an answer. To remedy this, Summers-Stay has developed a method for forming chains of “deductive logical reasoning,” which is akin to human “train of thought.”

Instead of attempting to find connective “paths between ideas” using a mathematical search technique to exhaustively search a knowledge graph, Summers-Stay uses a KB consisting of high dimensional vectors that are related to each via a predicate in form a triple (e_n, predicate, e_m). Let us imagine that there are 2 words, g and p, and you need to determine if g implies p. The goal becomes finding a chain of entities that links g to p, which means finding a sum of “fact vectors” (–e_n + e_m) that add up to (–g + p). As it turns out, in cases where such a chain exists, this method returns a set of facts that constitute the chain, and even when the the chain does not exist the method will return an approximation of the correct links in the path leading to potentially new knowledge of undefined relations between closely related entities.

Figure 8 provides a visual and somewhat intuitive view of the method. One can envision words as “vectors”. The word2vec knowledge base that was utilized for this research has a 300-dimensional vector for each word. By leveraging sparse approximation methods such as LASSO (least absolute shrinkage and selection operator) or Othogonal Matching Pursuit (OMP), to obtain a sparse weight vector which is multiplied by the list of fact vectors in order to find a chain of entities linking g to p.

Fig. 8.

Reasoning systems need to be able to give approximate answers when exact answers are impossible. Through sparse matrix techniques the intelligent agent is able to approximate reasoning with partial information and expand the knowledge base by exploiting distributional semantics at query time. In this simple example, terrorist is linked to Capitol [41].

An example from [40] is finding a path of relations between g: Michael Jackson and p: Music. The KB contains (Michael Jackson, is a, songwriter) and (musician, composes, music). Therefore, since songwriter and music are “nearby in semantic space,” therefore the connection between Michael Jackson and Music is made, demonstrating that this method can be used for analogical and associational connections. Another example is analogical and abductive reasoning where an analogy such as this is presented to the intelligent agent:

In this case the intelligent agent comes up with “snorkler”. How might this conclusion be made? Consider related contexts such as woods, sea, predator, and tourist. The vector for “bear” consists of a weighted average of all vectors related to “woods” and the average of all vectors related to “predators” plus noise. The analogy becomes:

So the vector between bear and hiker is –predator + tourist + noise. Likewise the vector from shark to snorkeler is: - predator + tourist + noise. This is similar to the vector from shark to snorkeler. The algorithm looks for any vector that will make the sum have as low an error as possible and, therefore, is able to use analogical relations to complete a chain of argument and connect the concept shark to the concept snorkeler.

This method has also been extended to ontology merging, where simple questions can be answered by connecting relating concepts and even more complex questions containing multiple concepts.

This basic research is important not only because of its usefulness in helping intelligent agents to reason, but also because it can potentially explain “why” a certain conclusion was provided. Depending on how the system is constructed, it may be possible to ascertain how “certain” an intelligent agent is of its conclusion.