Transformative Apps (TransApps) is a Defense Advanced Research Projects Agency (DARPA) funded program whose
goal is to develop a range of militarily-relevant software applications (“apps”) to enhance the operational-effectiveness
of military personnel on (and off) the battlefield. TransApps is also developing a military apps marketplace to facilitate
rapid development and dissemination of applications to address user needs by connecting engaged communities of endusers
with development groups. The National Institute of Standards and Technology’s (NIST) role in the TransApps
program is to design and implement evaluation procedures to assess the performance of: 1) the various software
applications, 2) software-hardware interactions, and 3) the supporting online application marketplace. Specifically, NIST
is responsible for evaluating 50+ tactically-relevant applications operating on numerous Android™-powered platforms.
NIST efforts include functional regression testing and quantitative performance testing. This paper discusses the
evaluation methodologies employed to assess the performance of three key program elements: 1) handheld-based
applications and their integration with various hardware platforms, 2) client-based applications and 3) network
technologies operating on both the handheld and client systems along with their integration into the application
marketplace. Handheld-based applications are assessed using a combination of utility and usability-based checklists and
quantitative performance tests. Client-based applications are assessed to replicate current overseas disconnected (i.e. no
network connectivity between handhelds) operations and to assess connected operations envisioned for later use. Finally,
networked applications are assessed on handhelds to establish baselines of performance for when connectivity will be
Proc. SPIE. 7345, Multisensor, Multisource Information Fusion: Architectures, Algorithms, and Applications 2009
KEYWORDS: Information fusion, Facial recognition systems, Sun, Sensors, Computing systems, Personal digital assistants, Environmental sensing, Global Positioning System, Standards development, Data fusion
In order to effectively evaluate information fusion systems or emerging technologies, it is critical to quickly, efficient,
and accurately collect functional and observational data about such systems. One of the best ways to test a system's
capabilities is to have an end user operate it in controlled but realistic field-based situations. Evaluation data of the
systems' performance as well as observational data of the user's interactions can then be collected and analyzed. This
analysis often gives insight into how the system may perform in the intended environment and of any potential areas for
improvement. One common method for collection of this data involves an evaluator/observer generating hand-written
notes, comments, and sketches. This often proves to be inefficient in complex sensor technology field-based evaluation
environments. Personnel at the National Institute of Standards and Technology (NIST) have been tasked with collecting
such evaluation data for emerging soldier-worn sensor systems. Lessons learned from the on-going development of
efficient field-based evaluation data collection techniques will be discussed. The most recent evaluation data collection
using a personal digital assistant (PDA)-style system and details of its use during an evaluation of a multi-team study
will also be described.
PRIDE (PRediction In Dynamic Environments) is a hierarchical
multi-resolutional framework for moving object
prediction. PRIDE incorporates multiple prediction algorithms into a single, unifying framework. To date, we
have applied this framework to predict the future location of autonomous vehicles during on-road driving. In
this paper, we describe two different approaches to compute long-term predictions (on the order of seconds into
the future) within PRIDE. The first is a cost-based approach that uses a discretized set of vehicle motions and
costs associated with states and actions to compute probabilities of vehicle motion. The cost-based approach
is the first prediction approach we have been using within PRIDE. The second is a fuzzy-logic-based approach
that deals with the pervasive presence of uncertainty in the environment to negotiate complex traffic situations.
Using the high-fidelity physics-based framework for the Unified System for Automation and Robot Simulation
(USARSim), we will compare the performance of the two approaches in different driving situations at
traffic intersections. Consequently, we will show how the two approaches complement each other and how their
combination performs better than the cost-based approach only.
PRIDE is a hierarchical multiresolutional framework for moving object prediction that incorporates multiple
prediction algorithms into a single, unifying framework. PRIDE is based upon the 4D/RCS (Real-time Control
System) reference model architecture and provides information to planners at the level of granularity that
is appropriate for their planning horizon. This framework supports the prediction of the future location of
moving objects at various levels of resolution, thus providing prediction information at the frequency and level of
abstraction necessary for planners at different levels within the hierarchy. To date, two prediction approaches have
been applied to this framework. In this paper, we provide an overview of the PRIDE (Prediction in Dynamic
Environments) framework and describe the approach that has been used to model different aggressivities of
drivers. We then explore different aggressivity models to determine their impact on the location predictions
that are provided through the PRIDE framework. We also describe recent efforts to implement PRIDE in
USARSim, which provides high-fidelity simulation of robots and environments based on the Unreal Tournament
4D/RCS is a hierarchical architecture designed for the control of intelligent systems. One of the main areas
that 4D/RCS has been applied to recently is the control of autonomous vehicles. To accomplish this, a hierarchical
decomposition of on-road driving activities has been performed which has resulted in implementation
of 4D/RCS tailored towards this application. This implementation has seven layers and ranges from a journey
manager which determines the order of the places you wish to drive to, through a destination manager which
provides turn-by-turn directions on how to get to a destination, through a route segment, drive behavior, elemental
maneuver, goal path trajectory, and then finally to servo controllers.
In this paper, we show, within the 4D/RCS architecture, how knowledge-driven top-down symbolic representations
combined with low-level bottom-up tasks can synergistically provide valuable information for on-road
driving better than what is possible with either of them alone. We demonstrate these ideas using field data
obtained from an Unmanned Ground Vehicle (UGV) traversing urban on-road environments.
We have developed PRIDE (Prediction In Dynamic Environments), a hierarchical multi-resolutional framework
for moving object prediction that incorporates multiple prediction algorithms into a single, unifying framework.
PRIDE incorporates two approaches for the prediction of the future location of moving objects at various levels
of resolution at the frequency and level of abstraction necessary for planners at different levels within the
hierarchy. These approaches, termed long-term (LT) and short-term (ST) predictions, respectively, are based
on situation recognition and vehicle models for moving object prediction using sensor data. Our recent efforts
have demonstrated the ability to use the results of the short-term prediction algorithms to strengthen/weaken
the estimates of the long-term prediction algorithms. Based on previous experiments, we have found that the
short-term prediction algorithms perform best when predicting on the order of a few seconds into the future
and that the longer-term prediction algorithms are best at predicting on the order of several seconds into the
future. In this paper, we explore the time window in which both the short-term and the long-term prediction
algorithms provide reasonable results. Additionally, we describe a methodology by which we can determine
the time point at which the short-term prediction algorithm no longer provides results within an acceptable
predefined error threshold. We provide experimental results in an autonomous on-road driving scenario using
AutoSim, a high-fidelity simulation tool that models details about road networks, including individual lanes,
lane markings, intersections, legal intersection traversability, etc.
The level of automation in combat vehicles being developed for the Army's objective force is greatly increased over the Army's legacy force. This automation is taking many forms in emerging vehicles; varying from operator decision aides to fully autonomous unmanned systems. The development of these intelligent vehicles requires a thorough understanding of all of the intelligent behavior that needs to be exhibited by the system so that designers can allocate functionality to humans and/or machines. Traditional system specification techniques focused heavily on the functional description of the major systems and implicitly assumed that a well-trained crew would operate these systems in a manner to accomplish the tactical mission assigned to the vehicle. In order to allocate some or all of these intelligent behaviors to machines in future vehicles it is necessary to be able to identify and describe these intelligent behaviors in detail. In this paper, we describe an effort to develop an intelligent systems (IS) ontology using Protege. The goal of this effort is to develop a common, implementation-independent, extendable knowledge source for researchers and developers in the intelligent vehicle community that will:
* Provide a standard set of domain concepts along with their attributes and inter-relations
* Allow for knowledge capture and reuse
* Facilitate systems specification, design, and integration, and
* Accelerate research in the field.
This paper describes the methodology we have used to identify knowledge in this domain and an approach to capture and visualize the knowledge in the ontology.
The Real-time Control System (RCS) Methodology has evolved over a number of years as a technique to capture task knowledge and organize it into a framework conducive to implementation in computer control systems. The fundamental premise of this methodology is that the present state of the task activities sets the context that identifies the requirements for all of the support processing. In particular, the task context at any time determines what is to be sensed in the world, what world model states are to be evaluated, which situations are to be analyzed, what plans should be invoked, and which behavior generation knowledge is to be accessed. This methodology concentrates on the task behaviors explored through scenario examples to define a task decomposition tree that clearly represents the branching of tasks into layers of simpler and simpler subtask activities. There is a named branching condition/situation identified for every fork of this task tree. These become the input conditions of the if-then rules of the knowledge set that define how the task is to respond to input state changes. Detailed analysis of each branching condition/situation is used to identify antecedent world states and these, in turn, are further analyzed to identify all of the entities, objects, and attributes that have to be sensed to determine if any of these world states exist. This paper explores the use of this 4D/RCS methodology in some detail for the particular task of autonomous on-road driving, which work was funded under the Defense Advanced Research Project Agency (DARPA) Mobile Autonomous Robot Software (MARS) effort (Doug Gage, Program Manager).
This paper describes NIST’s efforts in evaluating what it will take to achieve autonomous human-level driving skills in terms of time and funding. NIST has approached this problem from several perspectives: considering the current state-of-the-art in autonomous navigation and extrapolating from there, decomposing the tasks identified by the Department of Transportation for on-road driving and comparing that with accomplishments to date, analyzing computing power requirements by comparison with the human brain, and conducting a Delphi Forecast using the expert researchers in the field of autonomous driving. A detailed description of each of these approaches is provided along with the major finding from each approach and an overall picture of what it will take to achieve human level driving skills in autonomous vehicles.
Sensory processing for real-time, complex, and intelligent control systems is costly, so it is important to perform only the sensory processing required by the task. In this paper, we describe a straightforward metric for precisely defining sensory processing requirements. We then apply that metric to a complex, real-world control problem, autonomous on-road driving. To determine these requirements the system designer must precisely and completely define 1) the system behaviors, 2) the world model situations that the system behaviors require, 3) the world model entities needed to generate all those situations, and 4) the resolutions, accuracy tolerances, detection timing, and detection distances required of all world model entities.
In this paper, we present the PRIDE framework (Prediction In Dynamic Environments), which is a hierarchical multi-resolutional approach for moving object prediction that incorporates multiple prediction algorithms into a single, unifying framework. PRIDE is based upon the 4D/RCS (Real-time Control System) and provides information to planners at the level of granularity that is appropriate for their planning horizon. The lower levels of the framework utilize estimation theoretic short-term predictions based upon an extended Kalman filter that provide predictions and associated uncertainty measures. The upper levels utilize a probabilistic prediction approach based upon situation recognition with an underlying cost model that provide predictions that incorporate environmental information and constraints. These predictions are made at lower frequencies and at a level of resolution more in line with the needs of higher-level planners. PRIDE is run in the systems’ world model independently of the planner and the control system. The results of the prediction are made available to a planner to allow it to make accurate plans in dynamic environments. We have applied this approach to an on-road driving control hierarchy being developed as part of the DARPA Mobile Autonomous Robotic Systems (MARS) effort.
The realization of on- and off-road autonomous navigation of Unmanned Ground Vehicles (UGVs) requires real-time motion planning in the presence of dynamic objects with unknown trajectories. To successfully plan paths and to navigate in an unstructured environment, the UGVs should have the difficult and computationally intensive competency to predict the future locations of moving objects that could interfere with its path. This paper details the development of a combined probabilistic object classification and estimation theoretic framework to predict the future location of moving objects, along with an associated uncertainty measure. The development of a moving object testbed that facilitates the testing of different representations and prediction algorithms in an implementation-independent platform is also outlined.