The ASTM E57.02 Test Methods Subcommittee is developing a test method to evaluate the ranging performance of a
3D imaging system. The test method will involve either measuring the distance between two targets or between an
instrument and a target. The first option is necessary because some instruments cannot be centered over a point and will
require registration of the instrument coordinate frame into the target coordinate frame. The disadvantage of this option
is that registration will introduce an additional error into the measurements. The advantage of this option is that this type
of measurement, relative measurement, is what is typically used in field applications. A potential target geometry
suggested for the test method is a planar target. The ideal target material would be diffuse, have uniform reflectivity for
wavelengths between 500 nm to 1600 nm (wavelengths of most commercially-available 3D imaging systems), and have
minimal or no penetration of the laser into the material. A possible candidate material for the target is Spectralon1.
However, several users have found that there is some penetration into the Spectralon by a laser and this is confirmed by
the material manufacturer. The effect of this penetration on the range measurement is unknown. This paper will present
an attempt to quantify the laser penetration depth into the Spectralon material for four 3D imaging systems.
In 2006, ASTM committee E57 was established to develop standards for the performance evaluation of 3D imaging
systems. The committee's initial focus is on standards for 3D imaging systems typically used for applications
including, but not limited to, construction and maintenance, surveying, mapping and terrain characterization,
manufacturing (e.g., aerospace, shipbuilding), transportation, mining, mobility, historic preservation, and forensics.
ASTM E57 consists of four subcommittees: Terminology, Test Methods, Best Practices, and Data Interoperability.
This paper reports the accomplishments of the ASTM E57 3D Imaging Systems committee in 2007.
In June 2006, a new ASTM committee (E57) was established to develop standards for 3D imaging systems. This
committee is the result of a 4-year effort at the National Institute of Standards and Technology to develop performance
evaluation and characterization methods for such systems. The initial focus for the committee will be on standards for
3D imaging systems typically used for applications including, but not limited to, construction and maintenance,
surveying, mapping and terrain characterization, manufacturing (e.g., aerospace, shipbuilding), transportation, mining,
mobility, historic preservation, and forensics. This paper reports the status of current efforts of the ASTM E57 3D
Imaging Systems committee.
The NIST Construction Metrology and Automation Group (CMAG), in cooperation with the NIST Intelligent Systems
Division (ISD), is developing performance metrics and standard tests for the evaluation of 3D imaging systems used in
autonomous mobility applications. This work supports the broader effort to develop open, consensus-based
performance evaluation standards for a wide range of 3D imaging systems and applications through the ASTM E57
Committee on 3D Imaging Systems. This report presents initial efforts to characterize the range performance of a 3D
imaging sensor that will be used in a performance measurement system for crash prevention and safety systems. Factors
examined include range, target reflectance, target angle of incidence, and sensor azimuth.
This paper presents the status of an indoor artifact-based Performance Evaluation Facility at the National Institute of Standards and Technology (NIST) for 3D imaging systems, a terminology pre-standard, and a summary of the ranging protocol pre-standard. The indoor facility will be used to develop test protocols and performance metrics for the evaluation of terrestrial 3D imaging systems. The NIST facility was initiated in response to a workshop which was held at NIST in 2003 to determine future efforts needed to standardize 3D imaging system testing and reporting and to assess the need for a neutral performance evaluation facility. Three additional workshops have since been held at NIST with the most recent on March 2-3, 2006. These workshops provided further guidance in defining priorities and in identifying the types of measurements that are of most interest to the terrestrial scanning community. The two pre-standards were developed based on feedback from the workshops.
This paper presents a discussion of standards requirements for LADAR. Two specific questions are addressed: (1) is there a need for such standards and (2) what types of standards are required? LADAR standards development issues and current standardization efforts are also summarized.
The use and scope of LADAR (laser detection and ranging) applications continues to expand as the technology matures. This growth is reflected in the National Institute of Standards and Technology's (NIST) experience with research into the applications of LADARs for construction, manufacturing, and autonomous vehicle navigation. However, standard protocols or procedures for calibrating and testing LADARs have yet to be developed. Currently, selections of LADAR instruments are generally based on the manufacturer's specifications, the availability of standard test procedures would promote more uniform definitions of these specifications and provide a basis for a better informed differentiation between LADAR instruments.
Consequently, NIST's Construction Metrology and Automation Group (CMAG) has conducted exploratory experiments to characterize the performance of a LADAR instrument. The experiences gained in these efforts are summarized in this paper. These experiences also pointed to the need for an internal calibration/evaluation facility at NIST, as well as to the need for the development of uniform specifications and test procedures for characterizing LADARs. As a result, NIST convened a workshop on the establishment of a LADAR calibration facility. Discussions of some issues relating to the performance evaluation of LADARs, facility requirements, and similar efforts are presented in this paper.
We describe a project to collect and disseminate sensor data for autonomous mobility research. Our goals are to provide data of known accuracy and precision to researchers and developers to enable algorithms to be developed using realistically difficult sensory data. This enables quantitative comparisons of algorithms by running them on the same data, allows groups that lack equipment to participate in mobility research, and speeds technology transfer by providing industry with metrics for comparing algorithm performance. Data are collected using the NIST High Mobility Multi-purpose Wheeled Vehicle (HMMWV), an instrumented vehicle that can be driven manually or autonomously both on roads and off. The vehicle can mount multiple sensors and provides highly accurate position and orientation information as data are collected. The sensors on the HMMWV include an imaging ladar, a color camera, color stereo, and inertial navigation (INS) and Global Positioning System (GPS). Also available are a high-resolution scanning ladar, a line-scan ladar, and a multi-camera panoramic sensor. The sensors are characterized by collecting data from calibrated courses containing known objects. For some of the data, ground truth will be collected from site surveys. Access to the data is through a web-based query interface. Additional information stored with the sensor data includes navigation and timing data, sensor to vehicle coordinate transformations for each sensor, and sensor calibration information. Several sets of data have already been collected and the web query interface has been developed. Data collection is an ongoing process, and where appropriate, NIST will work with other groups to collect data for specific applications using third-party sensors.
Progress in algorithm development and transfer of results to practical applications such as military robotics requires the setup of standard tasks, of standard qualitative and quantitative measurements for performance evaluation and validation. Although the evaluation and validation of algorithms have been discussed for over a decade, the research community still faces a lack of well-defined and standardized methodology. The range of fundamental problems include a lack of quantifiable measures of performance, a lack of data from state-of-the-art sensors in calibrated real-world environments, and a lack of facilities for conducting realistic experiments. In this research, we propose three methods for creating ground truth databases and benchmarks using multiple sensors. The databases and benchmarks will provide researchers with high quality data from suites of sensors operating in complex environments representing real problems of great relevance to the development of autonomous driving systems. At NIST, we have prototyped a High Mobility Multi-purpose Wheeled Vehicle (HMMWV) system with a suite of sensors including a Riegl ladar, GDRS ladar, stereo CCD, several color cameras, Global Position System (GPS), Inertial Navigation System (INS), pan/tilt encoders, and odometry . All sensors are calibrated with respect to each other in space and time. This allows a database of features and terrain elevation to be built. Ground truth for each sensor can then be extracted from the database. The main goal of this research is to provide ground truth databases for researchers and engineers to evaluate algorithms for effectiveness, efficiency, reliability, and robustness, thus advancing the development of algorithms.