During the last ten years, the availability of images acquired from unmanned aerial vehicles (UAVs) has been continuously increasing due to the improvements and economic success of flight and sensor systems. From our point of view, reliable and automatic image-based change detection may contribute to overcoming several challenging problems in military reconnaissance, civil security, and disaster management. Changes within a scene can be caused by functional activities, i.e., footprints or skid marks, excavations, or humidity penetration; these might be recognizable in aerial images, but are almost overlooked when change detection is executed manually. With respect to the circumstances, these kinds of changes may be an indication of sabotage, terroristic activity, or threatening natural disasters. Although image-based change detection is possible from both ground and aerial perspectives, in this paper we primarily address the latter. We have applied an extended approach to change detection as described by Saur and Kruger,1 and Saur et al.2 and have built upon the ideas of Saur and Bartelsen.3 The commercial simulation environment Virtual Battle Space 3 (VBS3) is used to simulate aerial "before" and "after" image acquisition concerning flight path, weather conditions and objects within the scene and to obtain synthetic videos. Video frames, which depict the same part of the scene, including "before" and "after" changes and not necessarily from the same perspective, are registered pixel-wise against each other by a photogrammetric concept, which is based on a homography. The pixel-wise registration is used to apply an automatic difference analysis, which, to a limited extent, is able to suppress typical errors caused by imprecise frame registration, sensor noise, vegetation and especially parallax effects. The primary concern of this paper is to seriously evaluate the possibilities and limitations of our current approach for image-based change detection with respect to the flight path, viewpoint change and parametrization. Hence, based on synthetic "before" and "after" videos of a simulated scene, we estimated the precision and recall of automatically detected changes. In addition and based on our approach, we illustrate the results showing the change detection in short, but real video sequences. Future work will improve the photogrammetric approach for frame registration, and extensive real video material, capable of change detection, will be acquired.
Accurate geo-registration of acquired imagery is an important task when using unmanned aerial vehicles (UAV) for video reconnaissance and surveillance. As an example, change detection needs accurately geo-registered images for selecting and comparing co-located images taken at different points in time. One challenge using small UAVs lies in the instable flight behavior and using low-weight cameras. Thus, there is a need to stabilize and register the UAV imagery by image processing methods since using only direct approaches based on positional information coming from a GPS and attitude and acceleration measured by an inertial measurement unit (IMU) are not accurate enough. In order to improve this direct geo-registration (or pre-registration"), image matching techniques are applied to align the UAV imagery to geo-registered reference images. The main challenge consists in matching images taken from different sensors at different day time and seasons. In this paper, we present evaluation methods for measuring the performance of image registration algorithms w.r.t. multi-temporal input data. They are based on augmenting a set of aligned image pairs by synthetic pre-registrations to an evaluation data set including truth transformations. The evaluation characteristics are based on quantiles of transformation residuals at certain control points. For a test site, video frames of a UAV mission and several ortho images from a period of 12 years are collected and synthetic pre-registrations corresponding to real flight parameters and registration errors are computed. Two algorithms A1 and A2 based on extracting key-points with a floating point descriptor (A1) and a binary descriptor (A2) are applied to the evaluation data set. As evaluation result, the algorithm A1 turned out to perform better than A2. Using affine or Helmert transformation types, both algorithms perform better than in the projective case. Furthermore, the evaluation classifies the ortho images w.r.t. their degree of difficulty and even for the most unfavorable ortho image, the evaluation characteristics yield better results than those attached to the default pre-registration. Finally, the proposed evaluation methods have been proven to derive valuable results even
for input data with a high degree of difficulty.
Change detection is one of the most important tasks when unmanned aerial vehicles (UAV) are used for video
reconnaissance and surveillance. In this paper, we address changes on short time scale, i.e. the observations are taken
within time distances of a few hours. Each observation is a short video sequence corresponding to the near-nadir
overflight of the UAV above the interesting area and the relevant changes are e.g. recently added or removed objects.
The change detection algorithm has to distinguish between relevant and non-relevant changes. Examples for non-relevant
changes are versatile objects like trees and compression or transmission artifacts. To enable the usage of an
automatic change detection within an interactive workflow of an UAV video exploitation system, an evaluation and
assessment procedure has to be performed. Large video data sets which contain many relevant objects with varying scene
background and altering influence parameters (e.g. image quality, sensor and flight parameters) including image
metadata and ground truth data are necessary for a comprehensive evaluation. Since the acquisition of real video data is
limited by cost and time constraints, from our point of view, the generation of synthetic data by simulation tools has to
In this paper the processing chain of Saur et al. (2014)  and the interactive workflow for video change detection is
described. We have selected the commercial simulation environment Virtual Battle Space 3 (VBS3) to generate synthetic
data. For an experimental setup, an example scenario “road monitoring” has been defined and several video clips have
been produced with varying flight and sensor parameters and varying objects in the scene. Image registration and change
mask extraction, both components of the processing chain, are applied to corresponding frames of different video clips.
For the selected examples, the images could be registered, the modelled changes could be extracted and the artifacts of
the image rendering considered as noise (slight differences of heading angles, disparity of vegetation, 3D parallax) could
be suppressed. We conclude that these image data could be considered to be realistic enough to serve as evaluation data
for the selected processing components. Future work will extend the evaluation to other influence parameters and may
include the human operator for mission planning and sensor control.
Change detection is one of the most important tasks when using unmanned aerial vehicles (UAV) for video reconnaissance and surveillance. We address changes of short time scale, i.e. the observations are taken in time distances from several minutes up to a few hours. Each observation is a short video sequence acquired by the UAV in near-nadir view and the relevant changes are, e.g., recently parked or moved vehicles. In this paper we extend our previous approach of image differencing for single video frames to video mosaics. A precise image-to-image registration combined with a robust matching approach is needed to stitch the video frames to a mosaic. Additionally, this matching algorithm is applied to mosaic pairs in order to align them to a common geometry. The resulting registered video mosaic pairs are the input of the change detection procedure based on extended image differencing. A change mask is generated by an adaptive threshold applied to a linear combination of difference images of intensity and gradient magnitude. The change detection algorithm has to distinguish between relevant and non-relevant changes. Examples for non-relevant changes are stereo disparity at 3D structures of the scene, changed size of shadows, and compression or transmission artifacts. The special effects of video mosaicking such as geometric distortions and artifacts at moving objects have to be considered, too. In our experiments we analyze the influence of these effects on the change detection results by considering several scenes. The results show that for video mosaics this task is more difficult than for single video frames. Therefore, we extended the image registration by estimating an elastic transformation using a thin plate spline approach. The results for mosaics are comparable to that of single video frames and are useful for interactive image exploitation due to a larger scene coverage.
In the last years, there has been an increased use of unmanned aerial vehicles (UAV) for video reconnaissance and surveillance. An important application in this context is change detection in UAV video data. Here we address short-term change detection, in which the time between observations ranges from several minutes to a few hours. We distinguish this task from video motion detection (shorter time scale) and from long-term change detection, based on time series of still images taken between several days, weeks, or even years. Examples for relevant changes we are looking for are recently parked or moved vehicles. As a pre-requisite, a precise image-to-image registration is needed. Images are selected on the basis of the geo-coordinates of the sensor’s footprint and with respect to a certain minimal overlap. The automatic imagebased fine-registration adjusts the image pair to a common geometry by using a robust matching approach to handle outliers. The change detection algorithm has to distinguish between relevant and non-relevant changes. Examples for non-relevant changes are stereo disparity at 3D structures of the scene, changed length of shadows, and compression or transmission artifacts. To detect changes in image pairs we analyzed image differencing, local image correlation, and a transformation-based approach (multivariate alteration detection). As input we used color and gradient magnitude images. To cope with local misalignment of image structures we extended the approaches by a local neighborhood search. The algorithms are applied to several examples covering both urban and rural scenes. The local neighborhood search in combination with intensity and gradient magnitude differencing clearly improved the results. Extended image differencing performed better than both the correlation based approach and the multivariate alternation detection. The algorithms are adapted to be used in semi-automatic workflows for the ABUL video exploitation system of Fraunhofer IOSB, see Heinze et. al. 2010.1 In a further step we plan to incorporate more information from the video sequences to the change detection input images, e.g., by image enhancement or by along-track stereo which are available in the ABUL system.
Spaceborne SAR imagery offers high capability for wide-ranging maritime surveillance especially in situations,
where AIS (Automatic Identification System) data is not available. Therefore, maritime objects have to
be detected and optional information such as size, orientation, or object/ship class is desired. In recent
research work, we proposed a SAR processing chain consisting of pre-processing, detection, segmentation, and
classification for single-polarimetric (HH) TerraSAR-X StripMap images to finally assign detection hypotheses
to class "clutter", "non-ship", "unstructured ship", or "ship structure 1" (bulk carrier appearance) respectively
"ship structure 2" (oil tanker appearance). In this work, we extend the existing processing chain and are now
able to handle full-polarimetric (HH, HV, VH, VV) TerraSAR-X data. With the possibility of better noise
suppression using the different polarizations, we slightly improve both the segmentation and the classification
process. In several experiments we demonstrate the potential benefit for segmentation and classification.
Precision of size and orientation estimation as well as correct classification rates are calculated individually
for single- and quad-polarization and compared to each other.
It is expected, that ship detection and classification in SAR satellite imagery will be part of future downstream
services for various applications, e.g. surveillance of fishery zones or tracking of cargo ships. Due to the
requirements of operational services and due to the potential of high resolution SAR (e.g. TerraSAR-X), there
is a need for composing, optimization, and validation of specific fully automated image processing chains.
The presented processing chain covers all steps from land masking, screening, object segmentation, feature
extraction to classification and parameter estimation. The chain is base for experiments with both open sea
and harbor scenes for ship detection and monitoring. Within this chain, a classification component for SAR
ship and non-ship decision is investigated. Based on many extracted image features and numerous image chips
for training and test, some promissing results are presented and discussed. Since the classification can reduce
the false alarms of the screening component, the processing chain is expected to work on images with less
good weather and signal conditions and to extract ships with lower reflexions.
UAV have a growing importance for reconnaissance and surveillance. Due to improved technical capability also small
UAVs have an endurance of about 6 hours, but less sophisticated sensors due to strong weight limitations. This puts a
high strain and workload on the small teams usually deployed with such systems. To lessen the strain for photo
interpreters and to improve the capability of such systems we have developed and integrated automatic image
exploitation algorithms. An import aspect is the detection of moving objects to give the photo interpreter (PI) hints were
such objects are. Mosaiking of imagery helps to gain better oversight over the scene. By computing stereo-mosaics from
mono-ocular video-data also 3-d-models can be derived from tactical UAV-data in a further processing step. A special
instrument of gaining oversight is to use multi-temporal and multifocal images of video-sensors with different resolution
of the platform and to fusion them into one image. This results in a good situation awareness of the scene with a light-weight
sensor-platform and a standard video link.
For surveillance and reconnaissance tasks small UAVs are of growing importance. These UAVs have an endurance of
several hours, but a small payload of about some kilograms. As a consequence lightweight sensors and cameras have to
be used without having a mechanical stabilized high precision sensor-platform, which would exceed the payload and cost
An example of such a system is the German UAV Luna with optical and IR sensors on board. For such platforms we
developed image exploitation algorithms. The algorithms comprise mosaiking, stabilization, image enhancement, video
based moving target indication, and stereo-image generation. Other products are large geo-coded image mosaics, stereo
mosaics, and 3-D-model generation. For test and assessment of these algorithms the experimental system ABUL has
been developed, in which the algorithms are integrated. The ABUL system is used for tests and assessment by military
The miniature SAR-system MiSAR has been developed by EADS Germany for lightweight UAVs like the LUNASystem.
MiSAR adds to these tactical UAV-systems the all-weather reconnaissance capability, which is missing until
now. Unlike other SAR sensors, that produce large strip maps at update rates of several seconds, MiSAR generates
sequences of SAR images with approximately 1 Hz frame rate.
photo interpreters (PI) of tactical drones, now mainly experienced with visual interpretation, are not used to SARimages,
especially not with SAR-image sequence characteristics. So they should be supported to improve their ability to
carry out their task with a new, demanding sensor system. We have therefore analyzed and discussed with military PIs in
which task MiSAR can be used and how the PIs can be supported by special algorithms.
We developed image processing- and exploitation-algorithms for such SAR-image sequences. A main component is the
generation of image sequence mosaics to get more oversight. This mosaicing has the advantage that also non straight
/linear flight-paths and varying squint angles can be processed. Another component is a screening-component for manmade
objects to mark regions of interest in the image sequences. We use a classification based approach, which can be
easily adapted to new sensors and scenes. These algorithms are integrated into an image exploitation system to improve
the image interpreters ability to get a better oversight, better orientation and helping them to detect relevant objects,
especially considering long endurance reconnaissance missions.
Technological progress in the fields of computing hardware and efficient algorithms make it possible to set up real-time exploitation systems for a huge number of applications (e.g. assessment of camouflage effectiveness, or various surveil-lance applications, UAVs, as well as image sequence data reduction, indexing, archiving, and retrieval). The system in question has been developed to cope with highly dynamic situations. Such dynamic situations may be characterized by moving targets acquired by a static, trembling, or moving sensor system. The image sequences may stem from a visual-optical (VIS) or some forward looking infrared (FLIR) sensor. Except for wide-angle lenses (due to their optical distortions) neither sensor nor calibration parameters have to be known to the automatic exploitation system. Furthermore no human interaction is required. The algorithmic approach tries to digitally stabilize the movement of the sensor system. To accomplish this task the algorithm extracts 40-60 tie points from the static nonmoving background, then robustly matches the tie point constellations frame to frame for calculating the 8 parameters of a projective mapping. This is the basis for some sort of background stabilization. The difference image of two consecutive and matched image frames re-veals the moving targets. After the segmentation of the (moving) target signatures, additionally attached tracking and classification components have been tested.
For the exploitation of aerial and satellite imagery, human military photo interpreters need support by automatic image analysis components to meet the requirements of large data set analysis under strong time constraints. Extending the approaches of performance analysis of automatic target detection, a concept and an experimental study for the assessment of machine assisted vehicle detection is presented. This evaluation pursues the following goals: Extraction of a usability measure in terms of algorithm performance combined with user-oriented parameters. Secondly, an extraction of requirements for the image exploitation process concerning the algorithm performance, the man-machine interface and the training of the photo interpreters. A performance analysis concept for vehicle detection algorithms is presented as well as an experimental setup of the whole interactive exploitation process. This setup has been applied in an experiment with more than 100 real images and more than 40 military photo interpreters.
This contribution presents a comprehensive framework for algorithm evaluation. When we speak of evaluation, we have in mind that first the performance of an algorithm is measured and then the measured performance is assessed with regard to a given application. The performance assessment is done by applying an assessment function that uses desired values for the performance measures and weighting factors giving the importance of each measure, thus considering the application- specific requirements. The algorithm evaluation's goal is to verify the specification of an algorithm. This specification is mainly given by the definition of the input data and the expected output data, both of which are determined by the application. Prior to the evaluation process the algorithm specification has to be laid down by analyzing the application in order to deduce its requirements as well as by defining the application relevant data sets. To organize this sequence of preparatory steps and to formalize the accomplishment of the evaluation we have developed a 3-phase approach, consisting of the definition phase, the tuning phase, and the evaluation phase. An extensive software toolbox has been developed to support the evaluation process.