Multimodal data fusion for object recognition

Vladimir Knyaz

doi:10.1117/12.2526067

21 June 2019 Multimodal data fusion for object recognition

Vladimir Knyaz

Proceedings Volume 11059, Multimodal Sensing: Technologies and Applications; 110590P (2019) https://doi.org/10.1117/12.2526067
Event: SPIE Optical Metrology, 2019, Munich, Germany

Abstract

Multi-spectral imagery provides wide possibilities for improving quality of object detection and recognition due to better visibility of different scene features in different spectral ranges. To use the advantage of multi-spectral data the relation between different types of data is required. This relation is provided by capturing data using calibrated, aligned and synchronized sensors. Also geo-spatial data in form of geo-referenced digital terrain models can be used for establishing geometric and semantic relations between different types of data. The presented study considers the problem of object recognition based on two data sources: visible and thermal imagery. The main aim of the performed study was to evaluate the performance of different convolutional neural network models for multimodal object recognition. For this purpose a special dataset was collected. The dataset contains synchronized visible and thermal images acquired by several sensor based on unmanned aerial vehicle. The dataset contains synchronized color and thermal images of urban and suburb scenes gathered in different seasons, different times of day and various weather conditions. For convolutional neural network training the dataset was augmented by model images created using object 3D models textured by real visible and thermal images. Several convolutional neural network architectures were trained and evaluated on the created dataset using different splits to estimate the influence of training data on object recognition performance.

Citation Download Citation

Vladimir Knyaz "Multimodal data fusion for object recognition", Proc. SPIE 11059, Multimodal Sensing: Technologies and Applications, 110590P (21 June 2019); https://doi.org/10.1117/12.2526067

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available