Multi-object tracking with deep learning ensemble for unmanned aerial system applications

Wanlin Xie; Jaime Ide; Daniel Izadi; Sean Banger; Thayne Walker; Ryan Ceresani; Dylan Spagnuolo; Christopher Guagliano; Henry Diaz; Jason Twedt

doi:10.1117/12.2600209

12 September 2021 Multi-object tracking with deep learning ensemble for unmanned aerial system applications

Wanlin Xie, Jaime Ide, Daniel Izadi, Sean Banger, Thayne Walker, Ryan Ceresani, Dylan Spagnuolo, Christopher Guagliano, Henry Diaz, Jason Twedt

Author Affiliations +

Proceedings Volume 11870, Artificial Intelligence and Machine Learning in Defense Applications III; 118700I (2021) https://doi.org/10.1117/12.2600209
Event: SPIE Security + Defence, 2021, Online Only

Abstract

Multi-object tracking (MOT) is a crucial component of situational awareness in military defense applications. With the growing use of unmanned aerial systems (UASs), MOT methods for aerial surveillance is in high demand. Application of MOT in UAS presents specific challenges such as moving sensor, changing zoom levels, dynamic background, illumination changes, obscurations and small objects. In this work, we present a robust object tracking architecture aimed to accommodate for the noise in real-time situations. Our work is based on the tracking-by-detection paradigm where an independent object detector is first applied to isolate all potential detections and an object tracking model is applied afterwards to link unique objects between frames. Object trajectories are constructed using multiple hypothesis tracking (MHT) framework that produces the best hypothesis based on the kinematic and visual scorings. We propose a kinematic prediction model, called Deep Extended Kalman Filter (DeepEKF), in which a sequence-to-sequence architecture is used to predict entity trajectories in latent space. DeepEKF utilizes a learned image embedding along with an attention mechanism trained to weight the importance of areas in an image to predict future states. For the visual scoring, we experiment with different similarity measures to calculate distance based on entity appearances, including a convolutional neural network (CNN) encoder, pre-trained using Siamese networks. In initial evaluation experiments, we show that our method, combining scoring structure of the kinematic and visual models within a MHT framework, has improved performance especially in edge cases where entity motion is unpredictable, or the data presents frames with significant gaps.

Conference Presentation

Citation Download Citation

Wanlin Xie, Jaime Ide, Daniel Izadi, Sean Banger, Thayne Walker, Ryan Ceresani, Dylan Spagnuolo, Christopher Guagliano, Henry Diaz, and Jason Twedt "Multi-object tracking with deep learning ensemble for unmanned aerial system applications", Proc. SPIE 11870, Artificial Intelligence and Machine Learning in Defense Applications III, 118700I (12 September 2021); https://doi.org/10.1117/12.2600209

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available