Spatial-temporal attention in Bi-LSTM networks based on multiple features for video captioning

Chu-yi Li; Wei-yu Yu

doi:10.1117/12.2514651

29 October 2018 Spatial-temporal attention in Bi-LSTM networks based on multiple features for video captioning

Chu-yi Li, Wei-yu Yu

Proceedings Volume 10836, 2018 International Conference on Image and Video Processing, and Artificial Intelligence; 1083616 (2018) https://doi.org/10.1117/12.2514651
Event: 2018 International Conference on Image, Video Processing and Artificial Intelligence, 2018, Shanghai, China

Abstract

Automatically generating rich natural language descriptions for open-domain videos is among the most challenging tasks of computer vision, natural language processing and machine learning. Based on the general approach of encoder-decoder frameworks, we propose a bidirectional long short-term memory network with spatial-temporal attention based on multiple features of objects, activities and scenes, which can learn valuable and complementary high-level visual representations, and dynamically focus on the most important context information of diverse frames within different subsets of videos. From the experimental results, our proposed methods achieve competitive or better than state-of-the-art performance on the MSVD video dataset.

Citation Download Citation

Chu-yi Li and Wei-yu Yu "Spatial-temporal attention in Bi-LSTM networks based on multiple features for video captioning", Proc. SPIE 10836, 2018 International Conference on Image and Video Processing, and Artificial Intelligence, 1083616 (29 October 2018); https://doi.org/10.1117/12.2514651

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available