Learning human-object interaction detection via deformable transformer

Shuang Cai; Shiwei Ma; Dongzhou Gu

doi:10.1117/12.2606873

11 November 2021 Learning human-object interaction detection via deformable transformer

Shuang Cai, Shiwei Ma, Dongzhou Gu

Proceedings Volume 12076, 2021 International Conference on Image, Video Processing, and Artificial Intelligence; 1207602 (2021) https://doi.org/10.1117/12.2606873
Event: Fourth International Conference on Image, Video Processing, and Artificial Intelligence (IVPAI 2021), 2021, Shanghai, China

Abstract

The goal of human-object interaction (HOI) detection is to localize both the human and object in a picture and recognize the interactions between them. HOIs are always scattering in the image. The traditional methods based on CNNs are unable to aggregate the information scattered in the image. Many new methods utilizing the contextual features cropped from the outputs of the CNNs, which sometimes are not effective enough. To overcome the challenge, we utilize the deformable transformer to aggregate the whole features output form the CNNs. The attention mechanism and query-based predictions are the keys. In view of the success of the methods based on graph neural networks, the attention mechanism is proved to be effective to aggregate the contextual information image-wide. The queries can extract the features of each human-object pair without mixing up the features of other instances. The deformable transformer can extract effective embeddings and the prediction heads can be fairly simple. Experimental results show that the proposed method is effective in HOI detection.

Citation Download Citation

Shuang Cai, Shiwei Ma, and Dongzhou Gu "Learning human-object interaction detection via deformable transformer", Proc. SPIE 12076, 2021 International Conference on Image, Video Processing, and Artificial Intelligence, 1207602 (11 November 2021); https://doi.org/10.1117/12.2606873

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available

Members: $17.00

Non-members: $21.00 ADD TO CART

PROCEEDINGS
6 PAGES

DOWNLOAD PAPER SAVE TO MY LIBRARY

GET CITATION

RIGHTS & PERMISSIONS

Get copyright permission Get copyright permission on Copyright Marketplace

KEYWORDS

Transformers

Computer programming

Target detection

Visualization

Neural networks

Feature extraction

Network architectures

RELATED CONTENT

Summary of fine-grained image recognition based on attention mechanism
Proceedings of SPIE (February 16 2022)

Vehicle-mounted multi-object tracking based on self-query
Proceedings of SPIE (June 15 2022)

LDAM: line descriptors augmented by attention mechanism
Proceedings of SPIE (October 12 2022)

Water obstacle segmentation based on Swin transformer and sea sky...
Proceedings of SPIE (June 27 2022)

Event detection without trigger words on movie scripts
Proceedings of SPIE (November 10 2020)

Intermediate deep-feature compression for multitasking
Proceedings of SPIE (November 18 2019)

Efficient transformer for human-object interaction detection
Proceedings of SPIE (October 11 2023)

Subscribe to Digital Library

Receive Erratum Email Alert

Keywords/Phrases

Search In:

Publication Years