Skeleton-based human action recognition with a physics-augmented encoder-decoder network

Hongji Guo; Alexander Aved; Collen Roller; Erika Ardiles-Cruz; Qiang Ji

doi:10.1117/12.2664115

15 June 2023 Skeleton-based human action recognition with a physics-augmented encoder-decoder network

Hongji Guo, Alexander Aved, Collen Roller, Erika Ardiles-Cruz, Qiang Ji

Proceedings Volume 12525, Geospatial Informatics XIII ; 125250K (2023) https://doi.org/10.1117/12.2664115
Event: SPIE Defense + Commercial Sensing, 2023, Orlando, Florida, United States

Conference Poster

Abstract

Human action recognition is important for many applications such as surveillance monitoring, safety, and healthcare. As 3D body skeletons can accurately characterize body actions and are robust to camera views, we propose a 3D skeleton-based human action method. Different from the existing skeleton-based methods that use only geometric features for action recognition, we propose a physics-augmented encoder and decoder model that produces physically plausible geometric features for human action recognition. Specifically, given the input skeleton sequence, the encoder performs a spatiotemporal graph convolution to produce spatiotemporal features for both predicting human actions and estimating the generalized positions and forces of body joints. The decoder, implemented as an ODE solver, takes the joint forces and solves the Euler-Lagrangian equation to reconstruct the skeletons in the next frame. By training the model to simultaneously minimize the action classification and the 3D skeleton reconstruction errors, the encoder is ensured to produce features that are consistent with both body skeletons and the underlying body dynamics as well as being discriminative. The physics-augmented spatiotemporal features are used for human action classification. We evaluate the proposed method on NTU-RGB+D, a large-scale dataset for skeleton-based action recognition. Compared with existing methods, our method achieves higher accuracy and better generalization ability.

Citation Download Citation

Hongji Guo, Alexander Aved, Collen Roller, Erika Ardiles-Cruz, and Qiang Ji "Skeleton-based human action recognition with a physics-augmented encoder-decoder network", Proc. SPIE 12525, Geospatial Informatics XIII , 125250K (15 June 2023); https://doi.org/10.1117/12.2664115

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available