A Closer Look at Spatiotemporal Convolutions for Action Recognition

Abstract

In this paper we discuss several forms of spatiotemporal convolutions for video analysis and study their effects on action recognition. Our motivation stems from the observation that 2D CNNs applied to individual frames of the video have remained solid performers in action recognition. In this work we empirically demonstrate the accuracy advantages of 3D CNNs over 2D CNNs within the framework of residual learning. Furthermore, we show that factorizing the 3D convolutional filters into separate spatial and temporal components yields significantly gains in accuracy. Our empirical study leads to the design of a new spatiotemporal convolutional block "R(2+1)D" which produces CNNs that achieve results comparable or superior to the state-of-the-art on Sports-1M, Kinetics, UCF101, and HMDB51.

Keywords

Action recognitionComputer scienceResidualBlock (permutation group theory)Artificial intelligenceConvolutional neural networkPattern recognition (psychology)Action (physics)MathematicsAlgorithmClass (philosophy)

Affiliated Institutions

Related Publications

OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields

Zhe Cao , Gines Hidalgo , Tomas Simon +2 more

Realtime multi-person 2D pose estimation is a key component in enabling machines to have an understanding of people in images and videos. In this work, we present a realtime app...

2019 IEEE Transactions on Pattern Analysis... 4593 citations

Image Super-Resolution Using Very Deep Residual Channel Attention Networks

Yulun Zhang , Kunpeng Li , Kai Li +3 more

Convolutional neural network (CNN) depth is of crucial importance for image super-resolution (SR). However, we observe that deeper networks for image SR are more difficult to tr...

2018 Lecture notes in computer science 5131 citations

CBAM: Convolutional Block Attention Module

Sanghyun Woo , Jongchan Park , Joon‐Young Lee +1 more

We propose Convolutional Block Attention Module (CBAM), a simple yet effective attention module for feed-forward convolutional neural networks. Given an intermediate feature map...

2018 Lecture notes in computer science 20102 citations

Residual Dense Network for Image Super-Resolution

Yulun Zhang , Yapeng Tian , Yu Kong +2 more

A very deep convolutional neural network (CNN) has recently achieved great success for image super-resolution (SR) and offered hierarchical features as well. However, most deep ...

2018 3866 citations

Learning and Transferring Mid-level Image Representations Using Convolutional Neural Networks

Maxime Oquab , Léon Bottou , Ivan Laptev +1 more

Convolutional neural networks (CNN) have recently shown outstanding image classification performance in the large- scale visual recognition challenge (ILSVRC2012). The success o...

2014 3151 citations

Publication Info

Year: 2018
Type: article
Pages: 6450-6459
Citations: 3320
Access: Closed

External Links

View on DOI.org

Social Impact

Altmetric

A Closer Look at Spatiotemporal Convolutions for Action Recognition

PlumX Metrics

Social media, news, blog, policy document mentions

Citation Metrics

3320

OpenAlex

Cite This

APA Style

                            
                                    Du Tran, 
                                
                                    Heng Wang, 
                                
                                    Lorenzo Torresani
                                
                                et al.
                            
                            (2018). 
                            A Closer Look at Spatiotemporal Convolutions for Action Recognition. 
                            
                            , 6450-6459.
                            https://doi.org/10.1109/cvpr.2018.00675

Identifiers

DOI: 10.1109/cvpr.2018.00675