In:
PLOS ONE, Public Library of Science (PLoS), Vol. 17, No. 3 ( 2022-3-17), p. e0265115-
Abstract:
Most deep learning-based action recognition models focus only on short-term motions, so the model often causes misjudgments of actions that are combined by multiple processes, such as long jump, high jump, etc. The proposal of Temporal Segment Networks (TSN) enables the network to capture long-term information in the video, but ignores that some unrelated frames or areas in the video can also cause great interference to action recognition. To solve this problem, a soft attention mechanism is introduced in TSN and a Spatial-Temporal Attention Temporal Segment Networks (STA-TSN), which retains the ability to capture long-term information and enables the network to adaptively focus on key features in space and time, is proposed. First, a multi-scale spatial focus feature enhancement strategy is proposed to fuse original convolution features with multi-scale spatial focus features obtained through a soft attention mechanism with spatial pyramid pooling. Second, a deep learning-based key frames exploration module, which utilizes a soft attention mechanism based on Long-Short Term Memory (LSTM) to adaptively learn temporal attention weights, is designed. Third, a temporal-attention regularization is developed to guide our STA-TSN to better realize the exploration of key frames. Finally, the experimental results show that our proposed STA-TSN outperforms TSN in the four public datasets UCF101, HMDB51, JHMDB and THUMOS14, as well as achieves state-of-the-art results.
Type of Medium:
Online Resource
ISSN:
1932-6203
DOI:
10.1371/journal.pone.0265115
DOI:
10.1371/journal.pone.0265115.g001
DOI:
10.1371/journal.pone.0265115.g002
DOI:
10.1371/journal.pone.0265115.g003
DOI:
10.1371/journal.pone.0265115.g004
DOI:
10.1371/journal.pone.0265115.g005
DOI:
10.1371/journal.pone.0265115.g006
DOI:
10.1371/journal.pone.0265115.t001
DOI:
10.1371/journal.pone.0265115.t002
DOI:
10.1371/journal.pone.0265115.t003
DOI:
10.1371/journal.pone.0265115.t004
DOI:
10.1371/journal.pone.0265115.t005
DOI:
10.1371/journal.pone.0265115.t006
DOI:
10.1371/journal.pone.0265115.s001
DOI:
10.1371/journal.pone.0265115.s002
Language:
English
Publisher:
Public Library of Science (PLoS)
Publication Date:
2022
detail.hit.zdb_id:
2267670-3
Permalink