Action Recognition with Vision Transformers
Video action classification and spatio-temporal localisation — with an honest negative result.
- Status
- Advanced ML coursework — multi-seed results
- Role
- Solo — training pipeline, ablations, interpretability
- Timeline
- 2026
At a glance
A five-stage study of video action recognition using VideoMAE and TimeSFormer for classification, and a DETR-style head with a frozen backbone for localisation. VideoMAE reaches 92.55% top-1 with spatial augmentation; the localisation head converges but fails to generalise — a documented negative result.
- VideoMAE fine-tuned end to end beat TimeSFormer by ~6 points on a 25-class HMDB51 subset.
- Spatial augmentation was the single largest improvement; dense temporal sampling and temporal augmentation actually hurt.
- The DETR-style localisation head uses a frozen backbone to prevent catastrophic forgetting.
- Interpretability throughout: attention rollout, t-SNE of embeddings, cross-dataset transfer analysis.
- The localisation head is reported as a clear negative result, attributed to class imbalance and insufficient training epochs.
Key numbers
90.43%
Classification top-1
VideoMAE vs 84.57% for TimeSFormer
92.55% ± 1.57
Best config
VideoMAE + spatial augmentation, mean of 3 seeds
+1.6 pts
Biggest single gain
From spatial augmentation alone
0.069 video-mAP@0.5
Localisation
Negative result — documented, not hidden
Tech stack
- Backbones
- VideoMAETimeSFormerPretrained on Kinetics-400Hugging Face
- Localisation
- DETR-style decoderLearnable object queriesFrozen VideoMAE backbonePer-frame box + temporal extent
- Training
- PyTorchTensorBoardCosine / step / warmup schedulersUniform / random / dense sampling
- Analysis
- Multi-seed robustnessAttention rolloutt-SNE embeddingsCross-dataset transfer
The problem
Recognising actions in video needs reasoning over time, not just frames. Beyond classification, the harder task is localisation — saying where and when an action happens, not just that it happened.
This project works through both, in five progressive stages.
Classification
VideoMAE and TimeSFormer backbones pretrained on Kinetics-400 are fine-tuned on a 25-class HMDB51 subset. VideoMAE reached 90.43% top-1 against TimeSFormer's 84.57%. A systematic ablation over sampling strategies, augmentation, and learning-rate schedules found the best configuration at 92.55% ± 1.57 across three seeds — with spatial augmentation contributing the largest single gain and dense temporal sampling hurting.
Localisation
A DETR-style decoder with learnable object queries sits on a frozen VideoMAE backbone, predicting per-frame bounding boxes plus the temporal extent of each action on the 21-class JHMDB dataset. It converged during training but reached only 0.069 video-mAP@0.5 — the head failed to generalise, with only periodic actions showing any success.
Why the negative result matters
The localisation failure is reported in full and attributed to extreme class imbalance and too few training epochs. Being able to run an experiment, get a bad number, and explain it honestly is a core research skill — and more useful than a cherry-picked figure.
Want the details?
The full source, READMEs, evaluation harnesses, and results are on GitHub.
github.com/dariyaDeepanshu/Action_Recognition_Using-_Vision_transformer