Video Action Recognition & Localisation

Recognising actions in video requires reasoning over time, not just frames. Frame-level classifiers miss the motion cues that define actions like falling, waving, or running — and going beyond classification to localise where and when an action happens in a clip is a harder problem again.

Two Kinetics-pretrained backbones — VideoMAE and TimeSFormer — are fine-tuned end-to-end for 25-class classification on HMDB_simp (1,250 clips, seeded reproducible split). A one-factor-at-a-time ablation isolates the effect of temporal sampling strategy, augmentation type, and LR schedule; a 3-seed robustness check quantifies how much accuracy moves on identical settings.

For localisation, a DETR-style detector with learnable object queries and classification/bounding-box/temporal-extent heads is trained on top of the frozen VideoMAE backbone, evaluated on JHMDB (928 clips, 21 classes, per-frame boxes from joint annotations).

VideoMAE beats TimeSFormer by 5.86pp Top-1 (90.43% vs. 84.57%) while being the smaller, cheaper model — 86.3M vs. 121.3M params, 135.1 vs. 190.1 GFLOPs/clip. TimeSFormer overfits fast, hitting 100% training accuracy within a few epochs on 875 clips.

Ablations: spatial augmentation gives the largest single gain over baseline (+1.6pt Top-1); dense sampling and temporal augmentation both hurt accuracy relative to uniform sampling; warmup+cosine slightly beats plain cosine, and a step schedule underperforms both.

Across three seeds, VideoMAE's Top-1 ranges 92.02%–94.68% on the same configuration — evidence that a single-run number understates the true variance of the setup.

Sampling and augmentation strategy changed accuracy more than swapping backbones did. Multi-seed evaluation is what actually revealed that — a single lucky (or unlucky) run would have told a different, wrong story.