🤖 AI

On Occlusions in Video Action Detection: Benchmark Datasets And Training Recipes

This paper investigates the impact of occlusions on video action detection by introducing five new benchmark datasets and demonstrating that models enhanced with symbolic components and specific training recipes significantly outperform existing approaches in handling both static and dynamic occlusions.

Original authors: Rajat Modi, Vibhav Vineet, Yogesh Singh Rawat

Published 2026-07-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Rajat Modi, Vibhav Vineet, Yogesh Singh Rawat

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you're watching a soccer game on TV, but suddenly, a giant, fluffy dog runs right in front of the screen, blocking your view of the striker. If you're a human, you can still guess the striker is there because you know how the game works. But if you ask a computer to "see" the action, it often panics and says, "I can't find the player!"

That's exactly what this paper investigates: how badly current video-detecting AI fails when things get in the way.

The Big Problem: AI is Easily Distracted

The researchers set up a series of "toy experiments" to test how strong these AI models really are. They took videos of people doing actions (like jumping or running) and digitally pasted other objects—like buses or cats—over them.

The results were a wake-up call. When they blocked just a small part of the actor (20%), the AI's performance dropped by 20% to 50%. If they blocked the whole background, the drop was even worse. This proves that even the smartest, most "state-of-the-art" models today are not robust to real-world occlusions. They aren't naturally tough; they just haven't been trained to handle being blocked.

The New Playground: Five New Datasets

To study this properly, the team couldn't just use old videos. They built five new benchmark datasets from scratch:

O-UCF & O-JHMDB: These are like "training wheels" where they added static (still) or moving objects in a controlled way to test specific levels of blockage.
OVIS-UCF & OVIS-JHMDB: These use "semi-realistic" moving objects that mimic how things actually move in the real world.
Real-OUCF: This is the "final boss" level. These are real videos from YouTube where people are genuinely blocking each other in messy, real-world scenarios.

The Surprising Heroes: Transformers vs. CNNs

For a long time, the AI world thought that if you just taught a model to ignore blocked parts (by using "data augmentation"—basically training it on lots of blocked images), it would get tough. The paper suggests this isn't the whole story.

They found that Transformers (a type of AI architecture) are naturally better at this than the older CNN models, even if the Transformers never saw a single blocked image during training!

The Catch: These Transformers only become super-robust if they are pre-trained on massive datasets first. If you train a Transformer from scratch without that big head start, it performs poorly. It's like a student who needs to read a library of books before they can solve a puzzle; they aren't born knowing the answer, but they have the right "brain structure" to learn it fast.

The Secret Sauce: Capsules and "Islands of Agreement"

The researchers didn't just stop at finding the problem; they cooked up a recipe to fix it. They combined the Transformer backbone with something called Capsules.

Think of Capsules as a team of specialized detectives.

In a normal AI, if a chair blocks a person, the AI gets confused about what is a person and what is a chair.
In this new model, the "Capsule detectives" can spontaneously figure out, "Okay, this group of pixels is the person, and that group is the chair," even if they've never seen a chair blocking a person before.
The paper suggests that these models form "Islands of Agreement." Imagine a group of friends in a noisy room; even if they can't hear the whole conversation, they all agree on who is who. The AI's internal tokens (little pieces of data) start agreeing on where the actor is, creating a stable "island" of truth amidst the noise of the occlusion.

The Winning Recipe: Token Masking

To make these models even tougher, the authors tried a simple trick called Token Masking.

The Analogy: Imagine you are trying to learn a song, but every time you practice, you randomly cover up a few notes on the sheet music. You have to guess the missing notes based on the rest of the melody.
The Result: By randomly "blacking out" parts of the video data during training (without even adding fake objects), the model learned to fill in the gaps better.

The Scoreboard: How Much Better?

The numbers speak for themselves. The new recipe (Transformer + Capsules + Token Masking) beat the old champions by a huge margin:

On the synthetic datasets, it improved performance by 32.3% and 32.7%.
On the tricky, real-world videos, it still managed to beat existing methods by 2.6%.

What They Ruled Out

It's important to note what this paper says doesn't work or isn't the answer:

Just adding more parameters isn't the magic fix. While bigger models generally help, a smaller Transformer with the right architecture beat a much larger CNN.
Static training isn't enough. Models trained only on still images struggle when the blocking object starts moving. The paper shows that models perform significantly worse on dynamic occlusions (moving blockers) than static ones.
Pre-training isn't optional for Transformers. The paper explicitly shows that without pre-training on large datasets, Transformers lose their advantage.

The Bottom Line

The authors conclude that while we haven't "solved" occlusion perfectly (there's still a gap to bridge), we have found a much better way to handle it. By using Transformers that have been pre-trained, adding "capsule" layers to help group pixels correctly, and training them with a "guess the missing piece" strategy, we can build video detectors that are far more resilient to the messy reality of the world.

They've even released their code and datasets so anyone can try to build on this, because the goal is to make sure that when a dog runs in front of a pedestrian, the self-driving car doesn't just freeze—it keeps driving.

Technical Summary: On Occlusions in Video Action Detection: Benchmark Datasets And Training Recipes

Problem Statement
Deep learning has achieved significant success in video action detection, yet a critical gap remains regarding robustness to realistic occlusions. While applications like autonomous driving require reliable localization of actors (e.g., pedestrians) even when partially obscured, existing state-of-the-art (SOTA) models are typically evaluated on datasets where test splits lack realistic occlusions. Preliminary experiments in this work demonstrate that current models suffer significant performance drops (20–50%) when occluders are introduced, verifying that existing approaches lack inherent robustness to varying severities and types of occlusion. Furthermore, there is a lack of standardized benchmarks to systematically study these failures.

Methodology and Benchmark Construction
To address this, the authors introduce a comprehensive benchmark study and five novel datasets designed to evaluate spatio-temporal video action detection under occlusion:

Synthetic Static/Dynamic Datasets: O-UCF and O-JHMDB are generated by superimposing occluders (selected from Pascal-VOC) onto original UCF24 and JHMDB videos. These allow for controlled manipulation of occlusion severity (0–60% area coverage) and type (static vs. dynamic motions like linear, circular, and zoom).
Semi-Realistic Datasets: OVIS-UCF and OVIS-JHMDB utilize occluders with realistic motion trajectories derived from the OVIS dataset, superimposed on the original action videos.
Real-World Dataset: Real-OUCF consists of 1,743 hand-picked YouTube clips featuring natural, mutual occlusions among actors, manually refined to ensure precise action localization.

The study evaluates three SOTA baselines: MOC, YOWO, and VideoCapsuleNet (VCAPS), testing them across different backbones (CNNs like ResNet/ResNext and Transformers like MViTv2). The primary metric is vMAP (video Mean Average Precision) at 0.5 IoU, alongside absolute ( $\delta_a$ ) and relative ( $\delta_r$ ) robustness scores.

Key Findings and Analysis
The benchmarking reveals several critical insights:

Severity and Location: Performance degrades as occlusion severity increases on the actor region, whereas increasing severity in the background has minimal impact.
Static vs. Dynamic: Models generally perform better on static occlusions than dynamic ones, where both occluders and actors move.
Architecture and Pre-training: Transformers (specifically MViTv2) outperform CNN-based models, even when the CNNs are trained with occlusion as a data augmentation. However, this superiority is heavily dependent on large-scale pre-training (e.g., Kinetics-400); transformers trained from scratch perform poorly.
Parameter Count: Increasing parameter size within the same model family generally improves robustness, but efficient Transformers can outperform larger CNNs.

Proposed Solutions and Training Recipes
The paper proposes specific architectural and training strategies to enhance occlusion robustness:

Occlusion Augmentation: Training with synthetic occlusions improves robustness for all models, but the gains are most significant when combined with Transformer backbones.
Capsule Integration: Adding capsule layers to Transformer backbones (forming VCAPS-Mvitv2) enables the network to bind object-specific pixels to specific slots. Qualitative analysis shows these capsules can emergently separate actors from unseen occluders (e.g., chairs, sofas) without explicit supervision, effectively performing unsupervised object discovery.
Token Masking: Inspired by masked image modeling, the authors propose randomly masking input tokens during training. This forces the model to learn robust representations from partial information. This technique further boosts performance, allowing the model to approach its clean-data performance levels even under occlusion.
Emergent "Islands of Agreement": The study observes that without instance-level supervision or contrastive objectives, the transformer layers naturally group pixels into semantic objects, while capsule layers segregate these into instance-specific slots. This emergent behavior facilitates the "binding" mechanism necessary for robust localization.

Results
The proposed VCAPS-Mvitv2 model, trained with occlusion augmentation and token masking, achieves state-of-the-art results on the new benchmarks:

O-UCF: 32.3% improvement over existing detectors.
O-JHMDB: 32.7% improvement.
Real-OUCF: 2.6% improvement.
Clean Benchmarks: The model also achieves high scores on standard clean datasets (83.1% vMAP on UCF-24 and 92.8% on JHMDB-21), demonstrating that robustness does not come at the cost of clean performance.

Significance and Claims
The paper claims to provide the first systematic benchmark study on occlusions in spatio-temporal action detection. Its significance lies in:

Exposing Vulnerability: Formally confirming that current SOTA models are not robust to realistic occlusions, a critical gap for safety-critical applications.
New Standards: Introducing five diverse datasets (O-UCF, O-JHMDB, OVIS-UCF, OVIS-JHMDB, Real-OUCF) to facilitate future research.
Architectural Insights: Demonstrating that combining Transformers with symbolic components (capsules) and simple training recipes (token masking) can induce robustness that outperforms models relying solely on data augmentation.
Emergent Properties: Highlighting that complex behaviors like object/occluder separation can emerge in neural networks without explicit language or contrastive supervision, offering a new perspective on how visual grouping mechanisms function in deep learning.

The authors conclude that while their approach significantly narrows the gap, challenges remain, particularly in predicting missing regions (amodal segmentation) and handling the symmetry issues inherent in overlapping objects. The code and datasets are released to encourage further development in this area.