💻 computer science

Gen2Balance: Generative Balancing for Long-Tailed Video Action Recognition

Gen2Balance addresses the challenge of long-tailed video action recognition by augmenting imbalanced training sets with text-to-video generated clips conditioned on action profiles, utilizing a two-stage training strategy to achieve significant accuracy improvements on rare actions while demonstrating practical scalability through partial balancing.

Original authors: Prajwal Gatti, Simon Jenni, Fabian Caba Heilbron, Dima Damen

Published 2026-06-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Prajwal Gatti, Simon Jenni, Fabian Caba Heilbron, Dima Damen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to recognize different human actions, like "swimming," "cooking," or "dancing." You give it a massive library of videos to study. However, there's a huge problem: the library is heavily unbalanced.

Think of it like a music playlist where 99% of the songs are by one famous pop star, and only a few songs are by obscure indie bands.

The "Head" (Popular): The robot sees thousands of videos of "playing basketball." It becomes an expert at this.
The "Tail" (Rare): The robot sees only five videos of "cutting a keyboard" or "swinging on a tire." It barely learns anything about these, so when it sees them in real life, it gets confused and guesses wrong.

This is the Long-Tailed Problem. The robot is great at common things but terrible at rare ones.

The Solution: Gen2Balance (The "AI Chef")

The paper introduces a new method called Gen2Balance. Instead of just hoping to find more rare videos on the internet (which is hard because they are rare), the authors decided to cook up new videos using an AI generator.

Here is how they did it, broken down into simple steps:

1. The Smart Recipe Book (The Prompt Pipeline)

If you just tell an AI, "Make a video of 'Robot Dancing'," it might get confused. Does it mean a human dancing like a robot, or an actual metal robot dancing? If you just ask for "cutting a keyboard," the AI might make a weird, scary video that doesn't look like a real action.

To fix this, the authors built a three-step recipe system using a smart AI (a Large Language Model):

Step A: The Action Profile. They ask the AI to write a detailed definition of the action, including what it is and what it isn't (e.g., "A human moving like a robot, not a metal machine").
Step B: The Examples. They show the AI a few real videos of the action so it understands the context (e.g., "See this human? That's what we mean").
Step C: The Variety. They tell the AI to make 100 different versions of this video: change the lighting, the background (a park vs. a kitchen), the person's clothes, and the camera angle.

The Result: They created 140,000 brand new, synthetic videos that look real but cover the rare actions the robot was missing.

2. The Two-Stage Training (The Study Plan)

You can't just dump these new fake videos into the robot's brain and hope for the best. The robot might get confused because the fake videos look slightly different from real life (like a cartoon vs. a photo).

So, they used a two-stage study plan:

Stage 1: The "Balanced" Study. The robot studies the new, balanced library (Real Videos + Generated Videos). Because there are now plenty of "rare" examples, the robot learns to recognize them. However, the robot is told to pay extra attention to the real videos when deciding how important each category is.
Stage 2: The "Reality Check." The robot goes back to studying only the real videos for a short time. This is like a final exam review. It helps the robot forget the "fake" quirks and get its memory back on track with how the real world actually looks.

The Results: Did it Work?

The authors tested this on standard video datasets (UCF-101 and Kinetics) that they artificially made "long-tailed" to simulate the problem.

The Big Win: Gen2Balance beat all previous methods. On the Kinetics dataset, it improved accuracy by 7% over the best existing methods. On the UCF dataset, it improved by 5%.
The Rare Actions: For the hardest, rarest actions (like "cutting a keyboard"), the improvement was massive—31.9% better than before.
Cost Efficiency: They found you don't need to generate everything to get great results. If they only generated enough videos to fill up 27% of the missing data, they still got 79% of the total performance boost. This means the method is practical and doesn't require infinite computing power.

Why This Matters (According to the Paper)

The paper argues that while other methods try to "stretch" the few rare videos they have (which doesn't add new information), Gen2Balance creates new information. It fills in the gaps in the robot's knowledge by inventing diverse, realistic examples of rare actions, allowing the robot to finally learn what it was missing.

In short: Gen2Balance is like giving a student a textbook that has been rewritten to include hundreds of practice problems for the topics they were weak in, followed by a quick review of the original textbook to make sure they didn't get confused by the new examples.

Technical Summary: Gen2Balance

Problem Statement

Long-tailed video action recognition remains a fundamental challenge where class distributions are heavily skewed, with a few "head" classes dominating the dataset and many "tail" or "few-shot" classes having scarce samples. While long-tailed learning has seen progress in image recognition, video recognition lags significantly behind. Existing approaches, such as re-sampling, re-weighting, and feature-space augmentation, often fail to bridge the performance gap between balanced and long-tailed benchmarks. For instance, on the UCF-101 dataset, accuracy drops from 92% in balanced settings to 64% under long-tailed imbalance. Furthermore, current video generation models face challenges regarding temporal dynamics, physical plausibility, and object permanence, raising questions about whether synthetic videos can be sufficiently diverse and faithful to action classes to aid recognition.

Methodology: Gen2Balance

The authors propose Gen2Balance, a framework that addresses long-tailed video recognition by augmenting the training set with synthetic videos generated via a text-to-video model. The method consists of two primary components: a generative filling pipeline and a two-stage training strategy.

1. Generative Filling Pipeline

To synthesize diverse, class-faithful videos for under-represented classes, the authors employ a structured pipeline using a multimodal Large Language Model (LLM) and a text-to-video generator (WAN 2.1). The pipeline addresses semantic ambiguity and lack of diversity through four steps:

Diversifying Text Prompts: The LLM generates prompts varying along nine axes (e.g., environment, camera framing, actor demographics, lighting, social context) to ensure visual diversity.
Action Profiles: To resolve semantic ambiguities (e.g., "Robot Dancing" referring to a human dance style rather than a mechanical robot), the LLM generates an "Action Profile" containing a definition, positive constraints (key visual features), and negative constraints (common misconceptions).
In-Context Exemplars: To further ground the action profile in the correct semantics, the LLM is provided with a few-shot set of real training videos as in-context exemplars. Crucially, these exemplars inform the profile generation but not the final prompts directly, preserving prompt diversity.
Video Generation: The diversified prompts, conditioned on the grounded action profiles, are fed into the text-to-video generator to produce synthetic clips.

2. Two-Stage Training Strategy

Directly training on a mix of real and synthetic data introduces risks of overfitting to synthetic artifacts and domain shift. To mitigate this, Gen2Balance employs a two-stage training process:

Stage 1 (Learning from Augmented Data): The model is trained on the combined dataset ( $D_{train} \cup D_{gen}$ ) using Balanced Softmax (BSCE) loss. Crucially, the loss margins are calculated based on the real data frequencies ( $N_c$ ) rather than the augmented frequencies. This allows the model to leverage the abundance of synthetic data for feature learning while maintaining decision boundaries calibrated to the true class priors.
Stage 2 (Rehearsal on Real Data): The model is fine-tuned exclusively on the original real training set ( $D_{train}$ ) using the same BSCE loss with a reduced learning rate. This stage corrects domain shifts and prevents catastrophic forgetting of representations learned from synthetic data.

Key Contributions

Framework Proposal: Introduction of Gen2Balance, a generative balancing approach that converts imbalanced video datasets into balanced combinations of real and synthetic clips.
Automatic Pipeline & Dataset: Development of an automatic pipeline using a multimodal LLM (Gemini 2.5 Pro) and video generator (WAN 2.1) to synthesize class-faithful videos. The authors publicly release a complementary dataset of 140K synthetic videos with detailed prompts spanning 223 action classes across UCF-LT, K100-LT, and RareAct.
Training Strategy Analysis: Demonstration that a two-stage training strategy (augmented then real) combined with a class-balanced loss using real-data-derived margins yields superior results compared to standard training or single-stage approaches.
Comprehensive Evaluation: Extensive evaluation on long-tailed splits of Kinetics-100 (K100-LT) and UCF-101 (UCF-LT), as well as rare actions from the RareAct dataset.

Experimental Results

The authors evaluate Gen2Balance on long-tailed versions of standard benchmarks:

K100-LT: Gen2Balance improves class-average accuracy by 7.0% over the strongest long-tail baseline (BSCE), achieving 72.6% overall accuracy. It significantly boosts few-shot class accuracy by +38.8%.
UCF-LT: The method improves accuracy by 5.1% over the strongest baseline, reaching 88.9% overall, with a +48.0% gain on few-shot classes.
RareAct: On rare, compositional actions (e.g., "cut keyboard"), Gen2Balance improves accuracy by 31.9%, demonstrating effectiveness for genuinely scarce actions.
Ablation & Scaling:
- Partial Balancing: Achieving 79% of the performance gains with only 27% of the compute cost (by setting a partial filling threshold) highlights the method's scalability.
- Data Sources: Gen2Balance outperforms web-retrieval augmentation and static image generation, confirming the necessity of motion cues and the superiority of the specific prompting pipeline over naive templating.
- Robustness: The method remains robust even when few-shot classes are statistically rare in the generator's training distribution (Web-Popularity re-indexing).

Significance and Claims

The paper claims that Gen2Balance represents a significant step forward in long-tailed video recognition by leveraging generative models to synthesize missing diversity rather than relying on web retrieval or feature-space interpolation. The authors emphasize that:

Current generative models, when carefully prompted with grounded action profiles, produce videos of sufficient quality to achieve state-of-the-art performance in long-tailed settings.
The two-stage training strategy effectively mitigates domain shift, allowing models to learn from synthetic data without compromising performance on real data.
The approach is practical and scalable, as partial balancing offers substantial gains at a fraction of the computational cost.
The released dataset of 140K synthetic videos serves as a resource for future research in generative balancing for video understanding.

The work does not claim to solve all issues in video generation (acknowledging challenges with physics and object permanence) but argues that for the specific task of action recognition, the generated content is sufficiently diverse and semantically correct to overcome data scarcity.