🤖 machine learning

VideoNet: A Large-Scale Dataset for Domain-Specific Action Recognition

To address the decline in action recognition evaluation for vision-language models, this paper introduces VideoNet, a large-scale benchmark covering 1,000 domain-specific actions and a corresponding 500k video QA training dataset, demonstrating that fine-tuning on this data enables smaller models to surpass larger open-weight counterparts in recognizing complex, domain-specific actions.

Original authors: Tanush Yadav, Mohammadreza Salehi, Jae Sung Park, Vivek Ramanujan, Hannaneh Hajishirzi, Yejin Choi, Ali Farhadi, Rohun Tripathi, Ranjay Krishna

Published 2026-05-06

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Tanush Yadav, Mohammadreza Salehi, Jae Sung Park, Vivek Ramanujan, Hannaneh Hajishirzi, Yejin Choi, Ali Farhadi, Rohun Tripathi, Ranjay Krishna

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart robot that can read books, write poetry, and chat about almost anything. You might think, "Great! It can probably watch a video and tell me exactly what's happening, right?"

The paper VideoNet says: "Not so fast."

While these robots (called Vision-Language Models) are amazing at general tasks, they are surprisingly bad at recognizing specific, real-world actions that require expert knowledge. To prove this and fix it, the researchers built a new "gym" for robots called VideoNet.

Here is the story of what they did, explained simply:

1. The Problem: The Robot is a Generalist, Not a Specialist

Think of current AI models like a very well-read high school student. They know a little bit about everything. If you show them a video of someone playing basketball, they can tell you, "That's basketball."

But if you ask them to distinguish between a "Tomahawk Dunk" and an "Alley-Oop Dunk," or tell the difference between a "Triple Axal" and a "Triple Lutz" in figure skating, they get confused. They might guess, but they often get it wrong.

The researchers found that because there wasn't a big, diverse collection of videos showing these specific, tricky moves, the robots had never really practiced them. They were like a chef who knows how to cook a basic omelet but has never seen a soufflé.

2. The Solution: Building "VideoNet" (The Ultimate Trivia Quiz)

To test the robots, the team built VideoNet.

The Scale: It's a massive library containing 1,000 different actions across 37 different worlds (domains).
The Worlds: These range from the familiar (Cooking, Sports, Dancing) to the very specific (Pen Spinning, Neurological Exams, Crochet, and even Suturing).
The Difficulty: They didn't just grab random videos. They created "Hard Negatives."
- Analogy: Imagine a quiz where the question is "What is this?" and the options are "A Golden Retriever" and "A Golden Retriever wearing a tiny hat." That's hard. VideoNet is like that. They found videos where the actions look almost identical to an untrained eye, forcing the AI to look at the tiny details.

3. The Test: Robots vs. Humans

The researchers put the best AI robots (like Gemini and GPT) through a test using VideoNet.

The Result: The robots struggled. Even the smartest ones only got about 70% correct on a multiple-choice test. That sounds okay, but for a super-intelligent robot, it's a failing grade.
The "Few-Shot" Test: They tried to help the robots by showing them a few examples first (like showing a student a practice problem before the real test).
- The Twist: Humans got much better at the task when shown examples. The robots? They barely improved, and some actually got worse. It's as if the robots were confused by the examples rather than learning from them. They couldn't "connect the dots" the way humans do.

4. The Fix: Teaching the Robot from Scratch

The researchers realized the robots weren't failing because they were "stupid"; they were failing because they were untrained in these specific areas.

The Training Data: They built a huge training dataset of nearly 500,000 video clips. They used a clever trick: instead of hiring thousands of experts to label every video (which is too expensive), they used AI to find videos where the action was mentioned in the title or the spoken words (transcript).
The Result: They took a smaller robot (a 4-billion parameter model) and taught it using this new data.
- The Magic: After this training, this smaller robot became better at recognizing these specific actions than the larger, untrained robots (8-billion parameter models). It's like a dedicated apprentice who has practiced a specific craft for years beating a generalist genius who has never touched the tools.

5. The Big Takeaway

The paper concludes that to make AI truly understand the real world, we can't just rely on them "figuring it out" when we ask a question. We need to give them specific, high-quality training data for those specific tasks.

Current State: AI is like a tourist who can recognize a famous landmark but doesn't know how to fix a leaky faucet or perform a specific dance move.
VideoNet's Contribution: It provides the map and the practice drills to turn that tourist into a skilled local expert.

In short: The paper built a giant, difficult test to show that AI is bad at specific real-world skills, and then built a training manual that taught a smaller AI to beat the bigger ones at those specific skills.

Technical Summary: VideoNet

Problem Statement

Action recognition has long been a cornerstone of computer vision, yet modern Vision-Language Models (VLMs) have largely ceased to be evaluated on this task due to a scarcity of diverse, challenging, and domain-specific data. Existing benchmarks often focus on coarse-grained labels (e.g., "rock climbing") or fine-grained temporal attributes that lack real-world applicability (e.g., object rotation direction). Consequently, VLMs are not being tested on their ability to recognize domain-specific actions—such as a "triple flip jump" in figure skating or a "steppage gait" in neurology—which require not only perception but also compositional reasoning to distinguish subtle, expert-level variations. The absence of such data prevents researchers from evaluating the generalizability of VLMs to real-world, expert-level tasks.

Methodology

Benchmark Construction (VideoNet)

The authors introduce VideoNet, a benchmark covering 1,000 distinct actions across 37 domains (e.g., sports, medical, crafts, dance) organized into 7 categories. The construction process involved:

Taxonomy Generation: A top-down approach using expert-written sources and LLMs (augmented with web searches) to define actions and their visual differentiators.
Data Collection Pipeline: A three-stage human annotation process:
- Collection: Annotators find 7 clips per action from distinct web sources.
- Verification: Annotators verify if clips contain the action and are well-trimmed. Non-experts classify clips as "containing" or "not containing" the action to reduce complexity.
- Trimming: Poorly trimmed clips are refined by annotators trained on the action definitions.
Label Verification: Expert verification on a subset of 620 clips confirmed a 97.6% accuracy rate, validating the pipeline's robustness.
Hard Negative Generation: To prevent models from relying on scene-level cues, the authors generated "hard negatives" (actions visually similar to the target but distinct in technique or motion) using LLMs and reasoning models to filter out co-occurring or ambiguous actions.
Evaluation Settings:
- Multiple-Choice: 4 options (1 ground truth, 3 hard negatives) per clip.
- Binary Few-Shot: A binary "Yes/No" question with $k \in \{0, 1, 2, 3\}$ in-context video examples.

Training Dataset

Recognizing that human annotation is too expensive for training-scale data, the authors developed a fully automated pipeline to collect ~500k video question-answer (VQA) pairs (filtered down to ~160k high-quality clips).

Source: Web-crawled videos for the 37 domains.
Localization: Used Gemini 2.5 Flash to localize action timestamps.
Labeling Strategies: Three filtering strategies were tested:
1. TranscriptLocalized: Action name appears in the transcript near the clip.
2. TranscriptLocalizedTitleMatch: Action name in transcript and title.
3. SingleAction: If the title mentions an action and the localizer finds only one clip, that clip is labeled as the action.
Fine-Tuning: A Molmo2-4B model was fine-tuned on these datasets.

Key Results

VLM Performance on VideoNet

Multiple-Choice: Existing open-weight 8B models struggle, achieving only 45.0% accuracy (Qwen3-VL-8B), while the best proprietary model (Gemini 3.1 Pro) reaches 69.9%. Random chance is ~25%.
Binary Zero-Shot: In a relaxed binary setting (random chance 50%), open models reach 59.2% (Qwen3-VL-8B), while proprietary models plateau around 72%.
Few-Shot Learning: VLMs show limited ability to learn from visual in-context examples.
- Non-expert humans improve by +13.6% with 3-shot examples.
- VLMs improve by an average of only +2.9%, with some models (e.g., Gemini 3.1 Pro) actually declining in performance.
- This suggests VLMs lack the perceptual mechanisms for efficient visual few-shot learning compared to humans.

Impact of Training Data

Fine-tuning the Molmo2-4B model on the SingleAction dataset (the smallest but highest-quality subset) yielded an 11.5 percentage point improvement in the multiple-choice setting, reaching 53.5% accuracy.
Crucially, this 4B fine-tuned model surpassed all existing open-weight 8B models and approached the performance of proprietary 8B+ models, demonstrating that domain-specific training data is more critical than model scale for this task.
The fine-tuned 4B model also achieved 66.6% in the binary 0-shot setting, outperforming the 3-shot performance of Qwen3-VL-8B (66.2%).

Significance and Claims

The paper claims that VideoNet revitalizes action recognition as a relevant evaluation task for the VLM era by focusing on domain-specific, fine-grained actions. The authors argue that:

Current VLMs are deficient in recognizing domain-specific actions, struggling even with relaxed binary settings and failing to leverage visual few-shot examples effectively.
Data quality and domain specificity are paramount; a smaller, high-quality training dataset (SingleAction) enabled a 4B model to outperform larger 8B models, challenging the notion that scale alone solves these tasks.
Human vs. Model Gap: The significant gap between human and model few-shot performance highlights a fundamental limitation in current VLMs' ability to learn visual patterns from demonstrations.
Future Research: The dataset serves as a testbed for improving perception, visual reasoning, and real-world action understanding, potentially unlocking applications in expert-assisted domains like medical diagnostics or sports coaching.

The authors emphasize that their work provides the necessary infrastructure (benchmark and training data) to move beyond coarse-grained or synthetic temporal reasoning tasks toward genuine, expert-level video understanding.