VDCook:DIY video data cook your MLLMs

VDCook is a self-evolving, configurable video data operating system that enables researchers and domain teams to automatically generate, update, and manage specialized video training datasets for MLLMs through natural language queries, integrated retrieval-synthesis modules, and automated metadata annotation.

The Gist

Imagine you are a chef trying to make a perfect, custom dish for a very specific group of guests.

The Old Way (Traditional Datasets):
In the past, getting data for AI was like buying a giant, pre-packaged frozen meal from a supermarket. You get a box labeled "Video Data." It might have 10 million clips, but it's a mix of everything: cooking shows, cat videos, news, and cartoons.

• The Problem: If you want to train an AI to recognize only "dump trucks in construction zones" or "Chinese ink-wash paintings," you have to dig through the whole box, throw away 99% of it, and hope you find enough of what you need. Once you open the box, you can't add new ingredients. If the world changes, your frozen meal is stale.

The New Way (VDCook):
The paper introduces VDCook, which is like a smart, self-evolving kitchen for AI researchers. Instead of giving you a frozen meal, it gives you a fully equipped kitchen, a personal shopper, and a robot chef that works on demand.

Here is how VDCook works, broken down into simple steps:

1. You Order What You Want (The "Cooking" Request)

Instead of downloading a massive file, you just tell the system what you need in plain English.

• You say: "I need 5,000 videos of people falling down in the rain, but they must be high quality, and I want 20% of them to be generated by AI to fill in the gaps."
• The System: Instead of just searching, it understands your request like a smart sous-chef.

2. The Smart Shopping (Data Acquisition)

The system has two ways to get ingredients:

• The Web Crawler (MCP): It automatically scours the internet for videos that match your order, just like a shopper grabbing items from a store.
• Your Pantry: You can upload your own private videos (like a company's internal safety footage), and the system treats them with the same care as public videos.

3. The "Prep" Station (Metadata Enrichment)

This is the magic part. In the old days, chefs would throw away vegetables that looked a little weird. VDCook does the opposite.

• It takes every single video clip and tags it with a million details before deciding what to keep.
• It asks: "How fast is the camera moving?" "Is there text on the screen?" "Is the lighting good?" "Does the person look happy?"
• The Analogy: Imagine a librarian who doesn't just put books on a shelf; they read every book, summarize it, and write a detailed index card for it. Now, you can find any specific sentence instantly, even if it was in a book you didn't think you needed.

4. The "Cooking" Process (Retrieval & Synthesis)

Now the system "cooks" your dataset:

• Retrieval: It grabs the real videos that match your tags.
• Synthesis (The Secret Sauce): If you don't have enough videos of a rare event (like a specific type of car crash or a rare animal), the system uses a powerful AI to create new, realistic videos based on the real ones it found. It's like a baker who, realizing they are out of blueberries, uses a machine to bake perfect blueberry muffins that taste just like the real thing to fill the gap.

5. The Tasting Menu (Evaluation)

Before you serve the dish, the system runs a taste test. It trains a small AI model on your new dataset and checks: "Does this model actually get better at recognizing what we asked for?" If the answer is yes, the dataset is ready.

Why is this a Big Deal?

• It Never Gets Stale: The kitchen is always open. If new videos appear on the internet today, VDCook can add them to your dataset tomorrow. It's a "living" dataset, not a frozen one.
• It's Custom: You aren't forced to use a generic dataset. You can "cook" a dataset specifically for medical robots, self-driving cars, or art history.
• It Saves Time: Researchers don't need to spend months cleaning data. They just ask for it, and the system does the heavy lifting.

In a Nutshell:
VDCook turns data creation from a one-time shopping trip into a continuous, customizable cooking service. It allows anyone to build the perfect, high-quality video dataset for their specific needs, filling in the gaps with AI-generated ingredients, all while keeping a detailed recipe book so the process can be repeated perfectly anytime.

Technical Summary

Here is a detailed technical summary of the paper "VDCook: DIY video data cook your MLLMs".

1. Problem Statement

The construction of high-quality, domain-specific video datasets for Multimodal Large Language Models (MLLMs) faces significant challenges:

• Static Nature: Traditional datasets are "one-shot" offline releases. Once published, they are static, making adaptation to new domains or evolving research demands costly and requiring a full pipeline rebuild.
• Processing Barriers: Existing public corpora are often too large for local processing, while creating domain-specific subsets requires complex, non-reproducible indexing and curation pipelines.
• Data Loss in Filtering: Conventional pipelines aggressively filter data during preprocessing, discarding potentially useful clips and losing rich metadata signals that could be valuable for flexible downstream selection.
• Scarcity of Long-Tail Data: Rare events (e.g., specific urban risks, medical imaging) and niche artistic styles are underrepresented in generic web-crawled data.

2. Methodology: The VDCook System

VDCook is introduced as a self-evolving video data operating system that shifts dataset construction from a static artifact to a dynamic, configurable service. The system operates on a "cook-first" philosophy where data is enriched before being filtered.

Core Architecture

The system follows a closed-loop architecture comprising six key modules:

1. Multi-Source Data Acquisition:

   • Automated Crawling: Uses a Modular Crawling Protocol (MCP) to optimize user queries, perform domain-specific filtering, and periodically re-crawl sources for dynamic updates.
   • User Contributions: Supports the ingestion of proprietary or private-domain video data, processed through the same standardized pipeline to ensure consistency.

2. Unified Video Processing Pipeline (Enrichment-First):

   • Instead of immediate hard filtering, the system processes all videos (except clips <2s) through modular enrichment modules:
     • Scene segmentation.
     • Motion scoring and categorization.
     • OCR text ratio estimation.
     • Automatic video captioning.
     • Metadata standardization.
   • Key Design Principle: "Annotate-first, filter-later." Extracted signals are stored as structured attributes (indexing signals) rather than discarded, allowing downstream users to define selection criteria dynamically.

3. Indexing, Retrieval, and "Cooking":

   • Query Optimization: An agent expands natural language queries into multiple retrieval templates.
   • Retrieval & Synthesis: The system retrieves candidates based on user parameters (scale, quality thresholds) and a configurable retrieval-synthesis ratio.
   • Controllable Synthesis: If synthesis is requested, a controllable engine generates variants using seeds, keyframes, pose traces, and style labels.
   • Packaging: Clips are filtered based on job-specific policies and packaged into reproducible manifests with complete provenance.

4. Model Annotation Center:

   • Integrates an ensemble of large-scale models (Vision-Language, Action Recognition, OCR, Temporal Consistency) to provide multi-perspective labeling.
   • Supports custom model plugging for domain adaptation.

5. Baseline Evaluation Platform:

   • Provides benchmark models and metrics for video generation, action recognition, and temporal reasoning.
   • Allows users to fine-tune baselines on generated datasets to obtain quantitative feedback on data quality.

6. Long-Tail Data Augmentation (Self-Bootstrapping):

   • For scarce domains, the system synthesizes new samples conditioned on retrieved real clips and structured attributes.
   • Synthetic data is treated as a new source, re-annotated, and re-injected, creating a Data-Model Co-evolution loop.

3. Key Contributions

• System-Level Rethinking: Proposes a paradigm shift from fixed offline releases to an on-demand, configurable "cooking" process for video datasets.
• Decoupled Enrichment and Filtering: Introduces a modular pipeline that preserves rich metadata (motion, OCR, scene) as flexible indexing signals, minimizing information loss.
• MCP-Based Dynamic Ingestion: Integrates a standardized crawling protocol that enables continuous dataset evolution, reproducibility, and community contribution.
• Synthetic Data Integration: Establishes a mechanism where strong generative models amplify long-tail distributions, which are then validated and re-injected into the ecosystem.
• Open Infrastructure: Provides a platform with reproducible Notebooks, complete provenance tracking, and governance-enabled data expansion.

4. Results and Evaluation

The authors present a corpus of 100M+ video clips and demonstrate the system's flexibility through several domain-specific "cooked" datasets:

• Corpus Statistics:

  • Scale: >100M clips (exceeding many public datasets like Panda-70M or Koala-36M).
  • Richness: Average caption length of ~266 words (vs. 13–202 in other datasets), supporting dense captioning.
  • Quality: 32.7% at 1080p, with significant 4K content.
  • Metadata: High OCR coverage (avg 1.94 boxes/clip) and diverse motion intensity (mean score 61.6).

• Domain Demonstrations:

  • Long-Tail Events: Successfully curated rare scenarios like road waterlogging, dump trucks in construction, and fallen urban greenery.
  • Specialized Domains: Created datasets for medical imaging (Pulmonary CT angiography) and embodied sequential manipulation.
  • Artistic Styles: Synthesized and curated Chinese ink-wash style videos.

• Model Validation (Ink-Wash Adaptation):

  • Experiment: Fine-tuned the Wan-1.3B generative model on the curated ink-wash subset.
  • Outcome: The fine-tuned model showed significantly improved aesthetic alignment compared to the base model, exhibiting stronger brushstroke textures, reduced photorealistic artifacts, and better ink diffusion effects. This validates that VDCook-generated data effectively guides stylistic adaptation.

5. Significance

VDCook addresses the critical bottleneck of data scarcity and rigidity in MLLM development. Its significance lies in:

• Democratization: Lowering the technical barrier for researchers and vertical teams to build specialized video datasets without needing massive engineering resources.
• Reproducibility: Transforming dataset creation into a transparent, reproducible workflow with full provenance tracking.
• Adaptability: Enabling continuous updates and domain expansion, ensuring datasets remain relevant as research needs and model capabilities evolve.
• Infrastructure-Centric Research: Promoting a shift toward infrastructure-centered data practices, where the "cooking" process is as important as the final dataset, fostering a living ecosystem of multimodal data.

In conclusion, VDCook reframes video dataset construction as a continuous, configurable service, providing a robust foundation for the next generation of scalable and domain-specific video-language models.