🤖 AI

FineGen: A VLM-based Multi-Agent Framework for Fine-Grained Image-Text Dataset Construction

FineGen is a VLM-based multi-agent framework that automates the construction of high-quality, attribute-specific hard negative datasets through a collaborative generation-verification-correction pipeline, significantly enhancing fine-grained perception capabilities in downstream vision-language tasks.

Original authors: Chang Kong, Yuebing Li, Peng Mo, Haigang Zhang, Qiuming Luo

Published 2026-06-09

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Chang Kong, Yuebing Li, Peng Mo, Haigang Zhang, Qiuming Luo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to recognize the difference between a red apple and a green apple.

If you show the robot a picture of a red apple and say, "This is a red apple," and then show it a picture of a car and say, "This is a green apple," the robot will easily learn the difference. It doesn't need to look closely at the color; it just sees that one is a fruit and the other is a machine. This is how most current AI datasets work: they use "easy" examples where the differences are obvious.

But in the real world, we need the robot to spot subtle details. What if the robot needs to tell the difference between a red apple and a green apple that looks almost exactly like the red one? To learn this, the robot needs "hard" practice problems. It needs to see a red apple and be told, "No, this is a green apple," and then be corrected immediately.

The problem is that creating these "hard" practice problems by hand is incredibly expensive and slow. And if you ask a computer to write them automatically, the computer often starts "hallucinating"—making up details that aren't there or creating sentences that don't make sense.

Enter FineGen.

Think of FineGen as a highly organized, three-person team of expert editors working together to build a perfect practice book for AI. Instead of one person doing all the work, they split the job into three specialized roles, working in a loop until the work is perfect.

The Three-Step "Generate-Verify-Correct" Team

Imagine a factory assembly line for creating these tricky practice questions:

The Generator (The Creative Writer):
This agent looks at a photo and writes a description. It also tries to create "trick" versions of that description. For example, if the photo shows a black swan, the Generator might write a trick sentence: "A white swan."
- The Risk: The Generator might get lazy or confused and write something that is actually true (e.g., if the swan was actually white, or if it changed too many things at once).
The Verifier (The Strict Inspector):
This agent is the boss. It looks at the photo and the sentence the Generator made. It asks: "Is this sentence actually false for this picture?"
- If the sentence says "white swan" but the picture shows a black one, the Verifier says, "Good! This is a valid trick question."
- If the sentence is confusing or actually true, the Verifier says, "Nope, this is wrong," and sends it back.
The Corrector (The Editor):
This agent takes the "Nope" feedback from the Verifier and fixes the sentence. It doesn't just throw it away; it rewrites it to be a perfect "hard" example.
- The Loop: The team keeps passing the sentence back and forth (Generate → Verify → Correct) until the sentence is a perfect, logical trick that the AI must look closely at to solve.

What They Built: The "FineGen-100K" Dataset

Using this team of AI agents, the researchers built a massive new dataset called FineGen-100K.

They started with nearly 10,000 images from the famous ImageNet library.
For every single image, they created one perfect description of what is actually there.
Then, they created ten "hard" trick descriptions for each image.
This means for every "easy" example, there are ten "hard" challenges.

The dataset is like a gym for AI eyes. It doesn't just show the AI a picture; it forces the AI to pay attention to tiny details like color, material (is it metal or wood?), texture, and transparency.

The Results: Did It Work?

The researchers tested this new dataset by teaching a standard AI model (CLIP) using their "gym" data.

Quality Check: When humans checked the work, they found that 96.7% of the descriptions were perfect. The "hallucinations" (made-up details) were almost completely eliminated by the team's loop.
Performance Boost: When the AI was tested on a difficult challenge (the FG-OVD benchmark), the model trained on FineGen-100K got 14.4% better at spotting those subtle differences compared to models trained on standard data. It even beat the previous best methods by a significant margin.

In Summary

The paper argues that current AI is good at seeing the "big picture" but bad at seeing the "small details" because it hasn't been trained on enough tricky examples. FineGen solves this by using a team of AI agents that act like a self-correcting editorial board. They generate tricky questions, check if they are fair, and fix them until they are perfect. The result is a dataset that forces AI to stop guessing and start really seeing the fine details of the world.

Technical Summary: FineGen

Problem Statement

Current Vision-Language Pre-training (VLP) models, such as CLIP and ALIGN, excel at capturing global semantics but struggle with fine-grained attribute perception (e.g., distinguishing subtle differences in color, material, texture, or object state). This limitation hinders their deployment in precision-sensitive applications like open-vocabulary detection and detailed image editing.

The primary bottleneck is the scarcity of high-quality hard negative samples in existing datasets (e.g., MS COCO, LAION). Standard datasets rely on coarse-grained alignment and random batch sampling, where negative pairs are often semantically distant from the anchor image. Consequently, models learn to rely on global context shortcuts rather than discerning specific attribute details. Furthermore, constructing large-scale datasets with dense attribute annotations and valid hard negatives is prohibitively expensive for manual annotation. While automated generation using Large Language Models (LLMs) or Vision-Language Models (VLMs) offers a solution, unconstrained generation frequently suffers from hallucinations (generating plausible but visually incorrect details) and fails to ensure that generated negatives are both "hard" (minimal contrastive pairs) and "valid" (visually contradictory to the image).

Methodology: The FineGen Framework

To address these challenges, the authors propose FineGen, a VLM-based Multi-Agent framework designed for automated, fine-grained image-text dataset construction. Inspired by human collaborative workflows, FineGen decomposes the task into a modular pipeline featuring a closed-loop "Generation-Verification-Correction" feedback mechanism.

System Architecture:
The framework employs three specialized agent roles to prevent "self-deception" and ensure visual grounding:

Generative Agents (G): Act as the creative engine. They translate visual signals into initial positive descriptions or perform controlled linguistic perturbations to generate hard negative candidates.
Verification Agents (V): Act as "judges" with strong visual-grounding capabilities. They independently assess the consistency between the generated text and the raw image, detecting hallucinations, missing attributes, or logical inconsistencies, and output structured error logs.
Correction Agents (C): Act as "editors." Instead of regenerating from scratch, they perform targeted rewriting based on the explicit error logs provided by the Verifiers, ensuring convergence toward high-quality samples.

Pipeline Stages:
The construction process is divided into three phases:

Phase I: Visual-Grounded Positive Construction: A Multimodal Generator creates an initial description ( $T^+$ ) focusing on fine-grained attributes (texture, transparency, state). A Verifier checks alignment with the image. If inconsistencies are found ( $v=0$ ), a Corrector iteratively refines the text until a valid, visually grounded description is achieved.
Phase II: Visual-Aware Hard Negative Mining: Leveraging the validated $T^+$ , a Negative Generator performs controlled perturbations (e.g., changing "red" to "blue") to create candidates ( $T^-$ ). A Negative Verifier performs a dual-check: ensuring Visual Contradiction (the perturbed attribute is indeed false for the image) and Semantic Plausibility (the sentence remains logically coherent). Invalid samples are corrected iteratively.
Phase III: Atomic Sub-text Extraction: The global descriptions are decomposed into atomic noun-adjective pairs ( $S$ ). These units are filtered and labeled (binary $y \in \{0,1\}$ ) to support region-level dense alignment.

Constraint-Based Prompt Engineering:
To ensure strict adherence to the "minimal contrastive pair" assumption, the authors employ Constraint-based In-Context Learning (ICL). This includes:

Persona-based Role Alignment: Assigning specific personas (e.g., "Strict Logic and Visual Inspector") to stabilize output style.
Hard Structural Constraints: Enforcing strict syntactic rules (e.g., retaining original sentence structure, outputting JSON for sub-text) to prevent paraphrasing that would invalidate the contrastive signal.
Feedback-Injection Mechanism: Dynamically integrating verification error logs into the prompt for the Corrector, enabling "Reasoning-Action" (ReAct) updates to fix specific flaws without introducing new hallucinations.

Key Contributions

FineGen Framework: A novel VLM-based multi-agent system that automates fine-grained dataset construction by decoupling Generation, Verification, and Correction roles, achieving scalable synthesis without manual intervention.
Closed-Loop Verification: A "Generate-Verify-Correct" mechanism that enables agents to iteratively self-correct hallucinations and invalid hard negatives, ensuring synthesized samples are visually grounded minimal contrastive pairs rather than semantic noise.
FineGen-100K Dataset: A large-scale hierarchical dataset constructed from ImageNet, featuring:
- 9,997 unique images with a rigorous 1:10 positive-to-negative ratio.
- 147,381 total edits, covering four attribute dimensions: Color (66.48%), Material (24.04%), Pattern (6.59%), and Transparency (2.89%).
- Hierarchical annotations including global positive descriptions, hierarchical hard negatives (stratified by difficulty), and atomic sub-text units.

Results

The authors conducted extensive evaluations to validate the quality and utility of FineGen-100K:

Data Quality (Human Evaluation): A rigorous human inspection of 11,423 annotation units revealed an error rate of only 3.3%, corresponding to a 96.7% attribute validity rate. This confirms the multi-agent closed-loop mechanism effectively filters noise typically associated with VLM-generated data.
Downstream Performance (FG-OVD Benchmark): Fine-tuning a standard CLIP (ViT-B/16) model on FineGen-100K (referred to as SubCLIP) yielded significant improvements on the FG-OVD benchmark:
- Hard Split: Achieved 34.53% Top-1 Accuracy, a substantial +14.4% improvement over the zero-shot CLIP baseline and nearly 5% higher than the previous state-of-the-art method (Fine-CLIP).
- Mechanism Analysis: The authors attribute this gain to the dense, high-quality hard negatives forcing the model to abandon global "bag-of-words" shortcuts and strictly converge on localized visual features, thereby generating hard contrastive gradients.

Significance and Claims

The paper claims that FineGen successfully bridges the gap between fine-grained evaluation and model training by generating large-scale, reliable training data directly from raw images. The significance lies in demonstrating that automated, multi-agent collaboration with closed-loop verification can overcome the limitations of both manual annotation (scalability) and single-pass generation (hallucinations).

The authors assert that the resulting FineGen-100K dataset equips VLMs with precise fine-grained discriminative capabilities, enabling them to transcend coarse semantic matching. While acknowledging potential limitations regarding Verifier Bias (where the dataset might inherit blind spots from the underlying VLMs used for verification), the authors note that the substantial performance gains on diverse, real-world benchmarks suggest these biases do not currently degrade generalizability. Future work is proposed to extend this paradigm to complex modalities like video understanding.