💻 computer science

FineVision: Open Data Is All You Need

FineVision introduces the largest open corpus of 24 million meticulously curated vision-language samples, created through a rigorous semi-automated pipeline that unifies over 200 sources with human oversight to eliminate contamination and duplication, ultimately enabling models trained on this dataset to significantly outperform existing open alternatives.

Original authors: Luis Wiedmann, Orr Zohar, Amir Mahla, Xiaohan Wang, Rui Li, Thibaud Frere, Leandro von Werra, Aritra Roy Gosthipaty, Andrés Marafioti

Published 2026-05-21

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Luis Wiedmann, Orr Zohar, Amir Mahla, Xiaohan Wang, Rui Li, Thibaud Frere, Leandro von Werra, Aritra Roy Gosthipaty, Andrés Marafioti

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to see and understand the world, like a human does. To do this, you need to show it millions of pictures and teach it how to talk about them. This is what "Vision-Language Models" (VLMs) do.

However, until now, the "textbooks" (datasets) available to the public were a mess. They were like a library where some books were written in different languages, some pages were torn out, some had typos, and some were even copies of the test questions the robot would be graded on later. This made it hard for open-source robots to learn as well as the expensive, secret robots built by big tech companies.

FineVision is a new, massive, and meticulously cleaned-up library created by researchers from Hugging Face, the Technical University of Munich, and Stanford. Here is how they built it and why it matters, explained simply:

1. The Great Cleanup (Data Curation)

The researchers gathered over 200 different sources of data—everything from academic papers to cloud storage folders. But just dumping them together wouldn't work.

The Translator: They built a "semi-automated" system (using AI helpers) to translate all these different formats into one standard language. Imagine taking 200 different types of recipes (some in French, some in Japanese, some written on napkins) and converting them all into a single, easy-to-follow cookbook format.
The Human Editors: AI isn't perfect. So, human reviewers acted as editors. They checked the AI's work, fixed mistakes, and made sure the "recipes" were safe and accurate. If the AI messed up a conversion, the humans fixed it and told the AI to try again.
The De-Duplication: They used a special "copy detector" to find and remove duplicate images. If the same picture of a cat appeared 50 times in the library, they kept only one (or merged them into a single, richer conversation) so the robot didn't just memorize the same cat over and over.
The Test-Safe Zone: They checked the library against 66 standard tests used to grade these robots. If they found a picture in the library that was identical to a picture on a future test, they removed it. This ensures the robot is actually learning, not just cheating by memorizing the answers.

2. What's Inside the Library?

The final result is FineVision, a collection of 24 million samples (about 17 million images). It's not just a pile of pictures; it's organized into conversations.

Variety: It covers everything from simple "What is this?" questions to complex tasks like reading a chart, solving math problems, or understanding a document.
The "Action" Section: A special part of the library teaches the robot how to use computer interfaces (like clicking a mouse or typing on a phone screen). They standardized this so the robot learns a universal "language of action" that works on desktops, phones, and browsers.
Quality Control: Every single conversation in the library was scored by AI judges (and checked by humans) on four things:
1. Formatting: Is the text clean and readable?
2. Relevance: Does the answer actually match the question?
3. Visual Dependency: Does the answer need to look at the picture to be correct?
4. Correspondence: Does the picture actually show what the question is asking about?

3. The Results: Does It Work?

The researchers trained a small robot model using this new library and compared it to models trained on other popular, messy libraries.

The Scoreboard: The robot trained on FineVision scored significantly higher on 11 different tests. It beat the previous best open-source libraries by a wide margin (improving performance by roughly 11% to 38% depending on the competitor).
The "Cheating" Test: When they removed the few remaining "cheat" images (data that might have leaked from the tests) from the training data, the FineVision robot's performance barely dropped. In contrast, the other robots' performance fell significantly. This proves that FineVision taught the robot to understand, not just memorize.
The GUI Magic: The robot trained on FineVision was also much better at navigating computer screens (clicking buttons, typing) than other small models, performing as well as models that were four times larger.

The Bottom Line

The paper claims that clean, diverse, and well-organized data is the secret sauce. You don't necessarily need a bigger model or a secret dataset; you just need a better library. FineVision proves that by cleaning up the data and organizing it carefully, open-source models can finally catch up to the big, closed-source ones.

The researchers have released the library and the tools they used to clean it, hoping to help the whole community build smarter, more reliable AI vision systems.

Technical Summary: FineVision: Open Data Is All You Need

Problem Statement

The advancement of Vision-Language Models (VLMs) is currently hindered by a fragmented landscape of public datasets. While proprietary models leverage massive, curated corpora, open-source alternatives are forced to stitch together numerous smaller, specialized datasets. This approach leads to inconsistencies in annotation schemas, formatting, and data quality. Furthermore, existing open datasets often suffer from contamination, where training data overlaps with evaluation benchmarks, artificially inflating performance metrics and hindering robust, data-centric research. The field has shifted from simple aggregation to the need for principled, scalable curation that addresses diversity, hygiene, and emerging tasks like agentic GUI interaction.

Methodology

The authors introduce FineVision, a meticulously curated corpus of 24 million samples derived from over 200 public sources. The curation process employs a semi-automated, human-in-the-loop pipeline designed to unify heterogeneous data into a standardized conversational schema.

1. Data Curation Pipeline

Source Aggregation: Data was collected from Hugging Face, institutional cloud storage, GitHub repositories, and direct web downloads.
Semi-Automated Conversion: A hybrid approach combines Large Language Model (LLM) agents (specifically Claude) for bulk ingestion and strategy design with human reviewers. The pipeline breaks down conversion into annotation analysis, strategy design, script implementation, and quality verification.
Human-in-the-Loop Oversight: Reviewers audit mappings, spot-check outputs for faithful annotation consumption, and validate formatting and diversity. For agentic/GUI data, reviewers validate unified action schemas and inspect trajectories for executable fidelity.
Unified Schema: All datasets are converted into a standardized chat format (sample = {images, texts, source, metadata}). Non-conversational data (e.g., classification) is transformed into natural QA pairs. Single-turn QA datasets are often grouped into multi-turn conversations to create richer training signals.

2. Task-Specific Strategies

The pipeline applies six core conversion strategies to preserve semantic richness:

Visual QA: Grouping questions per image into multi-turn conversations with varied phrasings.
Captioning & Description: Wrapping ground-truth captions with randomized instructional prompts.
Grounding & Spatial Relations: Converting spatial annotations into yes/no questions with explanatory answers; coordinates are normalized to metadata.
Document Understanding: Processing multi-page documents as image lists with enriched answers (derivation steps, facts).
OCR & Transcription: Generating exact transcription turns alongside optional understanding turns.
Classification & Detection: Converting labels to decision questions with explanatory answers.
GUI/Agentic Unification: A specific pipeline normalizes heterogeneous function signatures and action taxonomies from GUI datasets into a unified, typed action space with resolution-agnostic coordinates.

3. Data Hygiene and Decontamination

Cleaning: Automated validation removes corrupted images, undecodable files, and malformed text. Text is normalized to UTF-8, control characters are stripped, and turns are capped at 8192 tokens.
De-duplication: Self-Supervised Copy Detection (SSCD) embeddings and cosine similarity are used to identify and merge near-duplicate images within and across sources.
Test-Set Decontamination: The pipeline identifies training images similar to those in 66 public VLM benchmarks. A conservative threshold ( $\tau = 0.95$ ) is applied to flag potential contamination, though the dataset is released in its original form with contamination statistics provided.

Key Contributions

FineVision Corpus: The release of the largest open resource of its kind, containing 17 million images, 89 million turns, and 9.5 billion answer tokens across 185 subsets.
Curation Workflow: A reproducible, semi-automated, human-in-the-loop pipeline that unifies 200+ sources while maintaining task structure and conversational diversity.
Unified Action Space: A novel contribution for agentic tasks, providing a standardized schema for GUI interactions across desktop, mobile, and browser environments.
Rigorous Hygiene: Comprehensive de-duplication and decontamination protocols using SSCD embeddings, with public release of the tools and precomputed embeddings.
Quality Analysis: A detailed characterization of the dataset along four axes (Formatting, Relevance, Visual Dependency, Image-Question Correspondence) using LLM-as-a-judge scoring.

Experimental Results

The authors trained a 460M-parameter SmolVLM on FineVision and compared it against models trained on The Cauldron, LLaVA-OneVision, and Cambrian-7M.

Performance Gains: Models trained on FineVision achieved state-of-the-art results among open-data VLMs, showing significant relative improvements:
- +27.7% over The Cauldron
- +11.4% over Cambrian-7M
- +38.4% over LLaVA-OneVision
- (Average improvement of 10.8 percentage points over baselines across 11 benchmarks).
Training Dynamics: While initial performance lagged slightly due to the inclusion of novel tasks, FineVision-trained models surpassed all baselines after approximately one epoch, demonstrating superior generalization.
Robustness to Contamination: When retrained on decontaminated versions of the datasets, baseline models suffered performance drops of 2.7–3.7 percentage points, whereas the FineVision model dropped by only 1.6 points, confirming that its gains are not driven by data leakage.
GUI Capabilities: On challenging GUI benchmarks (ScreenSpot-V2 and ScreenSpot-Pro), the FineVision-trained 0.5B model achieved performance comparable to an architecturally equivalent 2B model, outperforming base small models that failed to solve tasks without fine-tuning.
Diversity Metrics: FineVision exhibits high conceptual breadth (Effective Rank) and, crucially, a superior Participation Ratio compared to other large datasets, indicating a more uniform distribution of visual concepts.

Significance and Claims

The paper claims that data hygiene and thoughtful integration are as critical as scale. FineVision demonstrates that a principled approach to curation—combining massive scale with rigorous cleaning, de-duplication, and human oversight—can close the performance gap between open-source and proprietary VLMs.

The authors emphasize that the dataset is not merely a larger aggregation but a "foundation" engineered for emerging tasks, particularly in agentic/GUI settings via a unified action space. By releasing the corpus, conversion recipes, and de-duplication tools, the work aims to democratize access to high-quality training data and catalyze the next wave of innovation in open VLM development. The authors modestly acknowledge limitations, such as residual overlaps and the lack of integrated community benchmarks for GUI control, positioning FineVision as a starting point for future extensions into video, multilingual coverage, and longer-context reasoning.