💻 computer science

Visual Compositional Tuning

The paper introduces COMPACT, a compositional data curation method that synthesizes complex training examples by combining atomic visual capabilities, achieving superior data efficiency and performance on multimodal benchmarks compared to existing reduction techniques while reducing the required training data by 90%.

Original authors: Xindi Wu, Hee Seung Hwang, Polina Kirichenko, Esin Tureci, Olga Russakovsky

Published 2026-05-18

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xindi Wu, Hee Seung Hwang, Polina Kirichenko, Esin Tureci, Olga Russakovsky

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to understand the world through its eyes. Currently, the standard way to do this is to show the robot millions of pictures and ask it simple questions like, "What color is the car?" or "Is there a dog?"

The authors of this paper argue that this approach is a bit like teaching a child to swim by only having them practice floating in a shallow pool. It's safe and easy, but it doesn't prepare them for the ocean.

Here is the paper's core idea, broken down into simple concepts:

1. The Problem: Too Much "Easy" Practice

The current datasets used to train these AI models are huge, but they are filled with "low-complexity" questions.

The Analogy: Imagine a gym where everyone only lifts 5-pound weights. Even if you lift them a million times, you won't get very strong.
The Reality: Most questions in these datasets ask the AI to do just one or two things at once (e.g., "What is the object?" and "What color is it?"). The AI gets good at answering simple questions but struggles when things get complicated, like figuring out, "What color is the object to the left of the car?" which requires recognizing the car, finding the left side, and identifying the color all at once.

2. The Solution: "Atomic" Building Blocks

The researchers decided to stop just throwing more pictures at the AI. Instead, they started building better questions by mixing and matching "atomic capabilities."

The Analogy: Think of these capabilities as Lego bricks.
- Brick A: Recognizing an object (a car).
- Brick B: Understanding space (left/right).
- Brick C: Identifying a color (red).
The Old Way: Build a tower with just one brick.
The New Way (COMPACT): Build a complex castle by snapping three or four bricks together in a single instruction.

They created a recipe called COMPACT (COMPositional Atomic-to-complex Visual Compositional Tuning). This recipe takes an image and forces the AI to answer questions that require combining multiple "Lego bricks" (capabilities) simultaneously.

3. The Experiment: Quality Over Quantity

The team took a small slice of the massive standard dataset (only 5% of the original images) and used their new recipe to generate complex questions for them.

The Analogy: Instead of feeding the student 1,000 pages of easy reading comprehension, they gave them 100 pages of challenging puzzles that required deep thinking.
The Result: The AI trained on this small, "high-complexity" dataset performed better than the AI trained on the full, massive dataset of simple questions.
- On standard tests, the small, smart dataset achieved 100.2% of the performance of the huge dataset.
- On very hard tests (like understanding complex charts or reasoning about spatial relationships), the small dataset actually beat the huge one by a significant margin.

4. Why It Works

The paper suggests that by forcing the AI to juggle multiple concepts at once (like color + location + object), the model learns to pay closer attention to the details in the image. It stops guessing and starts actually "seeing" the relationships between things.

5. The Catch (Limitations)

The authors are honest about the limits of their method:

It's not magic for everything: The method is great for visual reasoning (looking at a picture and figuring things out). It doesn't help much with questions that require deep world knowledge (like "What is the history of the Roman Empire?") because those questions don't rely on the picture itself.
It's expensive to make: They used a very powerful, closed-source AI (Gemini) to generate these complex questions. This costs money and time, which might make it hard for others to copy exactly.

Summary

The paper claims that we don't need more data; we need smarter data. By mixing simple visual skills together to create complex, multi-step questions, we can train powerful AI models using a fraction of the data currently required, making them smarter and more efficient at understanding the visual world.

Technical Summary: Visual Compositional Tuning (COMPACT)

Problem Statement

Visual Instruction Tuning (VIT) for Multimodal Large Language Models (MLLMs) has traditionally relied on scaling dataset size to improve performance, often overlooking the informativeness of individual training samples. While recent works have demonstrated that smaller, curated subsets can achieve near-full performance, existing data selection methods largely treat complexity as a byproduct of scale rather than a controllable metric.

Current VIT datasets, such as LLaVA-665K, predominantly contain questions requiring only one or two fundamental visual capabilities (e.g., "What color is the car?"). These samples often under-utilize the rich visual information available in images by failing to combine multiple capabilities (e.g., object recognition, spatial relationships, and color attribution) into single, complex reasoning tasks. Consequently, models may struggle with complex visual reasoning tasks that require the integration of these atomic skills.

Methodology: COMPACT

The authors propose COMPACT (COMPositional Atomic-to-complex Visual Compositional Tuning), a data recipe designed to scale the complexity of training samples by combining multiple atomic visual capabilities within a single example. The methodology involves a four-step pipeline:

Atomic Capability Definition: The authors define a taxonomy of 10 "atomic visual capabilities" essential for visual reasoning, categorized into:
- Attribution: Color, Shape.
- Recognition: Object Recognition, Action Recognition, Text Recognition, Counting, Spatial Recognition.
- Relation: Spatial Relationship, Object Interaction, Scene Understanding.
  They introduce a $k$ -value metric, defined as the number of atomic capabilities required to answer a specific question.
Capability Sampling: Starting from a random sample of images (e.g., from LLaVA-665K), the system uniformly samples $k_{gen} \in \{1, 2, 3\}$ capabilities from the predefined pool. The sampling prioritizes capabilities not yet selected for a specific image to ensure diversity.
Conversation Generation: Using Gemini-2.0-Flash, the system generates conversational question-answer pairs that naturally integrate all sampled capabilities. The generation is constrained to ensure:
- Questions require visual information from the image (cannot be answered by text alone).
- Answers are concise.
- Capabilities are integrated naturally (not merely conjoined with "and").
- Questions reference objects actually present in the image.
Quality Verification & Dataset Assembly: Generated questions undergo verification to filter out uninformative answers, low-confidence outputs, near-duplicates, and cases where the capabilities were not successfully integrated. The final dataset is assembled by mixing the synthetically generated compositional tuning data with a small subset (5%) of the original LLaVA-665K data. This mixture ensures the model retains instruction-following capabilities while focusing the compositional data on visual reasoning.

Key Contributions

Complexity-Aware Data Curation: The paper demonstrates that increasing the complexity ( $k$ -value) of training samples allows for more effective use of information content within a dataset, addressing the limitations of conventional complexity-agnostic scaling.
Quantifiable Complexity Metric: The authors define the $k$ -value to quantify the complexity of vision-language tasks and analyze the optimal distribution of complexity for maximum efficiency.
Efficient Data Recipe: COMPACT provides a scalable synthetic data generation recipe that significantly reduces the data budget required for effective VIT.

Experimental Results

The authors evaluated COMPACT on the LLaVA-v1.5-7B model using eight multimodal benchmarks (including MM-Vet, MMStar, InfoVQA, and SeedBench2Plus).

Data Efficiency: Training on a dataset comprising only 10% of the volume of LLaVA-665K (specifically 32K compositional samples + 5% of LLaVA-665K for formatting) achieved 100.2% of the full LLaVA-665K performance across benchmarks.
Superiority on Complex Tasks: COMPACT outperformed the full-scale LLaVA-665K training on particularly complex benchmarks:
- MM-Vet: +8.6% improvement (31.7 vs. 29.2).
- MMStar: +2.9% improvement (36.1 vs. 35.1).
- InfoVQA: +13.8% improvement.
Comparison to Baselines: COMPACT significantly outperformed random subsets and other data reduction methods (e.g., ICONS, EL2N, Perplexity), which achieved only ~95-97% relative performance with similar data volumes.
Complexity Distribution Analysis: The LLaVA-665K dataset has a mean $k$ -value of ~1.95, with 77% of questions requiring $k \le 2$ . In contrast, the COMPACT dataset has a mean $k$ -value of ~2.89. Ablation studies confirmed that at least half of the performance gain is attributable to this higher complexity distribution.

Significance and Claims

The paper claims that visual compositional tuning offers a scalable and efficient pathway for improving MLLMs on vision-language tasks. By explicitly targeting higher $k$ -values during sample generation, COMPACT enables models to learn from more visual features in each image, leading to superior performance on complex reasoning tasks with a fraction of the data budget.

The authors acknowledge limitations, noting a reliance on closed-source models (Gemini) for data generation, which introduces potential biases and cost barriers for reproducibility. Furthermore, the approach focuses on vision-centric capabilities and may not be optimal for knowledge-intensive tasks requiring external world knowledge. Future work is suggested to explore the intersection of image and text complexity and the application of these principles to different learning algorithms.