🤖 machine learning

CRIT: Graph-Based Automatic Data Synthesis to Enhance Cross-Modal Multi-Hop Reasoning

The paper introduces CRIT, a graph-based automatically synthesized dataset and benchmark designed to address the lack of cross-modal multi-hop reasoning in current Vision-Language Models, demonstrating that training on this data significantly improves model performance on complex reasoning tasks and reduces hallucinations.

Original authors: Junyoung Sung, Seungwoo Lyu, Minjun Kim, Sumin An, Arsha Nagrani, Paul Hongsuck Seo

Published 2026-04-03

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Junyoung Sung, Seungwoo Lyu, Minjun Kim, Sumin An, Arsha Nagrani, Paul Hongsuck Seo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a mystery, but the clues are scattered across different rooms. Some clues are written on sticky notes (text), while others are hidden inside photographs (images). To solve the mystery, you can't just read the notes or just look at the photos; you have to walk back and forth, connecting a name on a note to a face in a photo, then linking that face to a date on another note, and finally finding the location in a third photo.

This is the kind of thinking humans do all the time. But for Artificial Intelligence (AI), this is incredibly hard.

Here is a simple explanation of the paper "CRIT: Graph-Based Automatic Data Synthesis to Enhance Cross-Modal Multi-Hop Reasoning" using everyday analogies.

1. The Problem: The AI is "Lazy" and Gets Lost

Right now, most AI models (like the ones that chat with you or describe pictures) are trained on data where the answer is usually right there in front of them.

The Old Way: You show an AI a picture of a cat and ask, "What is this?" The AI looks at the picture and says, "Cat." Easy.
The Real World: You show an AI a recipe (text) and a series of photos of a cooking process. You ask, "Which tool in Image 3 is the one mentioned in Paragraph 2 as being 'rusty'?"

Current AIs struggle here. They often:

Hallucinate: They make up facts because they are guessing.
Get Lazy: They try to answer using only the text or only the image, ignoring the other half of the puzzle.
Lose the Thread: They can't connect the dots across multiple steps (multi-hop).

The authors call this a lack of "Cross-Modal Multi-Hop Reasoning." In plain English: The AI can't walk the path from Text $\to$ Image $\to$ Text $\to$ Image to find the answer.

2. The Solution: Building a "Train Track" System (CRIT)

To fix this, the researchers built a new dataset called CRIT. But they didn't just hire humans to write thousands of questions (that would take forever and cost a fortune). Instead, they built a robotic factory to make the questions.

They used a clever trick called a "Graph."

The Analogy: The Train Station Map
Imagine a train station map.

Stations (Nodes): These are the things in the world (e.g., "a laptop," "a student named Elara," "the color silver").
Tracks (Edges): These are the connections between them (e.g., "Elara studies digital autonomy," "The laptop symbolizes digital autonomy").

In the old way, AI training data was like a pile of loose train cars with no tracks. The AI didn't know how to connect them.

In CRIT, the researchers built a digital map (a graph) first.

Step 1: They take real images and text.
Step 2: They turn them into a structured map. They make sure there are "tracks" connecting a word in the text to an object in the image.
Step 3: They use this map to automatically generate questions. Because they built the map first, they know exactly which tracks the AI must travel to answer the question.

Why is this special?
If you ask a normal AI to "make up a hard question," it usually makes up a question that is actually easy (because the AI is biased). But because CRIT builds the question from the map, it guarantees the question requires the AI to jump from text to image to text. It forces the AI to do the hard work.

3. The Factory in Action

The paper describes a three-step assembly line:

The Blueprint: They take images (like photos of a desk or a video frame) and turn them into a graph of objects.
The Storyteller: They use an AI to write a story (text) that connects these objects. For example, "The laptop in Image 1 is a symbol of freedom."
The Quiz Master: They use the map to ask a question that only makes sense if you combine the story and the photo.
- Question: "What color is the object in the image that symbolizes the concept studied by the student mentioned in the text?"
- The Path: You must read the text to find the student $\to$ find the concept $\to$ find the symbol in the text $\to$ find that symbol in the image $\to$ read the color.

4. The Results: Training the AI to Think

The researchers took this new dataset (CRIT) and used it to "train" (teach) several AI models.

Before Training: The models were like tourists with a map but no compass. They got lost easily, guessed wrong, or ignored half the clues.
After Training: The models became like detectives. They learned to:
- Look at the text and the image together.
- Follow the chain of clues (multi-hop).
- Stop making things up.

The paper shows that models trained on CRIT got much better at solving these complex puzzles. They even got better at other standard tests, proving that learning to "connect the dots" helps them in general, not just on this specific game.

5. Why This Matters

Think of the current AI as a student who is great at memorizing flashcards but terrible at solving a real-world mystery.

Old Data: "Here is a picture of a dog. What is it?" (Memorization).
CRIT Data: "Here is a diary entry about a dog named 'Buddy' who lost his collar. Here are three photos of a park. Which photo shows the collar Buddy lost?" (Reasoning).

By forcing AI to practice this "mystery solving" using their graph-based factory, the researchers are teaching AI to be more grounded, less likely to lie (hallucinate), and better at understanding the complex, messy world where text and images are mixed together.

In a nutshell: They built a machine that creates "connect-the-dots" puzzles for AI, forcing the AI to learn how to link words to pictures across multiple steps, making it a smarter, more reliable thinker.

1. Problem Statement

Real-world reasoning often requires cross-modal multi-hop reasoning, where a model must synthesize information from interleaved text and visual sources (images, videos, charts) to reach a conclusion. For example, following a DIY tutorial requires connecting textual instructions to specific visual steps.

However, current Vision-Language Models (VLMs) and benchmarks suffer from two critical limitations:

Benchmark Deficiencies: Existing multimodal benchmarks (e.g., MMQA, SPIQA) often rely on single images or static sets where answers can be inferred from a single modality alone. They fail to enforce true cross-modal grounding, where text and vision are complementary and both are strictly necessary.
Data Scarcity: Training data rarely contains high-quality, interleaved image-text content that enforces multi-hop reasoning. Consequently, VLMs frequently hallucinate or produce reasoning traces that are not grounded in visual evidence. Manually collecting such data is prohibitively expensive, and using VLMs to generate data automatically often leads to cyclical biases and the same grounding failures the researchers aim to solve.

2. Methodology: The CRIT Framework

The authors propose CRIT (Cross-modal multi-hop Reasoning over interleaved Image-Text), a new dataset and benchmark generated via a novel graph-based automatic data synthesis pipeline. This pipeline avoids direct VLM generation of questions to prevent hallucination and bias.

A. Core Pipeline Architecture

The generation process consists of three stages:

Multimodal Content Graph Construction:
- The system starts with source data (images with scene graphs, video captions, or scientific papers).
- It constructs a directed graph $G=(V, E)$ where nodes represent entities (visual objects or textual concepts) and edges represent relationships.
- Augmentation: An LLM is used to generate new textual entities and relations connected to visual nodes. These act as "bridges," linking entities across different images through text, ensuring the graph contains complex, multi-hop paths.
Textual Context Generation:
- Subgraphs are extracted to generate complementary textual descriptions (e.g., stories, scripts, or documentary styles).
- The text describes the augmented nodes and their relations but deliberately excludes visual attributes that must be inferred from the images, forcing the model to look at the visual input.
Question-Answer (QA) Pair Generation:
- The system samples a subgraph representing a chain of facts ( $1 \le h \le 5$ hops).
- Constraints: The terminal node of the reasoning chain must be a visual attribute or entity. The question is generated such that intermediate entities are not explicitly mentioned, requiring the model to perform multi-hop inference to connect the text to the image.
- CoT Generation: A Chain-of-Thought (CoT) trace is generated alongside the QA pair, explicitly mapping the reasoning steps to the graph structure.

B. Quality Control & Filtering

Filtering: Generated QA pairs undergo a staged filtering process using multiple LLMs. Questions are discarded if they can be answered using only text or only images (single-modality solvable) or if they explicitly mention intermediate reasoning steps.
Human Verification: The test set is manually verified by STEM-trained annotators to ensure the reasoning is valid, unambiguous, and truly cross-modal.

C. Domain Extensions

The pipeline is extended to handle:

Videos: Using dense video captions to construct temporal scene graphs.
Scientific Papers: Converting text-rich documents (figures, tables, paragraphs) into unified graphs, removing sentences that duplicate visual information to ensure the text complements rather than replaces the visual input.

3. Key Contributions

CRIT Dataset: A large-scale dataset (269k training QA pairs, 1.6k test QA pairs) covering three domains: Natural Images, Video Frames, and Scientific Papers. It is the first to systematically enforce interleaved, cross-modal multi-hop reasoning.
Graph-Based Synthesis Pipeline: A scalable framework that uses graphs to guarantee structural complexity and multi-hop dependencies, avoiding the pitfalls of direct VLM question generation.
Comprehensive Benchmarking: An evaluation of state-of-the-art (SOTA) models (including GPT-4o, Gemini, Qwen, and LLaVA) revealing significant performance gaps in cross-modal reasoning.
Transferability Evidence: Demonstrating that training on CRIT significantly improves performance not only on CRIT but also on other cross-modal benchmarks (SPIQA, VEGA, FCMR) and general multimodal tasks.

4. Experimental Results

Baseline Performance: Even proprietary models like GPT-4o and Gemini 2.0 Flash perform poorly on CRIT (Exact Match scores around 30-35% for natural images, dropping significantly for scientific papers). This highlights the difficulty of the task.
Fine-Tuning Gains: Models fine-tuned on CRIT (e.g., Qwen2.5-VL-7B-CRIT and Idefics2-8B-CRIT) show massive improvements:
- CRIT Benchmark: Qwen2.5-VL-7B-CRIT achieved 58.6% EM on Natural Images (vs. 28.3% baseline), surpassing proprietary models.
- Cross-Domain Transfer: Fine-tuning on CRIT improved performance on SPIQA (METEOR score doubled from 3.60 to 10.53) and VEGA, proving the learned reasoning capabilities generalize.
- General Benchmarks: Performance on standard benchmarks (MME, SeedBench, ChartQA) was preserved or improved, indicating no catastrophic forgetting.
Error Analysis: The primary failure mode for current models is Evidence Localization Error (55%), where models fail to identify the correct image or text segment. Visual perception errors (32%) are the second most common, while answer generation errors are rare (5%), suggesting models can reason well once the correct evidence is found.

5. Significance and Impact

Bridging the Gap: CRIT addresses the critical gap between current VLM capabilities and the complex, multi-hop reasoning required in real-world scenarios (e.g., medical diagnosis from reports and scans, legal analysis from contracts and evidence photos).
Scalable Data Synthesis: The paper proves that graph-based automatic synthesis is a viable, scalable alternative to manual data collection for complex reasoning tasks, provided strict structural constraints are applied.
Future Direction: The results suggest that to achieve robust multimodal intelligence, models must be trained on data that strictly enforces the interdependence of modalities, rather than just interleaving them loosely. CRIT provides the foundation and benchmark for this next generation of reasoning models.