Retrieval or Representation? Reassessing Benchmark Gaps in Multilingual and Visually Rich RAG

This paper demonstrates that improvements in multilingual and visually rich retrieval-augmented generation benchmarks are primarily driven by better document representation and preprocessing rather than advanced retrieval mechanisms, urging the field to adopt decomposed evaluation metrics to accurately attribute progress.

The Gist

Here is an explanation of the paper "Retrieval or Representation? Reassessing Benchmark Gaps in Multilingual and Visually Rich RAG," translated into simple, everyday language with some creative analogies.

The Big Idea: It's Not the Detective, It's the Evidence

Imagine you are a detective (the AI Retrieval System) trying to solve a mystery. You have a massive library of case files (the Documents), and you need to find the specific page that holds the answer to a question.

For a long time, the tech world believed that "old-school" detectives (using simple keyword matching like BM25) were terrible at solving complex cases involving foreign languages or documents full of charts and graphs. They thought these old detectives needed to be replaced by super-intelligent, high-tech detectives (using Multimodal AI) that could "see" the pictures and understand the context.

This paper says: "Wait a minute. We might be blaming the wrong thing."

The authors argue that the old detectives weren't actually bad at finding clues; they were just being handed bad evidence. The problem wasn't the detective's skill; it was that the evidence they were given was messy, incomplete, or written in a language the detective couldn't read.

---

The Two Main Problems

The paper looks at two specific types of "messy evidence":

1. The "Foreign Language" Problem (Multilingual)

Imagine you have a file written in Japanese or Arabic.

• The Old Way: The AI tried to read the file, but its "scanner" (OCR) was bad at recognizing the characters. It turned a clear sentence into gibberish like #@$%&. Then, the old detective tried to find keywords in that gibberish and failed.
• The New Finding: When the authors gave the old detective a better scanner that could perfectly read the foreign characters, the old detective suddenly became amazing. It didn't need a super-intelligent brain; it just needed to be able to read the text clearly.

2. The "Chart and Graph" Problem (Visually Rich)

Imagine a document with a pie chart showing "75% of sales come from Product A."

• The Old Way: The scanner looked at the chart and just saw a circle. It didn't know what the numbers meant. It told the detective, "There is a circle here." The detective looked for the word "Product A" in the text, couldn't find it (because it was inside the circle), and gave up.
• The New Finding: When the authors added a step where a smart assistant described the chart in plain English (e.g., "A pie chart showing 75% is Product A"), the old detective could suddenly find the answer. The detective didn't need to learn how to "see" the chart; it just needed someone to tell it what the chart said.

---

The Experiment: The "Fixed Detective" Test

To prove their point, the researchers ran a controlled experiment. They kept the "detective" (the retrieval algorithm) exactly the same and only changed the "evidence preparation" (the scanning and describing).

The Results were shocking:

• By simply using better scanners and adding descriptions for charts, the old, simple method (BM25) jumped from being a terrible detective to being almost as good as the fancy, expensive AI models.
• In some cases, the simple method caught up to the "super-detectives" by a huge margin just by cleaning up the data.

The Analogy: The Librarian vs. The Translator

Think of the Retrieval System as a Librarian who knows exactly where every book is on the shelf.

• The Problem: The books are written in a language the Librarian doesn't speak, and some pages have pictures instead of words.
• The Misconception: People thought the Librarian was stupid and needed to be replaced by a Polyglot Artist (the Multimodal AI) who could speak every language and interpret art.
• The Reality: The Librarian was actually very smart! They just needed a Translator (better OCR) to read the foreign text and a Describer (image captioning) to explain the pictures. Once the Translator and Describer did their job, the original Librarian could find the right book instantly.

Why Does This Matter?

1. Stop Wasting Money: We don't necessarily need to build massive, expensive AI models for every task. Sometimes, we just need to fix the "plumbing" (the text extraction and cleaning).
2. Better Benchmarks: The paper argues that when we test AI systems, we shouldn't just say "Model A is better than Model B." We need to ask: "Did Model A just get better text to read, or is it actually smarter at finding things?"
3. Focus on the Basics: Before we build the next generation of "thinking" AI, we should make sure the basic text extraction is perfect. A brilliant brain is useless if it's fed garbage data.

The Bottom Line

The paper's title, "I Can't Believe It's Not Better," is a play on the famous "I Can't Believe It's Not Butter" margarine.

The authors are saying: "You think you need a fancy new Multimodal AI to handle complex documents? No! If you just fix the text extraction and add descriptions for images, your old, simple system will perform just as well."

The gap wasn't in the Retrieval (finding the needle); the gap was in the Representation (making sure the needle was visible in the haystack).

Technical Summary

Here is a detailed technical summary of the paper "Retrieval or Representation? Reassessing Benchmark Gaps in Multilingual and Visually Rich RAG" by Asenov et al., presented at the ICLR 2026 "I Can't Believe It's Not Better" Workshop.

1. Problem Statement

Retrieval-Augmented Generation (RAG) systems face significant performance gaps when handling multilingual and visually rich documents (e.g., PDFs with complex layouts, tables, and figures).

• The Prevailing Narrative: Recent benchmarks (e.g., VisR-Bench) suggest that classical lexical retrieval methods (like BM25) are fundamentally inferior to modern dense and multimodal retrievers (e.g., ColPali, GME, VLMs) in these settings. The assumption is that lexical methods fail to capture semantic nuances or visual context.
• The Hypothesis: The authors challenge this narrative, proposing that the observed performance gaps are not primarily due to the limitations of the retrieval algorithms themselves, but rather due to upstream representation bottlenecks. Specifically, poor Optical Character Recognition (OCR) quality and inadequate language-specific text preprocessing (normalization) cause information loss before the retrieval step even occurs.

2. Methodology

The authors conducted a series of controlled experiments to decouple retrieval capability from transcription/preprocessing quality.

• Benchmark: They utilized VisR-Bench, a dataset for retrieval-augmented question answering over long, visually rich documents covering 15 languages.
• Experimental Design:
  • Fixed Retrieval: They held the retrieval mechanism constant (using BM25 for lexical retrieval and various dense/multimodal models for comparison) while varying the input data.
  • Variable 1: OCR/Transcription Pipelines: They compared the dataset's default extraction against four distinct OCR/transcription sources:
    • Adobe Document Extract (Dataset default).
    • EasyOCR (Open-source, text-only).
    • Mistral OCR 3 (Modern, annotates figures/tables).
    • Ministral 3B (A small Vision-Language Model used to generate markdown descriptions of visual content).
  • Variable 2: Language-Specific Preprocessing: For multilingual BM25, they applied specific linguistic normalizations:
    • Stemming (for Romance/Germanic languages).
    • Lemmatization (for highly inflected languages like Czech, Croatian).
    • Morphological Analysis (for Arabic, decomposing prefixes/stems/suffixes).
    • Script-aware Segmentation (for Japanese and Vietnamese).
• Metrics: Evaluation focused on Top-5 Accuracy across 15 languages and specific subsets (Figure-heavy, Table-heavy, Text-only).

3. Key Contributions

1. Disentangling Representation from Retrieval: The paper demonstrates that the performance gap between classical and modern retrievers is largely an artifact of poor data representation. When BM25 is fed high-quality transcriptions and proper linguistic normalization, it recovers a massive portion of the gap previously attributed to retrieval limitations.
2. Quantifying the OCR Bottleneck: The study identifies that for figure-heavy pages, the lack of semantic description is a critical failure mode. Simply transcribing text is insufficient; adding coarse semantic descriptions of figures (via VLMs) yields massive gains.
3. Language-Specific Optimization: The authors show that a "one-size-fits-all" preprocessing pipeline is suboptimal. Different languages require different strategies (e.g., morphology for Arabic vs. lemmatization for Czech) to maximize lexical matching.
4. Call for Decomposed Benchmarks: The paper argues that current benchmarks conflate transcription quality with retrieval ability. They propose that future evaluations must separate these components to correctly attribute progress.

4. Key Results

• BM25 Recovery: By improving OCR and applying language-specific preprocessing, BM25 recovered up to +8.9 Top-5 accuracy points on average across multilingual datasets. In some cases, optimized BM25 outperformed unoptimized dense retrievers.
• The "Figure" Gap: On figure-heavy pages, standard OCR often misses critical data (e.g., axis labels, legends).
  • Using a VLM (Ministral 3B) to generate semantic descriptions of figures recovered up to +31.1 Top-5 points for BM25.
  • This suggests that for visual reasoning, the bottleneck is the description of the visual, not the retrieval model's ability to handle images.
• Dense Retrievers Also Benefit: The study found that dense text retrievers (like SBERT and BGE-M3) also suffer from poor transcription. When the same high-quality OCR/preprocessing was applied to them, their performance improved significantly, proving the bottleneck is the input representation, not the embedding model.
• Multilingual Leaderboard Shift: In the full leaderboard (Table 1), the relative performance of text-based methods fluctuates wildly based on the OCR/preprocessing configuration, whereas multimodal methods (which bypass OCR) remain invariant. This highlights OCR as a major confounder in cross-modal comparisons.

5. Significance and Implications

• Re-evaluating SOTA: The paper suggests that many "state-of-the-art" multimodal retrievers may not be outperforming lexical methods due to superior retrieval logic, but because they implicitly rely on better internal representations or are evaluated on datasets where the OCR baseline is weak.
• Cost-Effectiveness: It implies that for many applications, investing in better OCR pipelines and language-specific normalization is a more cost-effective and immediate way to improve RAG performance than training or fine-tuning massive multimodal retrieval models.
• Benchmark Integrity: The authors urge the community to treat OCR and preprocessing as first-class variables in benchmarking. Future benchmarks should report results with standardized, high-quality transcription to ensure fair comparisons between retrieval architectures.
• Failure Modes: The research highlights a specific failure mode: when visual content lacks textual anchors or semantic descriptions, retrieval degrades sharply regardless of the model used.

Conclusion

The paper concludes that "Better document representation is the primary driver of benchmark improvements." The apparent superiority of modern multimodal retrievers in visually rich and multilingual settings is often an illusion created by the superior transcription capabilities of the systems they are paired with, rather than the retrieval algorithms themselves. To advance the field, researchers must focus on decomposed evaluation that isolates transcription quality from retrieval capability.