ViDoRe V3: A Comprehensive Evaluation of Retrieval Augmented Generation in Complex Real-World Scenarios

1. Problem Statement

Retrieval-Augmented Generation (RAG) systems are increasingly deployed for knowledge-intensive tasks, yet existing benchmarks fail to capture the complexity of real-world industrial scenarios. Current limitations include:

• Over-reliance on Text: Most benchmarks focus on textual data, ignoring critical visual elements (tables, charts, infographics) where information is encoded.
• Single-Document Focus: Many evaluations assume single-page comprehension, neglecting the need to synthesize information across multi-page corpora.
• Isolated Evaluation: Retrieval and generation are often evaluated separately, failing to assess end-to-end pipeline performance.
• Lack of Grounding: Few benchmarks require models to pinpoint the exact source of information (visual grounding) to mitigate hallucinations.
• Synthetic Bias: Many datasets rely on synthetic queries that do not reflect authentic user search behaviors or open-ended reasoning needs.

2. Methodology

The authors introduce ViDoRe V3, a comprehensive multimodal RAG benchmark designed to evaluate retrieval, generation, and visual grounding simultaneously.

A. Dataset Construction

• Corpora: 10 diverse, industry-relevant document collections (Finance, Computer Science, Energy, Pharmaceuticals, HR, Industrial Maintenance, Telecom, Physics) comprising 26,000 pages.
• Languages: Documents are in English and French; queries are available in 6 languages (English, French, Spanish, German, Italian, Portuguese) to test cross-lingual retrieval.
• Scale: 3,099 human-verified queries paired with relevant pages, bounding boxes, and reference answers.
• Annotation Effort: 12,000 hours of human annotation by 76 domain experts.

B. Human-in-the-Loop Annotation Protocol

To ensure high quality and realism, the authors employed a rigorous three-stage process:

1. Query Generation:
   • Taxonomy: Queries are classified by Type (Open-ended, Extractive, Numerical, Multi-hop, Compare-contrast, Boolean, Enumerative) and Format (Question, Keyword, Instruction).
   • Sources: Queries were generated via three streams to ensure diversity:
     • Human (Image): Annotators view raw pages to write queries (minimizing extractive bias).
     • Human (Summary): Annotators view summaries to write queries.
     • Synthetic (Summary): LLMs generate queries from clustered summaries, filtered by an LLM-as-a-Judge.
2. Query-Page Linking:
   • A VLM (Qwen2.5-VL-32B) pre-filters candidate pages to reduce the search space.
   • Human annotators rate relevance on a 3-point scale (Not Relevant, Critically Relevant, Fully Relevant) and select the final relevant pages.
3. Answer & Grounding Annotation:
   • Annotators write answers and draw bounding boxes around the supporting evidence.
   • Modality Labeling: Each bounding box is tagged with a modality (Text, Table, Chart, Infographic, Image, Mixed).
   • Aggregation: A VLM aggregates multiple annotator answers into a single "Golden Answer," verified for factual consistency.

3. Key Contributions

1. Realistic Query Methodology: A novel protocol that restricts annotator access to full document content during query formulation to simulate authentic search behaviors and reduce bias toward simple extractive queries.
2. The ViDoRe V3 Benchmark: The first large-scale, multilingual, multimodal RAG benchmark featuring:
   • Fine-grained visual grounding (bounding boxes).
   • Cross-lingual retrieval challenges.
   • A mix of 7 query types and 3 formats.
   • Integration into the MTEB ecosystem with a commercially permissive license.
3. Comprehensive Evaluation Framework: A unified evaluation of retrieval accuracy, answer generation quality, and visual grounding fidelity across different pipeline configurations (Text-only, Visual-only, Hybrid).

4. Key Results & Insights

The authors evaluated state-of-the-art models across 8 public datasets (2 held out as private test sets).

A. Retrieval Performance

• Visual > Textual: Visual retrievers (processing page images) consistently outperform textual retrievers (processing Markdown text) for the same parameter count.
• Late Interaction: Late-interaction models (e.g., ColBERT variants) significantly outperform dense single-vector models.
• Reranking Impact:
  • Textual Rerankers: Adding a textual reranker (e.g., zerank-2) to a visual pipeline yields massive gains (+13.2 NDCG@10), making the text-enhanced pipeline the strongest overall.
  • Visual Rerankers: Current visual rerankers provide marginal improvements (+0.2) and sometimes degrade performance, indicating a need for better multilingual visual reranking.
• Complexity & Modality:
  • Performance drops as the number of relevant pages increases (multi-hop difficulty).
  • Queries involving Mixed modalities (text + image + table) are the hardest to retrieve.
  • Cross-lingual retrieval degrades performance by 2–3 points compared to monolingual settings.

B. Answer Generation

• Visual Context: Using image-based context improves generation accuracy on "hard" queries by ~2.5% compared to text-only context, confirming that visual elements are crucial for complex reasoning.
• Hybrid Approach: A hybrid pipeline (combining top-5 visual and top-5 textual results) achieves the best performance on hard queries (54.7% accuracy), outperforming pure text or pure visual baselines.
• Model Limits: Even with oracle (ground-truth) context, models struggle with "hard" queries (multi-step reasoning), showing a >10% performance gap compared to "easy" queries.

C. Visual Grounding

• Significant Gap: There is a massive performance gap between human and model grounding.
  • Human Inter-annotator Agreement (F1): 0.602
  • Best Model (Qwen3-VL): 0.089
  • Best Model (Gemini 3 Pro): 0.065
• Failure Modes: Models frequently miss relevant pages (low recall) or produce bounding boxes that are either too tight (missing context) or too loose.

5. Significance

• Benchmarking Standard: ViDoRe V3 sets a new standard for evaluating RAG systems in realistic, multimodal, and multilingual environments, moving beyond simple QA on single documents.
• Actionable Insights: The results provide clear guidance for practitioners:
  • Prioritize visual retrievers and textual rerankers.
  • Adopt hybrid pipelines for robustness.
  • Recognize that visual grounding remains a major unsolved challenge requiring significant research.
• Community Resource: By releasing the dataset under a commercially permissive license and integrating it with MTEB, the authors encourage broad adoption and rapid iteration in the field of multimodal RAG.

In conclusion, ViDoRe V3 demonstrates that while RAG pipelines are advancing, they still struggle with the synthesis of complex visual information, cross-lingual retrieval, and precise source grounding, highlighting critical directions for future research.