IRPAPERS: A Visual Document Benchmark for Scientific Retrieval and Question Answering

1. Problem Statement

While AI systems have achieved significant success in processing text and relational data, visual document processing remains underexplored. Traditional systems rely on Optical Character Recognition (OCR) to transcribe document images into text and metadata before retrieval. However, recent advances in multimodal foundation models suggest a paradigm shift: retrieving and generating answers directly from document images without intermediate transcription.

The core research question is: How do image-based retrieval systems compare to established text-based methods for scientific literature? Scientific papers are inherently visual, containing figures, equations, and layouts that may be lost or distorted during OCR transcription. The authors aim to determine if image-based representations preserve signals lost in text transcription and whether multimodal approaches can outperform unimodal baselines.

2. Methodology

2.1 The IRPAPERS Benchmark

The authors introduce IRPAPERS, a new benchmark designed to evaluate visual document retrieval and Question Answering (QA).

• Corpus: 3,230 pages sourced from 166 scientific papers focused on Information Retrieval (IR). The papers are drawn from citations of a major survey on LLMs for IR, ensuring a semantically dense corpus where documents share vocabulary and techniques, making fine-grained discrimination difficult.
• Queries: 180 curated "needle-in-the-haystack" questions targeting precise methodological details (e.g., specific model architectures or hyperparameters).
• Representations:
  • Image: PDF pages converted to base64-encoded images (approx. 1.3 MB/page).
  • Text: Pages transcribed using GPT-4.1 multimodal API (approx. 4.5 KB/page), serving as the high-quality OCR baseline.

2.2 Retrieval Systems Evaluated

The study compares several retrieval strategies:

• Text-Based:
  • BM25: Sparse lexical retrieval.
  • Arctic 2.0: Dense single-vector text embeddings.
  • Hybrid Text Search: Fusion of BM25 and Arctic 2.0 using Relative Score Fusion (RSF).
• Image-Based (Late Interaction):
  • ColModernVBERT: A lightweight (250M params) multi-vector model combining a text encoder and vision encoder.
  • ColPali & ColQwen2: Larger multi-vector models (2.9B and 2.2B params respectively).
  • MUVERA: An efficiency optimization technique using Fixed Dimensional Encoding (FDE) to compress multi-vector representations into single vectors for approximate nearest neighbor search, followed by exact rescoring.
• Multimodal Hybrid Search: Fusing scores from text (Hybrid) and image (ColModernVBERT) retrievers.
• Closed-Source Baselines: Cohere Embed v4 (Image) and Voyage 3 Large (Text).

2.3 Question Answering (RAG)

The authors evaluate Retrieval-Augmented Generation (RAG) using an LLM-as-Judge protocol to assess semantic equivalence between generated answers and ground truth.

• ImageRAG: Retrieves page images (via ColModernVBERT) and passes them to a multimodal reader (GPT-4.1).
• TextRAG: Retrieves OCR text (via Hybrid Search) and passes it to the same reader.
• Baselines: No Retrieval, Hard Negative (top non-gold document), and Oracle (gold document provided).

3. Key Contributions

1. IRPAPERS Dataset: A specialized benchmark of 166 IR papers and 180 questions, designed to test fine-grained discrimination in a semantically dense domain.
2. Systematic Comparison: A rigorous evaluation of open-source multi-vector image models against hybrid text search, revealing complementary failure modes.
3. MUVERA Efficiency Analysis: Quantification of the performance-efficiency tradeoff in multi-vector image retrieval using MUVERA encoding with varying ef (candidate set size) parameters.
4. Closed-Source Benchmarking: Establishment of performance ceilings using Cohere Embed v4 and Voyage 3 Large, highlighting the gap between open and closed models.
5. Modality Limitations Analysis: Identification of specific scenarios where visual grounding is strictly necessary (e.g., t-SNE plots) versus where text is superior (exact lexical matching).

4. Key Results

4.1 Retrieval Performance (Recall@K)

• Open-Source Unimodal:
  • Text (Hybrid): 46% Recall@1, 78% Recall@5, 91% Recall@20.
  • Image (ColModernVBERT): 43% Recall@1, 78% Recall@5, 93% Recall@20.
  • Observation: Text excels at top-rank precision, while image retrieval recovers more relevant documents at deeper ranks.
• Multimodal Hybrid Search:
  • Achieved the highest performance: 49% Recall@1, 81% Recall@5, and 95% Recall@20.
  • The fusion exploits complementary failures: 22 queries succeeded only with text, while 18 succeeded only with images.
• Model Scaling: Larger models (ColQwen2) improved Recall@1 to 49% but showed diminishing returns at Recall@20 (94%), suggesting top-rank discrimination is the primary benefit of scale.
• MUVERA Efficiency:
  • With ef=1024, MUVERA achieved 41% Recall@1 (a 2-point drop from exact retrieval) with a 50x reduction in storage (33 MB vs 1.65 GB).
  • Lower ef values further reduced performance, demonstrating a tunable tradeoff.
• Closed-Source Performance:
  • Cohere Embed v4 (Image): 58% Recall@1.
  • Voyage 3 Large (Text): 52% Recall@1.
  • Closed-source image models significantly outperformed open-source baselines.

4.2 Question Answering (RAG)

• Baseline: Without retrieval, alignment scores were near zero (0.16), confirming questions cannot be answered from parametric knowledge alone.
• Modality Comparison:
  • TextRAG consistently outperformed ImageRAG (0.82 vs. 0.71 alignment at k=5).
  • TextRAG at k=1 (0.62) outperformed ImageRAG at k=1 (0.40).
• Depth Impact: Increasing retrieval depth to k=5 significantly boosted performance for both modalities, outperforming Oracle single-document retrieval. This suggests that synthesizing evidence across multiple related pages is crucial for scientific QA.
• Token Cost: Text inputs required ~11% more tokens than images but yielded higher accuracy.

4.3 Limitations of Unimodal Representations

• Visual Necessity: In a targeted adversarial study involving a t-SNE plot, image-based QA achieved 70% accuracy vs. 30% for text-based QA. Text transcriptions failed to capture spatial relationships and cluster geometries.
• Text Necessity: Image embeddings lack mechanisms for exact lexical matching (e.g., finding a specific acronym like "HyDE"). Text-based systems (specifically BM25) are superior for symbolic constraints.

5. Significance and Future Directions

• Paradigm Shift: The paper validates that direct image retrieval is a viable and often superior alternative to OCR-dependent pipelines, particularly for preserving layout and visual structure.
• Hybrid is King: No single modality is sufficient. Multimodal hybrid search is the optimal strategy for scientific retrieval, leveraging the precision of text for symbols and the structural fidelity of images for diagrams.
• Agentic Search: The authors propose future agentic systems that dynamically adjust retrieval weights based on query type (e.g., prioritizing images for visual questions and text for exact string matches) and only pass images to the LLM when visual grounding is strictly required to save compute costs.
• Benchmarking Standard: IRPAPERS fills a gap in visual document benchmarks by focusing on the "scientific" domain, where semantic density and fine-grained methodological details are critical, distinguishing it from broader enterprise benchmarks like ViDoRe.

In conclusion, IRPAPERS demonstrates that while text-based methods currently lead in top-rank precision and QA accuracy, image-based methods provide essential complementary signals. The future of scientific document processing lies in multimodal fusion and agentic architectures that intelligently combine these modalities.