Document-as-Image Representations Fall Short for Scientific Retrieval

1. Problem Statement

Scientific document retrieval requires locating evidence distributed across diverse modalities: text, equations, tables, and figures. While recent trends in document embedding favor document-as-image representations (processing rendered pages as pixels via Vision-Language Models), the authors argue this paradigm is suboptimal for text-rich scientific documents.

Key issues identified with the image-based approach include:

• Loss of Structure: Converting structured LaTeX/XML sources into images obscures explicit boundaries between content types (prose, equations, figures) and removes machine-readable cross-references.
• Inefficiency: Models must encode all content as pixels, which is computationally expensive and less effective for dense text.
• Benchmark Bias: Existing benchmarks (e.g., ArXivQA, ViDoRe) often treat documents as images or rely on OCR, implicitly favoring visual representations and failing to account for the structured nature of scientific papers.

The core research question is: Which document representations (text-only, image-only, or multimodal) are most appropriate for scientific retrieval, and how do they scale with document length and complexity?

2. Methodology

A. The ArXivDoc Benchmark

To address the lack of a controlled evaluation framework, the authors introduce ArXivDoc, a new benchmark constructed from raw LaTeX sources rather than rendered PDFs.

• Scale: 8,210 scientific papers (144,653 pages).
• Queries: 547 manually verified, context-independent queries grounded in specific evidence types:
  • 229 Text-based
  • 218 Table-based
  • 100 Figure-based
• Construction Pipeline: Queries are generated using LLMs (gpt-5.2) and undergo a rigorous filtering process (decontextualization, difficulty filtering via BM25, and human verification) to ensure they are natural, unambiguous, and require retrieval from a large corpus rather than simple keyword matching.

B. Document Representations

The study systematically compares five representation strategies derived from the same LaTeX source:

1. Text-Only: Raw LaTeX text (normalized to remove non-semantic markup) and Text + VLM Captions (augmenting text with AI-generated figure/table descriptions).
2. Image-Only: Rendered figures extracted from LaTeX (ignoring text).
3. Document-as-Image: Full pages rendered as images.
4. Interleaved Text + Image: A sequence preserving the original document flow, where text chunks are interleaved with their corresponding figure images.

C. Models and Evaluation

• Models: The authors evaluate both Single-Vector models (e.g., Qwen3-Embedding, Qwen-VL-Embedding) and Late-Interaction/Multi-Vector models (e.g., ColQwen2).
• Metric: Normalized Discounted Cumulative Gain at rank 10 (nDCG@10).
• Analysis: The study includes scaling experiments (varying context length) and cross-benchmark validation on ViDoRe to test generalizability.

3. Key Contributions

1. ArXivDoc Benchmark: The first large-scale, open-domain scientific retrieval benchmark that provides access to raw LaTeX sources, enabling a direct comparison between text-based and image-based representations without OCR noise.
2. Systematic Representation Comparison: A comprehensive evaluation showing that document-as-image approaches are consistently suboptimal for scientific retrieval, even for figure-based queries.
3. Interleaved Modality Discovery: Demonstration that interleaved text+image representations outperform document-as-image approaches without requiring specialized training, simply by preserving the document's native structure.
4. Efficiency vs. Performance: Evidence that single-vector models outperform complex late-interaction models (like ColQwen) in this domain while requiring significantly smaller index sizes (up to 40x reduction).

4. Key Results

• Suboptimality of Document-as-Image: Across all query types (text, table, figure), treating documents as images yielded lower retrieval performance. Even for figure-based queries, the best-performing model was Text + VLM Captions, not the image-based model.
• Text Sufficiency for Figures: Text-only models performed surprisingly well on figure-based queries (within 0.02 nDCG of image models). Analysis revealed that scientific papers heavily describe figures in surrounding text and captions; thus, the retrieval signal is often textual, not visual.
• Scaling Behavior: As document length (context) increases, document-as-image representations degrade significantly faster than text-based representations. Text-based methods remain robust for long, dense scientific papers.
• Interleaved Superiority: Interleaved text+image inputs outperformed document-as-image inputs even when using models originally trained on page-level images. This suggests that maintaining the alignment between text and figures is more critical than the visual rendering itself.
• Index Efficiency: Single-vector models achieved better nDCG@10 scores than late-interaction models while using up to 40x less storage for the index.

5. Significance and Implications

• Paradigm Shift: The paper challenges the prevailing trend of converting all documents to images for retrieval. It argues that for structured, text-dense domains like science, preserving the underlying structural markup (LaTeX) is superior to pixel-based processing.
• Cost-Effectiveness: The findings suggest that high-performance scientific retrieval does not require expensive multi-vector architectures or massive visual indices. Simple, text-centric single-vector models augmented with VLM captions offer the best trade-off between performance and storage cost.
• Future Directions: The work highlights a gap between current embedding model training (often on rendered images) and the actual structure of scientific data. It proposes that training embedding models on interleaved text-image sequences derived from source code is a promising direction for future research.
• Benchmarking Standards: ArXivDoc sets a new standard for evaluating multimodal retrieval by ensuring ground truth is derived from source code, eliminating the variability and errors introduced by OCR and PDF rendering.

In conclusion, the authors demonstrate that for scientific retrieval, structure and text are king. Relying on rendered images obscures the very evidence (equations, structured tables, specific text spans) that researchers need to find, making document-as-image approaches a step backward for this specific domain.