Technical Summary: ConceptFormer

Problem Statement

Visual document retrieval (VDR) aims to identify query-relevant pages from large collections of visually rich documents containing text, layouts, charts, and structural elements. While recent approaches leverage Vision-Language Models (VLMs) to encode full-page images and optimize query-page alignment via contrastive learning, they face significant limitations:

1. Coarse-Grained Supervision: Standard page-level contrastive objectives provide relevance signals but fail to specify which localized visual evidence within a page supports the query.
2. Limitations of Existing Fine-Grained Approaches: Attempts to introduce finer-grained supervision often rely on either:
   • Textual Descriptions: Which may fail to capture complex visual structures (e.g., charts, spatial layouts).
   • Localized Visual Regions: Which often lack the global context and document-level semantics necessary for accurate relevance modeling.
3. Fixed-Granularity Alignment: Existing methods typically use fixed-length representations or static alignment strategies that do not adapt to the varying scale and complexity of evidence required for different queries.

Methodology

The paper proposes ConceptFormer, a framework that models query-relevant evidence as continuous, query-conditioned latent concepts. These concepts serve as an intermediate representation bridging localized visual grounding and semantic relevance without requiring raw visual annotations or textual intermediates during inference.

Core Components

1. Adaptive Concept Capacity Allocation (Training Time):

   • ConceptFormer employs a strong pre-trained Vision-Language Model (VLM) as an "Evidence Proposer" during training.
   • Given a query-document pair, the VLM identifies relevant visual regions and generates structured bounding box annotations.
   • These regions are projected onto the retriever's visual token grid to calculate the number of overlapping patches.
   • The total number of overlapping patches determines the adaptive length ($T_i$) of the latent concept sequence for that specific pair. This ensures that complex queries requiring diverse evidence (e.g., large charts, cross-region relations) receive more latent capacity, while simpler queries receive less.

2. Latent Concept Representation Learning:

   • A sequence of special tokens <|lcon|> is allocated with length $T_i$.
   • The retriever generates continuous hidden states for these tokens, conditioned on the query, the positive document, and the preceding concept tokens.
   • These hidden states are aggregated via mean pooling to form a compact latent concept representation ($\hat{c}_i$).

3. Training Objectives:

   • Contrastive Loss ($L_{cons}$): Standard page-level contrastive learning to maintain global retrieval alignment.
   • Alignment Loss ($L_{align}$): A knowledge distillation objective where the latent concept representation is trained to mimic the ranking distribution induced by the original query. Specifically, the KL divergence between the query-induced ranking distribution and the concept-induced ranking distribution is minimized. This forces the latent concepts to encode the specific relevance factors required for document ranking.

4. Inference:

   • During inference, ConceptFormer follows the standard visual document retrieval pipeline. The latent concept generation process is not required, ensuring no additional latency or architectural changes for deployment.

Key Contributions

• Latent Concept Framework: Introduces a novel representation learning paradigm where query-relevant evidence is modeled as adaptive, continuous latent concepts, avoiding the pitfalls of fixed-length representations or reliance on explicit textual/visual proxies.
• Adaptive Capacity Mechanism: Proposes a method to dynamically determine the number of latent concept tokens based on the visual coverage of query-relevant evidence, addressing the heterogeneity of evidence scales in visual documents.
• Alignment Strategy: Develops a training objective that aligns the ranking behavior of latent concepts with the original query, effectively bridging the gap between localized visual grounding and semantic relevance.

Experimental Results

The authors evaluated ConceptFormer on six diverse visual document retrieval benchmarks (InfoVQA, ChartQA, SlideVQA, TQA, OWID Charts, and Wikimedia Maps), comparing it against OCR-based text retrievers and state-of-the-art visual retrievers.

• Performance Gains: ConceptFormer achieved significant improvements over the strongest baselines.
  • 16.7% relative improvement in average NDCG@10 over the strongest visual retrieval baseline.
  • 22.1% relative improvement in average NDCG@10 over the strongest OCR-based text retrieval baseline.
• Robustness: The model demonstrated consistent performance across different VLM backbones (Phi3V and Qwen2.5-VL), with the Qwen-based variant achieving the highest scores.
• Ablation Studies:
  • Representation Type: Latent concepts outperformed both textual proxies (which struggle with complex visuals) and visual proxies (which lack semantic abstraction).
  • Adaptive vs. Fixed: Adaptive capacity allocation outperformed all fixed-length token variants, confirming that evidence scale varies significantly across samples and requires dynamic allocation.
  • Loss Weight: The alignment loss weight ($\lambda$) was found to be optimal at 0.2; higher values degraded performance, suggesting a balance between concept alignment and query discrimination is necessary.
• Embedding Space Analysis: Visualization of the retrieval geometry showed that latent concepts achieved superior Query Discriminability, Page Discriminability, and Retrieval Alignment compared to visual or textual concept proxies.

Significance and Claims

The paper claims that ConceptFormer addresses a critical gap in visual document retrieval by providing a mechanism to capture fine-grained textual cues and complex document-level visual structures simultaneously. By learning adaptive latent concepts, the framework enables the retriever to:

1. Bridge Modalities: Effectively connect localized visual evidence with semantic relevance without relying on intermediate textual descriptions that may lose visual fidelity.
2. Adapt to Complexity: Dynamically allocate representational capacity based on the specific evidence requirements of a query-document pair, rather than using a one-size-fits-all approach.
3. Enhance Alignment: Improve the alignment between textual queries and visual documents by learning an intermediate representation that preserves the ranking preferences of the original query.

The authors conclude that adaptive latent concepts provide complementary relevance supervision, offering a more flexible and expressive mechanism for modeling query-relevant evidence in visually rich documents.