iDocV2: Leveraging Self-Supervision and Open-Set Detection for Improving Pattern Spotting in Historical Documents

1. Problem Statement

Historical documents are invaluable for preserving cultural heritage, but digitizing them introduces significant challenges in content retrieval. While text-based search (OCR) is common, it fails to address the prevalence of illustrations and graphical patterns in medieval manuscripts.

• Current Limitations: State-of-the-art (SOTA) methods for pattern spotting (locating a graphical query within a document) rely on dense correlation strategies (e.g., Curi et al., 2022). These methods suffer from:
  • Excessive Latency: Dense matching requires comparing query feature maps against every patch of a document, taking up to 7 seconds per query on the DocExplore dataset.
  • Poor Performance on Specific Queries: Existing models struggle significantly with small, non-square queries (which constitute 83% of the DocExplore test set), achieving low precision (0.427 mAP).
  • Scale Sensitivity: Dense correlation assumes the target pattern size matches the query exactly, limiting robustness to scale variations.

2. Methodology

The authors propose iDocV2, a novel framework that shifts from dense correlation to sparse matching combined with self-supervised learning. The architecture consists of three main components:

A. Semantic Region Proposal (Offline)

Instead of processing the entire document page, the model first extracts a set of candidate regions of interest (ROIs).

• Tool: Uses Grounding DINO, an open-set object detector, to identify semantic regions based on text prompts (e.g., "figure," "building," "symbol").
• Filtering: Regions are filtered using Non-Maximum Suppression (NMS) with an IoU threshold of 0.5 to remove duplicates.
• Benefit: This reduces the search space from thousands of image patches to a manageable number of semantic regions (average ~112 per page), drastically cutting computation time.

B. iDoc Encoder (Self-Supervised Adaptation)

The core innovation is iDoc, a visual encoder specifically adapted for historical document patterns.

• Base Architecture: Built upon iBOT (a Vision Transformer trained with non-contrastive self-supervised learning).
• Training Strategy:
  • Self-Supervision: Trained on the HORAE dataset (107k images of Books of Hours) without manual labels.
  • LoRA Adaptation: Instead of full fine-tuning (which risks "catastrophic forgetting" and is computationally expensive), the authors use Low-Rank Adaptation (LoRA). This adapts the pre-trained iBOT weights to the historical domain by updating only ~4% of the parameters.
  • Semantic vs. Random: The encoder is trained using the semantic regions extracted by Grounding DINO, rather than random crops, to ensure coherent semantic learning.
• Output: The encoder produces a single class token embedding for each region, representing the global semantic content.

C. Sparse Decoder (Similarity Search)

• Mechanism: The system computes the cosine similarity between the query's embedding and the embeddings of the pre-extracted semantic regions from the target documents.
• Ranking: Proposals are sorted by similarity score to generate the final detection ranking.
• Post-Processing: The paper introduces Non-Maximum Suppression (NMS) in the final version (iDocV2) to further reduce false positives.

3. Key Contributions

1. iDoc Encoder: A specialized, self-supervised visual encoder for historical documents that outperforms generic SOTA encoders (CLIP, DINOv2, VGG) in this specific domain.
2. Sparse Matching Paradigm: Replacing dense correlation with a two-stage process (Region Proposal + Similarity Search) reduces search time by 10x.
3. LoRA for Domain Adaptation: Demonstrates that LoRA is superior to full fine-tuning for adapting large vision transformers to niche historical domains, preserving pre-trained knowledge while adapting to new data.
4. Semantic Region Training: Proves that training encoders on semantically extracted regions (via Grounding DINO) yields significantly better results than training on random image crops.

4. Experimental Results

The model was evaluated on the DocExplore dataset (1,500 pages, 1,447 queries) and compared against SOTA models (En et al., Úbeda et al., Curi et al.).

• Pattern Spotting Performance:

  • Overall mAP: Achieved 0.620, surpassing the previous SOTA (Curi et al.) of 0.494.
  • Small Non-Square Queries: Achieved 0.588 mAP, a massive improvement over Curi's 0.427 (a 37.7% relative gain). This is the most challenging category in the dataset.
  • Comparison: Outperformed other encoders (DINOv2, CLIP, ResNet) significantly when combined with the semantic region proposal.

• Document Retrieval Performance:

  • Achieved an overall mAP of 0.789, statistically indistinguishable from Curi's model (0.799, p-value = 0.47) but 10x faster.

• Efficiency:

  • Speed: Reduced query time from 7 seconds (Curi) to 720 ms (iDocV2).
  • Computational Load: Reduced comparisons per page from ~22,080 (dense) to ~112 (sparse).

5. Significance

This work addresses a critical bottleneck in digital humanities: the ability to efficiently search large collections of historical documents for graphical patterns.

• Practical Application: The 10x speedup makes the system viable for real-time interaction in large digital libraries, whereas previous dense methods were too slow for practical use.
• Robustness: By solving the "small non-square" query problem, the model enables the retrieval of fine details and thin illustrations that were previously undetectable by automated systems.
• Methodological Impact: The paper validates the combination of open-set detection (for region proposal) and self-supervised adaptation (via LoRA) as a powerful paradigm for specialized document analysis, offering a blueprint for future research in historical document processing.