🤖 AI

KoVRE: Training an Efficient Embedding Model for Korean Visual Document Retrieval

The paper introduces KoVRE, an efficient single-vector embedding model for Korean Visual Document Retrieval that leverages targeted bilingual supervision and advanced training strategies to outperform larger backbones and multi-vector baselines without requiring massive scale.

Original authors: Yongbin Choi, Gyuho Shim, Youngjoon Jang

Published 2026-08-04

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Yongbin Choi, Gyuho Shim, Youngjoon Jang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to find a specific page in a massive, messy library where the books are made of pictures, not just words. If you ask a librarian for "a page with a red chart and a story about money," they might try to read the text out loud first. But what if the text is blurry, or the chart is drawn in a way that words can't describe? That's where Visual Document Retrieval comes in. Instead of just reading the words, this technology looks at the actual image of the page—the layout, the colors, the tables, and the pictures—to find exactly what you need. It's like having a librarian who can "see" the whole page, not just read the letters.

Usually, these smart librarians are trained mostly on English books. If you ask them about Korean documents, they might get confused because the shapes of the letters and the way the pages are designed are different. Also, making these librarians super smart often requires them to be huge, slow, and expensive to run, like a giant supercomputer trying to remember every single pixel of every book. The big question is: Can we build a smaller, faster, and cheaper librarian that is specifically trained to understand Korean visual documents without needing to be a giant?

This is exactly what the researchers behind KOVRE set out to do. They created a new, compact "librarian" (a computer model) specifically for Korean visual documents. Instead of making the librarian bigger, they taught it smarter. They trained it on a massive collection of 708,729 pairs of questions and document pages, mixing both Korean and English to keep the model sharp. They used a clever two-step training process: first, they showed the model tricky examples (hard negatives) to teach it what not to pick, and second, they had a "teacher" model (a very smart but slow expert) show the student model how to rank the best answers.

The results were surprising and promising. Their new 2-billion-parameter model (which is relatively small) didn't just do okay; it actually beat a much larger 8-billion-parameter model and even outperformed a powerful system that uses a complex, memory-heavy method to store information. The paper suggests that by carefully designing the training recipe—specifically how they handled difficult examples and how they normalized scores during the "teacher-student" learning phase—you can create a highly effective Korean document finder without needing a massive computer or a huge storage system. It's a proof that with the right training strategy, a small, efficient model can be just as good, or even better, than the giants.

Technical Summary: KOVRE – Training an Efficient Embedding Model for Korean Visual Document Retrieval

Problem Statement

Visual Document Retrieval (VDR) aims to match text queries directly against rendered document images, preserving critical visual and structural information (e.g., tables, figures, layout) that is often lost during Optical Character Recognition (OCR) and text extraction pipelines. While recent VDR models have achieved strong performance, they face three primary limitations:

English-Centric Bias: Existing training data and benchmarks are predominantly English, leaving a gap in high-performing systems for Korean.
Resource Intensity: High-performing models often rely on massive backbones (e.g., 8B+ parameters) or multi-vector representations (storing multiple embeddings per page), which incur significant storage and retrieval costs.
Underexplored Training Strategies: There is limited research on how to effectively train compact, single-vector retrievers for specific languages, particularly regarding training data composition, hard-negative mining, and knowledge distillation from rerankers.

Methodology

The authors introduce KOVRE (KOrean Visual Document Retrieval Embedding), a compact single-vector retriever adapted from the Qwen3-VL-Embedding-2B model. The training process is divided into two distinct stages:

1. Base Model and Representation

Architecture: The model retains the original Qwen3-VL-Embedding-2B vision-language architecture without modification.
Input: It accepts text queries and rendered document page images, mapping them to a shared dense space.
Matryoshka Representation Learning (MRL): The model is trained to output embeddings at multiple dimensions (2048, 1024, 768, 512, 256, 128), allowing for flexible trade-offs between retrieval quality and storage efficiency.
Similarity: Cosine similarity is used for both training and inference.

2. Stage 1: Contrastive Learning (CL)

Data: The model is trained on 708,729 query–page pairs comprising both Korean and English data.
- Korean: Combines public KoViDoRe data and private AI Hub collections.
- English: Aggregated from five existing VDR resources (reports, slides, tables) to prevent catastrophic forgetting of the backbone's multilingual capabilities.
Hard-Negative Mining: Seven hard negatives per query are mined using a larger Qwen3-VL-Embedding-8B model.
- Positive-Aware Filtering: Candidates scoring >95% of the annotated positive similarity are excluded to reduce false negatives.
- Self-Guide Filtering: Negatives scoring significantly higher than the positive (based on a stop-gradient threshold) are excluded from the loss calculation.
Loss Function: An InfoNCE objective is optimized with hardness weighting. Explicitly mined hard negatives receive an additional stop-gradient term weighted by a factor ( $\alpha=2.0$ ) to emphasize difficult examples, while in-batch negatives remain unweighted.
Objective: The loss is computed jointly across all Matryoshka dimensions.

3. Stage 2: Knowledge Distillation (KD)

Data: Uses only the Korean subset of the corpus.
Teacher Model: Qwen3-VL-Reranker-8B scores a candidate pool of 64 pages (32 mined hard negatives + 32 random samples) for each query.
Target Construction: The top 8 negatives (based on teacher scores) are selected alongside the positive to form the training instance.
Score Normalization: A critical step involves per-row min-max normalization of both teacher and student scores to the range [0, 1] before applying softmax. This addresses the unbounded, query-dependent scale of reranker scores, preserving the shape of the relevance distribution.
Loss Function: The student minimizes the KL-divergence between its score distribution and the teacher's normalized distribution across all Matryoshka dimensions.

Key Contributions

KOVRE Model: Development of a compact, single-vector retriever for Korean visual documents that achieves competitive performance without requiring a larger backbone or multi-vector indices.
Systematic Training Recipe: A comprehensive analysis of training components, demonstrating the efficacy of:
- Bilingual supervision (Korean + English) to preserve multilingual capabilities.
- Positive-aware hard-negative mining and self-guide filtering.
- Hardness weighting for mined negatives.
- Min-max score normalization for effective knowledge distillation.
- The utility of Matryoshka representations for compact retrieval.

Results

Evaluated on Korean VDR benchmarks (KoViDoRe and SDS KoPub VDR), KOVRE demonstrates significant improvements:

Performance Gains: The final 2B KOVRE model substantially outperforms its base Qwen3-VL-Embedding-2B backbone.
Comparison to Larger Models: KOVRE outperforms the 8B single-vector counterpart (Qwen3-VL-Embedding-8B) and a strong 4B multi-vector baseline (Jina Embeddings v4) in aggregate performance (nDCG@10).
Dimensionality Efficiency: Even at reduced dimensions (e.g., 256), KOVRE retains strong performance, slightly outperforming the 4096-dimensional Qwen3-VL-Embedding-8B.
Ablation Insights:
- Bilingual training outperforms Korean-only training.
- Adding extra Korean text datasets (without visual context) reduced performance.
- Min-max normalization in Stage 2 was critical; without it, distillation actively harmed performance.

Significance and Claims

The paper claims that targeted bilingual supervision and carefully designed training strategies can produce a highly effective Korean VDR model across diverse domains without scaling up model size or adopting storage-intensive multi-vector representations.

The authors position KOVRE as a practical solution for languages underserved by current models, demonstrating that a well-trained compact model can rival or exceed larger, more complex systems. The work provides a reproducible training recipe that emphasizes the importance of data composition, negative mining strategies, and score normalization in knowledge distillation for visual document retrieval.

Limitations Noted: The authors acknowledge that the Korean training corpus contains fewer distinct page images compared to English, potentially limiting the diversity of layouts and document types observed during training. They suggest future work should focus on expanding the diversity of Korean page images rather than just increasing query counts.