Technical Summary: Qwen Goes Brrr – Off-the-Shelf RAG for Ukrainian Multi-Domain Document Understanding

1. Problem Definition

The paper addresses the Fifth UNLP shared task, a Kaggle code competition focused on Ukrainian multi-domain document understanding. The system must process a collection of PDF documents and answer six-option multiple-choice questions. The task requires three coupled outputs:

1. The correct answer option (A–F).
2. The source document containing the evidence.
3. The specific page within that document grounding the answer.

Key Constraints & Challenges:

• Data Scarcity: Limited task-specific training data (40 initial questions, 461 development questions) and an unseen third domain in the hidden test set.
• Document Complexity: Inputs are full PDFs with varied formatting, tables, and section hierarchies, making page-level localization non-trivial.
• Runtime Budget: Solutions must execute within nine hours on Kaggle hardware (2×T4 GPUs) without internet access, favoring efficient pipelines over complex, multi-step agentic loops.
• Metric: The scoring function weights answer correctness (50%), document identification (25%), and page localization (25%), with page proximity only counting if the document is correct.

2. Methodology

The authors propose a retrieval-augmented generation (RAG) pipeline built around three core components: contextual chunking, question-aware dense retrieval, and answer-aware reranking.

2.1 Contextual Chunking

Instead of treating PDF pages as plain text, the authors employ structure-aware chunking:

• Processing: They replaced the slow/unstable Docling library with a custom pymupdf4llm-based converter to render PDFs to Markdown.
• Structure: Each chunk includes three nested context levels:
  1. A global document prefix (first 128 tokens).
  2. The current heading path (section context).
  3. The local chunk body.
• Configuration: Chunks are limited to 512 tokens with a 64-token overlap. Ablation studies confirmed that removing this document context caused a significant performance drop (Public score from 0.945 to 0.861).

2.2 Dense Retrieval (Stage 1)

• Model: The system uses Qwen3-Embedding-8B as the primary retriever.
• Query Construction: Queries are constructed from the full multiple-choice instance (question + all six answer options + instruction), rather than the question alone.
• Training Strategy: Fine-tuning the 8B model on the specific competition data provided no gain and risked overfitting. The pretrained model alone proved robust. Smaller models (0.6B) required fine-tuning on auxiliary data (UA-SQuAD + a new 80k corpus) to close the performance gap.
• Output: The top-20 chunks are passed to the reranker.

2.3 Option-Aware Reranking (Stage 2)

• Model: A fine-tuned Qwen3-Reranker-8B.
• Input: The reranker receives the question concatenated with all six answer options. This allows the model to distinguish passages based on which specific alternatives they support or refute, rather than just topical similarity.
• Training Data: The model was fine-tuned on a combination of:
  • 461 competition development examples.
  • 16,658 pairs from UA-SQuAD (Ukrainian extractive QA).
  • Hard negatives mined from impossible questions, dense retrieval errors, and confusing wrong-answer passages.
  • An auxiliary 80,000-example pretraining corpus (translated from English datasets: Natural Questions, HotpotQA, MS MARCO, GooAQ) to improve generalization.
• Selection: Only the top-2 reranked passages are passed to the final generator.

2.4 Answer Selection (Stage 3)

• Model: Qwen3-32B (AWQ quantized).
• Input: The question, all options, and the top-2 reranked passages.
• Generation: The model is constrained to generate a single token from the alphabet {A, B, C, D, E, F}.
• Localization: The predicted document and page are derived from the highest-ranked selected chunk.

3. Key Contributions

1. State-of-the-Art Pipeline: A retrieval-augmented pipeline for Ukrainian multi-domain document understanding that integrates structure-aware chunking, question-aware retrieval, and answer-aware reranking within strict code-competition constraints.
2. Off-the-Shelf Efficacy: Demonstration that highly competitive performance can be achieved using largely pretrained models with careful pipeline design (preserving document structure and conditioning relevance on the full multiple-choice instance) rather than complex downstream heuristics.
3. New Ukrainian Resources: Construction of new resources for retrieval-based multiple-choice QA, including:
   • A training set for dense retrieval and reranking combining competition data and UA-SQuAD.
   • A novel 80,000-example pretraining corpus for Ukrainian bi-encoder and cross-encoder models, derived from translated English datasets.

4. Results

• Leaderboard Performance: The final system achieved 0.9452 on the public leaderboard and 0.9598 on the private leaderboard.
• Impact of Reranking:
  • Reranking improved Recall@1 from 0.6957 to 0.7935.
  • Using the top-2 reranked passages increased answer accuracy from 0.9348 to 0.9674.
• Ablation Insights:
  • Document Context: Removing the document prefix from chunks caused the largest single performance drop.
  • Model Size vs. Fine-tuning: Larger pretrained embedders (8B) performed best out-of-the-box. Fine-tuning smaller models (0.6B) with diverse data (80k corpus + UA-SQuAD) recovered much of the gap, but fine-tuning on competition-specific data alone degraded performance due to overfitting.
  • Domain Routing: A domain-classifier routing strategy worked well on the public set (two visible domains) but failed on the private set (unseen third domain), confirming that domain-agnostic retrieval is more robust for generalization.
  • Summary Injection: Replacing raw document prefixes with generated summaries degraded performance, suggesting raw structural signals are superior to compressed summaries for this task.

5. Significance and Claims

The paper claims that for Ukrainian multi-domain document understanding, preserving document structure and making relevance estimation aware of the answer space are more effective than adding complex downstream heuristics or agentic loops.

• Efficiency vs. Complexity: The authors argue that under strict runtime budgets, a well-structured pipeline using modern pretrained components (Qwen3 family) is sufficient to achieve top-tier performance without needing task-specific architectural innovations.
• Resource Gap: By introducing the 80k pretraining corpus and fine-tuning strategies, the work addresses the scarcity of high-quality Ukrainian retrieval datasets, enabling better generalization across domains.
• Practicality: The results suggest that "off-the-shelf" models, when combined with careful data preparation (contextual chunking) and appropriate conditioning (question + options), can outperform more complex, custom-built systems in constrained environments.

The authors explicitly note that while their approach is robust for the competition setting, it relies on relatively large models (8B–32B) and machine-translated auxiliary data, which may present challenges for production deployments with stricter latency or manual review requirements. They identify bidirectional embedding models (e.g., diffusion-based) and late chunking as promising future directions if compute constraints are relaxed.