💬 NLP

A Comparative Study of Language Models for Khmer Retrieval-Augmented Question Answering

This paper presents a comparative study of a Retrieval-Augmented Generation system for Khmer-language telecom documents, identifying BGE-M3 as the superior retriever and demonstrating that while no single generator model dominates all performance metrics, different models excel in specific areas such as faithfulness, factual correctness, or semantic similarity.

Original authors: Sereiwathna Ros, Phannet Pov, Ratanaktepi Chhor, Kimleang Ly, Wan-Sup Cho, Saksonita Khoeurn

Published 2026-05-22

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Sereiwathna Ros, Phannet Pov, Ratanaktepi Chhor, Kimleang Ly, Wan-Sup Cho, Saksonita Khoeurn

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to answer a very specific question about Cambodian telecom laws, but you don't have the answer memorized. You have a massive library of documents (the "retriever") and a very smart, talkative assistant (the "generator") who can read those documents and write an answer for you.

This paper is like a taste test and a performance review of different tools used to build this system, specifically for the Khmer language (the language of Cambodia). Because Khmer uses a unique script and has fewer digital resources than English, the researchers wanted to see which tools actually work best together.

Here is the breakdown of their experiment, explained simply:

1. The Librarian (The Retriever)

First, the team needed a "Librarian" whose job is to find the right pages in the library when you ask a question. They tested three different librarians (AI models):

BGE-M3
Jina-Embeddings-v3
Qwen3-Embedding

The Result: BGE-M3 was the clear winner. It was like having a librarian who actually knows the Dewey Decimal system.

When asked to find the right document, BGE-M3 found the correct source file 70% of the time.
The other two librarians only found the right file about 50% of the time.
Interesting twist: One of the losing librarians (Jina) gave the highest "similarity score" (like saying, "These two pages look very similar!"), but it was actually the wrong page. This taught the researchers that a high similarity score doesn't always mean the answer is actually there.

2. The Writer (The Generator)

Once the Librarian finds the right pages, the "Writer" (a Large Language Model) reads them and writes the final answer. The team tested five different writers:

Qwen3 and Qwen3.5 (General multilingual writers)
Sailor2 (Specialized for Southeast Asia)
SeaLLMs (Specialized for Southeast Asia)
Llama-SEA-LION (Specialized for Southeast Asia)

They didn't just ask, "Is the answer good?" They used six different ways to grade the writers, like a teacher using a rubric:

Faithfulness (Did the writer stick to the facts?): Qwen3.5 was the most honest. It rarely made things up and stuck strictly to what the documents said.
Factual Correctness (Did the writer get the numbers and names right?): Qwen3 was the best at getting specific details (like phone numbers or law codes) exactly right.
Relevance & Similarity (Did the answer sound like what a human would say?): SeaLLMs was the best at sounding natural and matching the style of the "gold standard" answers.
The Loser: Llama-SEA-LION struggled the most, often making up facts or ignoring the documents.

3. The Big Takeaways

The researchers found that there is no single "perfect" robot. It depends on what you need:

If you need trust (making sure the AI doesn't lie), use Qwen3.5.
If you need precision (getting the exact numbers right), use Qwen3.
If you want the answer to sound natural and match human phrasing, use SeaLLMs.

The Bottleneck:
The biggest problem wasn't the Writer; it was the Librarian. Even the best Librarian (BGE-M3) only found the right document about 28% of the time when looking at the top 3 results. This means the whole system is held back because it's hard to find the right information in the first place.

4. The Catch

The paper admits a few limitations:

The Test: They only tested 200 questions. While carefully chosen, this might not cover every possible question a citizen might ask.
The Grader: They used another AI (GPT-4o-mini) to grade the answers, not real humans. While this is standard, human feedback would be the ultimate test.
The Setup: Some writers were tested on different computer setups, which might have slightly skewed the results.

Summary

In short, building a smart question-answering system for Khmer is like building a relay race team. You need a great runner to find the baton (the Librarian) and a great runner to carry it to the finish line (the Writer). The researchers found that BGE-M3 is the best runner to find the baton, but the best runner to carry it depends on whether you value honesty, precision, or style. Most importantly, the race is currently slowed down because finding the baton is still very difficult.

Technical Summary: A Comparative Study of Language Models for Khmer Retrieval-Augmented Question Answering

Problem Statement

Retrieval-Augmented Generation (RAG) has emerged as a critical paradigm for grounding Large Language Model (LLM) outputs in retrieved evidence, thereby reducing hallucinations and improving factual accuracy. However, the efficacy of RAG remains largely unexamined for low-resource, non-Latin-script languages like Khmer. Khmer presents unique challenges, including a complex Abugida script, limited annotated resources, weak standardization for word segmentation, and the presence of Optical Character Recognition (OCR) noise. In institutional settings, such as telecommunications, these challenges are compounded by the need for trustworthy, document-grounded responses. Existing evaluation practices are heavily shaped by English-centric settings and do not transfer cleanly to these environments. Consequently, it remains unclear which retrieval models, locally deployable generators, and automated evaluation signals are most suitable for Khmer-language RAG systems.

Methodology

The authors present a systematic study of a privacy-preserving RAG pipeline designed for locally hosted deployment over Khmer telecom-domain documents. The experimental setup follows a standard two-stage pipeline:

Dataset Construction:
- Corpus: Over 7,000 chunks derived from open-source ICT documents (notifications, laws, guidelines, etc.), mostly in Khmer with embedded English technical terms.
- Benchmark: A curated "golden dataset" of 200 question-answer pairs derived from the corpus, covering realistic citizen queries. Each entry includes the question, a domain-expert ground truth answer, and metadata.
Retriever Evaluation:
- Three dense embedding models were benchmarked for retrieving top- $k$ chunks: BGE-M3 (567M), Jina-Embeddings-v3 (570M), and Qwen3-Embedding (597M).
- Performance was measured using standard Information Retrieval (IR) metrics: Hit Rate@k, File-level Hit Rate@k, Mean Reciprocal Rank (MRR@k), and Precision@k.
Generator Evaluation:
- Using BGE-M3 (identified as the superior retriever), five generator backends were evaluated: Qwen3 (8B), Qwen3.5 (9B), Sailor2-8B-Chat, SeaLLMs-v3-7B-Chat, and Llama-SEA-LION-v2-8B-IT.
- Models included both general-purpose multilingual LLMs and those specifically focused on Southeast Asian languages.
Evaluation Metrics:
- Six RAGAS-inspired metrics were employed, utilizing GPT-4o-mini as an LLM judge and BGE-M3 for semantic similarity calculations:
  - Faithfulness: Ratio of generated statements present in the retrieved context.
  - Answer Relevance: Cosine similarity between the user question and generated questions derived from the answer.
  - Context Relevance: Semantic similarity between the question and the retrieved context.
  - Factual Correctness: F1-score of statements classified as true/false against the ground truth.
  - Answer Similarity: Embedding similarity between the generated answer and the ground truth.
  - Answer Correctness: A weighted sum of factual correctness and answer similarity.

Key Contributions

Retriever Benchmarking: The study provides the first dense retrieval benchmark for Khmer documents, analyzing effectiveness on noisy, domain-diverse institutional text.
Generator Comparison: It compares five locally deployable generator models to assess whether regional specialization (Southeast Asian-focused models) yields measurable gains over general-purpose multilingual models for Khmer QA.
Metric Adaptation: The paper examines the utility and limitations of six adapted RAGAS-style metrics in a low-resource, non-Latin-script setting, offering empirical evidence on their behavior for Khmer.

Results

Retrieval Performance

BGE-M3 consistently outperformed Jina-Embeddings-v3 and Qwen3-Embedding across all retrieval metrics.

Hit Rate@3: BGE-M3 achieved 0.285, more than double Jina v3 (0.135) and significantly higher than Qwen3-Embedding (0.175).
File Hit Rate@3: BGE-M3 retrieved a chunk from the correct source file 70% of the time, compared to 48.5% for Jina v3 and 52.5% for Qwen3-Embedding.
Observation: Interestingly, Jina v3 achieved the highest top-1 cosine similarity (0.759) but the worst retrieval accuracy, highlighting that raw cosine similarity is not a reliable indicator of retrieval quality for Khmer.

Generator Performance

No single model dominated across all metrics; performance varied based on the specific evaluation dimension:

Faithfulness: Qwen3.5-9B achieved the highest score (0.859), followed closely by SeaLLMs-v3-7B-Chat (0.846). Llama-SEA-LION-v2-8B-IT scored lowest (0.556).
Answer Relevance: SeaLLMs-v3-7B-Chat performed best (0.867), suggesting regional specialization aids topical responsiveness.
Context Relevance: Scores were tightly clustered (0.717–0.726) across all models, confirming that this metric is primarily driven by the retriever (BGE-M3) rather than the generator.
Factual Correctness: Qwen3-8B achieved the highest score (0.380), outperforming Qwen3.5-9B (0.303). This indicates a trade-off where the newer, larger Qwen3.5 model grounds better in context, while the slightly smaller Qwen3-8B reproduces factual details more accurately.
Answer Similarity & Correctness: SeaLLMs-v3-7B-Chat led in both Answer Similarity (0.836) and Answer Correctness (0.599), indicating the best overall balance between semantic closeness and factual adequacy.

Significance and Claims

The paper claims that its findings highlight critical bottlenecks and nuances in deploying RAG for low-resource languages:

Retriever Choice is a Major Bottleneck: The modest Hit Rate@3 (0.285) even for the best retriever indicates that retrieval quality remains the primary constraint on overall system performance. The study demonstrates that methods validated in English (e.g., relying on raw cosine similarity) cannot be assumed to behave identically for Khmer.
Metric Reliability: The authors argue that verdict-based metrics (Faithfulness and Factual Correctness) provide more reliable evaluation signals for Khmer text than similarity-based metrics (Answer Relevance, Answer Similarity), which are sensitive to embedding choices and inherent limitations of cosine similarity. This aligns with findings in the telecom domain regarding expert judgment.
Model Specialization is Nuanced: Regional specialization does not uniformly guarantee superior performance. While SeaLLMs excelled in relevance and similarity, Qwen3.5 was superior in grounding, and Qwen3-8B in factual precision. The impact of regional targeting is strongly model-specific rather than a guaranteed family-wide benefit.
Practical Deployment: The study offers evidence for practitioners that model selection must be driven by specific objectives: Qwen3.5 for grounding, Qwen3-8B for factual accuracy, and SeaLLMs for overall semantic quality.

The paper concludes by noting limitations, including the reliance on automated evaluation (without dedicated human evaluation for Khmer), fixed retrieval parameters, and potential biases from English-centric embedding models used in the evaluation pipeline. Future work is proposed to focus on hybrid retrieval, Khmer-specific tokenization, and human evaluation studies.