💬 NLP

Evaluating Prompt Scope and Demonstration Similarity in Local LLM Machine Translation

This paper evaluates how prompt scope and demonstration selection strategies affect the machine translation performance of local instruction-tuned LLMs across multiple language families, finding that while dedicated MT systems remain superior, embedding-based few-shot prompting and family-scope prompts can enhance larger models despite introducing structured-output challenges for smaller ones.

Original authors: Mihael Arcan

Published 2026-07-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mihael Arcan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart robot friend who can speak almost any language. You ask it to translate a sentence from English to French, and it does a great job. But what happens if you ask it to translate that same sentence into French, Spanish, and Italian all at once? Or what if you give it a few example sentences to show it how you like things done before it starts? This paper lives in the world of Machine Translation, which is the science of teaching computers to turn one language into another. For a long time, we tested these robots by asking them one simple question at a time: "Translate this." But real life is messier. People often ask for multiple languages at once, or they give the robot a "cheat sheet" of examples to help it understand the style they want. The big question is: does asking for more things at once, or giving the robot a cheat sheet, actually make it better, or does it just confuse it? This matters because more and more people are running these "robots" (called Large Language Models) on their own computers for privacy and speed, rather than using giant cloud servers. We need to know if these local robots are ready for the real world, or if they only work when the test is super simple.

The authors of this paper decided to put these local robots to the test in a very specific way. They didn't just ask them to translate one sentence into one language; they treated the way they asked the question as a variable. They tested three different "local" robots (smaller models you can run on your own computer) against some very serious, dedicated translation machines. They asked the robots to translate English into nine different European languages, grouped into two families: the "Romance" family (like French, Spanish, and Italian) and the "Germanic" family (like German, Dutch, and Swedish).

The researchers tried three different strategies for giving the robots "cheat sheets" (called few-shot prompting). First, they gave the robot zero examples. Second, they gave it five random examples. Third, they gave it five examples that were carefully picked to be very similar to the sentence they wanted translated, either by looking at the words used or by using a smart computer program to find similar meanings. They also tested two different "scopes": asking for just one language at a time, or asking for all five languages in a family to be translated at once, with the answers packed neatly into a JSON format (a specific computer file structure).

Here is what they found, and it's a bit of a mixed bag. First, the old-school, dedicated translation machines are still the champions. They are the most reliable, especially when translating into Germanic languages. The local robots are getting better, and they can hold their own for some Romance languages like Spanish and Portuguese, but they aren't quite beating the pros yet.

Second, the "cheat sheet" strategy depends entirely on which robot you are using. For the two stronger local robots (mistral:latest and qwen2.5:14b), giving them examples actually helped. They got better at translating when they saw examples first. But for the smallest robot (llama3.2:3b), giving it examples was a disaster. It got worse when it had to read the extra examples. It's like a student who does great on a test when they just read the question, but gets confused and makes mistakes when you show them a sample answer first.

Third, the way you ask the question matters a lot. When the researchers asked the robots to translate into all five languages at once (the family-scope prompt), the bigger robots handled it well, producing a neat list of translations for every language. But the smallest robot? It completely fell apart. It often forgot to translate some of the languages, or it couldn't follow the instructions to put the answers in the right format. It's as if the small robot got overwhelmed by the big order and just dropped half the items.

The paper also looked at whether the "smart" way of picking examples (finding the most similar sentences) was better than just picking random ones. For the robots that could use examples, the smart, similar examples did help a little bit more than random ones. But the difference wasn't huge. The most important thing was simply that the robot was capable enough to use the examples at all. If the robot was too small or confused, even the best examples couldn't save it.

In short, this paper suggests that we can't just say "AI translation is good" or "AI translation is bad." It depends on how you ask the question. If you want a local robot to translate for you, you need to pick a model that is big enough to handle complex instructions and multiple languages at once. If you pick a tiny model and ask it to do too much at once, or give it too much extra information, it might fail spectacularly. The authors conclude that when we test these translation tools, we shouldn't just look at the translation quality; we also need to check if the robot can follow complex instructions and deliver exactly what we asked for, without dropping any balls.

Technical Summary: Evaluating Prompt Scope and Demonstration Similarity in Local LLM Machine Translation

Problem Statement

Traditional machine translation (MT) evaluation treats translation as a single language-pair task, where a system translates one source sentence into one target language. However, Large Language Models (LLMs) are increasingly deployed as general-purpose translation tools where users may request multiple target languages simultaneously, provide in-context examples, or require structured outputs (e.g., JSON). This paper argues that current evaluation frameworks fail to capture these "extra degrees of freedom" inherent in LLM usage. Specifically, it investigates whether prompt scope (single-target vs. multi-target family requests) and demonstration selection strategies (random, lexical, vs. embedding-based retrieval) significantly impact the performance of local, consumer-hardware LLMs. The study focuses on local deployment due to its relevance for privacy, cost control, and offline use, noting that smaller local models may be more sensitive to prompt length and structured-output constraints than proprietary high-end systems.

Methodology

Experimental Design

The study evaluates English-to-Romance and English-to-Germanic translation across nine official EU languages (French, Spanish, Italian, Portuguese, Romanian, German, Dutch, Danish, Swedish) using the full FLORES devtest split (1,012 sentences). Demonstrations are drawn exclusively from the FLORES dev split to prevent data leakage.

The experimental variables include:

Prompt Scope:
- Single-target: The model translates one sentence into one language.
- Family-scope: The model translates one sentence into all languages of a family (Romance or Germanic) in a single JSON object keyed by FLORES language codes.
Demonstration Selection (Few-shot):
- Zero-shot: No examples provided.
- Random: $k=5$ examples selected uniformly from the dev set.
- Lexical-similarity: $k=5$ examples with highest cosine similarity based on bag-of-token source representations.
- Embedding-similarity: $k=5$ examples with highest cosine similarity based on multilingual sentence embeddings.

Models and Baselines

Local LLMs: Three instruction-tuned models run via Ollama (temperature 0): llama3.2:3b, mistral:latest, and qwen2.5:14b.
Dedicated MT Baselines: OPUS-MT (MarianMT) and NLLB-200 (distilled 600M and 1.3B versions). These are evaluated only in the single-target setting as they are not designed for structured multi-target instruction following.

Evaluation Metrics

Translation Quality: Corpus-level chrF++, BLEU (via SacreBLEU), and sentence-level COMET.
Compliance (Family-scope only): Target coverage, complete-output rate, missing-target count, and prompt echo count.
Statistical Analysis: Paired bootstrap resampling for confidence intervals and p-values comparing the best local LLM against the best MT baseline.

Key Results

1. Performance of Local LLMs vs. Dedicated MT

Dedicated MT systems remain the strongest overall baselines, particularly for Germanic languages where the performance gap is substantial. Local LLMs are competitive for specific Romance targets (notably Spanish and Portuguese), with qwen2.5:14b often approaching dedicated MT performance in single-target settings. However, no local LLM consistently matches the best MT baselines across all metrics and languages.

2. Impact of Few-Shot Prompting

The effect of few-shot prompting is highly model-dependent:

mistral:latest and qwen2.5:14b: Both models show significant improvements with few-shot prompting. Embedding-based retrieval yields the best average gains (e.g., mistral gains +4.60 chrF++ and +11.03 BLEU with embedding retrieval).
llama3.2:3b: This smaller model consistently degrades in performance with few-shot prompting across all retrieval strategies, suggesting that smaller models may struggle with the increased context length or instruction complexity.

3. Retrieval Strategy Efficacy

While embedding-based retrieval provides the best average results for models that benefit from examples, the margin over random and lexical retrieval is modest. Crucially, sentence-level analysis reveals that the similarity score of a retrieved example does not strongly predict whether that specific example will improve translation quality (correlation near zero). This suggests that examples help through multiple mechanisms (formatting, style anchoring, task clarification) rather than purely semantic similarity.

4. Family-Scope and Structured Output

Family-scope prompting acts as a stress test for structured output compliance:

Stronger Models (mistral, qwen): Maintain high coverage (>96%) and produce mostly complete JSON objects, though translation quality on produced outputs is slightly lower than single-target performance.
Smaller Model (llama3.2:3b): Suffers severe structured-output failures. In the Germanic family setting, its zero-shot coverage drops to 0.343, and even with embedding retrieval, it only reaches 0.493. For this model, the family-scope task is primarily a failure of output compliance rather than just translation quality.

5. Comparison with Dedicated MT

Paired significance tests confirm that the best local LLM conditions remain below the best dedicated MT baselines in chrF++ for all targets. The gap is smaller for Romance languages (average -2.55 chrF++) than for Germanic languages (average -7.26 chrF++). COMET scores sometimes narrow the gap or show non-significant differences (e.g., for Spanish), but they do not overturn the conclusion that dedicated MT systems are currently superior overall.

Significance and Claims

The paper does not claim to introduce a new translation model or a method to surpass dedicated MT systems. Instead, its primary contribution is an evaluation design and empirical analysis demonstrating that:

Prompt Scope is a Critical Variable: Evaluating LLMs solely on single-target translation is insufficient. Family-scope prompting reveals distinct failure modes (omitted targets, malformed JSON) that standard MT evaluation misses.
Demonstration Strategy Matters: The choice of few-shot examples (random vs. semantic) significantly impacts performance, but this effect is contingent on model capacity. Smaller models may be harmed by additional context.
Compliance is Part of Quality: For multi-target translation, "quality" must include structured-output compliance metrics (coverage, completeness). A model producing fluent but incomplete translations is not functionally equivalent to one producing complete outputs.
Evaluation Recommendations: LLM-based MT evaluation should report results across prompt scopes, retrieval strategies, and compliance metrics, not just language-pair specific scores.

The authors conclude that while local LLMs are useful and competitive in selected settings, they should be deployed with validation mechanisms rather than assumed reliability, especially in safety-critical contexts where missing outputs or structured failures pose real risks.