💬 NLP

Do LLMs Truly Benefit from Longer Context in Automatic Post-Editing?

This paper systematically evaluates LLMs for automatic post-editing and finds that while proprietary models achieve near-human quality with simple prompting, they fail to effectively utilize document-level context, suffer from high costs, and rely on metrics that poorly reflect their actual performance improvements.

Original authors: Ahrii Kim, Seong-heum Kim

Published 2026-03-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ahrii Kim, Seong-heum Kim

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a professional translator who just received a rough draft of a novel written by a clumsy robot. Your job is Post-Editing: you read the robot's draft and fix the mistakes to make it sound natural and accurate.

For a long time, translators worked sentence-by-sentence, like fixing one sentence in a paragraph and then moving to the next without looking back. But now, we have Large Language Models (LLMs)—super-smart AI brains—that can read an entire book at once. The big question this paper asks is: "Does letting the AI read the whole book (the 'long context') actually help it fix the translation better, or is it just a waste of time and money?"

Here is the story of what they found, explained with some everyday analogies.

1. The Experiment: The "Blind" vs. The "Bookworm"

The researchers set up a race between two types of AI editors:

The "Blind" Editor (APEseg): This AI only sees the specific sentence it needs to fix. It's like a mechanic fixing a car engine while wearing a blindfold, only looking at the one bolt in front of them.
The "Bookworm" Editor (APEdoc): This AI sees the sentence plus the entire surrounding document. It's like a mechanic who can see the whole car, the garage, and the owner's manual all at once.

They tested this on two types of AI:

The "Super-Models" (Proprietary): Like GPT-4o. These are expensive, closed-source giants (like a highly paid, elite consultant).
The "Open-Source" Models: Like LLaMA or Qwen. These are free to download but smaller and less polished (like a talented but inexperienced intern).

2. The Big Surprise: "More Context" Didn't Mean "Better Results"

You might think, "If the AI reads the whole book, it will understand the story better and fix the translation perfectly!"

The Reality:

For the Super-Models: They were already so good at fixing sentences that reading the whole book didn't really help them improve much. They were like a master chef who can make a perfect omelet with just one egg; giving them the whole fridge didn't make the omelet any tastier.
For the Open-Source Models: Giving them the whole book actually confused them. Instead of fixing the sentence, they started hallucinating (making things up) or rewriting the whole thing incorrectly. It's like giving a nervous intern a 500-page manual; they got overwhelmed and started scribbling nonsense on the page.

The Takeaway: Just because an AI can read a long document doesn't mean it knows how to use that information to fix a single sentence.

3. The "Data Poisoning" Problem

The paper found that when you feed these models a long document, it's like throwing a bunch of noise into a quiet room.

The Super-Models are like a noise-canceling headphone user. They ignore the background noise (the rest of the document) and focus on the task. They are robust but stubborn; they rarely use the extra context to fix subtle errors.
The Open-Source Models are like a person with no headphones. The background noise drowns out the instructions. They get distracted by irrelevant parts of the document and start "hallucinating"—saying things that weren't in the original text at all.

4. The Cost of "Reading the Whole Book"

This is the most practical part of the paper.

The Price Tag: Asking the AI to read the whole document is incredibly expensive. It's like asking a taxi driver to drive around the entire city to drop you off at a house just down the street.
- For the Super-Models, the cost went up by 4,300% (that's 43 times more expensive!) just to get the same result.
- For the Open-Source models, it took 10 times longer to process, and they crashed or gave bad answers more often.

5. The "Invisible" Improvements

The researchers found that sometimes the "Bookworm" editor did make the translation sound more natural (like changing a stiff phrase to a casual one). However, standard computer tests (metrics) couldn't see this improvement. It's like a human tasting a soup and saying, "This tastes better," but a machine measuring the salt content saying, "The salt is exactly the same."

This proves that humans are still needed to judge if a translation is truly good. Computers can't always tell the difference between a "good" edit and a "bad" edit when the meaning stays the same.

Summary: What Should We Do?

The paper concludes with a few key lessons:

Don't force the AI to read the whole book yet. Currently, the "naive" approach of just dumping the whole document into the AI's prompt is too expensive and often confusing.
The "Super-Models" are great, but expensive. They can fix translations almost as well as humans, but they cost too much to run for every single sentence.
The "Open-Source" models need training. They are currently too easily distracted by long texts. We need better ways to teach them how to focus on the right parts of a document without getting overwhelmed.
We need new tools. Instead of just feeding the AI the whole book, we need smarter ways to feed it only the relevant parts (like a librarian handing you just the specific chapter you need, not the whole library).

In a nutshell: Giving AI a "long context" sounds like a great idea, but right now, it's like giving a child a whole encyclopedia to help them tie their shoes. It's too much information, it costs too much to process, and it often leads to more mistakes than solutions. We need smarter ways to use that context before it becomes a practical tool.

1. Problem Statement

Automatic Post-Editing (APE) aims to refine Machine Translation (MT) outputs by correcting residual errors. While Large Language Models (LLMs) have shown remarkable translation capabilities at the sentence level, their effectiveness in document-level APE remains under-explored. Specifically, it is unclear whether providing full-document context (long-context) actually improves translation quality compared to segment-level processing, how different models utilize this context, and whether the computational overhead justifies the gains. The study addresses the gap in understanding how LLMs handle document-level context for error correction, particularly regarding robustness, efficiency, and the limitations of current automatic evaluation metrics.

2. Methodology

Experimental Setup

Task: The authors compare two prompting strategies:
- $APE_{seg}$ (Segment-level): The model edits a single sentence using only the source and draft translation for that segment.
- $APE_{doc}$ (Document-level): The model edits a sentence while having access to the full source document ( $D_S$ ) and full target document ( $D_T$ ) as context.
Models Evaluated:
- Proprietary: GPT-4o and GPT-4o-mini.
- Open-Weight: Qwen2.5-32B and LLaMA3-8B.
- Note: LLaMA3-8B was included as a baseline with limited context (8k) and no official Korean support to contrast with models optimized for long contexts.
Dataset: WMT24++ (English-Korean pair), comprising 7,974 segments across 59 documents in three domains: Literary, News, and Social. Korean was chosen for its morphological complexity and context-dependency.
Prompting Strategy: A "naive" full-document prompting approach was used. To mitigate distraction and hallucination, the prompt included:
- Strict instructions to avoid copying context text.
- An in-context learning (ICL) example.
- Document context placed at the end of the prompt (preliminary tests showed placing it at the beginning caused models to ignore the specific editing task).

Evaluation Metrics

Automatic Metrics:
- TER (Translation Edit Rate): Measures the amount of editing required (lower is better).
- COMET: A neural metric capturing semantic adequacy and fluency (higher is better).
- BLEU / chrF++: Surface-level overlap metrics.
Human Evaluation:
- Relative Ranking: Three professional translators ranked five candidates (Initial MT, $APE_{seg}$ , $APE_{doc}$ , Human PE, Human Reference) for 886 segments.
- Statistical Analysis: Friedman test and Nemenyi post-hoc test were used to determine statistical significance of rank differences. Inter-annotator agreement (IAA) was measured using Kendall's W and $\tau_b$ .

3. Key Contributions

Human Parity with Simple Prompting: The study demonstrates that proprietary LLMs (even lightweight ones like GPT-4o-mini) can achieve human-level APE quality using simple one-shot prompting, regardless of whether document context is provided.
Limited Benefit of Document Context: Introducing document-level context ( $APE_{doc}$ ) did not yield statistically significant improvements over segment-level APE ( $APE_{seg}$ ) for proprietary models. This suggests current naive prompting strategies fail to effectively leverage long-context information for error correction.
Robustness vs. Sensitivity:
- Proprietary Models: Exhibit high robustness against "data poisoning" (irrelevant context) but fail to utilize context for improvement.
- Open-Weight Models: Highly sensitive to long, noisy contexts. They suffer from severe performance degradation, excessive edits, and hallucinations when provided with full documents.
Metric Limitations: Standard automatic metrics (BLEU, TER) fail to capture the qualitative improvements (stylistic refinement) made by LLMs. Most edits are paraphrastic (semantic equivalence) rather than meaning-altering, leading to poor correlation between metric scores and human preference.
Efficiency Gap: Document-level APE incurs massive computational overheads (token usage, latency, and cost), making naive deployment impractical despite comparable quality.

4. Key Results

Quality Performance:
- Proprietary Models: $APE_{doc}$ and $APE_{seg}$ performed statistically indistinguishably in human rankings (both clustered around rank 2, comparable to Human PE).
- Open-Weight Models: Performance degraded significantly with document context. LLaMA3-8B showed a massive drop in BLEU (-44.47) and a spike in TER (+47.96) when moving from segment to document level, indicating it often rewrote sentences entirely or hallucinated.
Edit Behavior:
- GPT-4o-mini: As document length increased, the number of edits decreased, suggesting the model preserved the initial translation to avoid errors.
- LLaMA3-8B: As document length increased, edits increased drastically, often resulting in "context drift" where the model incorporated irrelevant information from the document into the translation.
Efficiency Overhead:
- Moving from $APE_{seg}$ to $APE_{doc}$ increased token usage by ~6,000% and latency by ~146% for GPT-4o-mini.
- For Qwen2.5-32B, latency increased by ~1,000%.
- The cost overhead for proprietary models was deemed prohibitive for real-world deployment.
Hallucination: A significant portion of $APE_{doc}$ outputs from open-weight models contained fluent but irrelevant content borrowed from other parts of the document (data poisoning effect), which standard metrics like COMET failed to penalize adequately.

5. Significance and Conclusion

The paper concludes that while LLMs have reached human parity in APE quality, naive document-level prompting is currently inefficient and often counterproductive.

The "Long-Context" Paradox: Simply feeding the entire document to an LLM does not automatically improve translation quality. In fact, for smaller or open-weight models, it introduces instability and hallucinations.
Need for Advanced Strategies: The findings suggest that future APE systems require adaptive context selection (retrieving only relevant sentences) or memory compression rather than full-document injection.
Evaluation Gap: The disconnect between automatic metrics and human judgment reinforces the necessity of human evaluation for assessing translation quality, particularly for stylistic and coherence improvements that metrics miss.
Practical Implication: For real-world deployment, the high cost and latency of document-level APE currently outweigh the marginal (or non-existent) quality gains, suggesting that segment-level processing remains the most viable approach for now.

This study serves as a critical reality check for the field, urging researchers to move beyond "longer context" as a default solution and focus on efficient, selective context utilization and robust evaluation methodologies.