💬 NLP

Which Tokens Need Context? A Reference-Based Analysis of Translation Responsibility Using Fertility and Entropy

This paper proposes a model-agnostic, post-hoc framework using word alignment-derived fertility and entropy to quantify how human translation selectively redistributes generative responsibility from source to context tokens, revealing that context primarily resolves ambiguity for function words without altering the overall information load of content words.

Original authors: Ramakrishna Appicharla, Baban Gain, Santanu Pal, Asif Ekbal

Published 2026-06-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ramakrishna Appicharla, Baban Gain, Santanu Pal, Asif Ekbal

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are translating a story from one language to another. When a human translator does this, they don't treat every single word the same way. Some words are like anchors; they stand on their own and don't need help from the rest of the story (like a specific name, "Paris"). Other words are like chameleons; their meaning changes completely depending on what was said just before or what comes next (like the word "it" or "he").

This paper asks a simple question: Which words actually need the "surrounding story" to make sense, and which ones don't?

The researchers wanted to find out if computer translation systems are using the surrounding story (context) the same way humans do. Instead of looking inside the computer's "brain" (which is complicated and varies by model), they built a new tool to look at the results of human translations.

Here is how they did it, using two simple concepts:

1. The Two Tools: "Fertility" and "Entropy"

To understand how words work, the authors used two metaphors:

Fertility (The "Workload"): Imagine every word in the original sentence has a job to do: it has to "give birth" to words in the new language.
- A word with high fertility is a hard worker that turns into many words in the new language.
- A word with low fertility might not turn into anything at all, or just one word.
- The Paper's Finding: When you add the previous sentence to the story, the original sentence's words suddenly have less work to do. They "offload" some of their job to the new context.
Entropy (The "Predictability"): Imagine how consistent a word is.
- Low Entropy means a word is predictable. It always does the same job, no matter the story. (Example: The name "Google" is always "Google").
- High Entropy means a word is unpredictable. Its job changes wildly depending on the story. (Example: The word "bank" could be a river bank or a money bank).
- The Paper's Finding: Words that rely on context (like pronouns) become more unpredictable (higher entropy) when you look at them in isolation, but the context helps settle them down.

2. The Big Discovery: "Responsibility Redistribution"

The most important finding of this paper is that adding context does not create new work; it just shifts the workload.

Think of it like a relay race.

Without Context: The runner holding the baton (the source word) has to run the whole distance alone.
With Context: The runner passes part of the distance to a teammate (the context word). The total distance of the race doesn't change, but the responsibility is shared.

The paper found that:

Function Words (The Chameleons): Words like pronouns ("he," "she"), helpers ("is," "are"), and connectors ("and," "but") are the ones that hand off the most work to the context. They rely heavily on the surrounding sentences to know what they mean.
Content Words (The Anchors): Words like nouns ("cat," "car") and proper names ("London") keep doing their own work. They rarely change their job, even when you add more sentences.

3. The "Random" Surprise

The researchers tested this with three types of context:

The sentence that came before.
The sentence that comes after.
A completely random sentence from a different story.

The Result: Even when they gave the translator a random, unrelated sentence, the "workload" still shifted slightly. The source words still offloaded some responsibility to the random words.

Why? The computer (and the alignment tool they used) found tiny, accidental connections between words, even if they didn't make logical sense. This suggests that simply having extra text changes how the translation is built, even if that text isn't perfectly relevant.

4. Why This Matters

The authors built a "diagnostic baseline." Think of this as a gold standard map of how humans actually use context.

Before this paper: We didn't have a clear way to tell if a computer was using context correctly or just guessing.
Now: We have a map. If a computer translation system shifts the workload exactly like the human map (offloading pronouns to the context, but keeping nouns steady), it is working like a human. If it tries to make every word work harder or ignores the context entirely, we know it's not behaving naturally.

Summary in a Nutshell

This paper didn't build a new translator. Instead, it built a magnifying glass to watch how human translators use the surrounding story. They discovered that humans are very selective: they only call in the "context team" to help with tricky, ambiguous words (like pronouns), while leaving the solid, clear words (like names) to do their own thing. They proved that context doesn't add new information; it just helps redistribute the effort of translating the sentence.

Technical Summary: Which Tokens Need Context? A Reference-Based Analysis of Translation Responsibility Using Fertility and Entropy

Problem Statement
While extra-sentential context is known to be crucial for resolving discourse-level ambiguities (e.g., anaphora, ellipses), current Document-level Machine Translation (DocMT) systems often encode context densely and sequentially, treating all tokens as potentially relevant. This approach contrasts with human translation, where context processing is selective and non-linear. Existing methods to analyze context utilization are limited: they either rely on narrow, discourse-specific test sets (e.g., pronoun translation benchmarks) or require access to model internals via Explainable AI (XAI) techniques like attention visualization or mutual information analysis. Consequently, there is a lack of a model-agnostic, ground-truth framework to quantify which specific tokens require context and how context influences the translation process in human-like ways.

Methodology
The authors propose a post-hoc, model-agnostic framework that analyzes human reference translations to quantify selective context usage. The method relies on two metrics derived from word alignments:

Fertility: Defined as the number of target tokens generated from a specific source or context token. It measures the structural contribution of a token to the translation.
Entropy: Measures the stability of fertility patterns across different contexts for a given Part-of-Speech (PoS) category. High entropy indicates variability in how a token participates in translation, while low entropy suggests consistent behavior.

The framework operates as follows:

Data: Reference translations for three language pairs (German $\leftrightarrow$ English, English $\to$ Hindi) are processed.
Alignment: Token-level alignments between source/context and target sentences are extracted using the awesome-align toolkit.
Context Conditions: Experiments compare four settings: No Context, Previous Sentence (Prev-1), Next Sentence (Next-1), and Random Sentence (Random-1).
Analysis: The authors compute average fertility and entropy for PoS categories (grouped into function words and content words) to observe how "generative responsibility" shifts when context is introduced.

Key Results
The analysis of reference translations yields several distinct findings regarding how humans utilize context:

Redistribution of Responsibility: Adding context does not alter the overall corpus-level fertility (the total number of target tokens generated per source token remains constant). Instead, context causes a redistribution of generative responsibility. Source token coverage decreases by 5–7%, while context token coverage increases by 9–20%.
Selective Impact on Function Words: The reduction in source fertility is not uniform. Function words (pronouns, particles, auxiliary verbs, conjunctions, determiners) exhibit the largest fertility reductions (up to ~13% for pronouns). In contrast, content words (nouns, proper nouns, adjectives) remain largely stable, with proper nouns showing almost no change. This suggests context resolves ambiguity for grammatical tokens rather than adding new lexical information.
Entropy and Stability: Function words exhibit higher entropy changes (greater variability in fertility patterns) when context is added, whereas content words remain stable. This indicates that context usage is highly selective and variable for discourse-related tokens.
Context Token Behavior: When context tokens do participate in translation, they typically generate approximately one target token with low entropy, indicating a focused, one-to-one contribution rather than complex generation.
Random Context: Even randomly sampled sentences introduce non-zero alignment coverage and reduce source fertility, though to a lesser extent than coherent previous/next sentences. This suggests alignment models may link semantically similar tokens even without discourse coherence, but coherent context drives more significant responsibility shifts.

Significance and Contributions
The paper claims to establish a diagnostic baseline for evaluating machine translation systems by characterizing "selective context usage" in human translation. Its primary contributions are:

A New Framework: It introduces a post-hoc, model-agnostic methodology using fertility and entropy to quantify context usage without requiring specific discourse test sets or model internals.
Concept of Responsibility Redistribution: It posits that context does not add new information but shifts generative responsibility from source to context tokens selectively.
Empirical Ground Truth: By analyzing human translations, it identifies that function words are the primary targets of selective context attention, providing a benchmark against which model behavior can be compared.
Interpretability: The joint analysis of fertility and entropy offers interpretable signals for understanding where contextual information contributes to translation and how consistently it is utilized.

The authors conclude that their framework provides a "ground-truth characterisation" of how humans process context, suggesting that effective translation models should emulate this selective, precise invocation of context rather than dense, indiscriminate encoding. Future work involves applying this framework to model-generated outputs to compare them against these human baselines.

1. The Two Tools: "Fertility" and "Entropy"

2. The Big Discovery: "Responsibility Redistribution"

3. The "Random" Surprise

4. Why This Matters

Summary in a Nutshell

More like this