🤖 AI

Answer Presence Drives RAG Rewriting Gains

This paper demonstrates through controlled interventions that the performance gains observed in retrieval-augmented QA pipelines are primarily driven by the presence of the gold answer string in the rewritten context rather than by the curation process itself, while also revealing the fragility of conventional leakage detection methods.

Original authors: Yuejie Li, Yueying Hua, Ke Yang, Li Zhang, Yueping He, Yueping He, Ruiqi Li, Bolin Chen, Tao Wang, Bowen Li, Chengjun Mao

Published 2026-06-05

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yuejie Li, Yueying Hua, Ke Yang, Li Zhang, Yueping He, Yueping He, Ruiqi Li, Bolin Chen, Tao Wang, Bowen Li, Chengjun Mao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Question: Is the "Rewriter" a Magician or a Cheater?

Imagine you are taking a difficult trivia quiz. You have a team of two people helping you:

The Researcher: They search through a massive library of books to find relevant pages.
The Editor: They take those messy pages, summarize them, clean up the noise, and write a neat "cheat sheet" for you.
The Student: They read the cheat sheet and write the final answer.

In recent years, researchers found that adding the Editor step makes the Student score much higher (sometimes by huge margins). The common belief was: "The Editor is great because they organize the information so well, remove distractions, and connect the dots."

This paper asks a suspicious question:
Is the Editor actually doing a great job of organizing? Or are they just accidentally (or intentionally) writing the answer right on the cheat sheet, and the Student is just reading it?

The Experiment: The "Controlled Edit" Audit

To find out, the authors didn't just look at the scores; they ran a "controlled audit." Think of it like a magic trick inspection. They took the Editor's cheat sheets and performed four specific tricks on them before showing them to the Student:

The "Erase" Trick: They found the actual answer on the cheat sheet and replaced it with a blank space (or a generic placeholder like [MASK]).
The "Placebo" Trick: They erased a random, unimportant sentence of the same length (just to see if any erasing hurts the score).
The "Insert" Trick: They took a cheat sheet that didn't have the answer, and they pasted the answer in at the very top.
The "Mid-Point" Trick: They pasted the answer in the middle of the text instead of the top.

The Shocking Results

The results were dramatic and changed how we should view these "Editors":

When they erased the answer: The Student's score crashed. It dropped by 28 to 64 points.
When they erased a random sentence (Placebo): The Student's score barely moved (dropped 0 to 13 points, or even went up slightly in some cases).
When they inserted the answer: The Student's score went back up significantly.

The Analogy:
Imagine a student taking a math test.

Scenario A: The teacher gives them a perfectly organized, beautiful summary of the math concepts. The student gets an A.
Scenario B: The teacher gives the same beautiful summary, but crosses out the final number (the answer). The student gets a F.
Scenario C: The teacher gives the same beautiful summary, but crosses out a random word like "the" or "and." The student still gets an A.

The Conclusion: The "beauty" of the summary (the curation) wasn't the main reason the student got an A. The reason was that the answer was right there in the text. The "lift" in performance wasn't due to better organization; it was due to the answer being surfaced.

The "Mask" Problem: Why Old Tests Failed

The paper also points out that previous methods used to check for this "cheating" were broken.

Previously, researchers used a single "magic token" (like [MASK]) to hide the answer and see if the score dropped. The authors found that this token itself was unreliable.

The Analogy: Imagine checking if a student is cheating by covering the answer with a specific sticker. If the student can read the sticker and guess the answer anyway, the test fails.
The paper showed that if you use a different "sticker" (like a word, a symbol, or a sentence saying "Answer Removed"), the results flip completely. The old method was like a faulty lie detector that gave false positives because it relied on one specific trick.

What This Means (and What It Doesn't)

What it means: On complex, multi-step questions (like "Who was the wife of the president who signed the Emancipation Proclamation?"), the "Editor" models are often just finding the answer and pasting it into the context. The huge score improvements we see are largely because the answer is visible, not because the information was perfectly curated.
What it doesn't mean: The authors aren't saying Editors are useless or that they never help. They are just saying that most of the score boost comes from the answer being present. They also aren't proposing a new AI model to fix this; they are just providing a new "audit kit" (a set of tools) so other researchers can test their own models to see if they are also "cheating" by surfacing the answer.

Summary

The paper reveals that in many AI reading-comprehension systems, the "smart" step of summarizing text is often just a fancy way of hiding the answer in plain sight. When you remove that answer, the system collapses. The "improvement" isn't magic; it's just the answer being there.

Technical Summary: "Answer Presence Drives RAG Rewriting Gains"

Problem Statement
Retrieval-augmented generation (RAG) pipelines increasingly utilize a "compiler" (a stronger LLM rewriter) to process retrieved passages before a smaller "reader" model generates the final answer. On multi-hop benchmarks, this rewriter typically lifts the reader's F1 score by tens of points. The prevailing explanation attributes this gain to improved evidence quality—better organization, denoising, and multi-hop chaining. However, the authors observe that in these same settings, the rewriter surfaces the gold answer string in the rewritten context in approximately 80% of records. This creates an observational entanglement: it is unclear whether the performance lift is causally driven by the curation of evidence or simply by the presence of the gold answer string in the context.

The conventional method to disentangle these factors involves substring-masking the gold answer with a sentinel token (e.g., [MASK]) and re-running the reader. A significant positive residual (performance remaining above raw retrieval) is typically interpreted as evidence of a "non-leakage" channel (i.e., genuine curation). The authors argue that this single-sentinel approach is unreliable and may produce artifacts.

Methodology
The paper proposes a controlled intervention audit to replace the conventional single-sentinel masking approach. The methodology consists of two layers:

Causal Intervention Layer:
- Design: For each rewritten context, the authors apply four controlled edits:
  - Remove: Remove the gold answer span and replace it with [MASK].
  - Placebo: Replace a length-matched random non-answer span with [MASK].
  - Insert (Prefix): Prepend the gold answer string to rewrites where it was absent.
  - Insert (Midpoint): Insert the gold answer string at a midpoint sentence boundary.
- Stratification: Interventions are applied based on whether the gold answer was present in the raw retrieval (ans_in_b1) and whether it was present in the compiled output (ans_in_compile).
- Causal Estimation: The primary metric is the "remove-minus-placebo" contrast ( $\Delta_{causal} = \Delta_{remove} - \Delta_{placebo}$ ) on the stratum where the gold answer is present in the compiled output ( $1 \to 1$ ). This isolates the effect of the gold answer's presence from the general effect of editing the text.
- Scope: The audit spans 12 intervention runs across three reader families (Qwen2.5-7B, Qwen3.5-35B, GLM-4.7), two datasets (HotpotQA, 2WikiMultihopQA), and three compiler configurations.
Sentinel-Fragility Audit:
- Design: To test the reliability of the conventional [MASK] token, the authors apply five different sentinel tokens to the same paired stratum: [MASK], [REMOVED], a natural-language deletion phrase, a generic word, and a symbol string.
- Equivalence Criteria: The audit checks if alternative sentinels produce results statistically equivalent to [MASK]. If the "non-leakage residual" flips sign or magnitude depending on the sentinel choice, the original [MASK] result is deemed fragile.

Key Results

Causal Dependence on Answer Presence: Across all 12 intervention runs, removing the gold answer from the rewritten context caused a dramatic drop in reader F1 (28 to 64 points) compared to the length-matched placebo. The placebo intervention (removing random text) resulted in minimal F1 changes (0 to 13 points, sometimes even positive). The calculated causal effect ( $\Delta_{causal}$ ) ranged from -28.2 to -64.1 F1 points, indicating that the presence of the gold answer string is the primary driver of the performance lift.
Insertion Effects: Prepending the gold answer to rewrites that originally lacked it recovered F1 scores by +0.7 to +9.7 points in 10 of 12 combinations. However, inserting the answer at a midpoint sentence boundary yielded mostly non-positive results, suggesting the reader's ability to utilize injected answers is position-sensitive.
Sentinel Fragility: The conventional [MASK] probe proved unreliable. On 2WikiMultihopQA, [MASK] reported a +4.12 F1 "non-leakage residual." However, under four alternative sentinels, this residual flipped to negative values (-3.33 to -7.81 F1). Only 1 out of 4 alternative sentinels passed the equivalence test, indicating the positive residual was largely an artifact of the [MASK] token itself.

Significance and Claims
The paper does not propose a new rewriter architecture or a mitigation strategy. Instead, its contribution is diagnostic:

First Controlled Audit: It presents the first controlled answer-presence intervention audit for "compile-then-read" RAG pipelines, demonstrating that the bulk of the F1 lift on multi-hop benchmarks is causally driven by the rewriter surfacing the gold answer string, rather than by evidence curation alone.
Negative Result on Masking: It provides evidence that single-sentinel masking diagnostics (specifically [MASK]) are sentinel-fragile and cannot reliably separate answer-string surfacing from genuine evidence curation.
Reproducible Standard: The authors release an intervention runner and a sentinel panel (audit kit) to allow future claims of rewriter gains to be tested against this controlled standard.

The authors conclude that while rewriters likely do perform some evidence curation, the magnitude of the performance gain cannot be attributed to "curation quality" without further controlled evidence, as the presence of the answer string accounts for the majority of the lift.