Technical Summary: EditPropBench

Problem Statement

Local factual edits in scientific manuscripts often trigger non-local revision obligations. When a specific value changes (e.g., a dataset size shifting from 215 to 80 documents), qualitative claims dependent on that value (e.g., "medium-scale," "a few hundred items") may become factually stale, even if they do not explicitly repeat the edited number. Current Large Language Model (LLM) editors often succeed at replacing the explicit string but fail to propagate the edit to these dependent qualitative claims, resulting in manuscripts that are locally updated but globally inconsistent. Existing benchmarks for knowledge editing or document revision lack the controlled fact graphs necessary to distinguish between successful local edits and failed cascade propagation of implicit dependencies.

Methodology

Benchmark Design: EditPropBench

The authors introduce EditPropBench, a benchmark designed to measure whether LLM editors propagate factual edits through dependent manuscript claims.

• Structure: Each item consists of a synthetic ML/NLP-style manuscript, a targeted edit instruction (changing a fact from an old value to a new value), and sentence-level annotations.
• Annotations: Sentences are partitioned into three sets:
  1. Direct-target units: Sentences explicitly mentioning the edited fact that must use the new value.
  2. Required-update units: Sentences whose meaning depends on the edited fact and must be revised (even if they do not repeat the number).
  3. Protected units: Unrelated sentences that must remain unchanged.
• Dependency Stratification: Required-update units are categorized by difficulty for deterministic baselines:
  • Number-mention: Direct value statements.
  • Literal-qualifier: Canonical qualitative labels (e.g., "large-scale").
  • Implicit: Paraphrased qualifiers from a closed set.
  • Free-form: Open-ended paraphrases (e.g., "occupying a workable middle ground").
    The hard stratum comprises implicit and free-form units, where simple substitution rules fail.

Corpus Construction

The corpus is generated in three stages to ensure "construct realism" (matching real dependency patterns) while maintaining exhaustive ground truth:

1. A symbolic generator creates templated manuscripts from fact graphs.
2. A canonical paraphraser (gpt-5-mini) rewrites sentences to introduce natural prose and implicit dependencies.
3. Edit instructions are paired with the manuscripts.
   The main corpus contains 122 paraphrased manuscripts (1,907 dependent sentences), with a specific focus on 461 hard-stratum scoring cells. A stress-test corpus variant rewrites implicit sentences with unique free-form phrasing to prevent enumeration by substitution tables.

Scoring Metrics

• Edit-Ripple Adherence (ERA): The headline metric, measuring the fraction of required-update units correctly revised.
• Local Edit Success (LES): The fraction of direct-target units correctly updated.
• Collateral Damage Rate (CDR): The fraction of protected units incorrectly modified.
• Global Consistency Score (GCS) & Coherence-Conditioned Gap (CCG): Metrics to penalize systems that fail to perform the direct edit or produce incoherent revisions.
• Scoring Pipeline: A two-stage process involving a deterministic matcher (Stage A) for explicit values and a three-call LLM-judge ensemble (Stage B, using Claude Sonnet) to evaluate semantic correctness of paraphrases and protected-unit damage.

Validation

The benchmark employs adversarial probes (e.g., Identity, Hedge-Bot, Truncation-Abuse) to ensure metrics cannot be gamed by partial editing or hedging. A human pilot (64 items) and judge-swap experiments validate the reliability of the LLM judge against human raters.

Key Results

Performance on Hard Stratum

On the main corpus hard stratum (implicit and free-form dependencies), five evaluated LLM editing systems span an ERA range of 0.148 to 0.705.

• The strongest system (Claude Opus 4-7, direct edit) achieves an ERA of 0.705, meaning it still misses roughly 30% of required cascade updates.
• Deterministic baselines (Find-Replace, Synonym-Table) score 0% on this stratum by construction.
• Plan-then-edit protocols underperformed direct-edit protocols for top-tier models (e.g., Opus: 0.555 vs. 0.705), primarily due to regressions on free-form and implicit units.

Stress-Test and Mixed Evaluation

In a stress-test aggregate that includes easier "number-mention" and "literal-qualifier" cases:

• Deterministic baselines reach an ERA of ~0.750.
• All evaluated LLM systems exceed this floor, with the best reaching 0.904.
• Crucially, the LLM advantage remains positive even when easy cases are included, confirming that LLMs provide value beyond simple substitution, though they do not solve the full cascade problem.

Real-World Relevance

An audit of 199 recent arXiv cs.CL benchmark and dataset papers found that 37.2% contain fact-dependent qualitative claims (e.g., describing a dataset size as "large-scale"). This confirms that the dependency pattern targeted by EditPropBench is common in the target genre.

Significance and Claims

The paper claims that EditPropBench fills a critical gap in evaluating LLM editors by isolating the capability to propagate factual edits through implicit, semantic dependencies rather than just explicit string replacements.

• Current State: While LLMs can repair many implicit consequences of factual edits, they are not yet reliable enough for fully automatic scientific revision. Even the strongest systems miss a significant portion of required cascade updates.
• Implication for Tools: Reliable scientific revision requires cascade-aware checking (explicit dependency tracking or human review) rather than relying solely on stronger local rewriting capabilities.
• Limitations: The authors acknowledge that the synthetic nature of the corpus prioritizes controlled supervision over surface indistinguishability from real papers. They note that human validation is preliminary and that the benchmark does not currently cover value-laden edits where a model might reasonably refuse to weaken a claim.

The work concludes that while progress is evident, the "cascade gap" remains a significant barrier to trustworthy, fully automated manuscript editing.