💬 NLP

DTO: a Differentiable Training Objective for Effective Counterfactual Story Rewriting

This paper proposes a novel Differentiable Training Objective (DTO) that uses an end-to-end backpropagation loss to jointly optimize for reference fidelity and semantic consistency, effectively addressing the challenges of localized modifications in counterfactual story rewriting and outperforming both traditional baselines and reinforcement learning approaches.

Original authors: Amelia Girard, Massimo Piccardi

Published 2026-05-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Amelia Girard, Massimo Piccardi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a story about a character named Julie who scores the winning goal in a soccer game. Now, imagine someone says, "What if Julie didn't play at all? What if she was just watching from the stands?"

Your job is to rewrite the ending of the story to match this new "what if" scenario. But here's the tricky part: you need to change only the parts that are affected by Julie watching, while keeping everything else (the other players, the score, the excitement) exactly the same. If you change too much, the story feels broken. If you change too little, the story doesn't make sense with the new rule.

This paper is about teaching a computer to do this "story editing" job perfectly.

The Problem: Computers Are Too Clumsy

The authors explain that big computer brains (called Large Language Models) are great at writing stories, but they struggle with this specific "edit" task.

The "Copy-Paste" Problem: If you just tell the computer to copy the original story as closely as possible, it will just copy the original ending where Julie scores the goal. It won't realize it needs to change anything.
The "Over-Edit" Problem: If you try to force the computer to change the story using complex, trial-and-error methods (like Reinforcement Learning), it often gets confused and changes too much, ruining the whole story.

It's like trying to fix a tiny scratch on a painting by either ignoring it completely or repainting the entire canvas.

The Solution: A "Soft" Training Objective (DTO)

The authors propose a new way to teach the computer, which they call DTO (Differentiable Training Objective).

Think of training a computer like teaching a student for a test.

Old Way (Maximum Likelihood): The teacher says, "Your goal is to match the answer key word-for-word." The student tries to memorize the exact words. If the answer key says "The cat sat," and the student writes "The feline sat," they get marked down, even though the meaning is the same.
The New Way (DTO): The teacher says, "Your goal is to make sure your story sounds like the edited version we want, but also sounds very different from the original version we don't want."

To do this, the authors use a special tool called BARTScore. Imagine BARTScore as a very smart, super-fast editor that can read two stories and tell you, "How similar are these?"

The magic trick in this paper is making this "editor" differentiable.

The Analogy: Usually, when a computer writes a story, it picks one specific word at a time (like "cat"). This is like a light switch: it's either ON or OFF. You can't smoothly adjust a light switch to "half-on."
The Innovation: The authors taught the computer to write with "soft" words instead. Instead of picking just "cat," the computer outputs a "cloud" of possibilities (50% "cat," 30% "feline," 20% "kitten"). This "cloud" is smooth and adjustable.
The Result: Because the output is smooth, the computer can look at the "editor's" feedback (BARTScore) and gently nudge its "cloud" to get a better score. It's like a sculptor smoothing clay with their hands, rather than chipping away at a rock with a hammer.

What They Found

The authors tested this new method on a dataset of 28,000 stories (called TIMETRAVEL).

Better than the Basics: Their new method beat the standard "copy-paste" training method. The stories it wrote were closer to the human-edited versions and kept the original story's flow better.
Better than Complex Methods: It also beat other advanced methods that try to learn by guessing and checking (like CPO), but it did so much faster and more stably.
Small Model, Big Results: They used a relatively small computer model (about 0.4 billion parameters). Even though massive models like GPT-4o are 100 to 500 times bigger, this small, smartly trained model performed just as well, if not better, at this specific task.

The Takeaway

The paper proves that you don't always need a giant, expensive computer to do complex tasks. If you give a smaller computer a better "teacher" (a training objective that directly rewards the specific kind of editing you want), it can learn to rewrite stories with the precision of a human editor, keeping the story coherent while making just the right changes.

In short: They found a way to teach computers to be precise editors rather than just generators, using a smooth, math-friendly method that avoids the guesswork of older techniques.

Technical Summary: A Differentiable Training Objective for Effective Counterfactual Story Rewriting

Problem Definition
Counterfactual story rewriting is a natural language processing (NLP) task requiring a model to update an existing narrative to reflect a specific alternative event (the counterfactual condition) while preserving all unaffected storyline elements and maintaining overall coherence. The core challenge lies in the nature of the required modifications: they are typically very small in size and highly localized. Consequently, models trained with conventional maximum-likelihood (negative log-likelihood, NLL) objectives tend to overlook these nuances, often failing to integrate the counterfactual condition or making excessive edits that compromise narrative flow. While reinforcement learning (RL) approaches offer a way to optimize for specific metrics, they are notoriously slow, difficult to set up, and suffer from high variance due to sampling noise.

Methodology
The authors propose a novel Differentiable Training Objective (DTO) that directly optimizes the generative model for the specific requirements of counterfactual rewriting without relying on non-differentiable policy gradients.

Metric Selection: The approach utilizes BARTScore, a text generation evaluation metric based on a pre-trained BART model. Unlike n-gram based metrics (e.g., BLEU, ROUGE), BARTScore estimates the log-likelihood of a hypothesis given a source, capturing semantic alignment and fluency.
Delta Scoring: To address the limitation that global similarity metrics may overvalue unchanged segments, the authors introduce a delta score ( $\Delta M$ ). This score is defined as the difference between the metric score of the generated output against the edited (counterfactual) reference and the score against the original ending. This contrastive measure explicitly rewards the model for moving closer to the counterfactual target rather than the original text. An adjusted score is also proposed to balance absolute accuracy with relative improvement.
Differentiability Mechanism: Since standard metrics and discrete token generation are non-differentiable, the authors employ a "softening" technique. Instead of sampling discrete tokens, the generative model outputs a probability distribution over the vocabulary. This distribution is multiplied by the embedding matrix of the BARTScore model to produce an "expected embedding" (a soft token). This continuous representation allows gradients to be backpropagated through the frozen BARTScore model directly to the generative model's parameters.
Training Process: The model is fine-tuned via end-to-end backpropagation using a loss function derived from the negative of the BARTScore (or the delta-adjusted BARTScore). This aligns the training signal directly with the evaluation criteria, avoiding the sampling-induced noise associated with RL methods.

Key Contributions

Novel Training Approach: The paper introduces DTO, a training strategy that explicitly optimizes a generative model using counterfactual rewriting metrics (specifically BARTScore-based delta scores) rather than standard likelihood.
Reward Function Exploration: The authors evaluate different reward functions based on BARTScore, exploring the trade-offs between adherence to the expected counterfactual ending and differentiation from the original ending.
Performance Validation: The study demonstrates that the DTO approach outperforms both a standard maximum-likelihood baseline and a preference-based approach (Contrastive Preference Optimization, CPO). Furthermore, it performs competitively against two contemporary large language models (LLMs): GPT-4o and Gemini 2.0.

Experimental Results
Experiments were conducted on the TIMETRAVEL dataset (nearly 28k instances) and the ART dataset (adapted for generative tasks).

Comparison with Baselines: The DTO-based models (specifically DTO-Score and DTO-Score+Delta) achieved superior performance in both predictive scores (fidelity to the reference) and counterfactual scores (ability to incorporate the new condition) compared to NLL and CPO baselines across BARTScore, ROUGE-L, and SacreBLEU metrics.
Comparison with LLMs: While the fine-tuned BART model (0.4B parameters) did not surpass the zero-shot performance of GPT-4o in all metrics, it significantly outperformed the LLMs in predictive accuracy. Notably, the small DTO model achieved competitive counterfactual scores to the much larger LLMs (estimated 200B and 70B parameters) while requiring a fraction of the computational footprint.
Qualitative Analysis: Case studies indicate that the DTO model produces minimal, accurate edits that respect the counterfactual premise, whereas LLMs (especially in one-shot configurations) tended to either deviate significantly from the task instructions or "embellish" the story with elements not present in the prompt.
Ablation: The DTO approach maintained strong performance even in an "ablated" setting where the original ending was removed from the input, suggesting robustness in predicting endings based solely on premises and events.

Significance and Claims
The paper claims that the proposed DTO approach substantiates the effectiveness of task-specific differentiable objectives for nuanced, controlled text-generation tasks. By treating an evaluation metric as part of the training signal, the method achieves stable convergence and aligns model gradients with the end goal of accurate yet minimal rewriting. The authors conclude that a small language model, when equipped with an effective differentiable training objective, can perform comparably to commercial-grade models that are 100–500 times larger, offering a computationally efficient alternative for specialized NLP tasks.

The Problem: Computers Are Too Clumsy

The Solution: A "Soft" Training Objective (DTO)

What They Found

The Takeaway

More like this