💻 computer science

Foresight Is Not Enough: Sentence-Level Future Signals, Self-Loop Hard Negatives, and the Calibration Gap in Small Language Models

This paper demonstrates that while the ForesightLM-v2 model successfully learns a stable sentence-level future prediction signal, this representation fails to directly control generation behavior, revealing that its apparent benefits stem primarily from semantic reranking rather than the learned future term itself and highlighting a critical gap between learned representations and calibrated behavioral outcomes in small language models.

Original authors: Ahmet Rıfat Öztürk, Yağız Ekrem Dalar, Ömer Faruk Aksoy, Nedim Mutlu Sezer, Feyzi Arda Salihoğlu

Published 2026-07-14

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ahmet Rıfat Öztürk, Yağız Ekrem Dalar, Ömer Faruk Aksoy, Nedim Mutlu Sezer, Feyzi Arda Salihoğlu

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a tiny, super-smart robot writer named Foresight. This robot is trained to write stories, but it has a special trick: it's been taught to peek slightly ahead and guess what the next sentence should be about before it even writes the current one. The big question the researchers asked was: "Does this 'peeking ahead' actually help the robot write better stories, or is it just a cool party trick that doesn't change anything?"

The answer they found is a bit of a plot twist: The robot learned the trick, but it didn't really use it to write better.

The "Crystal Ball" That Didn't Work

Think of Foresight as a student who has a crystal ball. In class (during training), the teacher shows the student a sentence, and the student uses the crystal ball to guess the next one. The student gets really good at this! They can look at a sentence and say, "I know what comes next!" with high confidence. The researchers measured this and found the crystal ball was indeed working: the robot's "future guess" score was consistently positive, rising from a small 0.0262 to a stronger 0.0614 as it practiced more.

But here's the catch: when the robot actually had to write a story, that crystal ball didn't help it pick the best next sentence. It was like having a GPS that knows the destination but the driver just ignores it and takes a random turn.

The "Magic Pool" vs. The "Magic Score"

To test this, the researchers set up a game. Instead of letting the robot write one sentence at a time, they asked it to generate five different possible next sentences (a "pool" of candidates). Then, they had to pick the best one.

They tried two ways to pick:

The "Future" Way: Use the robot's special crystal ball score to pick the winner.
The "Random" Way: Just pick a winner based on random numbers or shuffled scores.

The Shocking Result: It didn't matter which way they picked! The "Future" way and the "Random" way picked almost the exact same sentences. In fact, out of 120 different story prompts, removing the crystal ball entirely only changed the robot's choice in 2 cases.

The researchers realized that the robot wasn't actually using its "future sight" to make decisions. Instead, the improvement came simply from having five options to choose from in the first place. It's like if you were trying to pick the best apple from a basket. If you look at five apples instead of one, you'll almost always pick a better one, even if you don't have a special "apple-sense" to tell you which is best. The "future signal" was just along for the ride.

The "Hard Negatives" and the Loop

The robot was also trained to avoid "loops"—repetitive, boring sentences where it just says the same thing over and over. To teach this, the researchers made the robot generate its own bad, looping sentences and told it, "Don't pick these!" This worked well during training; the robot learned to separate good sentences from bad, looping ones.

However, when it came time to actually write, the robot still needed a little extra push to stop repeating words. The researchers found that adding a simple "punishment" for repeating words (a lexical penalty) worked much better than the fancy future-prediction trick. This punishment cut down on word repetition significantly, proving that sometimes a simple rule is better than a complex prediction.

The Bottom Line

The paper concludes that having foresight isn't enough. Just because a model learns to predict the future doesn't mean it will use that knowledge to write better stories. The improvements the researchers saw were mostly because they gave the robot more choices to pick from and used simple rules to stop it from repeating itself, not because the robot's "future-sight" was the hero.

The researchers are very careful to say this isn't a failure of the robot's brain—it definitely learned the signal! But the "interface" (the way the robot decides what to write) wasn't calibrated to listen to that signal. It's a reminder that in the world of AI, learning a skill and actually using it to do a better job are two very different things. The robot knows the way, but it needs a better map to get there.

Technical Summary: Foresight Is Not Enough

Problem Statement

Autoregressive language models (LMs) optimize for local next-token likelihood, which often leads to generation failures that are not token-local, such as semantic repetition, prompt drift, or a failure to advance discourse. While prior work suggests models can learn representations anticipating future text (e.g., via skip-thought vectors or contrastive predictive coding), a critical gap remains: does a learned future-oriented representation actually drive behavioral improvements in generation?

The paper investigates this "representation-to-behavior gap" using ForesightLM-v2, an approximately 82M-parameter autoregressive model. The central hypothesis is that even if a model successfully learns a signal distinguishing true future sentences from low-quality continuations, this signal may not be the primary driver of generation quality when compared to simpler mechanisms like multi-candidate sampling and generic reranking.

Methodology

Model Architecture and Objective

ForesightLM-v2 is based on a DistilGPT-2-scale causal language model. It augments standard next-token language modeling with two auxiliary components:

Sentence-Level Future Prediction Head: A trainable projection head ( $P_\theta$ ) maps the boundary representation of the context ( $h_c$ ) to a future vector ( $\hat{z}$ ). This vector is aligned with a frozen semantic target ( $z^+$ ) derived from the observed future sentence ( $s^+$ ) using a frozen sentence encoder ( $E$ ).
Self-Loop Hard Negatives: To provide supervision for low-quality outputs, the model mines "loop-like" continuations generated by itself. A margin-based loss ( $L_{hn}$ ) forces the model to separate the representation of the true future ( $s^+$ ) from these self-generated hard negatives ( $s^-$ ).

The total loss function is:
$L = L_{LM} + \lambda_f L_{future} + \lambda_h L_{hn}$
where $L_{LM}$ is the standard language modeling loss, and $\lambda_f, \lambda_h$ are weighting coefficients.

Experimental Design and Controls

The study employs a rigorous fixed-pool evaluation protocol to isolate the effect of the learned future signal from the benefits of sampling more candidates.

Dataset: WikiText-103 (three seeds: 7, 42, 123) and WritingPrompts (seed 42 as a sanity check).
Candidate Pool: For each prompt, a fixed pool of five candidate continuations is generated.
Selection Strategies: The paper compares several selectors on this same fixed pool:
- Learned-Future Reranking: Uses the future similarity score ( $f(y)$ ) in the selection formula.
- Generic Reranking: Uses a score based on prompt similarity, semantic repetition, progression, and length, but without the future term.
- Ablations: Random future scores, shuffled controls, and a "no-future" variant.
- Training-Free Baseline: A base-model reranker without any future-head training.
Metrics: Automatic diagnostics including Prompt Similarity (PSim), Topic Drift, Progression-band score (Prog), Lexical Loop (LexLoop), and Semantic Loop rates.

Key Results

1. Training-Side Signal is Stable

The model successfully learns a measurable future signal. Across three WikiText-103 seeds, the held-out hard-negative margin is consistently positive ( $0.0599 \pm 0.0015$ ). The signal emerges gradually during training, rising from a gap of 0.0262 at 250 steps to 0.0614 at 1000 steps.

2. Generation Gains are Driven by Candidate Selection, Not the Future Signal

When evaluating on the fixed pool of five candidates:

Indistinguishable Performance: The "Learned-Future" reranker performs nearly identically to generic reranking, random future-score controls, and shuffled controls.
Ablation Impact: Removing the learned future term from the selection score changes only 2 out of 120 selected continuations.
Strong Confound: A training-free base-model reranker (using only the base model's logits and generic heuristics) outperforms ForesightLM-v2 on several semantic diagnostics.
Conclusion: The aggregate improvements in generation quality are explained by the diversity of the candidate pool and generic semantic reranking, not the specific learned future signal.

3. The Calibration Gap

While the future score is not random (it correlates with prompt anchoring and progression at the population level), it is weakly calibrated for within-prompt decision-making.

The future-only selector performs poorly on prompt anchoring and progression.
The future score selects the "oracle-best" candidate (the best among the five) as the top-1 choice for only 18.3% of prompts.
The mean rank of the oracle-best candidate within the future-score ranking is between 2.87 and 3.31 out of five.

4. Lexical Repetition is Separable

Adding a lightweight lexical penalty to the selection score significantly reduces surface repetition (LexLoop) and semantic-loop rates without affecting progression scores. This suggests that controlling lexical degeneration is a distinct problem from semantic candidate selection.

Significance and Claims

The paper's primary contribution is methodological rather than a claim of solving long-form coherence. The authors argue that foresight is not enough; learning a future-oriented representation does not guarantee behavioral utility.

Representation vs. Behavior: The study demonstrates that a model can learn a stable, measurable signal that distinguishes future text from self-loop negatives, yet fail to utilize that signal as the primary driver for generation quality.
The Calibration Gap: There is a distinct gap between the existence of a semantic signal and its calibration for decision-making. The current behavioral interface (scoring function) does not make the future signal the decisive factor.
Importance of Controls: The paper emphasizes that without fixed-pool controls and strong negative baselines (like random scores or training-free rerankers), research risks attributing gains to learned signals when they are actually caused by candidate diversity or generic reranking effects.
Scope: ForesightLM-v2 is presented not as a standalone solution, but as a controlled case study highlighting the need for calibrated behavioral interfaces and separate treatment of lexical degeneration in small language models.

The authors conclude that for representation-based generation methods, the behavioral interface must be treated as a "first-class object of study," and that evidence of a signal's existence is weaker than evidence that it changes behavior beyond credible alternatives.