💬 NLP

Back to the Future: The Role of Past and Future Context Predictability in Incremental Language Production

Through two studies of naturalistic speech, this paper demonstrates that an information-theoretic measure quantifying the information shared between a word and its future context (constrained by the past) not only explains unique variance in phonetic reduction but also serves as the strongest predictor of substitution errors, thereby illuminating how speakers dynamically integrate past and future context during incremental lexical planning.

Original authors: Shiva Upadhye, Richard Futrell

Published 2026-04-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shiva Upadhye, Richard Futrell

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are driving a car down a winding road. You have two ways of looking at the road ahead:

Looking in the Rearview Mirror (Past Context): You see where you've already been. This helps you know how you got here.
Looking Through the Windshield (Future Context): You see the road stretching out in front of you. This helps you know where you are going.

For a long time, scientists studying how we speak thought that only the Rearview Mirror mattered. They believed that when we choose a word, we only look at what we just said to decide what comes next.

But this new paper argues that we are actually doing something much more impressive: We are looking through the windshield while we are still talking.

Here is a simple breakdown of what the researchers found, using some everyday analogies.

1. The "Backward Predictability" Mystery

Imagine you are telling a joke. You say, "It's not against the law to..."

The Rearview Mirror (Past): So far, you've said "It's not against the law to." This could be followed by anything: "eat a sandwich," "dance," or "sleep." The past doesn't narrow it down much.
The Windshield (Future): But let's say the next words you plan to say are "...send alligators through the mail."

Even though you haven't said "send" yet, your brain already knows that "send" is the only word that fits with "alligators through the mail." The researchers call this Backward Predictability. It turns out that our brains are so good at planning ahead that the future words actually change how we say the current word.

2. The "Speed vs. Clarity" Trade-off (Study 1)

The first study looked at how fast or slow we say words. Think of your speech like a stream of water.

High Predictability = A Smooth Stream: If the listener can easily guess what you're going to say (because the future context makes it obvious), you can "speed through" the word. You might mumble it or say it quickly because you know the listener will fill in the blanks.
Low Predictability = A Stagnant Pool: If the future is confusing or hard to predict, you have to "slow down" and speak clearly to make sure the listener understands.

The Big Discovery: The researchers found that looking at the future (the windshield) had a stronger effect on how fast we speak than looking at the past (the mirror). If the future is clear, we rush. If the future is foggy, we slow down to buy ourselves time to think.

3. The "Wrong Turn" Analysis (Study 2)

The second study looked at slips of the tongue (when you say the wrong word and correct yourself). Imagine you are trying to pack a suitcase (your message) but you keep grabbing the wrong items.

They analyzed two types of "wrong turns":

The "Lazy" Mistake (Goal-Invariant): Sometimes you grab a word because it's the most common one or because it rhymes with the previous word. This is like grabbing a sock because it's right on top of the pile, even if you needed a shirt. This happens when you rely too much on the Rearview Mirror (what you just said).
The "Smart" Mistake (Goal-Directed): Sometimes you grab a word that almost fits the future plan but isn't quite right. This is like grabbing a winter coat when you're actually going to the beach.

The Big Discovery: The researchers found that when we look at the future (the windshield), our brains actually prevent these mistakes. If the future context is very clear, it acts like a guardrail, stopping us from picking the wrong word.

Past Context tends to push us toward errors (by making common words too easy to grab).
Future Context tends to pull us away from errors (by reminding us of the specific goal).

4. The New "Super-Measure"

The authors realized that previous ways of measuring this "future looking" were flawed. They were like trying to measure the wind by only looking at the trees, ignoring the direction of the car.

They invented a new mathematical tool called Conditional PMI.

Old Way: "How likely is this word to appear next?" (Ignoring what came before).
New Way: "Given what I just said, how much does the future help me choose this specific word?"

Think of it like a GPS.

The old way just said, "You are near a gas station."
The new way says, "You are near a gas station, and since you just said you are out of fuel, the gas station is the most important thing for you right now."

The Takeaway

This paper tells us that human speech isn't just a chain reaction of "A leads to B." It's a dynamic dance between what we've already said and what we are planning to say next.

We are constantly planning ahead. Even when we are stumbling over a word, our brain is already looking at the next few words to see if they fit.
The future controls the present. The clarity of what we want to say next dictates how clearly we say what we are saying now.
We are resource managers. Our brains are constantly balancing "effort" (saying things quickly) with "accuracy" (making sure the listener understands). If the future is clear, we save energy. If the future is foggy, we work harder.

In short: We don't just speak to the past; we speak to the future.

1. Problem Statement

The paper addresses a gap in understanding how contextual predictability influences language production. While the effect of past context (forward predictability) on word choice and articulation is well-established, the role of future context (backward predictability) remains poorly understood and methodologically limited.

Key Issues Identified:

Methodological Flaw in Backward Predictability: Traditional measures of backward predictability ( $P(w_t | C_{>t})$ ) assume that the future context is generated independently of the past context. This ignores the fact that both past and future are derived from a single speaker's intended message ( $M$ ).
Collinearity: Backward predictability is often highly correlated with forward predictability, making it difficult to disentangle their unique contributions in regression models.
Theoretical Ambiguity: It is unclear whether future context influences production through anticipatory motor planning (prospective) or through retrospective facilitation of retrieval, and how these mechanisms interact with past context constraints.

2. Methodology

The authors propose a new theoretical framework and conduct two empirical studies using naturalistic speech data (Switchboard corpus).

A. Theoretical Framework & New Measures

The authors introduce two improved operationalizations of future context predictability to replace or refine standard backward predictability:

Conditional Pointwise Mutual Information (Conditional PMI):
- Formula: $\text{PMI}(w_t; C_{>t} | C_{<t}) = \log \frac{p(w_t | C_{>t}, C_{<t})}{p(w_t | C_{<t})}$
- Significance: This measure quantifies the information shared between the current word and the future context given the past context. It removes the independence assumption, capturing how the future constrains the present after accounting for what has already been said.
Relative Backward Predictability:
- Formula: $\log p(w_t | C_{>t}) - \log p(w_t | C_{<t})$
- Significance: A decorrelated measure that subtracts forward predictability from backward predictability to isolate the unique contribution of the future.

B. Language Modeling Approach

To estimate these probabilities accurately, the authors trained a single GPT-2 language model on the CANDOR corpus of spontaneous speech.

Data Augmentation: They used a "masking" or "infill" strategy where words were moved to the end of utterances with <PRE> (past) and <SUF> (future) tokens. This allowed the autoregressive model to learn bidirectional probabilities ( $P(w_t | C_{<t}, C_{>t})$ ) without needing separate forward and backward models, ensuring a consistent probability distribution.

C. Study 1: Probabilistic Reduction (Word Duration)

Goal: To test how past and future predictability affect articulatory duration (phonetic reduction).
Data: 435,795 words from Switchboard, excluding disfluencies.
Analysis: Linear mixed-effects regression modeling word duration (ms) as a function of unigram frequency, forward predictability, and the two new future-context measures.
Variables: Tested interactions with lexical class (Function vs. Content words).

D. Study 2: Substitution Errors (Word Choice)

Goal: To model the identity of lexical substitution errors (e.g., saying "becoming" instead of "having") to understand the trade-offs in lexical planning.
Data: 773 substitution utterances with self-repairs.
Analysis: Logistic regression (binary classification) predicting whether a candidate word is the observed error.
Predictors:
- Lexical: Unigram frequency.
- Contextual: Forward, Relative Backward, and Conditional PMI.
- Communicative Reward: "Noisy" semantic and phonetic distance to the intended target (self-repair), simulating uncertainty in the speaker's plan.
Error Categorization: Errors were manually coded into Semantic, Phonological, Morphosyntactic, and Mixed types.

3. Key Results

Study 1: Word Duration

Future Context Dominance: Predictability from the future (both Relative Backward and Conditional PMI) was the strongest predictor of word reduction, followed by unigram frequency, then past context.
Lexical Class Asymmetry: Contrary to previous findings (Bell et al., 2009) that suggested future context only affects content words, this study found that both function and content words are sensitive to future predictability. However, content words showed stronger sensitivity overall.
Measure Comparison:
- Relative Backward Predictability was the strongest single predictor of duration.
- Conditional PMI provided unique, non-redundant variance. The model including both measures fit the data best, suggesting that duration is influenced by both a de-contextualized aggregate of future plans (Relative Backward) and a specific, context-bound interaction (Conditional PMI).

Study 2: Substitution Errors

Opposing Effects:
- Past Context (Forward Predictability): Facilitates errors (positive coefficient). High predictability from the past makes a word more likely to be selected, even if it is the wrong word (goal-invariant processing).
- Future Context (Conditional PMI): Inhibits errors (negative coefficient). High predictability from the future makes a word less likely to be selected as an error. This suggests future context acts as a constraint or "conflict signal" that prevents incompatible words from surfacing.
Measure Superiority: Conditional PMI outperformed Relative Backward Predictability in predicting substitution errors. Once Conditional PMI was included, Relative Backward Predictability became redundant. This implies that lexical choice relies heavily on the specific interaction between past and future constraints.
Error Type Nuances:
- Semantic Errors: Strongly inhibited by future context predictability.
- Phonological Errors: Driven almost entirely by frequency and semantic proximity; context had little effect.
- Morphosyntactic Errors: Driven by frequency and past context; future context had no significant effect.

4. Key Contributions

Theoretical Refinement: The paper challenges the assumption that future context is independent of past context. It proposes Conditional PMI as a more cognitively plausible measure that captures the joint influence of past and future plans on the current word.
Methodological Innovation: Demonstrates a robust pipeline for estimating bidirectional and infill probabilities using a single autoregressive model (GPT-2) via data augmentation, avoiding the confounds of training separate forward/backward models.
Dissociation of Mechanisms:
- Word Duration: Influenced by both "global" future plans (Relative Backward) and "local" context-specific plans (Conditional PMI).
- Word Choice: Influenced primarily by "local" context-specific plans (Conditional PMI), where future context acts as a strong inhibitor of errors, while past context acts as a facilitator of errors.
Re-evaluation of Lexical Class Effects: Refutes the claim that future context only affects content words, showing that function words are also sensitive to future predictability, though with different magnitudes.

5. Significance and Implications

Incremental Planning: The findings support a view of sentence production that is not strictly linear. Speakers appear to maintain and integrate future representations into current planning, but the nature of this integration differs between articulatory encoding (duration) and lexical selection (word choice).
Goal-Invariant vs. Goal-Directed Processing:
- Past Context drives goal-invariant processing (automatic, accessibility-based), leading to errors if the accessible word conflicts with the message.
- Future Context drives goal-directed processing (controlled, utility-based), acting as a constraint that prevents errors and ensures communicative alignment.
Resource-Rationality: The results align with resource-rational theories of cognition. Speakers trade off processing effort (accessibility) against communicative utility (future constraints). When future constraints are strong, they inhibit errors; when they are weak or ignored, accessibility drives selection.
Future Directions: The paper suggests that uncertainty in future planning (rather than just the presence of a plan) may be a critical factor in production difficulty, opening avenues for research into how speakers manage working memory loads during non-linear planning.