Forecasting Future Language: Context Design for Mention Markets

This paper investigates how to optimize input context for large language models in mention markets by introducing Market-Conditioned Prompting (MCP) and its mixture variant (MixMCP), demonstrating that richer contextual information and treating market probabilities as priors significantly improve the accuracy and calibration of keyword-mention forecasts.

The Gist

Imagine you are trying to predict whether a specific word (like "inflation" or "AI") will be spoken by a CEO during their next quarterly earnings call. This is a game of "Future Language," and there are two main groups trying to guess the answer: The Market and The AI.

Here is a simple breakdown of the paper "Forecasting Future Language: Context Design for Mention Markets" using everyday analogies.

1. The Setup: The Prediction Market

Think of a Prediction Market like a giant, high-tech betting pool.

• The Scenario: A company is about to hold a conference call.
• The Bet: People buy "Yes" or "No" tickets on whether the CEO will say a specific word (e.g., "Quantum").
• The Price: If the "Yes" ticket costs $0.55, the market is saying, "We think there is a 55% chance the CEO will say this word."

Usually, this market price is a very smart guess because it aggregates the opinions of thousands of traders. But the researchers asked: Can a super-smart AI (Large Language Model) look at the news and the company's history to make this guess even better?

2. The Problem: How to Talk to the AI

The researchers tried to get the AI to predict the outcome, but they realized how you ask the question matters just as much as the question itself.

They tested three different ways to "feed" the AI information:

❌ Method A: The "Naive Reader" (Just giving the AI the numbers)

Imagine you hand the AI a stack of news articles and the company's past transcripts, and then you just whisper, "Oh, by the way, the betting market thinks it's 55%."

• The Result: The AI ignores the 55% or gets confused. It tries to guess from scratch, often doing worse than the market because it doesn't know how to use that 55% number. It's like giving a chef a recipe and a thermometer, but not telling them the thermometer is for the oven temperature.

✅ Method B: "Market-Conditioned Prompting" (MCP) – The "Editor"

This is the paper's first big breakthrough. Instead of just whispering the number, they tell the AI:

"The market thinks there is a 55% chance. This is your starting point (your 'prior'). Now, look at the news and the past transcripts. Do you see evidence that proves the market is wrong? If so, update your guess."

• The Analogy: Think of the Market Price as a Draft written by a committee. The AI is the Senior Editor. The Editor doesn't rewrite the whole article from scratch; they read the draft, check the facts (the news/transcripts), and say, "The draft says 55%, but I see a new press release that makes it 80%."
• The Result: This works much better. The AI respects the market's wisdom but uses its reading skills to correct it when the market is unsure.

🏆 Method C: "MixMCP" – The "Smart Mixture"

The researchers found that sometimes the AI gets too excited. If the news is a little vague, the AI might swing the prediction from 55% to 90% when it should have stayed closer to 55%. It "overreacts."

So, they created a MixMCP strategy.

• The Analogy: Imagine a Tug-of-War.
  • One team is the Market (very stable, hard to move).
  • The other team is the AI (very strong, but sometimes pulls too hard).
  • MixMCP is the referee who says, "Let's pull the rope 70% toward the Market's side and 30% toward the AI's side."
• The Result: This creates the most accurate prediction. It keeps the stability of the market but adds the AI's "aha!" moments when the market is confused.

3. Key Findings (The "Aha!" Moments)

1. More Context is Better: Just like a detective needs more clues, the AI performs better when it has both news articles AND past earnings transcripts. The past transcripts are especially helpful because they teach the AI how that specific CEO usually talks.
2. The "Mid-Confidence" Sweet Spot: The AI is most useful when the market is unsure (e.g., when the market says 50–60%).
   • If the market is 95% sure, the AI usually agrees (no need to change).
   • If the market is 50/50, the AI can look at the text and say, "Actually, looking at the CEO's recent tone, I think it's 75%." This is where the AI adds the most value.
3. Don't Replace the Market, Refine It: The best results didn't come from the AI trying to beat the market alone. It came from the AI acting as a refinement tool for the market's existing guess.

The Bottom Line

This paper teaches us that to get the best prediction from an AI, you shouldn't just ask it to guess. You should:

1. Give it the Market's current guess as a starting point.
2. Give it all the relevant text (news, history).
3. Tell it to update that starting point based on the text.
4. Finally, mix the AI's new guess with the original market guess to smooth out any wild swings.

It's like having a wise old investor (the Market) and a brilliant young analyst (the AI) working together. The investor provides the steady baseline, and the analyst spots the new trends, resulting in a smarter decision than either could make alone.

Technical Summary

1. Problem Definition

The paper addresses the challenge of improving prediction markets for specific events known as mention markets. In these markets, contracts resolve based on whether a specific keyword is mentioned verbatim in a future public event (specifically, corporate earnings call transcripts).

• The Core Question: While Large Language Models (LLMs) have shown promise in forecasting open-domain events, it is unclear how to design input contexts to maximize their accuracy when combined with existing market signals.
• The Gap: Current approaches often treat LLMs as standalone predictors competing with market prices. The authors investigate whether LLMs can instead act as complementary refinements that update market-implied probabilities using textual evidence (news and prior transcripts), rather than replacing the market forecast entirely.
• The Task: Given a market price ($p_{MKT}$) and pre-call textual evidence (news $N$, prior transcripts $T$), the goal is to output a calibrated probability ($p$) for a binary outcome ($y \in \{0, 1\}$).

2. Methodology

The authors propose a framework where the LLM updates a market prior based on textual evidence, rather than generating a prediction from scratch.

A. Data and Setup

• Dataset: 856 Kalshi earnings-call mention markets (50 companies, 70 events, April–Dec 2025).
• Inputs:
  • Market Signal: The YES-contract price ($p_{MKT}$) 7 days before the call.
  • Textual Context: Prior quarter earnings transcripts ($T$) and retrieved company news ($N$).
• Model: GPT-5.1 (fixed parameters, no fine-tuning) with structured output.

B. Proposed Approaches

The paper compares several prompting strategies:

1. Baseline (Market Signal as Plain Text): The market price is inserted into the prompt as a simple text string alongside news and transcripts, without specific instructions on how to use it.
2. Market-Conditioned Prompting (MCP):
   • Concept: Explicitly treats the market probability as a prior.
   • Mechanism: The prompt instructs the LLM to evaluate the market prior and update it using the provided textual evidence. The LLM is asked to output a revised probability based on this reasoning.
   • Formalism: $p_{MCP} = \text{LLM}(T, N \mid p_{MKT})$.
3. Mixture Forecast (MixMCP):
   • Concept: A convex combination of the market prior and the LLM's updated posterior to prevent overreaction to noisy cues.
   • Formula: $p_{MixMCP} = \alpha p_{MKT} + (1 - \alpha) p_{MCP}$.
   • Optimization: $\alpha$ was tuned to 0.7, meaning the final forecast relies 70% on the market and 30% on the LLM's evidence-based update.

3. Key Contributions

1. Framework Formalization: The paper formalizes text-grounded forecasting as a market-conditioned updating problem. It shifts the paradigm from "LLM vs. Market" to "LLM updating Market."
2. Methodology (MCP): Introduces a prompting protocol that explicitly conditions the LLM on market prices as priors, leading to significantly better calibration than standard prompting.
3. Performance Validation: Demonstrates that a conservative mixture (MixMCP) consistently outperforms the market baseline, proving that LLMs add value even in efficient markets by resolving ambiguity.

4. Experimental Results

The study evaluates performance using the Brier Score (lower is better), Expected Calibration Error (ECE) (lower is better), Accuracy, and F1 Score.

• Impact of Context: Richer context consistently improves performance. Combining prior transcripts ($T$) and news ($N$) yields the best results among text-only baselines (Brier: 0.1991 vs. 0.2635 for no context).
• MCP vs. Naive Inclusion:
  • Simply inserting the market price as text ("W/O Prompting") resulted in poorer calibration (ECE: 0.0705) and lower accuracy (74.4%) than the market baseline alone.
  • MCP significantly improved calibration (ECE: 0.0514) and accuracy (78.2%), proving that the instruction to reason about the prior is crucial, not just the presence of the number.
• Mid-Confidence Regimes: MCP provides the most value when the market is uncertain (probabilities between 50–70%). In these "mid-confidence" bins, MCP outperformed the market in 56.7% to 62.5% of disagreement cases, effectively resolving ambiguity.
• MixMCP Superiority:
  • MixMCP achieved the best overall performance:
    • Brier Score: 0.1392 (vs. 0.1402 for Market alone).
    • Accuracy: 80.3% (vs. 79.8% for Market alone).
    • F1 Score: 0.842.
  • This confirms that dampening the LLM's update with the market prior (via $\alpha=0.7$) creates a more robust predictor than either component alone.

5. Significance and Conclusion

• Complementarity: The paper demonstrates that LLMs do not need to replace prediction markets to be useful. Instead, they serve as effective refinement tools that incorporate semantic information (news, transcripts) to adjust market priors.
• Context Design Matters: The way information is presented to an LLM is critical. Treating market prices as a "prior to be updated" yields superior results compared to treating them as mere data points.
• Robustness: The MixMCP approach suggests that in efficient markets, the safest and most accurate strategy is to trust the market's aggregate wisdom as a baseline while allowing LLMs to make targeted corrections based on specific textual evidence, particularly when market signals are ambiguous.

Limitations: The study is specific to earnings-call keyword mentions and offline historical data. Future work is needed to test generalizability to other contract types and real-time deployment dynamics.