📊 statistics

Pitwall: Faithful Natural-Language Race-Strategy Briefings from a Calibrated Real-Time Monte Carlo Engine

The paper introduces Pitwall, a production system that generates faithful, multilingual Formula 1 race-strategy briefings by enforcing an architectural verification loop where every factual claim is validated against a calibrated, real-time Monte Carlo engine, ensuring grounded generation while revealing critical trade-offs between vividness and hallucination in fine-tuned models.

Original authors: Juan S. Santillana (Independent Researcher)

Published 2026-07-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Juan S. Santillana (Independent Researcher)

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a high-stakes Formula 1 race. The cars are zooming around, tires are wearing down, and the team strategists on the "pit wall" have to make split-second decisions: Should our driver pit now? Wait two laps? Switch tires? If they guess wrong, they lose the race.

This paper introduces Pitwall, a computer system designed to act like a super-smart, super-accurate assistant for these strategists. It doesn't just guess; it calculates, predicts, and then explains its reasoning in plain English, Spanish, or Portuguese.

Here is how it works, broken down into simple concepts:

1. The Crystal Ball (The Monte Carlo Engine)

Think of the race as a giant game of "Choose Your Own Adventure," but with 2,000 different storylines happening at once.

The Simulation: The system runs 2,000 virtual copies of the rest of the race every single second. In some copies, a Safety Car comes out; in others, a tire blows; in others, a driver gets stuck in traffic.
The Calibration: Just like a weather forecaster checks past storms to predict rain, Pitwall has been "trained" on 126 real races from 2018 to 2024. It learned to adjust its "crystal ball" so that when it says there is a 70% chance of winning, it actually happens 70% of the time.
The Result: It gives the team a probability board: "If you pit now, you have a 65% chance of finishing in the top 3. If you wait, it drops to 40%."

2. The Strict Editor (The Verifier)

This is the most unique part of the system. Usually, AI chatbots are great at writing but sometimes they "hallucinate" (make things up). In a race, making up a fact like "Driver X is 5 seconds ahead" when they are actually behind is dangerous.

Pitwall treats truth as a hard rule, not a suggestion.

The Gatekeeper: Before the system writes a single sentence for the broadcast, it breaks that sentence down into tiny facts (e.g., "Driver is in 4th place," "Tire is Medium," "Gap is 2.1 seconds").
The Check: It checks every single fact against the live data. If the data says the driver is in 5th place, but the AI wants to write "4th place," the system blocks that sentence.
The Safety Net: If the AI tries to be too creative and invents facts, the system forces it to fall back to a boring, 100% accurate template. This ensures that every sentence published is 100% grounded in reality.

3. The "Two Paths" Philosophy

The authors discovered something interesting: Being realistic doesn't always mean being accurate.

Path A (The Oracle): This path is for giving probabilities. It uses the simplest, most stable models to ensure the percentages (like "90% chance to win") are mathematically correct. It ignores some "fancy" details that might mess up the math.
Path B (The Decision Maker): This path is for giving advice. It uses the "fancy" details (like how tires degrade in specific ways) to tell the team what to do.

The system keeps these two paths separate. It doesn't let the "fancy" details ruin the math, and it doesn't let the "simple" math give bad advice.

4. The "Rich vs. Sparse" Trap

The team tried to make the system sound more like a human commentator by training it on rich, exciting race descriptions.

The Problem: When the race data was full (lots of cars, lots of info), the AI sounded great. But when the data was "sparse" (early in the race, or a glitch in the data feed), the AI started hallucinating. It would invent drivers that didn't exist or make up gap times to fill the silence.
The Fix: They realized that the type of AI model matters. They tested four different "base brains." One specific brain (Phi-4-mini) was smart enough to say, "I don't have enough info to be specific," instead of making things up. The others tried to fill the silence with lies. They chose the brain that knew when to stay quiet.

5. Real-World Proof

The paper claims this isn't just a theory. The system was tested live during the 2026 Austrian and British Grands Prix.

It successfully ingested live data, ran the 2,000 simulations, and generated verified commentary in real-time.
At the British Grand Prix, the system tracked the race probabilities lap-by-lap. It correctly identified the eventual winner 10 laps before the race ended, even while the race was still chaotic and the outcome was unknown.

Summary

Pitwall is a race-strategy tool that combines a super-calculator (to predict outcomes) with a strict fact-checker (to prevent lies). It learned that to be truly useful, it must separate its "math brain" from its "advice brain," and it must choose an AI model that knows when not to speak if the facts aren't clear. It proved that you can build a professional-grade sports prediction tool using public data and standard computers, as long as you prioritize truthfulness over flashiness.

Technical Summary: Pitwall – Faithful Natural-Language Race-Strategy Briefings

Problem Statement
Live sports commentary, particularly in Formula 1, represents a demanding instance of grounded generation. The system must generate statements about real, named athletes where the grounding state (positions, gaps, tyre compounds, track status) mutates every few seconds. Unlike standard data-to-text tasks, there is no reference text at generation time, and the cost of hallucination is high: a confidently incorrect statement about a specific driver is more damaging than silence. Furthermore, while teams utilize proprietary simulation tooling for strategy, these systems lack public calibration, and academic literature has not yet addressed the real-time, probability-calibrated, decision-focused setting required for live strategy briefings. The core challenge is to build a system that ingests live timing data, simulates race outcomes with calibrated probabilities, and verbalizes these findings into natural language without fabricating facts, even when the state is sparse.

Methodology
Pitwall is an end-to-end production system comprising four stages: Data Pipeline, Monte Carlo Engine, Decision Layer, and Language Layer.

Data Pipeline & State Reconstruction:
The system ingests the official F1 SignalR live-timing stream at a 1-second cadence. It reconstructs a canonical probabilistic race state incrementally, mirroring the schema used for historical data (157 races from 2018–2026, excluding 2022 due to data reliability issues). This ensures the live engine and backtest engine run identical code paths.
Vectorized Monte Carlo Engine:
The core simulator advances $N=2,000$ race continuations simultaneously using vectorized array operations (NumPy). Key components include:
- Lap Time & Degradation (H5): Models tyre wear as a linear slope with optional circuit-specific "cliff" knots. Crucially, it decomposes fitted slopes to remove fuel-burn and track-evolution confounds, clamping wear to physically plausible bounds.
- Overtaking (H7): A logistic kernel fitted on 131,000 battles predicts pass probability based on pace delta and tyre age. However, experiments showed this improved individual pass prediction but degraded field-level outcome calibration.
- Safety Car/Virtual Safety Car (H13): Simulates stochastic deployment with distinct pace multipliers and pit-loss discounts.
- Rival Policy (H1): Assumes rivals play a pace-optimal, open-loop strategy. A reactive component (H17) allows rivals to pull stops forward during neutralizations.
- Undercut Window (H14): Evaluates counterfactuals (box now, +1 lap, +2 laps, stay out) using Common Random Numbers (CRN) to isolate strategy effects from sampling noise.
- Compound-Base Normalization (H9): Projects fitted tyre paces onto a canonical ordering (soft $\le$ medium $\le$ hard) to prevent spurious "fast tyre" recommendations caused by race-phase confounding.
Dual-Path Principle:
A central methodological contribution is the separation of the Oracle Path (for calibrated outcome probabilities) and the Decision Path (for recommendations). Components are gated independently:
- The Oracle path uses the configuration that minimizes the Brier score on held-out data (e.g., excluding H9 normalization, which improved decision fidelity but degraded calibration).
- The Decision path uses configurations that maximize decision fidelity (e.g., including H9 and feasibility caps), even if they worsen outcome calibration.
- Every component is "parity-gated," shipping with a default setting that reproduces the incumbent engine bit-for-bit until a held-out metric proves improvement.
Verifier-Gated Language Layer:
To ensure faithfulness, the system employs a typed claim schema (10 types: position, gap, tyres, etc.).
- Verification: Every generated sentence is decomposed into claims and verified against the live state snapshot.
- Faithfulness-Gated Fine-Tuning: The fine-tuning corpus (3,045 targets) is filtered by the verifier itself. Only model-generated targets where every claim is supported by the state are admitted (81.9% admission rate); the rest fall back to provably faithful templates. This ensures the generator is never trained on ungrounded data.
- Base Model Selection: Experiments across four base models (1B–7B) revealed that faithfulness under sparse contexts is not a function of scale but of the base model's instruction adherence. The Phi-4-mini model was selected because it abstains from hallucinating on sparse data, whereas larger models (7B) fabricated entities to fill the register.

Key Contributions

Validated, Calibrated Real-Time Engine: A vectorized simulator calibrated on 126 races (2018–2024) and validated on fully held-out 2025–2026 seasons, achieving a Brier score of 0.0745 and 90.3% winner-in-top-3 accuracy.
Learned Overtake Kernel with Honest Ablation: A logistic model that improves pass prediction but is explicitly shown to degrade field-level calibration, demonstrating that task-level accuracy does not guarantee system-level calibration.
Undercut Window as CRN Counterfactual: A formal treatment of pit-stop timing using Common Random Numbers to provide variance-controlled probability deltas for strategic decisions.
Compound-Base Normalization & Calibration/Decision Split: A principled projection method that fixes pathological live calls, coupled with the finding that this correction must be isolated to the decision path to preserve probability calibration.
Verifier-Gated Language Layer: A system where faithfulness is an architectural property enforced at three levels: on every published sentence, on every fine-tuning target, and via the selection of the base model itself.
End-to-End Live Deployment: Successful operation at the 2026 Austrian and British Grands Prix, demonstrating live ingestion, simulation, and verified trilingual commentary on commodity cloud infrastructure.

Results

Calibration: The system achieves a combined Brier score of 0.0745 on the held-out 2025–2026 seasons. The winner is identified as the modal simulated winner in 64.5% of races and placed in the top-3 in 90.3%.
Decision Fidelity: The feasibility-capped decision path reduces infeasible stint recommendations from 90.5% (unconstrained) to 0.0% and improves stop-count agreement with executed modal plans from 59.5% to 69.0%.
Live Performance: At the 2026 British Grand Prix, the system locked onto the eventual winner ten laps before the flag with a win probability of 97.7–99.5%. A timestamped probability trace committed to disk before the outcome was known confirmed the system's ability to track race dynamics in real-time.
Faithfulness: The Phi-4-mini adapter, selected for its instruction adherence, successfully avoided hallucinating drivers or gaps in sparse contexts, whereas richer models failed. The system successfully filtered 18.1% of generated targets to template fallbacks during training to ensure no ungrounded data was learned.

Significance and Claims
The paper claims that Pitwall is the first publicly documented end-to-end system to ingest official live timing, maintain a calibrated probabilistic state, and generate verified natural-language briefings in real-time. Its primary significance lies in the methodological argument that virtues trade off and must be gated separately. The authors demonstrate that:

Calibration-optimal configurations are not necessarily decision-optimal.
Richness in generation targets buys vividness but can collapse into hallucination when the grounding state is sparse.
Base model selection (instruction adherence) is a critical gate for faithfulness, independent of parameter scale.

The system treats faithfulness not as an aspiration but as a structural constraint, routing components to the specific path (Oracle vs. Decision) they improve and using verifiers to gate both training data and inference outputs. The paper positions these findings as a template for decision-support systems in other sports, emphasizing the necessity of reporting negative results (e.g., components that improve realism but degrade calibration) to maintain credibility. The deployment confirms that team-grade decision support can be built from public data on commodity infrastructure, provided that calibration, verification, and dual-path routing are rigorously enforced.