🤖 machine learning

Likelihood scoring for continuations of mathematical text: a self-supervised benchmark with tests for shortcut vulnerabilities

This paper introduces a self-supervised benchmark that evaluates whether model-generated forecasts of mathematical text continuations improve likelihood scores compared to context-only controls, effectively distinguishing model capabilities and reasoning efforts while identifying potential shortcut vulnerabilities in scoring mechanisms.

Original authors: Daniel Ranard

Published 2026-05-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Daniel Ranard

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to guess the ending of a very complex, technical story written by a brilliant mathematician. The story is full of strange symbols and equations. You can see the beginning of the story, but the last few sentences are hidden behind a curtain.

This paper introduces a new way to test if a computer is actually "thinking" about the story or just guessing randomly. It does this without needing a human teacher to grade the answers.

Here is how the test works, broken down into simple parts:

1. The Setup: The "Curtain" Game

The researchers take a real scientific paper and cut off the end of a math equation.

The Context (X): The computer sees everything leading up to the cut.
The Hidden Ending (Y): This is the part we want to predict. It's the "answer key."
The Forecast (Z): The computer under test is asked to write a "hint" or a "forecast" about what comes next. It's like the computer whispering, "I think the next part is going to look like this..."

2. The Judge: The "Probability Meter"

Instead of a human reading the forecast and saying "Good job" or "Bad job," the researchers use a different computer program as a Judge.

The Judge looks at the hidden ending (Y) and asks: "How likely is this to be the real ending?"
The Judge does this twice:
1. Without the hint: Just looking at the story so far.
2. With the hint: Looking at the story plus the computer's forecast (Z).

If the computer's forecast is smart and actually helps the Judge understand the hidden ending, the Judge will say, "Ah, this ending makes much more sense now!" The score goes up. This increase in score is called "Likelihood Lift."

3. The Trap: The "Context Stuffer"

The researchers were worried about a trick. What if the computer doesn't actually predict the future, but just copies and pastes the last few sentences of the story into the "hint" box?

Imagine a student taking a test. Instead of solving the problem, they just copy the question back onto the answer sheet.
To catch this, the researchers created a Control Group. They forced the computer to use its "hint" space to paste the recent history of the story instead of making a prediction.
The Test: If the computer's real prediction scores higher than the pasted history, it means the computer is actually doing some reasoning, not just copying.

4. The Results: Who Passed?

The researchers tested several different AI models (like GPT-5.5, Opus 4.7, and a smaller "nano" version).

The Big Winners: The smarter, more powerful models (like GPT-5.5) were able to write forecasts that helped the Judge significantly more than just pasting the history. Even when the researchers made the Judge very strict (by training it to love pasted history), the smart models still won.
The Losers: The smaller, weaker models (like GPT-5.4 nano) couldn't beat the "pasted history" trick. Their forecasts didn't help the Judge understand the hidden ending any better than just reading the recent text again.
The "Reasoning" Factor: They also tested if telling the AI to "think harder" (using more computing power) helped. They found that when the AI was told to use more reasoning, it got better at predicting the math, just like a human student who takes more time to solve a problem gets a better grade.

5. Why This Matters (According to the Paper)

The paper claims this is a new, automatic way to test AI reasoning in math and science.

No Human Teachers Needed: You don't need a professor to grade every single math problem. The "Probability Meter" does the grading automatically.
Catching Cheaters: It helps researchers see if an AI is actually solving the problem or just finding a "shortcut" (like copying the question) to get a high score.
A New Benchmark: It creates a standard test using real, recent scientific papers to see which AI models are truly getting smarter at technical tasks.

In short: The paper built a game where AI models try to guess the next part of a math equation. If their guess helps a computer judge understand the answer better than just reading the recent text, the AI passes. The smartest models passed; the weaker ones failed, proving this test can tell the difference between a smart thinker and a copycat.

Technical Summary: Likelihood Scoring for Continuations of Mathematical Text

Problem Statement
Evaluating language models on technical reasoning is challenging because technical continuations (e.g., derivations, proofs, equations) often have multiple roughly equivalent valid forms, making item-by-item judgment noisy. Conversely, standard next-token prediction on raw text rewards surface-level modeling (syntax, notation) rather than deep reasoning. There is a need for an automatically generated benchmark that sits between these poles: abundant enough to be renewable, yet sensitive enough to detect reasoning capabilities without relying on human labels or brittle verifiable answers (like competition math). Furthermore, there is a concern that reward signals used for Reinforcement Learning with Verifiable Rewards (RLVR) can be "hacked" by models exploiting shortcuts (e.g., context stuffing) rather than generating genuine forecasts.

Methodology
The paper introduces a static benchmark based on equation-suffix prediction.

Task Setup: A predictor model is given visible context $X$ (preceding paper text) and a prefix of a displayed equation. It must generate an auxiliary string $Z$ (a forecast) to complete the equation.
Scoring Mechanism: A separate, fixed autoregressive language model (the "scorer," $J$ $J$ ) assigns a next-token likelihood to the true hidden continuation $Y$ $Y$ . The core metric is the likelihood lift ( $\Delta_J$ $Δ_{J}$ ), defined as the increase in the average log-likelihood of $Y$ $Y$ when the scorer conditions on $Z$ $Z$ versus when it conditions only on $X$ $X$ (or a control string).
- Formula: $\Delta_J(X, Z, Y) = \frac{1}{T} \log p_J(Y | X, Z) - \frac{1}{T} \log p_J(Y | X)$ .
Softened Metric: To prevent rare, catastrophic token mismatches (common in LaTeX/TeX) from dominating the score, the authors use clipLL2. This clips per-token log-likelihoods from below at $-2$ before averaging, ensuring that a forecast remains useful even if it makes a few local errors, provided it helps the scorer recover later.
Controls and Stress Tests:
- Same-Budget Context Control ( $C_B$ ): Replaces the forecast $Z$ with raw source text from immediately preceding the equation, using the same character budget. This tests if the signal comes from prediction or merely from providing more context.
- Context-Only SFT Control: The scorer is fine-tuned (via LoRA) on a "context-only" route (no forecasts) and then evaluated on held-out papers. This tests if a predictor can exploit the scorer's weights via scaffold priming rather than transmitting information about $Y$ .
- True-Suffix Upper Bound: A theoretical ceiling where $Z$ literally contains the hidden suffix $Y$ .

Key Contributions

Automated Benchmark: A dataset of 1,363 equation-suffix cuts from 138 recent arXiv papers (quant-ph, hep-th, math-ph), generated automatically without human selection of "favorable" equations.
Cross-Model Evaluation: Demonstrates that the benchmark can distinguish between model families (GPT-5.5, Opus 4.7, GPT-5.4 nano) and reasoning-effort settings (high, medium, low, none) using fixed scorers (Qwen3-8B and Kimi K2.6) without human labels.
Shortcut Vulnerability Audit: Provides a rigorous framework for testing whether likelihood-based rewards are susceptible to non-forecast strategies. The paper shows that while strong models (GPT-5.5) survive a fine-tuned context-only control, weaker models (GPT-5.4 nano) do not, suggesting the signal is genuine for the former but potentially exploitable or non-existent for the latter.
Metric Analysis: Validates the use of softened scoring rules (clipLL2) over raw log-likelihood, showing that raw metrics are dominated by catastrophic token errors that obscure the utility of otherwise correct reasoning.

Results

Model Differentiation: On the equation-suffix benchmark, forecasts from GPT-5.5 (high reasoning) consistently achieve the highest likelihood lift, followed by Opus 4.7, and then GPT-5.4 nano. This ordering holds across both Qwen and Kimi scorers.
Reasoning Effort: Within model families, higher reasoning-effort settings (e.g., GPT-5.5 high > medium > low > none) generally yield higher lift, demonstrating a monotone gradient. However, adjacent contrasts (e.g., high vs. medium) are sometimes noisy and not always statistically significant on a per-item basis, reinforcing the need for statistical aggregation over many examples.
Control Performance:
- All predictor settings beat the same-budget context control.
- GPT-5.5 forecasts maintain a positive lift over the context-only SFT control (the strongest static control), indicating they transmit genuine information about the hidden suffix.
- GPT-5.4 nano forecasts fail to beat the context-only SFT control, suggesting their signal may be indistinguishable from context-stuffing shortcuts under this stress test.
Longer Continuations: In a harder regime involving longer mixed prose/TeX continuations, the lift remains positive but is noisier and more sensitive to the scorer's design and the length of the target. The reasoning-effort gradient is less robust here than in equation suffixes.

Significance and Claims
The paper positions this work as a static benchmark and a diagnostic tool, not a demonstration of successful RL training. Its primary significance lies in:

Feasibility of Likelihood Rewards: It supports the hypothesis that likelihood lift on technical continuations can serve as a dense, reference-grounded reward signal that correlates with model capability and reasoning effort.
Safety and Robustness: It highlights the necessity of auditing reward signals against strong static controls (like SFT-based context shortcuts) before deploying them in RLVR loops. The fact that GPT-5.5 survives these controls while GPT-5.4 nano does not suggests that not all "likelihood lift" is equal; some may be artifacts of model-specific quirks or shortcuts.
Methodological Middle Ground: It offers a middle ground between brittle "LLM-as-a-judge" rubrics (prone to style bias) and simple string similarity (too rigid for technical text), leveraging the probabilistic nature of autoregressive models to evaluate the usefulness of a forecast rather than its literal correctness.

The authors explicitly state that they do not perform reinforcement learning here; the goal is to validate the signal's properties and identify potential vulnerabilities (shortcut exploitation) that would need to be addressed before such a signal could be used for post-training optimization.