💬 NLP

Judging What We Cannot Solve: A Consequence-Based Approach for Oracle-Free Evaluation of Research-Level Math

This paper proposes Consequence-Based Utility, an oracle-free evaluation method that assesses research-level math solutions by measuring their effectiveness as in-context exemplars for solving related verifiable problems, demonstrating superior ranking performance compared to existing reward models and LLM judges.

Original authors: Guijin Son, Donghun Yang, Hitesh Laxmichand Patel, Hyunwoo Ko, Amit Agarwal, Sunghee Ahn, Kyong-Ha Lee, Youngjae Yu

Published 2026-02-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Guijin Son, Donghun Yang, Hitesh Laxmichand Patel, Hyunwoo Ko, Amit Agarwal, Sunghee Ahn, Kyong-Ha Lee, Youngjae Yu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Expert Bottleneck"

Imagine a team of brilliant AI students (Large Language Models) trying to solve the world's hardest math problems—problems that even real human professors struggle with. The AI can generate many different attempts at solving these problems.

However, there is a massive bottleneck: Who checks the work?
To know if an AI's math proof is correct, you usually need a human expert (a professor) to read it. But experts are expensive, rare, and slow. If you ask an AI to check another AI's work, it often fails because the AI gets tricked by fancy-sounding but wrong arguments, or it gets confused by the complexity.

The paper asks: How can we tell if a math solution is good without needing a human expert to read every single line?

The New Idea: "Judge by the Aftermath"

The authors propose a method called Consequence-Based Utility (CBU).

Instead of asking, "Does this solution look right?" (which is what current AI judges do), they ask, "Does this solution help solve other related problems?"

The Analogy: The Master Chef and the Recipe

Imagine you have a mysterious, untested recipe for a complex dish (the "Research-Level Math Problem"). You have three different versions of this recipe written by different chefs (the AI candidates). You don't know if any of them are actually correct because you haven't tasted the final dish yet.

The Old Way (LLM Judges): You ask another AI to read the recipes and guess which one looks the most professional. It might get fooled by a recipe that uses fancy words but has a missing ingredient.
The New Way (Consequence-Based Utility): You take each recipe and use it to cook a simpler, related dish (a "neighborhood question") that you can verify easily.
- If Recipe A helps you cook the simple dish perfectly, it likely contains the right "flavor logic" and is probably a good recipe.
- If Recipe B makes the simple dish taste terrible, it likely has a fundamental flaw, even if it sounded fancy.

The paper argues that a correct solution contains the right "logic" or "method." If you use that method as a guide (an "in-context exemplar") to help solve easier, related problems, the AI will do much better. A wrong solution might look good on paper, but it will confuse the AI when it tries to use that logic for other tasks.

How They Tested It

The researchers created a special dataset called EXPERTMATH.

They gathered 192 extremely hard math problems written by real university professors.
They generated 9 different AI attempts for each problem (some correct, some wrong).
They created "neighborhood questions" for each problem—slightly easier versions that rely on the same core logic.

They then compared their new method (CBU) against the standard methods:

Reward Models: AI trained to give a score based on "quality."
LLM Judges: AI asked to read the solution and give a grade (1–10).

The Results

The new method (CBU) was consistently better at picking the right answer.

Better at spotting fakes: It was much better at realizing that a solution was wrong, even if the solution looked very convincing.
The "Solver-Evaluator" Gap: Usually, if an AI can't solve a problem, it can't judge it either. But CBU broke this rule. Even when the AI couldn't solve the hard problem itself, it could still tell which solution was the best helper for related problems.
Ignoring Style: LLM judges often get tricked by long, verbose answers that sound authoritative. CBU didn't care about the "style" or "tone"; it only cared if the logic actually worked when applied to other tasks.

Key Takeaways for the Reader

Don't judge the book by its cover: Just because a math proof looks long and confident doesn't mean it's right.
Test the utility: The best way to judge a difficult idea is to see if it helps you solve easier, related problems.
No human needed (yet): This method allows us to evaluate research-level AI work without needing a human professor to read every single line, saving time and money.

What the Paper Does Not Claim

It does not claim this method works for every type of problem (it is specifically designed for research-level math).
It does not claim that AI can now solve these problems on its own; it only claims we can now evaluate the attempts better.
It does not suggest using this for medical diagnoses or legal advice; the scope is strictly mathematical reasoning.

In short, the paper introduces a clever "stress test" for AI math solutions: If your solution is truly good, it should make the AI smarter at solving related puzzles. If it doesn't, it's probably wrong.

Technical Summary: Judging What We Cannot Solve

Problem Statement

Recent advancements in Large Language Models (LLMs) have demonstrated the ability to generate plausible attempts at research-level mathematics. However, a critical bottleneck remains: verification. Validating these solutions typically requires scarce expert time or relies on "oracle-free" evaluators like LLM judges and reward models, which are often unreliable, biased, and prone to being misled by stylistic cues or authority-like statements.

The core challenge addressed is how to evaluate the correctness of candidate solutions for research-level questions where ground-truth answers are unavailable, the problems are currently unsolved by state-of-the-art models, and expert validation is not scalable. Existing oracle-free methods (e.g., majority voting, reward models, LLM judges) struggle to distinguish correct from incorrect solutions in this high-difficulty regime, often failing to separate valid reasoning from "plausible but wrong" hallucinations.

Methodology: Consequence-Based Utility (CBU)

The authors propose Consequence-Based Utility (CBU), an oracle-free evaluation framework grounded in the mathematical concept of "support by consequences." The central hypothesis is that a correct solution contains method-level information that transfers effectively to related problems, whereas an incorrect solution does not.

Core Mechanism

Neighborhood Construction: For a target research-level question $Q$ and a candidate solution $C$ , the system constructs a set of neighborhood questions $\mathcal{N}(Q)$ . These are variants of $Q$ (e.g., modified parameters or related theorems) that are designed to be verifiable (having known ground truths) and slightly easier than the original, yet solvable using the logic from $C$ .
In-Context Transfer: The candidate solution $C$ is used as an in-context exemplar (demonstration) for a solver model $M_\theta$ . The solver is prompted to solve each question $Q^* \in \mathcal{N}(Q)$ conditioned on $(Q, C)$ .
Utility Scoring: The utility score $U(C)$ is defined as the average accuracy of the solver on the neighborhood questions when $C$ is provided as context.
$U(C) = \frac{1}{|\mathcal{N}(Q)|} \sum_{Q^* \in \mathcal{N}(Q)} \mathbb{E}_{\tilde{C} \sim M_\theta(\cdot|Q,C,Q^*)} [v(Q^*, \tilde{C})]$
where $v$ is a verifier checking the correctness of the solver's output.
Ranking: Candidates are ranked by their induced utility scores. A higher score implies the candidate provides more helpful, transferable reasoning, suggesting it is more likely to be correct.

Dataset: EXPERTMATH

To evaluate this approach, the authors introduce EXPERTMATH, a benchmark consisting of:

192 expert-written research-level problems (spanning representation theory, algebraic combinatorics, geometry, and homotopy theory). Half of these remain unsolved by leading models (e.g., GPT-5, Gemini-3-Pro).
425 LLM-generated problems derived from existing literature.
Ground Truth: Each problem is paired with an expert-written solution and nine LLM-generated candidate solutions (four correct, five incorrect), manually verified by humans.

Key Contributions

Consequence-Based Utility (CBU): A novel, oracle-free validation method that scores solutions based on their downstream performance on related, verifiable questions rather than direct inspection of the solution text.
EXPERTMATH Benchmark: A high-difficulty dataset of research-level math problems with expert-verified solutions and LLM-generated candidates, specifically designed to test evaluation methods on problems that current models cannot solve directly.
Empirical Superiority: Demonstration that CBU consistently outperforms existing oracle-free baselines, including LLM judges, reward models (RMs), and generative reward models (GenRMs).
Error Analysis & Failure Modes: Identification of specific failure modes in LLM judges (e.g., over-reliance on verbosity, unjustified compression, and authority cues) that CBU successfully avoids by penalizing solutions that fail to transfer reasoning.

Experimental Results

Evaluated on EXPERTMATH using models like GPT-OSS-120B and Qwen3-235B-A22B:

Ranking Quality: CBU significantly improves ranking metrics over baselines.
- For GPT-OSS-120B acting as the solver/evaluator, CBU improves Acc@1 (accuracy of the top-ranked solution) from 67.2% to 76.3% and AUC (Area Under the Curve) from 71.4% to 79.6%.
- Similar gains are observed for smaller models (e.g., GPT-OSS-20B AUC increases from 69.0 to 79.2).
Solver-Evaluator Gap: CBU maintains a larger separation between correct and incorrect solutions even on questions the underlying solver cannot solve itself. While LLM judges' ability to distinguish correct from incorrect solutions degrades sharply as problem difficulty increases, CBU remains robust.
Predictive Power: Logistic regression classifiers trained on CBU scores achieve higher accuracy in predicting binary correctness compared to those trained on LLM judge scores (e.g., +6 to +18 percentage point gains across backbones).
Robustness to Style: Unlike LLM judges, which are often misled by verbose or "authoritative" sounding incorrect solutions, CBU downranks solutions with unjustified compression (missing steps) and incorrect reasoning because these fail to provide transferable information for the neighborhood questions.

Significance and Claims

The paper claims that Consequence-Based Utility offers a scalable, reliable alternative to human expert validation for research-level scientific outputs. Its significance lies in:

Oracle-Free Validation: It enables the evaluation of "unsolvable" problems without requiring a ground-truth oracle for the specific target question.
Shift in Evaluation Paradigm: It moves evaluation from "static inspection" (judging the text of a proof) to "dynamic utility" (judging the proof's ability to enable the solution of related problems).
Practical Applicability: The authors provide a practitioner's guide, showing that CBU remains effective even when neighborhood questions are generated by LLMs (provided the target questions are sufficiently difficult) and that stable estimates can be achieved with a moderate number of rollouts (e.g., $n \ge 8$ ).

The authors conclude that CBU is particularly well-suited for high-stakes scientific validation where high-confidence separation of correct and incorrect hypotheses is required, and where traditional LLM judges fail due to subtle reasoning errors or stylistic biases.