🤖 machine learning

Escaping the Cognitive Well: Efficient Competition Math with Off-the-Shelf Models

This paper introduces a cost-efficient inference pipeline using off-the-shelf models that achieves state-of-the-art performance on IMO-style math problems by overcoming the "Cognitive Well" failure mode through context-detached conjecture extraction and independent verification.

Original authors: Xingyu Dang, Rohit Agarwal, Rodrigo Porto, Anirudh Goyal, Liam H Fowl, Sanjeev Arora

Published 2026-06-16

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xingyu Dang, Rohit Agarwal, Rodrigo Porto, Anirudh Goyal, Liam H Fowl, Sanjeev Arora

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a incredibly difficult math puzzle, like the ones given to the world's best high school math students at the International Olympiad.

For a long time, computers (specifically Large Language Models or "LLMs") have been terrible at this. They could solve easy puzzles, but when things got hard, they would get stuck, make mistakes, and keep making the same mistakes over and over.

To fix this, researchers tried a "brute force" approach: they would ask the computer to try solving the problem thousands of times, having one part of the computer act as a "Solver" (trying to write the answer) and another part act as a "Grader" (checking the work). If the Grader found a mistake, the Solver would try again.

The Problem: The "Cognitive Well"
The authors of this paper discovered a funny but dangerous trap in this process. They call it the "Cognitive Well."

Imagine you are walking in a deep, foggy valley. You think you are climbing up a hill, but you are actually just walking in circles at the bottom.

The Solver writes a proof that looks almost right.
The Grader (which is the same AI model) looks at it. Because the AI is "contaminated" by the context of the proof it just read, it gets fooled. It thinks, "Oh, that looks mostly correct! Just a tiny typo here and there."
The Grader gives it a high score.
The Solver thinks, "Great! I'm almost done!" and keeps refining the same wrong idea.
The AI is now stuck in a "well" of its own making, convinced it is solving the problem when it is actually completely wrong.

Previous methods tried to escape this well by throwing massive amounts of money at the problem—running the computer thousands of times in parallel. One method cost about $3,000 per single math problem. That is too expensive for most people to use.

The Solution: The "Detective" Pipeline
The authors built a new, much cheaper system (costing about $9 per problem) that escapes the Cognitive Well using three clever tricks:

Don't Just Guess, Isolate the Clues (Conjecture Extraction):
Instead of just asking the AI to "try harder," the system stops and asks: "What specific piece of logic is missing?"
Imagine a detective looking at a broken alibi. Instead of just saying "This doesn't make sense," the detective pulls out the specific claim: "He said he was at the park at 5 PM." The system isolates that single claim and treats it as a separate, tiny math problem.
The "Fresh Eyes" Test (Context Detachment):
This is the most important part. The system takes that isolated claim (the "conjecture") and puts it in a brand new, clean room. It asks the AI to prove the claim and prove the opposite of the claim, without seeing the original messy proof.
- If the AI proves the claim is true, it adds it to a "Cheat Sheet" of facts.
- If the AI proves the claim is false (by proving the opposite), it realizes the original proof was built on a lie.
- This breaks the AI out of the "Cognitive Well" because the AI can no longer be tricked by the confusing context of the original wrong proof.
The "Dialectic" Team:
The system doesn't just ask one AI to do the work. It creates a virtual meeting room with different "personas."
- The Architect: Tries to build the solution.
- The Skeptic (Momus): Constantly attacks the plan, looking for holes.
- The Grader: Checks the logic line-by-line.
  By forcing these different "personalities" to argue with each other, the system avoids the lazy thinking that leads to the Cognitive Well.

The Results
The authors tested this new pipeline on the hardest math problems available (the "IMO-ProofBench").

Performance: Their system solved 87.6% of the problems correctly. This is better than the "Gold Medal" winning systems from big tech companies (which are secret and unreleased) and far better than other public methods.
Cost: While other methods cost thousands of dollars per problem, theirs cost about $9.
Versatility: It works on many different types of AI models, not just one specific brand.

In Summary
The paper shows that you don't need to spend a fortune or run millions of simulations to solve hard math problems. Instead, you need a smarter strategy: when the AI gets stuck in a loop of confidence about a wrong answer, you need to pull the specific "suspect" (the logical gap) out of the crowd, test it in isolation, and use the result to guide the next step. This turns a confused, expensive brute-force approach into an efficient, logical detective story.

Technical Summary: Escaping the Cognitive Well: Efficient Competition Math with Off-the-Shelf Models

Problem Statement

Recent advances in mathematical reasoning have seen custom, unreleased models (e.g., Google DeepMind's "Deep Think") and large-scale agentic pipelines achieve gold-medal performance on the International Mathematical Olympiad (IMO). However, these successes rely on prohibitive computational budgets, often costing thousands of dollars per problem (e.g., ~$3,000/question for DeepSeekMath V2) and requiring massive parallelism (up to 64,000 LLM calls per problem). Furthermore, existing pipelines using publicly available models often fail to solve difficult problems reliably in a single call, necessitating iterative refinement.

The authors identify two critical failure modes in current solver-grader pipelines that necessitate such expensive parallelism:

Cognitive Plateau: The solver makes progress, but the grader identifies flaws the solver cannot fix, leading to a stagnation in score.
Cognitive Well: A more insidious failure where iterative refinement converges on a logically consistent but fundamentally incorrect solution. The internal grader (often sharing the solver's parameters and context) is "contaminated" by the plausible-but-wrong context, rating the flawed solution highly (e.g., 7/7) and overlooking fatal errors. This is likened to "reward hacking" or getting stuck in a local minimum.

Methodology

The paper proposes a new inference pipeline, the "Momus pipeline," designed to achieve best-in-class performance on IMO-style problems using only general-purpose, off-the-shelf models at a fraction of the cost. The methodology relies on three core design principles:

1. Narrow Width (Limited Parallelism)

Instead of brute-force parallelism (e.g., 64 branches), the pipeline limits parallelism to a modest level (typically $K=4$ concurrent solutions). New solutions are generated conditioned on the current state, which includes previous solutions, grader feedback, and results from a conjecture process.

2. Conjecture Extraction

To address the inability of internal graders to distinguish between "minor slips" and "fatal gaps," the pipeline employs a Solver with Conjectures prompt. When proofs stall, this module forces the model to explicitly formalize logical gaps as self-contained conjectures (and their negations). This transforms vague uncertainty into falsifiable mathematical statements.

3. Contextual Detachment

To escape the "Cognitive Well," the pipeline strips extracted conjectures from their parent proof context. These conjectures (and their negations) are fed as new, independent problems to fresh solver instances.

If a conjecture is proven, it is added to a Global Memory ( $M_{lemma}$ ) to guide future solvers.
If a conjecture's negation is proven, the pipeline uses this to actively discredit the stalled proof's logic, forcing the solver to abandon the flawed path.

Pipeline Architecture

The inference process is a stateful loop with three phases:

Phase I (Limited Exploration): Generates a small set of candidate solutions ( $K=4$ ) using a "Dialectic Solver" (a multi-persona prompt engineering framework). An adversarial grader checks for specific logical gaps.
Phase II (Contextual Detachment): If solutions are plausible but unverified, the system extracts load-bearing conjectures. It then runs parallel instances to prove both the conjecture and its negation in a fresh context.
Phase III (Refinement with Global Memory): New solvers are launched. Most are injected with the verified lemmas from Phase II, while one "zero-shot" solver remains to maintain diversity and avoid falling back into a Cognitive Well.

The pipeline also includes a Post-Enhancement step where, if no verified solution is found, the system rewrites proofs based on conjectures and runs a final aggregation step to select the best output.

Key Contributions

Efficient Pipeline: A new solution pipeline using only off-the-shelf models that achieves best-in-class performance on IMO-style problems at an average cost orders of magnitude lower than competing methods (e.g., ~$9/question vs. $3,000/question).
Identification of "Cognitive Wells": The paper identifies and provides an effective solution for the "Cognitive Well" failure mode via a conjecture-based context-detachment mechanism. This reduces false-positive wells and measurably boosts performance.
Grader Design and Evaluation: The authors propose new methods for designing effective graders (using a "guilty until proven innocent" framework) and evaluate the disconnect between a grader's performance as a standalone scorer versus its utility as a feedback provider in a pipeline.
Open-Source Accessibility: The prompts and methodology are open-sourced, demonstrating efficacy on frontier models (Gemini 3.1 Pro, Opus 4.6, DeepSeek-V3.2) at affordable costs, lowering the barrier to entry for researchers.

Results

The pipeline was evaluated on IMO-ProofBench Advanced (PB-Adv), a benchmark of 30 curated, difficult IMO-style questions.

Performance: Using Gemini 3.1 Pro, the pipeline achieved 87.6% performance (expert-graded), surpassing the DeepThink IMO gold-winning model and more than tripling the success rate of the next best publicly accessible pipeline (Huang and Yang [2025]) operating at an affordable cost.
Cost Efficiency: The average cost per question was approximately $9. This is significantly cheaper than the Huang and Yang pipeline (~~$371/question) and DeepSeekMath V2 (~~$3,000/question).
Generalization: The method generalized across different backbones. For instance, on Claude 4.6 Opus, performance jumped from a 23.8% base call to 85.7% with the pipeline. On DeepSeek-V3.2, it improved from 15.2% to 66.7%.
Ablation Studies: Removing the conjecture extraction stage resulted in a significant drop in performance (from 55% to 34.3% on PB-Adv with Gemini 2.5 Pro), even when token budgets were matched, confirming that the gains are not merely due to increased sampling diversity.

Significance

The paper claims that reliable LLM theorem proving does not necessarily require massive parallelism or thinking budgets. By identifying and addressing the "Cognitive Well" and "Cognitive Plateau" failure modes through conjecture extraction and contextual detachment, the authors demonstrate that off-the-shelf models can reach state-of-the-art performance on difficult competition math problems at a fraction of the cost.

The work suggests that current solver-grader pipelines may be deep-rooted in these failure modes, which could extend to broader AI-for-Science systems. The authors emphasize that their approach significantly lowers the barrier to entry for mathematical reasoning research, making high-level problem solving accessible to researchers without access to proprietary, custom-trained models or massive compute resources.