OEIS Open: How many conjectures can language models turn into theorems?

Technical Summary: OEIS OPEN

Problem Statement
The paper addresses the lack of systematic benchmarks for evaluating the capabilities of AI systems in solving open research problems in mathematics. Existing benchmarks, such as HorizonMath and FrontierMath: Open Problems (FM:OP), rely on "generator–verifier" gaps where solutions are concrete objects (e.g., graphs, polynomials) that are hard to find but cheap to verify. The author argues these approaches fail to cover the majority of research mathematics, which often requires proving general statements rather than exhibiting objects. Furthermore, existing benchmarks often accept numerical evidence rather than formal proofs, rely on hand-crafted verification code prone to errors, and include problems that may be unsolvable.

The author proposes OEIS OPEN, a benchmark designed to test whether Language Models (LMs) can autonomously resolve open mathematical conjectures by producing formal proofs (or disproofs) within the Lean proof assistant.

Methodology

• Dataset Construction: The benchmark consists of 492 open conjectures drawn from the Online Encyclopedia of Integer Sequences (OEIS). These conjectures were originally collected and formalized in Lean by Tsoukalas et al. [6]. The selection process involved filtering 2,649 OEIS conjectures to find non-trivial, mathematically interesting, yet not famous problems suitable for automated theorem proving. The focus on integer sequences minimizes the risk of misformalization, as these problems typically involve elementary operations rather than complex definitions requiring long chains of library dependencies.
• Evaluation Framework: Unlike previous attempts that used bespoke, complex agents (e.g., AlphaProof Nexus using evolutionary search), this evaluation uses a simple, open-source agent harness built on the Inspect library. The base agent utilizes a ReAct-style loop with three tools: a bash shell, a text editor, and a resource monitor. The environment includes Lean 4 with Mathlib, SageMath, and Python libraries (sympy, mpmath, numpy).
• Security and Verification: To prevent "unauthorized modifications" (e.g., axiom injection, tampering with the environment), the evaluation splits each attempt across three isolated Docker containers:
  1. Agent Container: Where the model generates the proof.
  2. Compile Container: A clean environment where the submitted source is compiled to a binary artifact (.olean).
  3. Scorer Container: Runs SafeVerify (an adaptation of lean4checker) to verify the binary against the target theorem.
  • SafeVerify ensures the submission proves the exact target statement without introducing extra axioms beyond the standard three (propext, Quot.sound, Classical.choice) and without relying on unverified metaprogramming tricks.
• Experimental Variants: The study evaluates models under different conditions:
  • Budgets: A $50 cap per conjecture for the full set (492 problems) and a $200 cap for a random subset of 100 problems (OEIS OPEN LITE).
  • Agent Complexity: Comparing the simple base agent against a "DeepAgent" variant (featuring delegation, persistent memory, and todo-lists) and a variant with access to an offline snapshot of 476,000 arXiv mathematics papers.

Key Results

• Performance:
  • On the full OEIS OPEN set ($50 budget), Claude Opus 4.8 resolved 30% (147/492) of the conjectures. GPT-5.5 resolved 26%, and Gemini 3.5 Flash resolved 22%.
  • These results significantly outperform the previous state-of-the-art system, AlphaProof Nexus (Tsoukalas et al.), which resolved only 9% (44/492) of the same conjectures at a similar cost.
  • On the OEIS OPEN LITE subset ($200 budget), performance improved, with Claude Fable 5 achieving a 44% success rate and GPT-5.6 Sol reaching 43%.
• Cost Efficiency: The average cost per resolved conjecture was modest, ranging from $6 (GPT-5.5) to $10 (Claude Opus 4.8), with a maximum of $47.
• Impact of Complexity and Data:
  • Providing agents access to the mathematics literature (476k arXiv papers) did not improve performance on the LITE set.
  • Using the more complex DeepAgent architecture did not improve performance over the simple base agent.
• Scaling: Solve rates appear to rise log-linearly with the spending budget. The author estimates that with a $200 budget, the best current models could resolve approximately 216 of the 492 conjectures (up from 147 at $50), suggesting that further scaling of inference compute would yield more results, though at exponential cost.

Significance and Claims

The paper claims that OEIS OPEN demonstrates that current language models can autonomously resolve open research conjectures at a modest cost, without the need for highly specialized, bespoke agent architectures.

• Autonomy: The results suggest that LMs can push the frontier of research mathematics independently, resolving problems that had previously only been attempted by complex, custom-built systems.
• Simplicity: The success of the minimal "ReAct" agent supports the "bitter lesson" in AI: providing models with simple, general tools often outperforms prescribing complex, problem-specific structures.
• Benchmark Utility: The work establishes formal proofs as a viable and rigorous standard for AI benchmarking in mathematics, avoiding the pitfalls of numerical evidence and hand-crafted verifiers.
• Nature of Conjectures: The author is modest regarding the mathematical significance of the specific conjectures solved. They note that while these are open problems proposed by mathematicians, most have likely received little prior attention. The dataset serves as a proxy for "open research" rather than a test of solving famous, high-profile problems (like the Riemann Hypothesis).

Limitations
The author acknowledges several limitations:

1. Misformalization Risk: While integer sequences reduce this risk, some conjectures in the dataset may be misformalized in Lean, potentially making them easier to solve than intended.
2. Scope: The benchmark is limited to problems formalizable in Mathlib; it cannot cover all of research mathematics.
3. Data Leakage: Once a conjecture is resolved, its proof may enter the training data of future models, potentially inflating future performance metrics.
4. Reductions: The benchmark only accepts proofs of the conjecture or its negation, not reductions to other famous open problems, which mathematicians might value highly.

In conclusion, the paper presents OEIS OPEN as a robust, secure, and scalable benchmark that reveals the growing capability of general-purpose language models to perform autonomous mathematical research, resolving a significant portion of open conjectures with minimal agent complexity.