Can LLMs Build a MaxSAT Solver from Papers? The CoreForge Experience
The paper reports on CoreForge, an experiment demonstrating that large language models can successfully assist in building a functional unweighted MaxSAT solver directly from research papers through an iterative workflow, though the resulting system still requires human guidance and validation while lagging behind hand-engineered solvers in performance.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are a detective trying to solve a massive, tangled knot of clues. In the world of computer science, this knot is called a "constraint problem." You have a list of rules (like "the red ball must be on the left" or "the blue ball cannot be next to the green one"), and your job is to find a way to arrange everything so that every single rule is happy. Sometimes, it's impossible to make everyone happy, so the goal shifts to finding the arrangement that makes the most people happy. This is the job of a "MaxSAT solver"—a super-smart computer program that acts like a master puzzle-solver, finding the best possible solution when a perfect one doesn't exist.
For decades, building these puzzle-solvers has been a job for human experts. They have to read complex research papers, understand deep mathematical theories, and then spend years writing millions of lines of code, tweaking tiny details to make the solver fast enough to be useful. But recently, a new kind of digital assistant has arrived: Large Language Models (LLMs). These are the AI brains behind tools like ChatGPT, capable of writing stories, poems, and even computer code. The big question is: Can an AI read a research paper and build a working puzzle-solver from scratch, without a human writing a single line of the code? This is the mystery that the "CoreForge" project sets out to solve.
The CoreForge Experiment: Teaching AI to Build a Puzzle-Solver
In a project called CoreForge, a researcher named Ruben Martins from Carnegie Mellon University decided to test the limits of AI. Instead of asking an AI to fix a broken puzzle-solver that already existed, he challenged it to build one from the ground up, using only research papers as its instruction manual. Think of it like handing a robot a stack of cookbooks written by famous chefs and asking it to build a restaurant kitchen and cook a five-star meal, without ever having seen a stove before.
The process wasn't a "one-shot" magic trick where the AI just spat out a perfect program. Instead, it was a long, iterative dance between human and machine. The human researcher would pick a research paper, discuss the main ideas with the AI, and then ask the AI (specifically a coding tool called Codex) to write the code. But the AI didn't get a free pass. The human would then run tests, check for bugs, and ask the AI to fix its mistakes. They repeated this cycle over and over, like a student and a tutor working through a difficult math problem, until they had a working solver.
What Did They Build?
The result was a massive codebase of over 25,000 lines of C++ code. The AI successfully translated complex mathematical ideas from papers into working software. It implemented several advanced strategies, including:
- Core-guided optimization: A method where the solver learns from its mistakes (called "cores") to avoid repeating them.
- Preprocessing: Cleaning up the puzzle before trying to solve it to make it easier.
- Lookahead: A new feature the AI helped invent, where the solver takes a few quick "practice runs" to guess the best strategy before committing to the real solution.
Did It Work?
The most surprising part of the story is that the AI didn't cheat. The researchers ran thousands of tests, including "fuzzing" (throwing random, chaotic data at the solver to see if it breaks) and comparing it against official competition benchmarks. They found zero wrong answers in the tested configurations. The solver was mathematically correct. It didn't just guess; it actually solved the puzzles correctly.
But Is It Fast Enough?
Here is the catch. While the AI built a correct solver, it wasn't the fastest one. When compared to the top human-engineered solvers that have been refined for years, CoreForge was slower and solved fewer puzzles. It's like the AI built a car that drives perfectly and never crashes, but it only goes 40 miles per hour when the best human-built cars are going 120.
The researchers found that the AI was great at understanding the high-level ideas and turning them into code. However, it struggled with the "fine-tuning" details—the tiny, aggressive optimizations that make a solver lightning-fast. The AI also sometimes added extra steps that slowed things down, like taking a detour when a straight road would have been better.
The Verdict
The CoreForge experience suggests that AI can definitely help build complex software from research papers, but it's not ready to replace human engineers just yet. The AI needs a human guide to check its work, decide which ideas to keep, and help it avoid getting stuck in slow loops.
The paper concludes that while we haven't reached a point where AI can autonomously build a world-class solver, we are getting close. The AI proved it can read the "cookbooks" and build the "kitchen." The next step is teaching it how to be a master chef who knows exactly how to tune the heat and timing to win the competition. For now, the best results come from a team: a human providing the strategy and the AI doing the heavy lifting of writing the code.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.