Imagine you are trying to teach a robot how to be a mathematician. You don't just want it to chat about numbers; you want it to write the actual code that solves a puzzle. This is the world of combinatorial code synthesis. Think of "combinatorics" as the study of how things can be arranged, shuffled, or counted—like figuring out how many ways you can stack blocks, arrange a deck of cards, or color a map without two touching colors being the same. "Code synthesis" is simply the act of an AI writing a computer program to do the math.

For a long time, we've tested AI on general coding tasks, like "write a function to sort a list of names" or "fix this bug in a website." But those tests are a bit like asking a chef to chop onions; they show if the chef knows how to hold a knife, but not if they can invent a new recipe from scratch. This paper asks a harder question: If you give an AI a strict, abstract mathematical rule and a few examples, can it write a perfect program that follows that rule for any possible input, even ones it has never seen before? It's the difference between a robot that can follow a recipe and a robot that can understand the chemistry of cooking well enough to invent a dish that has never existed.

The Great Math Code Challenge: FindStatBench

Enter FindStatBench, a new, super-tough gym for AI models. Created by researchers at Scale AI, this benchmark is designed to stress-test Large Language Models (LLMs) on their ability to turn pure mathematical descriptions into working Python code. Instead of asking the AI to write a simple script, the researchers gave it 2,329 distinct math puzzles. These puzzles come from a real-world database called FindStat, which is like a library of "mathematical recipes" for things like permutations (shuffling lists), graphs (networks of dots and lines), and partitions (breaking numbers into sums).

The rules of the game were strict and "closed-book." The AI got a description of a math problem and up to five examples of inputs and outputs. It had to write a single Python function called solve(obj) that could handle any object, not just the examples it saw. The AI couldn't use a calculator, search the internet, or ask for help. It had to get it right on the first try. To make sure the AI wasn't just memorizing the answers, the researchers tested the code against 5.52 million hidden test cases—millions of math problems the AI had never seen before. If the code failed even one of these hidden tests, it was marked wrong.

The Results: A Surprising Tie and a Few Weird Glitches

When the dust settled, the results were fascinating and a little bit counter-intuitive.

1. The "Big" and "Small" Models are Neck-and-Neck
You might expect the most expensive, powerful "closed-source" models (the ones you have to pay big companies to use) to crush the open-source models (the ones anyone can download and run). But in this math-heavy arena, they didn't. The best open-source model, gpt-oss-120b, scored almost exactly the same as the best closed-source model, Claude Sonnet 4.6. They were separated by less than 1 percentage point. It's like two runners in a marathon: one is sponsored by a giant corporation with a fancy team, and the other is a local hero with a homemade shoe. They crossed the finish line at the exact same time.

Even more surprising, if you took the best answers from all eleven models tested and combined them, you only improved the total score by about 10%. This suggests that all these models are hitting a similar "ceiling" of ability. They are all struggling with the same hard parts of the puzzle.

2. More Examples Can Actually Make Things Worse
Here is where it gets weird. In most AI tests, giving the model more examples helps it learn. But on FindStatBench, for certain types of math puzzles called "bijections" (which are fancy ways of saying "perfect matching rules"), giving the model five examples actually made it perform worse than giving it zero examples.

Imagine you are teaching a student to fold an origami crane. If you just say "fold the paper this way," they might remember the classic, perfect fold. But if you show them five examples where someone made a slightly messy fold, the student might get confused and try to copy the messy folds, forgetting the perfect rule. The paper found that for some classic math rules, the examples "crowded out" the AI's internal knowledge, causing it to write broken code that failed even the examples it was shown. It's a case of "too much information, too little understanding."

3. The "Thinking" Trap
Another major finding was that some models, especially the ones designed to "think" before they speak, often ran out of space. These models have a limit on how much text they can generate. Sometimes, they spent their entire "budget" writing out their internal reasoning (their thought process) and never got around to writing the actual code. It's like a student spending the whole exam time writing a diary about how hard the test is, and then running out of time to actually solve the problems. The researchers found that simply giving these models more space to write their thoughts (and the code) fixed a lot of these failures.

4. The Hard Stuff is Still Hard
While the AI got pretty good at some tasks (like counting things or working with simple shapes), it completely failed at others. For tasks involving "set partitions" (grouping items in every possible way) and "binary trees" (hierarchical structures), the accuracy was near zero for every model, whether open or closed source. It seems that no matter how big the model is, or how much it costs, these specific types of structural puzzles are currently beyond their reach.

What This Means

The paper concludes that while AI is getting very good at writing "plausible" math code, it is still brittle when it comes to exact, symbolic rules. The models can guess the right answer often, but they struggle to consistently derive the perfect logic for complex structures.

The biggest takeaway? Cost doesn't always equal capability. In this specific test, a cheaper, open-source model performed just as well as the most expensive, proprietary ones. And sometimes, giving an AI more examples or more "thinking time" doesn't help—it can actually confuse it. The researchers suggest that to truly master math, AI needs to get better at understanding the rules of the game, not just memorizing the moves. For now, the AI is a talented apprentice who can follow instructions well but still gets lost when asked to invent a new kind of geometry.