🤖 AI

PuzzleClone: A DSL-Powered Framework for Synthesizing Verifiable Data

The paper introduces PuzzleClone, a DSL-powered framework that synthesizes over 83,000 diverse and verifiable logic puzzles to enhance large language model reasoning, demonstrating significant performance gains across multiple mathematical and logical benchmarks after post-training.

Original authors: Kai Xiong, Yanwei Huang, Rongjunchen Zhang, Kun Chen, Haipang Wu, Yingcai Wu

Published 2026-05-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Kai Xiong, Yanwei Huang, Rongjunchen Zhang, Kun Chen, Haipang Wu, Yingcai Wu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to solve logic puzzles. You want the robot to get really good at it, but you only have a few hundred practice puzzles. If you just ask the robot to "make up more puzzles," it might invent nonsense or get the answers wrong, which would confuse the robot even more.

This paper introduces PuzzleClone, a clever system designed to solve that problem. Think of it as a "Lego Blueprint Generator" for logic puzzles.

Here is how it works, broken down into simple steps:

1. The Master Blueprint (The "Seed")

Instead of asking the robot to invent a puzzle from scratch, a human expert takes one high-quality, tricky puzzle (like a Sudoku or a logic riddle about who bought which food) and turns it into a structured blueprint.

The Analogy: Imagine a cake recipe. The "Seed Puzzle" is a specific chocolate cake. The "Blueprint" (called a DSL in the paper) isn't the cake itself; it's the recipe structure. It says: "You need a list of ingredients (variables), a set of rules (conditions), and a question to ask."
Crucially, this blueprint separates the logic from the specific numbers or names. It knows that "Alice" is just a placeholder for a name, and "5" is just a placeholder for a number.

2. The Factory Machine (Randomization)

Once the blueprint is ready, PuzzleClone acts like a factory machine. It takes that single blueprint and starts shaking the dice.

The Analogy: Imagine a machine that takes your chocolate cake recipe and automatically swaps "chocolate" for "strawberry," changes "5 eggs" to "7 eggs," and swaps "Alice" for "Bob."
Because the machine follows the strict rules of the blueprint, it can generate 83,000+ unique variations of that one original puzzle. It doesn't just change the words; it changes the underlying math and logic in a way that creates a brand new, valid puzzle every time.

3. The Quality Control Inspector (Verification)

This is the most important part. In other systems, if the machine makes a mistake, the puzzle might be broken (unsolvable or have the wrong answer). PuzzleClone has a built-in Inspector.

The Analogy: Before the factory sends out a new cake, it runs a test. It asks: "If we put these exact ingredients back into the recipe, do we get the original cake back?"
The system uses a math solver (a computer program that is perfect at logic) to solve the new puzzle instantly. If the solver finds an answer, the puzzle is "verified." If the system can't solve it, it throws the puzzle in the trash. This ensures that every single one of the 83,000 puzzles is 100% correct.

What Did They Find?

The researchers used this system to build a massive test bank called PC-83K.

The Challenge: They tested the world's smartest AI models (like ChatGPT-4o and DeepSeek) on these puzzles. The results were surprising: even the best AIs struggled. They got the puzzles wrong quite often, showing that current AI still has trouble with deep, complex logic.
The Training: They then took a smaller AI model and "fed" it these 83,000 verified puzzles to study.
The Result: After studying this high-quality data, the AI got much smarter. Its ability to solve logic puzzles jumped from 14.5% to 66.0%. It also got better at other math and logic tests it had never seen before.

Why Is This Important?

The paper argues that to make AI smarter at reasoning, we need high-quality, verified data, not just a lot of data.

Old Way: Ask an AI to write 10,000 puzzles. It might write 1,000 good ones and 9,000 bad ones. The bad ones confuse the learning process.
PuzzleClone Way: Use a strict blueprint to generate 83,000 puzzles where the answers are mathematically guaranteed to be correct.

In short, PuzzleClone is a tool that turns a few good logic puzzles into a massive, error-free library of practice problems, which helps train AI to become a much better logical thinker.

Technical Summary: PuzzleClone

Problem Statement

High-quality mathematical and logical datasets with verifiable answers are critical for enhancing the reasoning capabilities of Large Language Models (LLMs). While recent data augmentation techniques have enabled the creation of large-scale benchmarks, existing LLM-generated datasets frequently suffer from three primary limitations:

Reliability: Without robust, end-to-end verification, synthesized data often contains flaws, inaccuracies, or biases.
Diversity: Current pipelines disproportionately depend on the generative capabilities of a single underlying LLM, which tends to explore only a narrow range of variations in assumptions, conditions, and parameters.
Scalability: Involving LLMs in every step of the synthesis pipeline incurs substantial computational costs, severely constraining the ability to generate data at scale.

Manual curation of high-difficulty reasoning problems remains a bottleneck due to the extensive effort required to compose and validate items, resulting in relatively small and homogeneous corpora.

Methodology

The authors introduce PuzzleClone, a formal framework designed to synthesize verifiable data at scale using a novel Domain-Specific Language (DSL) driven approach. The framework operates through a three-stage pipeline:

1. Puzzle Encoding

Seed puzzles are manually encoded into two components:

Structured Problem Specification ( $Q_s$ ): A DSL definition that captures the core logic of the puzzle, including variables, symbols, conditions, queries, and natural language templates. Unlike prior works relying on Prolog or SMT-LIB, this DSL is designed to be human-readable while remaining machine-parsable. It explicitly encodes constraints for data augmentation, such as value domains and the number of constraints.
Configuration File ( $Q_c$ ): A file containing the specific parameter values for the seed puzzle.

The DSL structure includes:

Variables ( $V$ ): Parameters that can be varied (e.g., number of students, food types), characterized by type, value domain, and a difficulty factor.
Symbols ( $S$ ): Quantities to be solved, mapped from variables with associated natural language templates.
Conditions ( $C$ ): A pool of static and dynamic constraints. Dynamic conditions are randomly generated from templates with parameters drawn from specified pools.
Queries ( $Q$ ): The questions and options, defined by templates and parameter pools.
Description ( $D$ ): A natural language template assembling the components.

2. Puzzle Generation

A puzzle generator translates the DSL specification into new instances by:

Randomization: Systematically varying parameters and combinations of constraints within their defined domains to create diverse mathematical configurations.
Solving: Automatically generating ground-truth answers for each instance using symbolic solvers (e.g., Z3 for SMT problems) or custom solvers.
Rendering: Synthesizing the final puzzle text by integrating the randomized parameters into the natural language templates.

3. Config-Based Validation

To ensure fidelity, the framework includes a reproduction mechanism:

A validation script is generated to reproduce the original seed puzzle by extracting specific values directly from the configuration file ( $Q_c$ ) rather than randomizing.
The system computes a deterministic reference answer and compares it against the ground-truth solution of the seed puzzle. This verifies the accuracy of both the problem formulation and the encoding process.

Key Contributions

PuzzleClone Framework: A formal, DSL-driven pipeline for synthesizing verifiable reasoning data that decouples the logic definition from the specific instance generation, ensuring programmatic validity.
PC-83K Benchmark: The construction of a dataset comprising 83,657 diverse, programmatically validated puzzles. The dataset covers various formats including Satisfiability Modulo Theories (SMT), Linear Programming (LP), and classical logic puzzles (e.g., Sudoku-style problems).
- The dataset includes 86 high-quality seed puzzles curated from competitions and established benchmarks.
- It features a stratified difficulty distribution and supports multiple question types (multiple-choice, fill-in-the-blank, short answer) and answer structures.
Verification Mechanism: A unique reproduction step that guarantees the correctness of the synthesis pipeline by verifying that the DSL specification can regenerate the original seed puzzle.

Results

The authors evaluated the PC-83K benchmark and the efficacy of post-training on it:

Benchmark Difficulty: Current state-of-the-art LLMs (including ChatGPT-4o, DeepSeek-R1, and various Qwen and GLM models) face significant challenges on PC-83K. For instance, Qwen2.5-7B-Instruct achieved an average accuracy of only 14.5% on the test set.
Post-Training Performance:
- SFT and RL: Fine-tuning Qwen2.5-7B-Instruct on PC-83K yielded substantial improvements. Post-training raised the average performance on PC-83K from 14.5 to 66.0.
- Generalization: The training data improved performance across 7 external logic and mathematical benchmarks. Notably, on the SATBench, accuracy improved by 18.4 percentage points (from 51.6 to 70.0).
- RL vs. SFT: Reinforcement Learning (RL) demonstrated superior alignment with the target distribution (PC-83K), while Supervised Fine-Tuning (SFT) showed slightly better cross-benchmark generalization on average, though RL captured deeper compositional reasoning on specific tasks like BBEH-mini.
Generalizability: A case study reproducing the SynLogic dataset using PuzzleClone demonstrated that the framework could synthesize data (PC-SL-35K) that was at least as effective as the original SynLogic data in enhancing model reasoning performance.

Significance

The paper claims that PuzzleClone provides a principled, scalable path to generating reliable reasoning data indefinitely. By moving away from LLM-dependent annotation pipelines toward a formal DSL approach with rigorous programmatic verification, the framework addresses the critical issues of data reliability, diversity, and scalability. The resulting PC-83K dataset serves as a challenging benchmark that exposes the limitations of current models and demonstrates that high-quality, verifiable synthetic data can significantly boost the logical reasoning capabilities of LLMs through post-training.