🤖 AI

Agentic Synthesis against Counterexample-Supplemented Sketches

This paper introduces "Agentic Synthesis against Counterexample-Supplemented Sketches," a repository-native method where human-approved policy corrections are integrated into an evolving code sketch rather than accumulated examples, demonstrating that this approach reduces developer effort and improves the transfer of learned rules to unseen cases compared to traditional replay-based regeneration.

Original authors: Muness Castle, Eric Rubeck

Published 2026-07-20

📖 8 min read🧠 Deep dive

CC BY 4.0

Original authors: Muness Castle, Eric Rubeck

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where you have a super-smart robot assistant that can write computer code for you. It's like having a genius apprentice who can type faster than anyone else and knows a million programming tricks. But here's the catch: this robot is a bit of a "people pleaser." If you tell it, "Fix this broken button," it will happily fix the button. But if the button was broken because of a secret rule you never told it (like "never let the button turn red on Tuesdays"), the robot might fix the button while accidentally breaking that secret rule. If you don't catch it, the robot might make the exact same mistake again next time, because it only remembers the specific fix, not the reason behind it. This is the problem of "coding agents" in the real world: they are great at patching holes, but terrible at learning the deep, unwritten laws of how a system should behave unless someone explicitly teaches them.

This paper, written by Muness Castle and Eric Rubeck, tackles that exact problem. It introduces a new way of working called "Agentic Synthesis against Counterexample-Supplemented Sketches." Think of it as a rigorous training camp for AI coders. Instead of just letting the AI guess and check, this method forces a human to act as a strict coach. When the AI makes a mistake that breaks a hidden rule, a human doesn't just say "fix it." They say, "Here is the mistake, here is the correct answer, and here is the rule you broke." The AI then has to rewrite its own "instruction manual" (called a sketch) to include that new rule before it's allowed to try again. The paper tests this idea with a fake browser app called "CatSynth" and finds that this method helps the AI learn the rules better than just showing it a list of past mistakes, though the authors are careful to say this is based on one specific experiment, not a magic bullet for every situation.

The Story of the Sketch and the Rulebook

Imagine you are trying to build a robot that picks the perfect cat for a family. You give the robot a "Sketch." In this story, a Sketch isn't a drawing; it's a rough, half-written instruction manual. It says things like, "The cat should be fluffy," and "The cat should be friendly." But it's missing the most important part: "The cat cannot be a Persian because the family has a cat allergy."

At first, the robot (the coding agent) looks at the Sketch and says, "Okay, Persians are fluffy and friendly! I'll pick a Persian!" It writes the code to do this. But then, a human expert (the Subject-Matter Expert) looks at the result and says, "Whoa, stop! That's a disaster. The family has an allergy. You picked the wrong cat."

In the old way of doing things, the human might just say, "Okay, fix it, pick a different cat." The robot would then pick a different cat, maybe a Siamese. But the robot wouldn't actually learn the rule about allergies. It would just remember "Don't pick Persians for this specific family." Next time, if the family changed their mind about the allergy but the robot didn't know, or if a new family with an allergy came along, the robot might make the same mistake again.

This paper proposes a different, stricter loop. When the human expert says, "Wrong cat!", they don't just fix the output. They act like a judge in a courtroom. They say:

The Mistake: "You picked a Persian."
The Correction: "The correct cat is a Siberian."
The Rule: "The rule you broke is: 'Hard rules (like allergies) must always beat soft preferences (like fluffiness).'"

Then, a human "Operator" (the boss) has to explicitly sign off on this. They say, "Yes, this is a real rule now." Once signed, the robot must rewrite its Sketch to include this new rule. The robot can't just patch the code; it has to update the manual.

The Two-Step Check: The Replay and The Compare

To make sure the robot actually learned the lesson, the paper uses a clever two-step test, like a security guard checking a visitor at a gate.

The Replay (The "Did you fix the mess?" check): This checks if the robot's new answer actually solves the immediate problem. Did the robot pick a cat that is fluffy and friendly? If yes, the "Replay" passes.
The Compare (The "Did you follow the rules?" check): This is the tricky part. It checks if the robot picked the right cat based on the new rule. Did it pick the Siberian instead of the Persian? Did it remember the allergy? If the robot picked a fluffy Persian that solved the "fluffy" problem but broke the "allergy" rule, the "Compare" fails.

The paper calls this the "Gate." If the robot passes both, it moves on. If it fails the Compare, it means the robot fixed the surface problem but ignored the deep rule. The robot has to go back and fix its Sketch again.

The CatSynth Experiment

The authors tested this with a fake project called CatSynth. They created a scenario where a robot had to pick a cat based on a list of preferences (big, fluffy, affectionate) and a hard rule (no cats that trigger allergies).

They ran a simulation where the robot tried to solve 14 different "frozen" cases (scenarios that were set in stone before the experiment started).

The Result: In 8 out of 14 cases, the robot's initial guess was wrong because it missed a rule. These 8 cases became "counterexamples."
The Process: For each of these 8 mistakes, a human approved the correction, the robot updated its Sketch, and the robot tried again.
The Test: After the robot finished learning, the authors tested it on 21 new, hidden cases (cases it had never seen before).

Here is what they found:

When they rebuilt the robot's code from scratch using only the final, evolved Sketch (the one that had been updated with all the rules), the robot got 19 out of 21 of the new cases right.
When they tried to rebuild the robot by showing it the original Sketch plus a long list of all the past examples (the "replay-all" method), the robot only got 15 out of 21 right.

This suggests that having the robot learn the rules and write them into its Sketch was more effective than just showing it a history of mistakes. The robot that learned the rules generalized better to new situations.

What This Means (and What It Doesn't)

The paper is very careful not to overhype the results. They don't claim this is the "perfect" way to fix all AI coding problems forever. They admit that this was a simulation with one specific model (GPT-5.4-mini) and one specific order of problems. They don't know if it would work exactly the same way with a different robot or a different set of rules.

However, the experiment does show something important: It's better to teach an AI the "Why" (the rule) than just the "What" (the example).

If you just show an AI a list of "Don't do X," it might memorize the list but miss the pattern. But if you force the AI to update its own instruction manual (the Sketch) every time it breaks a rule, and then test it by throwing away its old code and rebuilding it from that manual, you get a smarter, more reliable AI.

The authors also point out that this method requires a lot of human effort. You need a human expert to spot the mistake, a human operator to approve the new rule, and a human to curate the list of tests. It's not a "set it and forget it" system. It's a partnership where the human is the teacher and the AI is the student who has to take notes and rewrite its own textbook.

The Takeaway

In the end, this paper is a guide on how to stop AI from being a "plausible but wrong" fixer. It argues that when an AI makes a mistake, we shouldn't just patch the code. We should treat the mistake as a lesson that changes the AI's fundamental understanding of the world. By using a "Sketch" to hold the rules, a "Gate" to check the work, and a human to approve the lessons, we can build AI that doesn't just guess the right answer, but actually understands why it's right.

The authors conclude that while this method didn't solve every problem in their simulation (the robot still got 2 out of 21 hidden cases wrong), it proved that the "evolved Sketch" carried the knowledge better than a simple history of examples. It's a step toward making AI coding more like a disciplined apprenticeship and less like a lucky guess.

Technical Summary: Agentic Synthesis against Counterexample-Supplemented Sketches

Problem Statement

Coding agents can generate plausible patches that satisfy immediate signals (e.g., failing tests, state deltas, or prompt instructions) while violating underlying domain rules that have not yet been encoded in the repository. When a Subject-Matter Expert (SME) rejects a repair due to a policy violation, that rule often remains in human memory or chat transcripts rather than the codebase. Consequently, subsequent agent runs may repeat the same "plausible mistake." The core problem is that standard repair loops lack a mechanism to convert a single rejected repair into a durable, repository-native governing policy that prevents the recurrence of that specific failure mode.

Methodology: The Sketch-CE Loop

The paper proposes Agentic Synthesis against Counterexample-Supplemented Sketches (Sketch-CE), a repository-native workflow designed to evolve a governing policy (the "sketch") through explicit operator approval of counterexamples. The method relies on six specific artifacts and a strict role separation.

Core Artifacts

Sketch ( $S$ ): A reviewable, agent-editable artifact containing prose, code structures, and policy rules. It defines the strategy space, priority order, and holes to be filled. Crucially, the sketch evolves; every accepted counterexample must revise the sketch to encode the new rule.
Accepted-Counterexample Archive ( $A$ ): The complete provenance of approved failures. Each record includes the input, the authoritative corrected output, the "tempting" wrong output, the distinguishing rule, and the operator's approval.
Regression Set ( $R \subseteq A$ ): A curated subset of executable cases selected to reject known wrong implementations and protect specific policy boundaries.
Known-Code Anchors ( $K$ ): Existing repository shapes (APIs, conventions, fixtures) that the agent must preserve to prevent the invention of parallel local conventions.
Implementation Surface: The versioned code (Oracle A) or prompt text (Oracle B) that applies the sketch rules. These are replaceable; a fresh implementation should be regenerable from the evolved sketch and anchors without replaying the full archive.
Gate ( $G$ ): An executable validation bundle with two distinct layers:
- Replay: Checks if the candidate output repairs the encoded state gap (e.g., does the math work?).
- Semantic Compare: Checks if the output matches the approved expectation on policy-bearing fields (e.g., was the correct operation selected?).

Roles and Workflow

The loop involves distinct actors to prevent self-justification:

SME: Judges concrete outputs and provides corrections with explanations.
Operator: Explicitly approves counterexamples that contradict the current sketch. This approval grants the case "specification authority" and mandates a sketch revision.
Maintainer: Curates the regression set and maintains the machinery.
Coding Agent: Repairs implementation surfaces (code or prompts) under the constraints of the current sketch, anchors, and the single active failure.

The Loop Sequence:

An observed case fails the current implementation.
If the failure contradicts the sketch, the SME provides a correction.
The Operator explicitly approves the counterexample.
The Developer (agent) revises the sketch to encode the new rule and repairs the implementation surface for that specific failure.
The Gate runs the active case and the regression set ( $R$ ).
If the gate passes, the case is curated into $R$ (if not already covered), and the next case is revealed. If it fails, the agent repairs the specific regression.
Periodic Clean Regeneration: To verify that the sketch carries the learned policy, the implementation is discarded and regenerated from the evolved sketch and anchors alone. If this fails, the sketch has not sufficiently synthesized the policy.

Key Contributions

Repository-Native Artifact: The introduction of the "counterexample-supplemented sketch" as a central artifact that binds operator decisions, archive records, and reviewed policy clauses.
Process Model: A workflow that transforms operator-approved corrections into an evolved sketch, a complete archive, and a curated regression set, separating the governing model from the implementation history.
Dual-Oracle Implementation: A hybrid approach using deterministic code (Oracle A) for encodable rules and prompt-mediated completion (Oracle B) for narrative or under-encoded policy, both constrained by the same sketch and gate.
Finite-Regression Theorem: A formalization of the "Gate Pass" condition, proving that a passing gate guarantees $R$ -correctness (satisfaction of state repair and policy fields) for the specific regression set, bounded by the current checkers and approved rows.
Empirical Validation (CatSynth): A synthetic browser application experiment demonstrating the method's mechanics and comparing it against control conditions.

Experimental Results (CatSynth)

The authors evaluated the method using CatSynth, a synthetic application involving pet selection with hard policy rules (e.g., allergies) and soft preferences. The experiment used GPT-5.4-mini as the Developer.

Setup: An open-world run where the agent discovered rules via counterexamples, compared against two controls:
- Replay-All: Rebuilding from the initial sketch and replaying all accepted examples as context.
- Evolved-Sketch Rebuild: Rebuilding from the evolved sketch and anchors only (clean regeneration).
Findings:
- Policy Retention: The "Evolved-Sketch Rebuild" passed 19 of 21 withheld cases, outperforming "Replay-All" which passed 15 of 21. This suggests the evolved sketch successfully synthesized the learned policy better than the model rediscovering rules from raw examples.
- Efficiency: Retaining code across counterexamples (Sketch-CE) required 9 Developer calls and 719 lines of artifact churn, compared to 15 calls and 2,394 lines for the Replay-All approach.
- Counterexample Density: In the open-world run, 8 of 14 frozen candidate cases became counterexamples. All three paths (Sketch-CE, Replay-All, Evolved-Sketch) passed the 8 accepted cases.
- Token Usage: The Sketch-CE approach consumed more tokens during the visible acceptance phase (1,021,822) compared to the controls, largely due to the cost of evaluating external candidates and proposing general rules, rather than generating them.

Claims and Significance

The paper makes modest, bounded claims regarding its significance:

Inspectability: The primary contribution is an inspectable workflow where every accepted counterexample connects an operator's decision, an archive record, and a reviewed sketch clause. This prevents "plausible wrong repairs" from disappearing into chat history.
Policy vs. Implementation: The method demonstrates that a repository-native sketch can carry the "learned policy" such that a fresh implementation can be regenerated without replaying the entire history of examples.
Reduced Rework: In the specific CatSynth run, retaining code across counterexamples reduced implementation churn and developer calls compared to replaying all examples.
Bounded Correctness: The authors explicitly state that a passing gate does not prove general correctness or superiority. The claim is bounded to the current regression set, the encoded replay/compare predicates, and the approved expected rows. The results do not establish general model superiority or correctness beyond the encoded checks.
Limitations: The paper acknowledges that the evidence is limited to one model, one reveal order, and synthetic data. It does not claim to solve the problem of formal verification or to replace human judgment in policy definition.

In summary, the paper argues that for policy-heavy repairs, the repository must treat the sketch as the durable governing model and the counterexample archive as the source of policy evolution, rather than relying on the accumulation of prompt history or test cases alone.