💬 NLP

CRAFT: Learn the Schema, Execute the Plan

CRAFT is a two-stage post-training framework for enterprise coding agents that learns schema-grounded planning and execution behaviors without exhaustive prompt-time schema injection, significantly improving analytical performance, consistency, and efficiency while reducing token overhead.

Original authors: Aakash Kolekar, Sahika Genc, Shahriar Shariat, Bunyamin Sisman, Tibor Mezi, Barbara Poblete, Shree Vandana Kachroo, Calvin Chi, Parth Parmar, Ari Singer, Prayaas Jain, Cindy Barker, Benoit Dumoulin

Published 2026-07-28

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Aakash Kolekar, Sahika Genc, Shahriar Shariat, Bunyamin Sisman, Tibor Mezi, Barbara Poblete, Shree Vandana Kachroo, Calvin Chi, Parth Parmar, Ari Singer, Prayaas Jain, Cindy Barker, Benoit Dumoulin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where you can talk to a super-smart robot assistant and ask it to do complex math, analyze huge piles of data, or write computer programs just by using plain English. This is the dream of "coding agents." But here's the catch: to do its job, this robot needs to know the secret rules of the database it's working with. Think of it like a chef trying to cook a gourmet meal. If the chef doesn't know what ingredients are in the pantry or how the stove works, they can't cook. In the past, to help the robot, engineers would shove a massive, dusty encyclopedia of every single rule, table name, and column definition right into the robot's brain every time it asked a question. It was like forcing the chef to read the entire encyclopedia before chopping a single onion. It was slow, expensive, and if the pantry changed even a little, the whole system would break.

Now, imagine if instead of reading the encyclopedia every time, the chef could just remember the layout of the kitchen after a few practice sessions. That's the big question this paper tackles: Can we teach these AI assistants to learn the rules of the database once and for all, so they don't need to be reminded of them every single time? The researchers wanted to see if they could train the AI to "internalize" the schema (the map of the data) and the tools it needs to use, making it faster, smarter, and less likely to get confused when the conversation gets long and complicated.

The Paper: CRAFT – Learning the Map, Then Running the Race

The paper introduces a new system called CRAFT (which stands for a two-stage recipe for learning). The authors, a team from Amazon, wanted to build an AI that could help advertising experts analyze their campaign data without drowning in technical details. They found that the old way of "stuffing" the AI's prompt with endless documentation was like trying to run a marathon while carrying a backpack full of bricks. It slowed everything down and made the AI prone to mistakes.

So, they tried a different approach: Learn the Schema, Execute the Plan.

Stage 1: The "Schema-Stripped" Classroom

First, they taught the AI in a special classroom where the "encyclopedia" was taken away. They gave the AI thousands of examples of how to solve problems, but they hid the raw database rules (the DDL) from the input. Instead of reading the rules, the AI had to learn the patterns of how to think and plan.

Think of it like teaching a student to solve a math problem by showing them the steps and the logic, but not letting them look at the formula sheet. They have to memorize the way to think. The researchers call this PLAN Supervised Fine-Tuning (SFT). It's like the AI is learning the "muscle memory" of how to structure a plan and write code, without needing the cheat sheet every time.

Stage 2: The "Reinforcement Learning" Gym

Once the AI had the basic muscle memory, they put it in a gym for Reinforcement Learning (RL). This is where the AI actually tries to run the code and use the tools. If it makes a mistake—like calling the wrong tool or writing code that crashes—it gets a "negative score." If it succeeds, it gets a "positive score."

The researchers used a special scoring system that didn't just care if the code ran; it cared if the code was good, if the plan matched the code, and if the AI could recover when things went wrong. It's like a coach watching a runner not just finish the race, but checking their form, their speed, and how they handle a stumble. This stage, called Execution-Shaped GRPO, taught the AI to be reliable and consistent, even when the user asked follow-up questions or changed the rules mid-conversation.

The Results: Faster, Smarter, and Less Cluttered

The results were pretty impressive. When they tested CRAFT against the old "backpack of bricks" method:

Speed: The new system reduced the amount of text (tokens) the AI had to read by about 9 times. That's a massive weight loss for the AI's brain.
Smarts: The overall "Agent Score" (a measure of how well it did the job) went up by 9.6 percentage points.
Consistency: The AI became 4.1 percentage points more consistent, meaning it gave the same good answer even if you asked the same question twice.
Less Confusion: It reduced the number of times the AI had to "search" for the right tool by up to 5 times.

The researchers also found that the AI got much better at handling multi-turn conversations. If a user said, "Show me the sales," and then followed up with, "Now filter for only the red shirts," the CRAFT AI remembered the context and kept the thread going, whereas the old system often got lost.

What It's Not (And What to Watch Out For)

It's important to note what this paper doesn't claim. The authors are very clear that this system is designed for stable, known schemas. If the database suddenly invents a completely new type of data or a brand-new tool that the AI has never seen, CRAFT might get stuck. It's like a chef who knows the kitchen perfectly but doesn't know how to use a brand-new, weird gadget they've never seen before. In those cases, you might still need to look up the rules (retrieval) or retrain the chef.

Also, the paper admits that while the AI is great at following the plan, it doesn't yet fully optimize for things like how much computer memory it uses or how much it costs to run on the server. That's a job for future upgrades.

The Bottom Line

CRAFT suggests that instead of constantly shoving information into an AI's face, we can teach it to learn the rules once and then trust it to execute the plan. By combining a "no-cheat-sheet" training phase with a "practice-and-get-corrected" phase, they built an agent that is lighter, faster, and more reliable. It's a step toward making AI assistants that feel less like robots reading a manual and more like expert colleagues who just know how the business works.

Technical Summary: CRAFT – Learn the Schema, Execute the Plan

Problem Statement

Enterprise coding agents are increasingly used to translate natural-language analytical requests into executable code over proprietary APIs and schemas. However, the prevailing deployment pattern relies on "context stuffing," where exhaustive schema definitions (DDL), API documentation, and tool instructions are injected into every prompt. This approach presents three critical challenges for production analytics:

Inference Overhead: Long schema prompts compete with user turns and intermediate results for working memory, increasing token costs significantly.
Schema Evolution Fragility: Physical schema changes (e.g., column renames, table migrations) necessitate frequent updates to prompt pipelines and regression testing.
Multi-Turn Reliability: The constant re-injection of static context undermines consistency and coherence in multi-turn analytical dialogues, where the agent must retain constraints and advance the analysis rather than treating each turn independently.

The authors investigate whether stable schema knowledge and tool-use behaviors can be acquired through post-training rather than transient prompt context, thereby preserving the consistency required for production-facing analytics.

Methodology

The paper proposes CRAFT, a two-stage post-training recipe designed for schema-grounded coding agents. The system is evaluated within a controlled enterprise environment using beta APIs and schema-linked advertising data, avoiding exposure to customer-sensitive production data.

1. Trajectory Distillation and Filtering

To address the lack of human ground truth for long-horizon, multi-tool code generation, the authors treat trajectory generation as a verifiable search problem. Teacher models generate candidate trajectories using a three-block format: a reasoning scratchpad, a business-level <PLAN> block (explaining entities/metrics without raw DDL), and an executable <CODE> block.

Tri-Gate Filter: A three-stage rejection filter curates the training data from over 100K generated trajectories down to 15.1K accepted trajectories:
- Gate 1: Rejects execution failures, syntax errors, and timeouts.
- Gate 2: Rejects empty or incomplete outputs.
- Gate 3: Uses an LLM-judge ensemble to detect semantic mismatches between the prompt, plan, and code.

2. Stage I: Schema-Stripped PLAN Supervised Fine-Tuning (SFT)

The first stage trains a 120B MoE base model to learn domain-structured plans and executable behaviors without exhaustive prompt-time schema injection.

Schema Stripping: The 50K+ token schema context is removed from the input. The model must rely on internalized domain structure.
Loss Masking: Tool outputs are masked from the loss function to prevent the model from memorizing unpredictable runtime values (identifiers, timestamps, dataframe contents).
Objective: To provide domain familiarity and teach common entity relationships and code idioms, serving as a necessary initialization for the agent.

3. Stage II: Execution-Shaped Reinforcement Learning (RL)

The second stage applies Group Relative Policy Optimization (GRPO) to align the agent's policy with execution feedback.

Reward Function: The total reward ( $R_{total}$ $R_{t o t a l}$ ) is a weighted combination of:
- $R_{outcome}$ : Execution success and unit test correctness.
- $R_{process}$ : Tool-call structure, code quality, and avoidance of anti-patterns (e.g., redundant joins).
- $R_{consistency}$ : Agreement between the stated plan and generated code.
- $R_{judge}$ : LLM-judge assessment for business alignment.
Infrastructure: The system utilizes extensions to VERL and SLIME to support asynchronous multi-turn rollouts, propagating step-level tool rewards and handling stateful tool-response turns.

Key Contributions

Planned-Rollout Enterprise System: A description of a deployment setting for advertising analytics covering 25 schema-linked entities and 30 agentic workflows, evaluated against beta APIs.
Schema-Stripped Post-Training Recipe: A novel combination of PLAN SFT and execution-shaped RL that reduces reliance on exhaustive prompt-time schema injection.
Production-Oriented Evaluation: A comprehensive evaluation measuring task quality, consistency, multi-turn interaction, and efficiency, reporting confidential deltas over a schema-stuffed baseline.
Deployment Lessons: Documentation of design tradeoffs, failure modes, and infrastructure extensions (VERL/SLIME) required for multi-turn tool-use RL in enterprise settings.

Results

Evaluated against a schema-stuffed baseline, CRAFT demonstrates significant improvements in quality and efficiency:

Performance Metrics:
- Agent Score: Improved by +9.6 percentage points (pp).
- Consistency: Improved by +4.1 pp.
- Multi-turn Coherence: Improved by +4.2 pp.
Efficiency Metrics:
- Input Token Burden: Reduced by approximately 9x (0.11x of the baseline) due to schema stripping.
- Schema-Discovery Loops: Reduced by up to 5x (0.20x of the baseline), indicating the agent requires fewer exploration rounds to find valid execution paths.
Component Analysis: Improvements were distributed across execution success, tool-use correctness, plan-code alignment, and final-answer accuracy, rather than being concentrated in a single metric.
General Capabilities: Public benchmarks (IFEval, GSM8K, GPQA) showed near-zero deltas, indicating no material regression in general reasoning capabilities.

Significance and Claims

The paper claims that CRAFT offers a viable path for deploying reliable coding agents in enterprise environments where schemas are proprietary and stable. By moving stable schema knowledge from the prompt context into the model weights via post-training, the system achieves:

Scalability: Drastic reduction in token overhead, making multi-turn interactions economically feasible.
Robustness: Enhanced ability to handle multi-turn refinements and recover from execution failures without re-injecting massive context.
Maintainability: Decoupling the agent's core reasoning from specific schema versions, reducing the need for prompt-pipeline updates during schema evolution.

The authors modestly position CRAFT as a "post-training recipe for coding agents operating over stable proprietary schemas in a planned-rollout setting," explicitly noting it is not a replacement for retrieval or in-context tool grounding in arbitrary open-world domains. The work emphasizes that while SFT provides necessary domain familiarity, execution-shaped RL is critical for aligning the agent's interactive decision policy regarding tool selection and ambiguity resolution.