💻 computer science

PlanCompiler: A Deterministic Compilation Architecture for Structured Multi-Step LLM Pipelines

The paper introduces PlanCompiler, a deterministic compilation architecture that separates planning from execution via a typed node registry and static validation to generate executable Python code, achieving significantly higher reliability, lower costs, and competitive latency than free-form code generation baselines on structured multi-step LLM workflows.

Original authors: Pranav Harikumar

Published 2026-04-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Pranav Harikumar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a complex machine, like a coffee maker that also sorts your mail and writes a thank-you note.

If you ask a very smart but slightly chaotic assistant (a standard Large Language Model, or LLM) to "just build this," they might do something like this:

They write a recipe for the coffee part.
They write a recipe for the mail part.
They write a recipe for the note part.

The Problem: The assistant is great at writing individual recipes, but they are terrible at making sure the recipes fit together. Maybe the coffee recipe says "pour hot water," but the mail recipe expects "cold water." Maybe they use different names for the same cup. Maybe they invent a new type of screw that doesn't exist. When you try to run the whole machine, it breaks because the pieces don't fit, and the errors pile up until the whole thing crashes.

The Solution: PlanCompiler
The paper introduces PlanCompiler, which acts less like a "creative writer" and more like a strict construction foreman or a compiler (a tool that turns blueprints into a working machine).

Here is how it works, using a simple analogy:

1. The "Lego Box" (The Registry)

Instead of letting the AI invent new tools or materials, PlanCompiler gives it a fixed box of Lego bricks.

You can only use the bricks in the box (e.g., "Red Block," "Blue Block," "Connector").
The AI cannot invent a "Flying Brick" or a "Magic Glue."
Every brick has a specific shape and a specific job.

2. The "Architect" (The Planner)

The AI's only job is to be the Architect. It looks at your request ("Build a coffee mail sorter") and says:

"Okay, I will use a Red Block, then a Blue Block, then a Connector. I will connect them in this order."

It doesn't write the code for how the blocks work; it just decides which blocks to use and how to snap them together.

3. The "Safety Inspector" (The Validator)

Before anyone starts building, a Safety Inspector checks the Architect's blueprint. This is the magic part. The inspector runs seven strict checks:

"Did you use a brick that isn't in the box?" (No.)
"Do the shapes of the bricks match where you connected them?" (Yes.)
"Is the order logical, or did you create a loop where the machine chases its own tail?" (No loops.)
"Did you forget to add a necessary screw?" (No.)

If the blueprint fails any of these checks, the project is rejected immediately. No building happens. This stops the "silently broken machine" problem.

4. The "Factory" (The Compiler)

Once the blueprint passes inspection, a Factory Robot takes over. It doesn't ask the AI for help anymore. It simply looks at the approved blueprint and snaps the pre-made Lego pieces together exactly as drawn. Because the pieces are pre-made and the connections are strict, the machine is guaranteed to be built correctly.

Why is this better? (The Results)

The paper tested this against standard AI methods (letting the AI write everything from scratch) on 300 different tasks.

Standard AI (The Chaotic Writer): It was fast to start but often failed. It made up fake tools, used the wrong names, or got confused halfway through. It succeeded about 67% of the time.
PlanCompiler (The Foreman): It was slightly slower to plan, but because it checked the blueprint first, it succeeded 93% of the time.

The Cost:

The standard AI was expensive because it kept trying to write huge, complex code that often failed, wasting money on every attempt.
PlanCompiler was 76 times cheaper per successful task. It spent a little bit of money planning carefully, which saved a massive amount of money on failed attempts.

The "Glitches" That Remain

Even with this strict system, the AI still made mistakes in two specific ways:

The "Loophole" Trick: Sometimes the AI would try to do a complex math task (like counting items) by sneaking a raw SQL query (a database command) into a part of the machine that was supposed to be a simple block. It worked, but the result didn't look exactly like the boss wanted.
The "Confused Label" Error: Sometimes the AI got confused about what a "database handle" was versus a "file path," like confusing a key with a door. The Safety Inspector caught most of these, but a few slipped through.

The Big Takeaway

The paper argues that for complex, multi-step tasks, we shouldn't just ask AI to "be creative." Instead, we should force it to work within a strict, pre-defined system where it plans the steps, a computer checks the plan for errors, and then a machine builds it.

It turns "hoping the AI gets it right" into "guaranteeing the AI gets it right by checking the math first." It's the difference between asking a friend to "fix the car" and giving a mechanic a specific checklist of parts to swap out.

1. Problem Statement

Large Language Models (LLMs) exhibit significant brittleness when tasked with structured, multi-step workflows (e.g., data ingestion $\rightarrow$ transformation $\rightarrow$ SQL persistence $\rightarrow$ export).

The Core Issue: In free-form code generation, errors compound across sequential steps. A hallucinated import, a mismatched column name, or a naming drift in step two propagates through the pipeline, causing silent failures or runtime crashes in subsequent steps.
Limitations of Existing Approaches:
- Tool-orchestration frameworks (e.g., LLMCompiler) focus on scheduling and parallelism but often accept execution failures as a runtime concern rather than preventing them structurally.
- NL-to-pipeline systems generate intermediate representations but often allow the model too much freedom in defining the logic, leaving room for semantic drift.
The Gap: There is a lack of a system that enforces a hard structural correctness boundary before execution, ensuring that the generated plan is valid by construction rather than relying on post-hoc repair or prompting tricks.

2. Methodology: PlanCompiler Architecture

PlanCompiler introduces a deterministic compilation architecture that strictly separates planning (LLM-driven) from execution (deterministic code generation). The system operates on a typed node registry and follows a five-stage pipeline:

A. System Components

Typed Node Registry: A fixed set of callable primitives (nodes) defined as Pydantic models. Each node has fixed input_type, output_type, required_params, and a pre-written template_path (Python file). The LLM cannot invent new primitives; it can only select from this registry.
Planner (LLM): The only generative component. It receives a natural language task and the registry description. It outputs a JSON plan containing:
- nodes: Ordered list of selected node names.
- edges: Directed pairs defining data flow.
- parameters: Key-value pairs for node arguments.
- Note: The LLM does not write execution logic; it only selects nodes and parameters.
Static Validator: A deterministic layer that runs seven structural checks on the JSON plan before any code is generated:
- Node existence (against registry).
- Edge validity (referenced nodes exist).
- Type compatibility (input/output types match, with ANY wildcard support).
- Acyclicity (topological sort via Kahn's algorithm).
- No orphans (all nodes connected).
- Input arity (single-stream pipeline constraint).
- Required parameter presence.
- Result: If any check fails, the plan is rejected immediately. No code is compiled.
Compiler: If validation passes, the compiler assembles an executable Python script (app.py) by:
- Performing a topological sort to determine execution order.
- Reading pre-written template files verbatim for each node.
- Injecting planner-supplied parameters using repr() (treating them as Python literals).
- Generating a deterministic execution block that threads data between nodes.
Execution: The compiled app.py is run in a single subprocess. There is no runtime LLM intervention, no repair loops, and no autoregressive chaining.

B. Design Philosophy

Determinism: Given a valid plan and input fixtures, the compiler always produces identical code.
Separation of Concerns: The LLM is restricted to selection and parameterization. Logic, wiring, and code generation are handled by the deterministic compiler.
First-Pass Reliability: The system aims for success on the first attempt without iterative correction.

3. Key Contributions

Typed Intermediate Representation (IR): A system that eliminates hallucinated imports and naming drifts by construction. The model cannot generate arbitrary code; it must map to a closed registry.
Reliability-Focused Evaluation Protocol: A benchmark of 300 tasks across six sets (varying depth, schema traps, and SQL roundtrips) with explicit success criteria and a structured failure taxonomy.
Failure Characterization: Identification of two specific, interpretable residual failure modes in constrained systems:
- Late Semantic-Output Failures: "Constraint evasion" where the model routes aggregation through an unconstrained SQL surface (QueryEngine) instead of the constrained Aggregator node, missing output contracts.
- Early Persistence-Boundary Failures: Type mismatches at the SQLite boundary where the planner confuses database handles, file paths, and materialized tables.

4. Experimental Results

The system was evaluated against GPT-4.1 and Claude Sonnet 4.6 (free-form code generation baselines) across 300 tasks.

A. Accuracy & Reliability

PlanCompiler: Achieved 92.67% overall first-pass success rate.
- Perfect (100%) on shallow (Set A) and medium (Set B) pipelines.
- 88% on long pipelines (Set C) and 96% on maximum complexity (Set D).
- 88% on schema traps (Set E) and 84% on SQL roundtrips (Set F).
Baselines:
- GPT-4.1: 67% success rate.
- Claude Sonnet 4.6: 62% success rate.
Improvement: PlanCompiler outperformed GPT-4.1 by +25.67 percentage points and Claude by +30.33 percentage points.

B. Cost Efficiency

Total Cost: PlanCompiler cost $0.356 for the entire 300-task suite.
- GPT-4.1 cost: $2.140.
- Claude cost: $18.391.
Cost per Success: PlanCompiler is 8.27x cheaper than GPT-4.1 and 76.75x cheaper than Claude per successful task. This is due to the use of a smaller model (GPT-4o-mini) for planning and the elimination of redundant token generation seen in free-form baselines.

C. Latency

PlanCompiler latency scales linearly with pipeline depth (5.79s for Set A to 12.05s for Set D).
While slightly slower than GPT-4.1 on average (9.37s vs 5.05s), it is significantly faster than Claude (16.15s) and offers reproducible results, whereas baselines showed multi-run variance even at temperature=0.

D. Failure Analysis

PlanCompiler Failures (22 total): Highly localized.
- 59% were "QueryEngine Evasion" (semantic routing errors).
- 41% were "DBHandle Type-Confusion" (structural validation errors).
- Significance: Failures are narrow, interpretable, and architectural, not random hallucinations.
Baseline Failures: Distributed across 5+ error types (column naming drift, schema hallucinations, deprecated APIs, import errors, structural inconsistency), making them harder to fix systematically.

5. Significance and Implications

Reliability as a Compiler Problem: The paper argues that for structured multi-step tasks, reliability should be treated as a compilation problem (enforcing structural constraints) rather than a prompting problem.
Shift in Failure Modes: The architecture successfully eliminates "noisy" errors (imports, naming drift) and localizes remaining errors to specific semantic boundaries. This makes failures interpretable and addressable (e.g., by tightening the SQL surface or clarifying type definitions).
Cost-Performance Trade-off: By offloading correctness to a deterministic validator and compiler, the system can achieve high reliability using a smaller, cheaper LLM for the planning phase, drastically reducing inference costs compared to relying on massive models to generate correct code end-to-end.
Future Directions: The authors suggest that future work should focus on tightening the "semantically open" surfaces (like raw SQL strings) and extending the architecture to support branching control flows and automatic repair loops.

In conclusion, PlanCompiler demonstrates that constraining LLMs to a typed registry and enforcing strict structural validation before compilation yields significantly higher reliability, lower costs, and more predictable behavior for structured data workflows compared to free-form code generation.