🤖 AI

Compile, Then Page: Executable SOP Programs and a Capability-Gated Runtime for Procedural LLM Agents

This paper introduces a capability-gated runtime system that compiles standard operating procedures into executable pseudo-code and uses program-guided paging to significantly enhance the safety and performance of procedural LLM agents, particularly for models with strong state discipline.

Original authors: Chenglin Yu, Li Yin, Ying Yu, Hongxia Yang, Ming Li

Published 2026-07-14

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Chenglin Yu, Li Yin, Ying Yu, Hongxia Yang, Ming Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you're trying to teach a super-smart but slightly scatterbrained robot how to run a bank. You have a massive, 10,000-character rulebook (the Standard Operating Procedure, or SOP) that tells the robot exactly what to do, what to check, and when to say "No, absolutely not."

The old way of doing this was like shoving that entire 10,000-character rulebook into the robot's brain every single time it tried to answer a question. The robot would get overwhelmed, miss a crucial step, or accidentally approve a loan it shouldn't have. It's like trying to read a whole encyclopedia while someone is shouting a question at you; you might miss a page.

This paper suggests a smarter way: Compile, Then Page.

The "Recipe Book" vs. The "Chef's Station"

The authors realized that instead of giving the robot the whole messy text, they could first compile the rulebook into a clean, executable program. Think of this like turning a long, confusing paragraph of instructions into a precise, step-by-step recipe card.

But here's the magic trick: they don't show the robot the whole recipe at once. They use a system called paging. Imagine a chef who only puts the current step of the recipe on the counter (the "active frame"), while the rest of the book stays safely on a shelf. The robot only sees the 570 characters it needs for the next move, not the whole 10,000-character mess. This keeps the robot focused and prevents it from getting lost in the details.

The "Gated" Surprise: It Depends on the Robot

Here is where things get really interesting. The paper tested this on six different AI models, ranging from "super-smart" to "struggling."

For the Smart Robots: The new method was a huge win. On a banking task, the best model went from getting 70.4% of things right to 92.8%. Even better, it got 100% of the "refusal" tasks right (meaning it correctly said "no" when it wasn't supposed to do something). The smart robots loved having the recipe card and the focused counter.
For the Struggling Robots: The results were the opposite. The weaker models actually got worse. One model dropped from 60.8% to 48.8% right answers.

Why? The paper argues it's not because the weak robots can't "read" the recipe. When the researchers tested if the robots could reconstruct the state of the program, they could! The problem is discipline. The smart robots naturally follow the steps on the counter and stop when they're done. The weak robots, however, tend to keep checking things even after the job is finished, or they get confused by the structure. The "gate" isn't about intelligence; it's about self-control.

What This Rules Out

The authors were very careful to prove what didn't work:

It's not just about the format: They tried giving the smart robots the same information but in "code" format versus "prose" format. The code format helped the smart robots, but it actually hurt the weak ones. So, just switching to code isn't a magic fix for everyone.
It's not about hiding information: They tested if the robots failed because they couldn't see the whole program. They found that even if they showed the whole program to the smart robots, the "paging" (showing just the active step) still helped. The weak robots, however, couldn't be fixed by just showing them the whole thing; they still struggled with the discipline of following the steps.
It's not a "hard" block: The system doesn't physically stop the robot from making a mistake. It just highlights the right path. If the robot decides to ignore the path, it can still do it, but the system will record the mistake in an "audit trail." This means the system relies on the robot choosing to be good, which is why it fails with the undisciplined ones.

The Bottom Line

The paper concludes with a simple rule for anyone building these agents: Compile first, then page.

Compile: Turn your messy rulebook into a clean, executable program. This helps almost everyone, even if it doesn't change the weak robots much.
Check the Model: Before you turn on the "paging" feature (showing only the current step), you must check if your robot has the discipline to follow it.
Page Only if Ready: If your robot is smart and disciplined, paging will boost its performance significantly. If your robot is weak or undisciplined, paging might actually make it mess up more.

In short, giving a robot a focused view of the task is a powerful tool, but it only works if the robot has the self-control to look at the right thing at the right time. For the smart ones, it's a superpower; for the others, it's a trap.

Technical Summary: Compile, Then Page: Executable SOP Programs and a Capability-Gated Runtime for Procedural LLM Agents

1. Problem Statement

Deploying language agents in enterprise environments (e.g., banking, clinics) requires strict adherence to Standard Operating Procedures (SOPs). These SOPs are characterized by long-horizon workflows, conditional branches, and safety-critical refusal rules. Current dominant practices treat SOPs as unstructured text prompts ("resident prose"), which competes with the dialogue for attention and strips away executable structure. This leads to specific failure modes: skipped verification checks, premature actions, and incorrect refusals. On the SOPBench benchmark, even top configurations fail to meet compliance standards, with error rates unacceptable for compliance offices.

The core challenge is to transform SOPs from static text into executable logic that an LLM can follow without losing the flexibility required for dynamic dialog, while ensuring that safety constraints are enforced without hard-blocking the model (which can cause unrecoverable errors when user goals shift).

2. Methodology

The authors propose a two-stage architecture that separates the representation of the SOP from its runtime execution.

2.1 Offline Compilation (SOP-to-Program)

Instead of feeding the SOP as natural language, an offline, deterministic compiler (LLM-free) translates machine-readable dependency constraints into a two-layer pseudo-code program:

Process Functions: One function per user goal.
Rule Subroutines: Each constraint is paired with a specific verification recipe and an evidence-bearing return value {holds, evidence}.
Key Transformations:
- Stateful Subroutines: Multi-turn constraints persist across turns.
- Gate Hoisting: Authentication gates required for verification are moved before guarded checks.
- Evidence Discipline: Every verdict is tied to specific tool evidence; no verdict is issued without it.
- Frame Factoring: Large groups are modularized into named functions.
Output: A structured program where a false required result branches to a fail state before the goal action is executed.

2.2 Online Program-Guided (PG) Runtime

The compiled program is executed by a stack-based virtual machine that manages the context window:

Stack Paging: The runtime maintains a call stack, cursor, and variable store. It pages only the active frame (approx. 570 characters) into the LLM's context, rather than the full program (approx. 10,000 characters).
Execution Flow: The LLM acts as the semantic executor. It converses, selects tool arguments, interprets evidence, and reports verdicts. The symbolic runtime handles bookkeeping (stack transitions, cursor movement, variable binding).
Soft Enforcement: The runtime manages attention, not permissions. Deviations are possible but observable in the stack audit trail. This flexibility allows for goal changes but relies on the model's ability to faithfully execute the local frame.

2.3 Experimental Design

The study uses a three-arm design on the SOPBench (Bank domain and six others) across six models of varying capability:

Offtext: Official verbalized SOP text (baseline).
Flat: Full compiled program text resident in every turn (isolates representation).
PG: Compiled program executed with the stack-paged runtime (isolates runtime).
Full-Cursor (Ablation): Active frame first, but the full program retained below (isolates active-frame salience vs. global visibility).

3. Key Contributions

SOP-to-Program Compiler & Stack-Paged Runtime: A system that makes procedures executable and auditable, reducing context load from ~10k to ~570 characters per turn while preserving logical structure.
Capability-Gated Attribution Study: A six-model, seven-domain analysis demonstrating that the benefits of this approach are strictly capability-dependent.
Mechanism Disentanglement: Separation of generic bookkeeping, active-frame salience, and selective visibility, alongside diagnostics for state reconstruction and attention density.
Deployment Rule: A practical guideline: "Compile first; enable active-frame paging only after a model-level discipline check."

4. Results

4.1 Representation vs. Runtime

Compilation (Flat vs. Offtext): The compiled representation never significantly hurts performance. It yields significant gains (+16.0 points on Bank) where official prose underperforms.
Runtime (PG vs. Flat): The runtime effect is capability-gated.
- Strong Models: Two strong models (DeepSeek-V4-Flash and Qwen3.7-Plus) show independent, positive gains across seven domains (e.g., 58:19 and 75:31 discordant pairs in favor of PG). On the Bank domain, DeepSeek-V4-Flash improved from 86.4% (Flat) to 92.8% (PG) with 100% refusal correctness.
- Weak Models: Weak models (e.g., GPT-4o-mini, Qwen2.5-7B) are significantly harmed by the runtime, losing 14–26 points on Bank.

4.2 Mechanism Analysis

Active-Frame Salience: An ablation (Full-Cursor) that retains the full program but highlights the active frame recovers most of the strong-model refusal gain. This suggests the primary benefit is making the current instruction salient, not hiding global context.
State Discipline vs. Reconstruction:
- Reconstruction: All models, including those harmed by the runtime, can reconstruct state when explicitly probed (near-ceiling scores).
- Discipline: The "gate" is spontaneous state discipline. Weak models fail because they over-verify or continue checks after the goal is achieved (positive $\Delta R^2$ in post-goal calls), whereas strong models naturally stop.
- Attention: Strong models allocate attention to the active frame proportional to its size; weak models under-allocate attention to the active frame when it is part of a larger context.

4.3 Content vs. Format

Format: Re-syntactizing official text into code format helps strong models (+14.4 points) but harms weak models (-12.8 points).
Content: Adding verification recipes and evidence discipline helps strong models significantly but shows no significant gain for weak models.
Conclusion: The "code" format is not universally beneficial; it is a tool for models that already possess the discipline to follow structured instructions.

5. Significance and Claims

The paper claims that the primary barrier to reliable SOP-following agents is not the lack of information, but the representation and the model's spontaneous discipline in executing that representation.

Rejection of "More Scaffolding is Better": The results show that adding complex runtime guidance (paging) to weak models degrades performance.
The Capability Gate: The effectiveness of the PG runtime is strictly bounded by the model's ability to maintain state discipline. The system does not "fix" weak models; it amplifies the performance of strong ones while exposing the limitations of weak ones.
Practical Guidance:
1. Compile first: Convert SOPs to executable pseudo-code to improve structure and reduce context noise.
2. Check discipline: Only enable active-frame paging (runtime guidance) after verifying the model can follow instructions without the runtime's structural aid.
3. Soft Enforcement: The system relies on observability (audit trails) rather than hard blocking, preserving flexibility for dynamic user interactions.

The authors conclude that while their approach achieves 92.8% pass rates and 100% refusal correctness on the Bank domain for strong models, it is not a universal solution. It requires a "model-level discipline check" before deployment to avoid harming weaker models.