Technical Summary: LACUNA: Safe Agents as Recursive Program Holes

Problem Statement

Current Large Language Model (LLM) agents face a fundamental architectural split between the runtime that drives the agent loop and the code the model writes. In dominant approaches like ReAct, the runtime owns the control flow, context management, and tool dispatch, while the model merely emits individual tool calls or reasoning steps. In "code-as-action" agents, the model writes code to compose tools and logic, but this code remains distinct from the runtime.

This separation creates a safety dilemma. If the runtime strictly controls the loop, the agent lacks expressiveness. If the model shapes the runtime itself to increase expressiveness, safety risks escalate: a model diverted by prompt injection, a tool call error, or a partial failure can leave the environment in an inconsistent state or compromise the runtime. Existing defenses (sandboxes, restricted interpreters, policy languages) are often piecemeal, operating at runtime or on inputs, and none verify a whole generated action against the surrounding program context before execution begins.

Methodology: LACUNA

The authors propose LACUNA, a programming model that closes the split between agent runtime and model-written code while preserving safety through static typing. The core primitive is a typed "hole" in the host program:

def agent[T](task: String): T

Core Mechanism

1. Typed Holes: An agent call acts as a placeholder where the surrounding program statically expects a value of type T. The LLM is prompted to generate Scala code that produces this value.
2. Context-Aware Compilation: When execution reaches the agent call, the LLM generates code based on the natural language task, the expected type T, and the live lexical scope (variables, functions, tools, and imports available at that specific point).
3. Pre-Execution Verification: The generated code is compiled by the host compiler (Scala 3) in the context of the surrounding program before it ever runs.
   • Type Checking: The code must produce a value of type T.
   • Scope Checking: All names used must be in scope.
   • Capability Checking: If enabled, capture checking ensures the code only accesses resources (files, network, tools) explicitly granted in the lexical scope.
4. All-or-Nothing Execution: If the code passes all checks, it executes. If it fails (e.g., type mismatch, missing tool, unauthorized access), the entire snippet is rejected. No partial execution occurs, preventing inconsistent states. Compiler diagnostics are fed back to the LLM to drive a retry loop.

Implementation

The prototype is built in Scala 3, leveraging two specific features:

• In-process recompilation: The ability to invoke the compiler within a running program to splice and compile a string of source code against the current context.
• Capture Checking: A mechanism to track which effects and resources a code snippet is allowed to use, enforcing capability-based security.

The agent primitive is a thin wrapper around a lower-level eval operation. It manages the prompt construction (including the enclosing source and available bindings), calls the LLM, passes the result to eval, and handles the retry loop upon compilation failure.

Key Contributions

1. A Code-as-Action Model with Static Guarantees: A model where the LLM writes actions as code that runs as part of its own runtime, verified at the call site before execution.
2. Safety Analysis: Demonstrates that pre-execution rejection of unavailable names and type mismatches, combined with atomic execution (no partial runs), provides strong safety guarantees. It further shows how capability scoping bounds the agent's authority and information flow.
3. Expressiveness via Ordinary Control Flow: Shows that nested agent calls and standard code constructs can express complex agent patterns (ReAct loops, sub-agents, skills, parallel decomposition, multi-model planning) without needing a separate agent protocol.
4. Empirical Evaluation: A Scala 3 realization evaluated on verifier test cases, BrowseComp-Plus (hard information-seeking), and $\tau^2$-bench (multi-turn tool-using conversations).

Results

The evaluation highlights the trade-off between safety, expressiveness, and model capability:

• Safety and Rejection Rates: On BrowseComp-Plus, 8.6% of generations were rejected before execution due to type or scope errors. The system averaged 0.7 retries per query, with a 91.4% end-to-end compile success rate. This confirms that the compiler effectively blocks ill-typed or out-of-scope actions.
• Performance on Complex Tasks:
  • BrowseComp-Plus: Using the deepseek-flash model, LACUNA achieved 27.1% accuracy, comparable to baseline agents, while driving genuine multi-step research (avg. 5.9 rounds, 15.5 searches).
  • $\tau^2$-bench: Across 392 tasks in four domains (Retail, Airline, Telecom, Tel.-WF), LACUNA solved 76.0% of tasks with a capable model, performing on par with a reference tool-calling agent.
• Model Dependency: The system's success is heavily dependent on the model's coding ability. Weaker models (e.g., gemini-3.1-flash-lite) struggled with complex domains (e.g., 89% rejection rate in telecom), leading to lower solve rates. The safety guarantee holds regardless of model competence (broken attempts never run), but progress halts if the model cannot generate valid code.
• Adversarial Robustness: In prompt injection tests (AgentDojo), the capability layer successfully confined the agent's actions to its granted scope. Even when influenced by injected content, the agent could not access resources outside its lexical scope.

Significance and Claims

The paper positions LACUNA not as a replacement for existing agent architectures, but as a safer, more flexible foundation.

• Unification of Execution and Reasoning: By fusing program execution and model reasoning into a single process where the model writes the control flow, LACUNA allows agents to be more expressive while maintaining the safety guarantees of the host language.
• Safety via Compilation: The primary contribution is shifting safety from runtime sandboxes to pre-execution static verification. A rejected snippet leaves the environment untouched, and the compiler's diagnostics drive self-correction.
• Capability Safety: The system demonstrates that "least authority" can be enforced through lexical scoping and capture checking, bounding what an adversarial or erroneous model can do even if it is influenced by prompt injection.
• Modest Limitations: The authors acknowledge that "well-typed" does not mean "correct" (semantic errors are not caught), that the system relies on the model's coding proficiency (which varies by language and domain), and that resource exhaustion (latency, infinite recursion) must be managed via runtime budgets rather than type systems.

In summary, LACUNA proposes that treating agent actions as typed holes in a statically typed host language allows for a recursive, expressive agent design where safety is guaranteed by the compiler's soundness rather than runtime heuristics.