🤖 AI

Governed Metaprogramming for Intelligent Systems: Reclassifying Eval as a Governed Effect

This paper proposes "governed metaprogramming," a language design that reclassifies the unrestricted `eval` primitive into a governed effect requiring structural inspection and policy compliance before materializing symbolic forms into executable code, thereby ensuring safety and authority control in self-modifying AI systems.

Original authors: Alan L. McCann

Published 2026-05-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Alan L. McCann

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart robot assistant that can write its own instructions. It can look at a problem, think about it, and then type out a new set of rules for itself to follow.

In the old world of programming, if this robot wrote a new set of rules, the computer would just say, "Okay, I'll run them right now." It wouldn't ask, "Wait, are these rules safe? Do they try to steal your bank password? Do they try to delete your files?" The computer treated the act of turning written words into action as a magic button that always worked.

This paper argues that for AI systems that can rewrite themselves, that "magic button" is dangerous. The author, Alan L. McCann, proposes a new way to handle this called Governed Metaprogramming.

Here is the core idea, broken down with simple analogies:

1. The Problem: The "Magic Button" is Broken

Think of a computer program like a recipe.

The Code: The written recipe (the ingredients and steps).
The Execution: The act of cooking the meal.

In traditional systems, if you write a new recipe on a piece of paper, you can immediately hand it to the chef, and the chef starts cooking. The paper-to-cooking transition is instant and unrestricted.

But in modern AI, the "chef" (the AI) can write new recipes while it is cooking. If the AI writes a recipe that says "Eat the whole kitchen," and the system just lets it cook immediately, disaster happens. The paper calls this transition "Authority Amplification." It's the moment a piece of paper (data) suddenly gets the power to change the real world (execute).

2. The Solution: The "Security Guard" for Recipes

The paper suggests we stop treating the act of turning a recipe into a meal as a simple computer function. Instead, we treat it as a governed effect—a special action that requires a security guard's approval.

The system introduces a new concept called Machine Forms.

Machine Forms are like blueprints or diagrams of the recipe. They are just data. They can't cook anything. They can't call a phone number. They are just pictures of instructions.
Manipulating Blueprints (drawing on them, erasing lines, combining two blueprints) is safe. It's like a child playing with Legos. No real-world damage can happen just by moving the plastic bricks around.

3. The Critical Step: "Materialization"

The dangerous moment is called Materialization. This is when you take the blueprint and say, "Okay, build this."

In this new system, you cannot just say "Build this." You have to hand the blueprint to a Governance System (the Security Guard). The Guard does three things before letting the chef cook:

Inspects the Blueprint: Does this recipe try to use a forbidden ingredient (like a dangerous AI model)?
Checks the Rules: Does this recipe fit within the budget? Does it try to access files it shouldn't?
Decides: If it passes, the Guard gives the "Go" signal. If it fails, the blueprint is thrown in the trash.

4. Why This Matters for Self-Improving AI

Imagine an AI that wants to get smarter. It looks at its own code, realizes it's bad at math, and writes a new version of itself that is better at math.

Without this system: The AI writes the new code, hits "Run," and suddenly the new code has full power. If the AI made a mistake or was tricked, it could accidentally give itself the power to delete everything.
With this system: The AI writes the new code (the blueprint). It then has to ask the Security Guard, "Can I become this new version?" The Guard checks the new code. If the new code tries to do something the AI isn't allowed to do (like access a secret database), the Guard says "No." The AI cannot bypass the guard, even if it wrote the code itself.

The Big Takeaway

The paper claims that eval (the command that turns code into action) is not just a tool; it is a power grant.

By reclassifying it as a "governed effect," the system ensures that:

Writing code (manipulating blueprints) is safe and pure.
Running code (turning blueprints into action) always goes through a checkpoint.
No shortcuts exist. You cannot sneak a dangerous program past the guard by hiding it inside a normal calculation.

The author implemented this in a system called MashinTalk and proved mathematically (using 454 formal theorems) that it is impossible to bypass this security guard. It creates a world where AI can build new versions of itself, but only if those new versions pass a strict safety inspection first.

Technical Summary: Governed Metaprogramming for Intelligent Systems

1. Problem Statement

Modern AI systems, including Large Language Models (LLMs) and autonomous agents, increasingly synthesize executable structures at runtime. These systems generate workflows, tool-calling plans, and self-modifying code. In classical homoiconic (e.g., Lisp) and staged languages (e.g., MetaOCaml), the transition from a code representation (data) to an executable program is handled by primitives like eval or run. These transitions are traditionally unrestricted: any valid representation can immediately become running code.

In the context of governed intelligent systems—which require explicit effect boundaries, capability control, and auditability—this unrestricted transition creates a critical vulnerability. If a system can generate a new program that bypasses the governance constraints of the original, the entire governance model is compromised. The paper identifies this as a "metaprogramming escape hatch": a system governed at the execution level but ungoverned at the code generation level is fundamentally insecure.

2. Core Methodology: Reclassifying `eval`

The central methodological contribution of the paper is the reclassification of eval (or the transition from form to execution) from a language primitive into a governed effect.

The authors propose a framework called Governed Metaprogramming, built on the following pillars:

Machine Forms: Program representations are treated as first-class values (structured data maps) rather than opaque strings or bytecode.
Pure Form Manipulation: All operations on these forms (inspection, transformation, composition, diffing) are classified as pure computation. They emit no directives and possess no capability to perform I/O or invoke models.
Materialization as Authority Allocation: The transition from a machine form to an executable machine is defined as materialization. This is not merely computation; it is the allocation of executable authority (the power to invoke models, perform I/O, and consume resources). Consequently, materialization is classified as a governed effect.
Structural Inspection: Before materialization is permitted, a governance system performs a decidable structural inspection of the form. This inspection analyzes capability requirements, policy compliance, model authorization, and resource estimates.

3. Formal Operational Semantics

The paper formalizes the system using two distinct judgments to enforce the boundary between pure computation and governed execution:

Pure Form Evaluation ( $\Gamma \vdash pure\ e \Downarrow v$ ):
- Operates on form values.
- Emits zero directives ( $Dir(e) = \emptyset$ ).
- Includes quote construction, splice evaluation, navigation, and transformation.
- Guarantees that no sequence of pure operations can produce an authorized machine.
Governed Materialization ( $\Gamma; \Pi; \sigma \vdash materialize(v_f) \rightsquigarrow \langle Machine, d \rangle$ ):
- Takes a form, a policy context ( $\Pi$ ), and a capability set ( $\sigma$ ).
- Requires a governance decision record ( $d$ ) for every outcome (approval or rejection).
- Emits exactly one governed directive.
- The governance system inspects the form's structure to verify:
  - Capability containment ( $CapSet(f) \subseteq \Pi.allowed\ caps$ ).
  - Model authorization.
  - Structural policy compliance.
  - Resource bounds.

4. Key Contributions

The paper presents five primary contributions:

Reclassification of eval: A precise characterization of materialization as authority allocation, moving it from a primitive to a governed effect.
Machine Forms: A concrete data type for structured program representation with constructs for quote, splice, and reflection.
Formal Semantics: A definition of the system using two judgments (pure evaluation and governed materialization) that strictly separates data manipulation from execution.
Machine-Checked Properties: Three core properties are formally verified in Rocq:
- Purity of Form Manipulation: Form operations emit no directives.
- The No-Bypass Theorem: No sequence of pure operations can directly cause execution.
- Boundary Preservation: Adding metaprogramming capabilities does not split the expressiveness and governance boundaries (they remain coterminous).
Governed Self-Modification: A concrete mechanism allowing machines to propose structural changes to themselves via an evolution ledger, subject to the same governance inspection as external code generation.

5. Implementation and Results

The design is implemented in MashinTalk, a domain-specific language for AI workflows that compiles to BEAM (Erlang VM) bytecode.

System Architecture: The implementation consists of three modules: Form (construction and manipulation), FormInspector (governance checks), and FormMaterializer (governed compilation).
Verification: The core properties are mechanized in Rocq (formerly Coq) with 26 theorems and 662 lines of code, featuring zero admitted lemmas. These theorems integrate with an existing formalization of 454 theorems across 36 modules.
Testing: A test suite of 51 unit tests validates functional correctness and governance properties. All tests pass.
Performance: On Apple M-series processors, form construction takes <1 µs, and structural inspection takes <100 µs for typical machines. The overhead is negligible compared to the compilation to BEAM bytecode.

6. Significance and Claims

The paper claims to close the "metaprogramming escape hatch" in governed intelligent systems. Without this mechanism, a system governed at the execution level remains vulnerable because it cannot govern the generation of new executable structures.

The authors argue that in AI systems synthesizing their own executable structure, the allocation of executable authority must be:

Visible: The transition from data to code is a distinct, observable event.
Mediated: It must pass through a governance interpreter that inspects the structure.
Recorded: Decisions are logged in an append-only ledger (the evolution ledger).

The paper explicitly acknowledges limitations:

Rice's Theorem: Structural inspection is decidable but cannot reason about semantic behavior. Governance guarantees mediation, not the semantic soundness of the generated program.
Scope: The boundary preservation argument is architectural; a full conservativity proof regarding effect signature sets is deferred to future work.
Applicability: The design is specific to languages with effect-boundary structures (like MashinTalk); its applicability to general-purpose languages is an open question.

In conclusion, the paper establishes that eval is not merely a computational function but an authority-allocation event. By treating it as a governed effect, intelligent systems can safely synthesize and modify their own executable structures without compromising their governance models.