🔢 mathematics

Harness Engineering as Categorical Architecture

This paper establishes categorical architecture as the formal theoretical foundation for LLM agent harness engineering by mapping the four pillars of agent externalization to the ArchAgents framework's (G, Know, Phi) triple, thereby enabling structural guarantees and cross-framework compilation verified through identity and replay rather than output-layer correctness.

Original authors: Bogdan Banu

Published 2026-05-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Bogdan Banu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: The "Harness" vs. The "Brain"

Imagine you have a brilliant, super-smart assistant (the AI Model). This assistant knows everything in the world, but it's a bit chaotic. It might forget what you asked five minutes ago, it might try to use tools it doesn't have, or it might get confused about the order of operations.

In the world of AI, the Model is the brain. But the Harness is everything else: the notebook where it writes things down (Memory), the toolbox it uses (Skills), the rules it follows to talk to you (Protocols), and the manager that tells it what to do next (Orchestration).

The paper argues that for a long time, engineers built these "managers" (harnesses) by guessing and trying things out (trial and error). They didn't have a formal rulebook to prove their manager would work reliably.

This paper says: "We have a math-based rulebook for building these managers, and we can prove it works."

The Three-Part Blueprint: The "Architecture Triple"

The authors introduce a mathematical framework called the Architecture Triple. Think of this as a blueprint for building a reliable AI manager. It has three parts:

The Wiring Diagram (G): This is the flowchart. It shows how information moves from one step to the next. Analogy: The plumbing in a house. It shows where the water (data) flows, but not what the water is.
The Rulebook (Know): This is the most important part. It lists the "structural guarantees" or promises the system makes. Analogy: The building codes. It promises things like "The roof will never leak" or "The fire escape will always be open," regardless of who lives in the house.
The Deployment Map (Φ): This is the instruction on which specific brain (AI model) to use for which job. Analogy: The staff roster. It says, "Use the junior chef for chopping vegetables, but the head chef for the main course."

The Four Pillars of the Manager

The paper connects this math blueprint to four real-world things engineers already build:

Memory: The system's ability to remember. In the math world, this is treated like a "state machine" that updates over time.
Skills: The tools the agent can use. In the math world, these are like Lego blocks that can be snapped together in specific ways (in a line, side-by-side, or in a loop).
Protocols: How the agent talks to itself or others. In the math world, this is the "wiring" that ensures the right type of message goes into the right slot.
The Harness: The whole system itself.

The Magic Trick: "Certificate Preservation"

The paper's biggest claim is about portability.

Imagine you build a complex machine (a harness) in a factory in Germany. You want to ship the blueprints to a factory in Japan to build the exact same machine. Usually, when you translate blueprints, you might accidentally lose a safety feature or change a gear ratio.

This paper claims that because they use this "Architecture Triple" math, they can translate the harness from one software framework to another (e.g., from LangGraph to Swarms) without losing the safety guarantees.

They call these guarantees "Certificates."

Example Certificate: "If the quality of the answer is too low, the system will automatically switch to a smarter, more expensive AI model."
The Test: When they translated the harness to a new framework, they didn't just check if the code ran. They checked if the Certificate still held true. They proved that the "safety switch" still worked, even though the underlying code looked different.

The Experiments: Did it actually work?

The authors didn't just talk about math; they built a prototype and ran tests.

1. The "Escalation" Test
They set up a task where a "fast but dumb" AI model tried to solve a problem.

The Setup: The fast model tried to write a code review.
The Rule: If the quality score was too low, the system had to "escalate" to a "slow but smart" model.
The Result: The fast model failed. The system checked the score, saw it was too low, and automatically switched to the smart model.
Why it matters: This proved that the rule (the harness) worked perfectly, even though the brain (the model) changed. The harness is in control, not the model.

2. The "Code Fixing" Test (SWE-bench)
They tried to use their system to fix bugs in real software (Python code).

The Result: They hit a wall. The AI models they used (which were small, local versions) were simply not smart enough to write the code correctly, no matter how good the harness was.
The Lesson: A great harness can't fix a broken brain. If the AI model is too small or weak, it will fail to format the code correctly, and the harness can't save it. This is a "ceiling" on what the current small models can do.

The Takeaway

This paper is a bridge between math theory and engineering practice.

Before: Engineers built AI managers by guessing. "Let's add a safety check here."
Now: Engineers can use a formal math language to design the manager, prove that the safety checks will survive when they change software tools, and ensure that the system behaves reliably regardless of which AI model is plugged in.

In short: The paper provides the "instruction manual" and the "quality control test" for building AI systems that are reliable, portable, and safe, proving that the structure of the system is just as important as the intelligence of the model inside it.

Technical Summary: Harness Engineering as Categorical Architecture

1. Problem Statement

The engineering of "agent harnesses"—the system layer comprising prompts, tools, memory, and orchestration logic surrounding Large Language Models (LLMs)—has become the primary determinant of agent reliability. However, current harness design remains ad hoc, lacking a formal theory for composition, property preservation under compilation, or systematic comparison across frameworks. Practitioners rely on trial-and-error rather than structural guarantees, leading to a gap where engineers build without formal verification and theorists formalize without practical grounding.

2. Methodology

The paper proposes a formal unification of two existing frameworks:

The Externalization Pillars (Zhou et al.): Memory, Skills, Protocols, and Harness Engineering.
The ArchAgents Category (De los Riscos et al.): An Architecture triple $(G, Know, \Phi)$ representing syntactic wiring, structural knowledge, and deployment maps.

The core methodology involves mapping the four externalization pillars directly to the components of the Architecture triple:

Memory $\rightarrow$ Coalgebraic State: Modeled as a dynamical system with update rules (e.g., bi-temporal memory tracking valid time vs. record time).
Skills $\rightarrow$ Operad-Composed Objects: Procedural capabilities composed via an operad (serial, parallel, trace operations) to ensure typed composition preserves properties.
Protocols $\rightarrow$ Syntactic Wiring ( $G$ ): A graph of modules with typed ports and integrity labels (lenses, prisms, traversals).
Harness $\rightarrow$ Full Architecture $(G, Know, \Phi)$ : The complete system where structural guarantees are encoded as Know-level certificates.

Key Mechanism: Certificate Preservation
The paper defines a structural guarantee as a certificate $(\tau, \sigma, evds)$ —a theorem, parameter mapping, and replayable evidence. A compiler (morphism) between frameworks is valid only if it preserves these certificates. Preservation is verified not by checking model output correctness, but by structural replay: ensuring the theorem, parameters, and evidence survive compilation and can be mechanically re-verified in the target framework.

3. Key Contributions

Formal Correspondence: Establishes that the four pillars of agent externalization map precisely to the categorical Architecture triple, providing a mathematical foundation for harness engineering.
Certificate Preservation Invariant: Introduces a mechanism where structural guarantees (e.g., integrity gates, quality-based escalation) are treated as replayable artifacts independent of the underlying model or framework.
Reference Implementation (Operon): A concrete implementation featuring:
- Five compiler functors targeting Swarms, DeerFlow, Ralph, Scion, and LangGraph.
- A LangGraph compiler that creates one node per stage, calling the native run_single_stage() method to ensure zero reimplementation of harness logic while maintaining observability.
- An atomic skills catalog based on Ma et al.'s five coding skills, mapped to operad compositions.
Empirical Validation:
- Compiler Verification: Demonstrated that three named certificate types (priority gating, no false activation, no oscillation) are preserved with 100% identity and verification across all five compiler targets.
- Model Parametricity: An escalation experiment showed that a quality-based control path (escalating from a fast model to a deep model based on a verifier score) functions correctly regardless of the specific model used, confirming the harness is model-parametric.

4. Results

Structural Preservation: Across five different orchestration frameworks, the compiler functors successfully preserved graph structure and certificate invariants. The structural guarantees survived compilation because they rely on preserved hooks and parameters, not specific graph topologies.
Escalation Control: In a two-model, one-task experiment, a quality-based escalation mechanism successfully triggered a switch from a Phi-3 Mini (fast) to a Gemma 4 (deep) model when quality fell below a threshold. This confirmed that the control logic resides in the harness structure ($Know$), not the model.
SWE-bench-lite Limitations (Negative Results):
- Experiments on SWE-bench-lite using 8B parameter models (Gemma 4 and DeepSeek-R1 distilled) revealed a format-discipline ceiling.
- Despite grounding (providing repository context) and a format-correction retry loop, 0/30 submissions were resolved.
- The dominant failure mode was the inability to produce valid git apply-compatible diffs (e.g., empty extractions, malformed hunk headers), not file selection errors.
- The paper concludes that at the 8B scale, the model's ability to emit correct diff formats is the binding constraint, and three-stage decomposition does not relax this constraint. The harness's value in this context is limited to preservation and primitive reuse, not task-resolution gains on this specific benchmark with these model sizes.

5. Significance and Claims

The paper claims that categorical architecture is the formal theory behind harness engineering. Its significance lies in:

Shifting the Abstraction: Moving the focus from "model-centric" to "harness-centric" design with formal guarantees.
Portability: Proving that structural guarantees can be preserved across different software frameworks (e.g., from a custom "Organism" to LangGraph) via structure-preserving morphisms.
Model Independence: Demonstrating that structural certificates (Know) are distinct from deployment maps ( $\Phi$ ), allowing the same harness logic to run on different models without losing its integrity properties.

The authors explicitly state modesty regarding the results: the demonstrated value is in preservation (certificate identity across compilers) and primitive reuse (escalation logic), not in achieving superior task resolution on SWE-bench-lite with current 8B local models. The work bridges the gap between theoretical category theory and practical agent engineering by providing a vocabulary and a toolset for defining and verifying harness properties.