💻 computer science

ARCHER: Agentic Rule and Compliance Harness for Executable Regulations

The paper introduces ARCHER, a deterministic multi-agent program-synthesis harness that generates auditable verification code from regulatory documents to achieve scalable, transparent, and highly accurate building compliance checking, demonstrating that self-hosted open-weights models can match frontier API performance at a fraction of the cost.

Original authors: Chiraag Singh Anand, Xue Wen Tan, Lionel Teo, Eric Tan

Published 2026-07-29

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Chiraag Singh Anand, Xue Wen Tan, Lionel Teo, Eric Tan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to follow a very strict set of rules for building a house. The problem isn't that the robot is lazy; it's that the rulebook is written in human language, full of vague phrases like "make sure the hallway is wide enough," while the robot only understands math and geometry. In the world of architecture, these rules are called "building codes," and the blueprints are digital 3D models known as BIM (Building Information Modeling). For decades, checking if a design follows the rules has been a massive, expensive headache, usually requiring armies of human experts to manually scan thousands of lines of code and geometry. It's like trying to find a single typo in a library of a million books by reading every single page with a magnifying glass.

Recently, scientists have started using "AI agents"—smart computer programs that can think, plan, and write code—to help with this. But here's the catch: if you just ask a smart AI to "write a rule checker," it often gets confused, makes up rules that don't exist, or writes code that crashes. It's like asking a brilliant but chaotic chef to bake a cake without a recipe; they might make something delicious, or they might burn the kitchen down. The big question researchers are asking is: How do we organize these AI agents so they don't just guess, but actually build reliable, working tools that we can trust? This is the puzzle that a team from Singapore's Infocomm Media Development Authority decided to solve.

The ARCHER: A Team of AI Architects

The paper introduces a new system called ARCHER (Agentic Rule and Compliance Harness for Executable Regulations). Think of ARCHER not as a single super-brain, but as a highly organized construction crew where every member has a specific job, and they follow a strict, unbreakable schedule.

In the past, researchers tried to get AI to do this by giving a single AI model a big prompt and hoping for the best. The authors call this the "Blind" approach. It's like handing a complex math problem to a student and saying, "Figure it out," without letting them check their work. The paper found that this method is unreliable. Even the smartest AI models often fail to get the answer right because they can't see their own mistakes until it's too late.

ARCHER changes the game by using a multi-agent system with a deterministic orchestration. Let's break that down with an analogy:

Imagine you are trying to build a perfect LEGO castle based on a vague instruction manual.

The Planner: First, a "Planner" agent reads the manual and draws a step-by-step sketch of how to build it. It doesn't touch the bricks yet; it just makes a plan.
The Generator: Next, a "Generator" agent follows that sketch and starts snapping the LEGO bricks together, writing the actual computer code.
The Evaluator: Then, a strict "Evaluator" agent checks the castle. It measures every wall and counts every brick against the original rules. If the wall is too short, the Evaluator doesn't just say "Wrong." It points exactly at the short wall and says, "Fix this specific brick."
The Loop: The Generator then fixes the mistake and tries again. This cycle repeats up to 30 times. If the Evaluator realizes the plan itself was wrong (not just the bricks), it sends the work back to the Planner to redraw the sketch.

This "Plan-Generate-Evaluate" loop is the secret sauce. The paper shows that by forcing the AI to check its own work repeatedly and fix specific errors, it can turn a vague rule like "parking spots must be 2.4 meters wide" into a perfect, working computer program that checks thousands of 3D models instantly.

What They Found: The Power of Structure

The researchers tested this system using a new dataset of ten real-world building rules and 3D models. They compared their "ARCHER" crew against six other ways of using AI, ranging from a single confused robot to a team of robots that argue with each other.

Here are the big takeaways, told in plain numbers:

Structure beats raw intelligence: The most surprising finding is that a well-organized team of "weaker" AI models performed better than a single "super-smart" AI model working alone. When the AI was allowed to just guess once (the "Blind" method), it got the answer right only about 47% of the time on average. But when the team used the ARCHER loop, the accuracy jumped to 85% on average. That's an 82% improvement over the basic method.
The "Orchestrator" matters: The team tested two ways to run the crew. One way let an AI decide who talks to whom (like a chaotic meeting). The other way used a fixed, unchanging schedule (like a military drill). They found the fixed schedule (ARCHER) was much more reliable. It improved accuracy by up to 24.5 percentage points for some models compared to the chaotic version.
Cheaper isn't always cheaper: The team tested four different AI models, from expensive, top-tier "frontier" models (which cost a lot of money to run) to cheaper, self-hosted models (which you can run on your own computer).
- The expensive models were great, but the paper found that a cheaper, self-hosted model running on the ARCHER system could reach 97.8% of the expensive model's accuracy.
- Even better, this cheaper setup cost only one-quarter of the price.
- This suggests that organizations don't need to spend a fortune on the most expensive AI to get great results; they just need the right system (ARCHER) to guide the cheaper AI.

Why This Matters

The paper argues that building compliance checking has been stuck in a "manual" era for too long. Existing tools are often locked behind expensive, proprietary software that no one can see or fix. If a rule is ambiguous, the software just fails, and you have to call a human.

ARCHER suggests a new way forward: Transparent, adaptable, and scalable. Because the system writes standard Python code (a common programming language), humans can actually read the code the AI wrote to see how it decided a building was safe or unsafe. If the AI gets something wrong, a human can look at the code, see the logic, and fix it.

The authors are careful to note that this isn't a magic wand that solves every problem. Some building rules are just too complex for a computer to measure perfectly (like judging if a weirdly shaped staircase is "safe" in a way that requires human intuition). For those, the system is designed to flag them for a human to check (labeled as "MANUAL_CHECK"). But for the thousands of standard rules—like "is the door wide enough?" or "is the parking spot the right size?"—ARCHER suggests we can now automate the heavy lifting with high accuracy and low cost.

In short, the paper proves that if you give AI a strict job description, a team of specialists, and a loop to check its homework, it can turn messy, confusing building rules into clear, working code. It turns the "black box" of AI into a transparent, auditable tool that could one day make building safer, faster, and cheaper for everyone.

Technical Summary: ARCHER (Agentic Rule and Compliance Harness for Executable Regulations)

1. Problem Definition

Building Information Modeling (BIM) has transitioned architectural design toward semantically rich, interoperable data models via the Industry Foundation Classes (IFC) schema. However, verifying compliance against these models remains a manual, costly, and unscalable bottleneck, particularly in jurisdictions like Singapore where projects must satisfy thousands of regulatory rules.

Existing Automated Compliance Checkers (ACCs) face three primary limitations:

Lack of Generalizability: They are typically hard-coded for specific rule sets and use cases, making them difficult to adapt to new regulations.
Proprietary Opacity: Many commercial ACCs do not release underlying verification code, preventing users from verifying if their regulatory intent is accurately captured.
Ambiguity in Operationalization: Natural language regulations are often ambiguous regarding geometric measurement (e.g., how to measure the width of a parking lot represented as a solid vs. 2D lines). Domain experts possess regulatory intent but lack programming expertise to formalize it, while developers lack the domain context to interpret the rules faithfully.

Current Large Language Model (LLM) approaches have reduced some barriers but often rely on predefined APIs or extracted metadata, failing to capture the spatial and contextual reasoning required for manual expert review. Furthermore, they lack the reliability of test-driven feedback loops found in broader agentic code-generation research.

2. Methodology: The ARCHER Framework

The authors introduce ARCHER, a test-driven, deterministically orchestrated multi-agent program-synthesis harness. Its goal is to generate auditable, executable Python compliance checkers from regulatory "Codes of Practice" and expert-labelled BIM models.

Core Architecture

ARCHER operates on two expert-authored artifacts:

Rule Interpretation (RI): A plain-text document translating regulatory logic into an unambiguous flow (e.g., defining what constitutes a "pass," "fail," or "manual check" without specifying the geometric computation method).
Labelled BIM Models: Training ( $D_{train}$ ) and test ( $D_{test}$ ) pairs where domain experts have assigned verdicts ( $L = \{PASS, FAIL, ALERT, MANUAL\_CHECK, NA\}$ ) to specific elements.

The Synthesis Loop

ARCHER employs a deterministic multi-agent orchestration (Harness 5) involving three specialized agents:

Planner: Drafts a coding plan based on the RI.
Generator: Writes or edits Python code (main.py) to implement the plan.
Evaluator: Executes the generated code against $D_{train}$ , calculates Union Accuracy ( $A_\cup$ ), and provides feedback.

The system iterates up to 30 times. If the Evaluator identifies a failure rooted in the logic plan rather than a coding bug, it triggers a Replan (up to 2 times), sending control back to the Planner. The loop terminates when the generated checker achieves perfect agreement ( $A_\cup = 1.0$ ) on the training set or hits iteration limits. The final checker is evaluated once on the held-out test set ( $D_{test}$ ).

Evaluation Taxonomy

To characterize the efficacy of agentic synthesis, the authors evaluated a taxonomy of six harnesses ranging from simple to sophisticated:

H0 (Blind): Single agent, no training data access.
H1 (BIM Model Aware): Single agent, execution against model only.
H2 (Full TDD): Single agent with full scoring feedback.
H3 (Planning Step): Single agent with explicit planning.
H4 (Multi-Agent, Non-Deterministic): LLM-orchestrated pipeline.
H5 (Multi-Agent, Deterministic - ARCHER): Fixed control loop orchestrating Planner, Generator, and Evaluator.

3. Key Contributions

First Application of Agentic Synthesis: The authors claim to be the first to apply agentic program synthesis specifically to automated building compliance checking.
ARCHER Harness: A novel multi-agent architecture that orchestrates planning, generation, and evaluation to produce inspectable, executable compliance checkers.
Comprehensive Cost–Accuracy Analysis: A study spanning four backbone models (from frontier APIs to on-premise open-weights models) across different deployment tiers (third-party, private cloud, on-premise).
Novel Benchmark Dataset: A dataset of ten real-world regulatory requirements paired with expert-labelled BIM models, designed to test robustness against out-of-distribution (OOD) variations in geometry and metadata.

4. Experimental Results

The study evaluated the six harnesses across four backbone models: GPT-5.5 (Frontier API), GPT-5.4 Mini (Budget API), DeepSeek-v4-flash (Self-hosted), and GPT-OSS-120B-Q4KM (On-premise).

Accuracy Findings

ARCHER (Harness 5) is Uniformly Best: It achieved the highest union accuracy for every backbone model.
Significant Improvement: ARCHER improved the mean union accuracy by 82% over the naïve single-pass prompting baseline (H0), raising the overall mean from 0.468 to 0.850.
Deterministic vs. LLM Orchestration: The deterministic control loop (H5) outperformed the LLM-orchestrated pipeline (H4) across all models, particularly for weaker backbones that struggled to sequence their own sub-agents.
Feedback Structure: Exposing full test-driven feedback to a single agent (H2) degraded performance for open-weights models. Structured multi-agent harnesses were required to convert dense scoring feedback into targeted code revisions.

Cost and Efficiency

Token Consumption: Token usage scaled inversely with model capability. Weaker models (e.g., GPT-OSS-120B) consumed significantly more tokens (up to 223.7× more input tokens than GPT-5.5 in Harness 5) due to repeated refinement loops.
Cost–Accuracy Trade-off:
- Harness Sophistication > Model Price: Increasing the harness level within a model family yielded better returns than switching to a more expensive backbone.
- Data Sovereignty: A self-hosted open-weights model (DeepSeek-v4-flash) under ARCHER achieved 97.8% of the frontier API's accuracy at roughly one-quarter of the cost.
- Pareto Frontier: The most cost-effective configurations were found among self-hostable models using the ARCHER harness, making data-sovereign compliance checking practical.

Scenario Difficulty

Requirements demanding relational or topological reasoning (e.g., fire-escape routes, sewer connectivity) were significantly harder and more expensive to synthesize than simple dimensional checks.
While the frontier model (GPT-5.5) solved the hardest scenarios (e.g., R5, R10) where others failed, the self-hosted DeepSeek model matched or exceeded the frontier model on 7 out of 10 requirements.

5. Significance and Claims

The paper positions ARCHER as a step toward transparent, adaptable, and scalable compliance checking. Its primary significance lies in:

Democratizing Compliance: By generating open, inspectable Python code, it removes the "black box" nature of proprietary ACCs, allowing domain experts to audit the logic.
Feasibility of On-Premise Solutions: It demonstrates that organizations with strict data sovereignty requirements (e.g., government or defense) can achieve near-frontier accuracy using self-hosted open-weights models, provided they use a sophisticated agentic harness.
Human-Agent Collaboration: The system is framed as a collaborative tool where experts define intent (via RI) and the agent handles the computational implementation, bridging the gap between regulatory intent and executable logic.

Limitations

The authors acknowledge that ARCHER is not fully autonomous. It requires upfront effort from experts to author Rule Interpretation documents and label training data. Additionally, the system treats rules in isolation, does not model inter-rule dependencies, and remains bounded by the geometric limitations of BIM schemas (hence the inclusion of MANUAL_CHECK for ambiguous cases). Future work is suggested to address rule dependencies via semantic knowledge graphs and to integrate vision-language models for non-deterministic geometric checks.