🤖 AI

Toward a Modular Architecture for Embedded AI Agent Systems at the Edge

This paper proposes a modular, tiered reference architecture for embedded AI agent systems that bridges the gap between deterministic real-time control and agentic intelligence by decoupling on-device execution from cloud-augmented reasoning while integrating a governance layer to ensure safety and observability in resource-constrained environments.

Original authors: Marcus Rüb, Michael Gerhards

Published 2026-06-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Marcus Rüb, Michael Gerhards

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where every device, from a tiny soil sensor in a farm to a smart speaker in your living room, has a "brain" that can think, plan, and make decisions on its own. This is the promise of Agentic AI. However, there's a major problem: the super-smart brains we have today (Large Language Models) are like massive, hungry elephants. They need huge amounts of electricity and memory to run, which fits perfectly in a data center but is impossible for a tiny, battery-powered sensor.

This paper proposes a solution: Don't force the elephant into the mouse hole. Instead, build a flexible system where the "brain" adapts to the size of the device.

Here is the paper's proposal, broken down into simple concepts and analogies:

1. The Core Problem: The "Memory Wall"

Think of a standard microcontroller (the chip inside a simple sensor) as a tiny notebook. It has very little space to write things down and a very small brain.
Think of a powerful Edge Gateway (like a smart home hub) as a large library. It has shelves full of books and a big desk to work on.

Currently, AI researchers are trying to put the entire library's worth of knowledge into the tiny notebook. It doesn't work. The paper argues we need to stop trying to force one size to fit all.

2. The Solution: A Two-Tiered "Team" System

The authors propose a modular architecture that splits the work based on what the hardware can handle. They call these two approaches "Flavors":

Flavor A: The Autonomous Gateway (The "Local Brain")

Where it lives: On powerful devices like smart home hubs or industrial gateways (the "Library").
How it works: This device runs a Small Language Model (SLM) locally. It's like having a smart assistant sitting right in the room with you.
The Benefit: It can think, reason, and make decisions without needing the internet. It's fast, private (your data never leaves the house), and works even if the power goes out.
The Trade-off: It costs more money to build and uses more battery power because it's doing the heavy lifting itself.

Flavor B: The Tethered Agent (The "Messenger")

Where it lives: On tiny, cheap devices like soil sensors or simple switches (the "Tiny Notebook").
How it works: This device is too small to think deeply. Instead, it acts like a messenger. It gathers data (like "the soil is dry") and sends a quick text message to the Cloud (the "Super Brain") to ask, "What should I do?"
The Benefit: It is incredibly cheap and energy-efficient. You can put thousands of them in a field without breaking the bank.
The Trade-off: It needs an internet connection. If the internet cuts out, the device goes "lobotomized"—it can only do simple, pre-programmed reflexes (like "if it gets too hot, shut off") but can't make complex new decisions.

3. The "Governance Layer": The Safety Manager

Whether you are using the "Local Brain" or the "Messenger," the paper introduces a crucial third component: a Governance Layer.

Think of this as a strict safety manager or a referee that watches over all the devices.

Observability: It keeps a log of what every agent is doing so humans can check the records later.
Safety: It acts as a "seatbelt." Before any device (whether it's a smart hub or a tiny sensor) executes a dangerous command (like turning on a boiler or moving a robot arm), the Governance Layer checks: "Is this safe? Does this match our rules?"
Why it matters: This ensures that even if an AI makes a mistake or "hallucinates," the safety manager can stop it from causing physical harm.

4. Real-World Examples from the Paper

The paper uses three scenarios to show when to use which "Flavor":

Smart Agriculture (Flavor B): Imagine a huge farm with thousands of soil sensors. You can't afford a $200 smart hub for every single plant. You use cheap "Messenger" sensors (Flavor B) that send data to a central cloud brain. They save water based on simple rules, but ask the cloud for big decisions like "It's going to be a drought, change the irrigation schedule."
Predictive Maintenance (Hybrid): Imagine a factory machine. A tiny sensor (Flavor B) listens to the vibration 24/7. If it hears a weird noise, it sends a snapshot to a powerful local gateway (Flavor A). The gateway then uses its "Local Brain" to figure out exactly what's wrong and how to fix it.
Privacy-First Smart Home (Flavor A): Imagine a voice assistant in your bedroom. You don't want your private conversations sent to the cloud. So, you use a powerful local hub (Flavor A) that processes your voice and commands right there in your house. It works even if the internet is down, and your secrets stay in your home.

5. The Bottom Line

The paper concludes that there is no "one-size-fits-all" solution.

If you need privacy, speed, and offline reliability, use the Local Brain (Flavor A).
If you need scale, low cost, and massive numbers of devices, use the Messenger (Flavor B) connected to the cloud.

The genius of this architecture is that it treats both types of devices as part of the same team. They speak a common language, follow the same safety rules, and work together to create a world where even the smallest sensor can be part of an intelligent system.

Technical Summary: Toward a Modular Architecture for Embedded Agent Systems at the Edge

Problem Statement

The rapid advancement of Large Language Models (LLMs) has enabled "agentic AI" capable of complex reasoning, planning, and tool use. However, a significant disconnect exists between these powerful AI systems and the constraints of pervasive computing environments. While cloud-based agents assume abundant memory, compute power, and continuous connectivity, the physical world of embedded systems is dominated by deeply embedded microcontrollers (MCUs) and sensor nodes operating under strict resource limitations (e.g., kilobytes of RAM, limited Flash storage) and often unreliable network conditions.

Current frameworks typically assume server-class resources, leaving a gap for deeply embedded systems. Running fully capable reasoning agents on standard microcontrollers is technically infeasible due to the "memory wall," while existing embedded development prioritizes deterministic control loops over the dynamic, context-aware nature of modern agentic AI. There is a lack of a unified architectural framework that bridges deterministic real-time control with agentic intelligence across heterogeneous hardware tiers.

Methodology

The paper proposes a Modular Embedded Agent Architecture designed to unify heterogeneous deployment patterns. Rather than attempting to force a single agent definition onto all devices, the methodology adopts a differentiated, tiered approach based on hardware capabilities:

Hardware Differentiation: The architecture distinguishes between:
- Edge Gateways (MPUs): Devices with gigabytes of RAM capable of hosting quantized Small Language Models (SLMs).
- Deeply Embedded MCUs: Devices with strict memory constraints (SRAM < 512 KB) that cannot host generative models.
Dual Deployment "Flavors": The architecture defines two distinct operational modes that share a common structural DNA but differ in execution location:
- Flavor A (Autonomous On-Device Agent): Runs on capable edge hardware. It executes local SLMs for reasoning, ensuring privacy and offline capability.
- Flavor B (Tethered Cloud Agent): Runs on constrained MCUs. It acts as an intelligent interface, delegating complex reasoning to Cloud Agents (LLMs) while maintaining local deterministic reflexes for safety.
Modular Decomposition: The agent logic is decoupled into functional modules (Perception, Memory, Planning, Reasoning, Action, Learning, Tool Registry, Security). The implementation of these modules adapts to the hardware tier (e.g., probabilistic Chain-of-Thought planning on Gateways vs. deterministic Finite State Machines on MCUs).
Cross-Cutting Governance: A mandatory Governance Layer is introduced to ensure observability, policy enforcement, and safety across distributed fleets, regardless of whether the agent is local or cloud-tethered.

Key Contributions

The paper makes four primary contributions:

Hardware-Specific Constraint Analysis: It analyzes the specific limitations of embedded agents, clearly distinguishing between SLM-capable edge devices and MCU-based cloud interfaces, highlighting the "memory wall" as a primary constraint.
Modular Embedded Agent Architecture: It proposes a framework that decouples the execution layer (local vs. cloud) from the logical agent definition. This allows for hybrid deployment strategies where the definition of an "agent" adapts to the underlying hardware.
Governance Layer Integration: It introduces a cross-cutting Governance Layer as a mandatory component for managing distributed fleets of semi-autonomous agents, addressing critical needs for observability and safety in enterprise environments.
Conceptual Trade-off Evaluation: Instead of purely empirical benchmarks, the paper provides a conceptual evaluation of design trade-offs (latency, energy, privacy, cost) across three application domains: Smart Agriculture, Predictive Maintenance, and Smart Home.

Results and Analysis

The paper presents a conceptual analysis rather than empirical performance benchmarks, deriving the following conclusions regarding the trade-offs between the two deployment flavors:

Latency and Determinism:
- Flavor A (Gateway): Offers low-latency local perception-action loops (<10 ms) but incurs high inference latency (200 ms to seconds) for generative reasoning.
- Flavor B (MCU): Provides extremely fast local reflexes (<1 ms) but introduces unpredictable network jitter (100–1000 ms) for complex reasoning, making it unsuitable for high-frequency cognitive loops.
Energy Consumption:
- Flavor A: Computation-dominant; running SLMs locally consumes significant power (Watts), limiting battery life.
- Flavor B: Communication-dominant; MCUs consume milliwatts in sleep/sensing modes but experience power spikes during transmission. For sporadic tasks, this is more energy-efficient than continuous local processing.
Memory and Cost:
- Flavor A: Requires gigabytes of RAM and higher Bill of Materials (BOM) costs ( $50–$ 200+), restricting scalability for massive sensor deployments.
- Flavor B: Operates within kilobyte RAM limits and low BOM costs ( $2–$ 10), enabling scalability to billions of endpoints but relying on cloud connectivity.
Privacy and Resilience:
- Flavor A: Offers superior privacy (data stays local) and resilience (fully functional in air-gapped environments).
- Flavor B: Relies on the cloud; while data can be encrypted, loss of connectivity degrades the agent to a "lobotomized" state with only basic safety reflexes.

Application Scenarios:

Smart Agriculture: Favors Flavor B due to cost and energy constraints; local rules handle immediate control, while cloud handles strategic decisions.
Predictive Maintenance: Favors a Hybrid approach; MCUs handle high-frequency sampling, while Gateways/Cloud handle complex anomaly diagnosis.
Privacy-First Smart Home: Favors Flavor A to ensure voice data remains local and the system functions during internet outages.

Significance and Claims

The paper claims that its primary significance lies in providing a unified blueprint for bringing agentic behavior to the edge by decoupling the "Agent Concept" from the underlying execution hardware.

Unified Abstraction: It allows system architects to treat a $2 microcontroller and a $100 edge gateway as nodes within the same agentic system, enabling principled design decisions where latency-critical reflexes are assigned to MCUs and privacy-sensitive reasoning to Gateways.
Modularity as an Enabler: By decomposing agents into technology-agnostic modules, the architecture prevents "feature creep" and allows lightweight devices to participate in agentic workflows without being overloaded with unsustainable software stacks.
Governance as a First-Class Citizen: The introduction of a cross-cutting Governance Layer addresses the "Black Box" problem of AI at the edge, providing a uniform audit trail essential for deploying autonomous agents in regulated industries like energy and healthcare.

The authors conclude that there is no "one-size-fits-all" solution. The proposed architecture emphasizes hybrid composability, enabling simple microcontrollers to participate in agentic workflows via standardized interfaces and robust safety fallbacks, while acknowledging that the "memory wall" currently restricts full generative reasoning to higher-end gateways. Future work is identified as developing open-source reference implementations, bridging the semantic gap between text-based AI protocols and binary IoT protocols, and prototyping governance tooling for distributed tracing and "model armor."