Skilled AI Agents for Embedded and IoT Systems Development

1. Problem Statement

The application of Large Language Models (LLMs) and agentic systems to Hardware-in-the-Loop (HIL) embedded and Internet-of-Things (IoT) development faces significant hurdles that do not exist in pure software development:

• Hardware-Software Coupling: Embedded code must adhere to strict physical constraints (timing, peripheral initialization, pin multiplexing). Code that compiles successfully often fails at runtime due to hardware-specific behaviors.
• Ecosystem Fragmentation: The embedded landscape is highly heterogeneous, involving diverse MCUs (AVR, Xtensa, ARM), toolchains (Arduino, ESP-IDF, Zephyr), and communication protocols (I2C, SPI, UART).
• Limitations of Current Approaches:
  • Context Window Overhead: Augmenting agents with full SDK documentation or datasheets consumes excessive tokens and dilutes the model's focus.
  • Simulation Gaps: Existing emulators (e.g., QEMU, Wokwi) lack fidelity in peripheral timing and real-world sensor-actuator interactions, failing to catch hardware-specific bugs.
  • Verification Gap: Successful compilation and flashing do not guarantee correct runtime behavior.

2. Methodology

The authors propose a skills-based agentic framework and introduce a new benchmark, IoT-SkillsBench, to evaluate AI agents in real embedded environments.

A. The Skills-Based Framework

Instead of feeding raw documentation, the framework utilizes structured "skills"—concise, human-readable documents that distill essential programming patterns, initialization constraints, and known failure modes for specific hardware combinations.

• Agent Architecture: Built on LangGraph with a three-node pipeline:
  1. Manager Node: Acts as a planner. When skills are enabled, it selects relevant skills from a library based on the task and passes them to the coder.
  2. Coder Node: Acts as an expert engineer, generating firmware logic based on the task and selected skills.
  3. Assembler Node: Formats the output into a compilable project structure (e.g., generating CMakeLists.txt, prj.conf, or device tree overlays) specific to the target framework.
• Skill Types Evaluated:
  1. No-Skills: Baseline using only the model's parametric knowledge.
  2. LLM-Generated Skills: Skills synthesized by an LLM (Claude Sonnet 4.6) without external documentation.
  3. Human-Expert Skills: Concise, curated skills authored by embedded engineers, focusing on failure modes and constraints.

B. IoT-SkillsBench Benchmark

A comprehensive HIL benchmark designed to systematically evaluate agents:

• Platforms: Three representative MCU/Framework pairs:
  1. ATmega2560 + Arduino
  2. ESP32-S3 + ESP-IDF
  3. nRF52840 + Zephyr
• Peripherals: 23 distinct peripherals (GPIO, ADC, UART, I2C, SPI, etc.).
• Tasks: 42 tasks across three difficulty levels:
  • Level 1: Basic control (GPIO, ADC).
  • Level 2: Protocol-level reasoning (I2C, SPI timing).
  • Level 3: System integration (interrupts, multi-device coordination).
• Evaluation Protocol: Each of the 126 task-platform combinations (42 tasks × 3 platforms) is tested under 3 skill configurations (No-Skills, LLM-Skills, Human-Skills), with 5 runs per configuration.
  • Total Experiments: 378 hardware-validated runs.
  • Validation: Code is compiled, flashed, and physically tested on real hardware. Outcomes are classified as Compile Failure (CF), Behavior Failure (BF), or Behavior Correct (BC).

3. Key Contributions

1. IoT-SkillsBench: The first benchmark specifically designed for HIL evaluation of AI agents in embedded systems, covering diverse platforms, peripherals, and task complexities with real hardware validation.
2. Skills-Based Framework: A novel approach that replaces bloated documentation with compact, structured "skills" to improve agent reasoning and reduce token overhead.
3. Empirical Analysis of Knowledge Quality: A rigorous comparison demonstrating that the source and quality of knowledge (Human vs. LLM-generated) are more critical than the sheer volume of information.

4. Experimental Results

The study evaluated 378 instances across the three skill configurations:

• No-Skills Baseline:

  • Performed well on simple Level 1 tasks for well-documented platforms (Arduino: 100% success).
  • Performance degraded significantly on complex tasks (Level 3) and less common platforms (Zephyr: only 28/42 total tasks resolved).
  • Low token usage (~1,500 tokens/task).

• LLM-Generated Skills:

  • Inconsistent Results: Often failed to improve performance and sometimes degraded it (e.g., ESP-IDF success dropped from 31/42 to 27/42).
  • Token Inefficiency: Significantly increased token usage (~10,000+ tokens/task) as the model reiterated or reasoned aloud about the generated skills before coding.
  • Issue: LLMs tended to reinforce incorrect platform-specific assumptions rather than correcting them.

• Human-Expert Skills:

  • Near-Perfect Performance: Achieved 42/42 success on Arduino, and 41/42 on both ESP-IDF and Zephyr.
  • Failure Analysis: The few failures were due to fundamental hardware ambiguities (e.g., voltage incompatibility or non-standardized encoder behavior) rather than agent limitations.
  • Efficiency: Maintained moderate token overhead (~2,500–7,500 tokens/task), striking a balance between performance and cost.

5. Significance and Conclusion

The paper concludes that raw LLM capabilities are insufficient for reliable embedded development. The key findings are:

• Quality over Quantity: Providing structured, expert-curated knowledge (skills) is far more effective than feeding raw documentation or relying on LLM-synthesized knowledge.
• Grounding is Essential: For hardware-constrained environments, knowledge must be grounded in real physical behavior and failure modes.
• Scalability: The skills-based approach scales naturally; adding support for new hardware only requires authoring new skills without retraining the agent or changing the pipeline.

IoT-SkillsBench serves as a foundational testbed for future research in AI-assisted embedded development, highlighting that the path to reliable autonomous hardware programming lies in structured, expert-grounded procedural knowledge rather than simply scaling model size.