From Human Guidance to Autonomy: Agent Skill System for End-to-End LLM Deployment on Spatial NPUs

Technical Summary: From Human Guidance to Autonomy: Agent Skill System for End-to-End LLM Deployment on Spatial NPUs

Problem Statement
Spatial Neural Processing Units (NPUs) offer an energy-efficient platform for edge Large Language Model (LLM) inference. However, deploying LLMs end-to-end on these resource-constrained hardware accelerators remains a labor-intensive process requiring substantial expert engineering. While AI coding agents have begun to reduce engineering costs, existing research primarily focuses on single-kernel optimization rather than full end-to-end deployment. Furthermore, no existing framework encodes the complete deployment workflow as composable "Agent Skills" that can be invoked autonomously to handle the explicit hardware management, data-movement scheduling, and tile-level kernel placement required by spatial NPUs.

Methodology
The authors propose a two-stage methodology instantiated on the AMD XDNA™2 NPU, designed to transition from human-guided development to agent autonomy.

• Stage 1: Human-Guided Agent Assistance (Reference Deployment):
  The team deployed a BF16 Llama-3.2-1B model end-to-end using an AI coding agent (Claude Code with Claude Opus 4.7) as a copilot. To manage the complexity of sparse tooling documentation and opaque error messages, they employed a "documentation-first" discipline. A structured set of Markdown documents (e.g., plan.md, issues.md, perf_opt_traj.md) tracked the optimization trajectory, debugging steps, and design rationale. The agent performed code edits, profiling, and tool runs under human supervision, with every decision logged. This process achieved a 2.2× speedup on prefill and a 4.0× speedup on decode compared to a hand-optimized baseline (IRON). Key optimizations included vectorization, shape-specific tile tuning, fused kernel design (merging 15 kernels into 3 dispatches), and host-side optimizations like zero-copy buffer mapping and skipping redundant data transfers.

• Stage 2: Agent Skill System (Autonomous Deployment):
  The documentation and optimization trajectory from Stage 1 were distilled into a reusable "Agent Skill System." This system orchestrates the deployment of new, unencountered LLMs through an eight-phase pipeline:

  1. Phase 0–3 (Correctness): Establishes end-to-end functional correctness by decomposing the model into a CPU FP32 oracle, validating individual NPU kernels, wiring them into a single transformer block, and cascading to the full N-layer model.
  2. Phase 4–5 (Optimization): Applies specific optimization skills (e.g., multi-kernel merging, buffer reuse, layout alignment) to address prefill and decode bottlenecks.
  3. Phase 6 (Integration): Integrates prefill and decode into a single runner and harvests new findings.
  4. Phase 7 (Independent Evaluation): Spawns an independent evaluator agent (without prior context) to re-audit the deployment, re-run numerical gates, and verify performance reports to prevent reward hacking.

  Crucially, Phases 0–6 enforce strict numerical correctness gates against a CPU FP32 reference (measuring absolute error, relative error, and Pearson correlation). If a gate fails, the system auto-invokes debug skills based on known failure patterns or prompts for human intervention.

Key Contributions

1. High-Performance Reference Deployment: An end-to-end deployment of Llama-3.2-1B on the AMD XDNA 2 NPU that outperforms the hand-optimized baseline by 2.2× (prefill) and 4.0× (decode), with the entire optimization trajectory documented to form the foundation of the skill system.
2. Agent Skill System for NPUs: A novel system comprising an eight-phase skill chain with strict numerical gates, a set of optimization and debug skills that auto-trigger on common patterns, and an independent evaluator agent. This is the first skill set targeting end-to-end LLM deployment on spatial NPUs.
3. Autonomous Deployment of Eight New Models: The first open-source end-to-end deployment of eight additional decoder-only LLMs (including Llama-3.2-3B, SmolLM2-1.7B, and various Qwen2.5/3 models) on the AMD XDNA 2 NPU. These deployments were completed autonomously in 0.5–4 hours of agent wall time with almost no human guidance.

Results
Using the agent skill system, the team successfully deployed eight additional LLMs. All models passed numerical correctness gates and were manually verified.

• Efficiency: Three of the eight models (Llama-3.2-3B, SmolLM2-1.7B, Qwen2.5-3B) achieved a sustained performance ($\eta_{scale}$) matching or exceeding the Llama-3.2-1B reference (ranging from 0.94 to 1.27). These models share similar transformer block structures with the reference, allowing existing kernels to handle them effectively.
• Limitations: The remaining five models showed lower efficiency ($\eta_{scale}$ 0.24–0.94). The authors attribute this to smaller per-kernel workloads (failing to amortize dispatch overhead) and unsupported operations (e.g., QKV bias, per-head Q/K normalization) that required padding or prevented kernel fusion.
• Deployment Time: Autonomous deployments were completed in 0.5 to 4 hours, significantly reducing the engineering effort compared to manual deployment.

Significance
The paper claims to demonstrate "functional generalization" to previously unencountered LLMs, proving that a skill system distilled from human-guided optimization can autonomously deploy competitive LLM implementations on spatial NPUs without additional model-specific human engineering. It addresses the gap in existing research by moving beyond single-kernel optimization to full end-to-end deployment workflows. The work establishes a pathway for reducing the engineering barrier to edge LLM deployment on energy-efficient spatial accelerators, suggesting that agent autonomy can eventually replace the need for extensive expert engineering in this domain.