Technical Summary: ExecuTorch – A Unified PyTorch Solution to Run AI Models On-Device

Problem Statement

The local execution of AI on edge devices is critical for applications requiring low latency and offline operation, such as live translation, autonomous vehicles, and patient monitoring. However, the transition from research to production remains fragmented. While PyTorch powers over 70% of AI research, deploying models to diverse hardware often forces developers to leave the PyTorch ecosystem. Current solutions suffer from several key limitations:

• Semantic Gaps: Model conversion between disconnected environments (e.g., to ONNX or TensorFlow Lite) introduces semantic mismatches that often only surface after deployment.
• Reimplementation Costs: Frameworks like llama.cpp or vendor-specific tools (e.g., Qualcomm SNPE, Apple CoreML) often require complete model reimplementation or tight coupling to specific hardware vendors.
• Runtime Overhead: Existing PyTorch-native solutions like TorchScript and PyTorch Mobile suffer from prohibitive runtime overhead and memory footprints.
• Experimentation Friction: The lack of "experimentation parity" means researchers cannot validate deployment behaviors (quantization, hardware delegation) within PyTorch before committing to production, leading to numerical mismatches and resource-intensive debug cycles.

Methodology: ExecuTorch Architecture

ExecuTorch is introduced as a unified, PyTorch-native deployment framework designed to bridge the gap between research and production. It leverages core PyTorch technologies (torch.compile, torch.export) to construct a portable representation of models that can be deployed as binaries across heterogeneous hardware.

Core Components

1. AOT Export Stack:

   • torch.export and IRs: The system captures computation graphs from torch.nn.Module into a static graph data structure called Export IR. This IR is a Torch FX graph with strong guarantees regarding shape soundness, graph normalization (no Python semantics), and metadata availability.
   • Edge Dialect: A more restrictive Intermediate Representation (IR) built on top of Export IR. It enforces fully functional graphs (no mutations/aliasing), restricts operators to a set of <300 "Core ATen" primitives, and explicitly defines data types and memory formats.
   • Graph Transforms: Operations such as quantization, subgraph delegation, and memory planning are performed ahead-of-time (AOT). This allows the runtime to remain lean, focusing solely on executing pre-optimized graphs.

2. Runtime Stack:

   • Lean Execution: The runtime is a lightweight C++ library (C++17) with minimal dependencies. It executes instruction lists encoded in the PTE (PyTorch Edge) file format.
   • Backend Delegation: The architecture supports selective backend delegation. A partitioner identifies subgraphs composed of operators supported by specific hardware accelerators (e.g., Qualcomm Hexagon, Apple Neural Engine) and routes them to delegate compilers. The remaining graph runs on the CPU.
   • Memory Management: A memory planner allocates space within fixed-size memory arenas before serialization, eliminating dynamic memory allocation at runtime. This is crucial for resource-constrained environments like microcontrollers.

3. Serialization and Storage:

   • PTE Format: A custom file format designed for minimal overhead, separating program data (instructions) from weight data (segments).
   • Weight Sharing: Supports multi-method PTE files (sharing data between model methods like prefill and decode) and program-data separation (sharing weights across models via external PTD files).

Key Technologies

• Quantization: Built on TorchAO, supporting both Post-Training Quantization (PTQ) and Quantization-Aware Training (QAT). It supports static, dynamic, and weight-only quantization, including 4-bit group-wise quantization for Large Language Models (LLMs).
• Custom Kernels: Developers can implement custom kernels and selectively build operators to reduce binary size.
• DevTools: Includes ETRecord (capturing Edge Dialect graphs and debug metadata) and ETDump (capturing runtime events, latencies, and memory lifetimes) to link eager-mode behavior with on-device execution.

Key Contributions

The paper outlines three principal contributions:

1. Unified Framework: ExecuTorch is presented as the first PyTorch framework to achieve experimentation parity through locally executable export graphs. It enables unified deployment from microcontrollers to smartphones without model conversion or reimplementation.
2. Design Principle: The paper introduces "experimentation parity" as a core design principle. Through torch.export and capability-driven backend integration, researchers can validate quantization, hardware delegation, and performance entirely within PyTorch before production deployment.
3. Production Viability: The framework is demonstrated at scale, powering billions of daily inferences across Meta's family of apps and Reality Labs (e.g., Ray-Ban smart glasses). It supports vision, audio, and LLMs on 12 hardware backends.

Performance Results

The authors evaluated ExecuTorch against widely used alternatives (ONNX Runtime, llama.cpp, LiteRT, CoreML) on mobile devices (Samsung Galaxy S25 Ultra, Google Pixel 9 Pro XL, iPhone 15 Pro).

• LLM Performance:
  • CPU: ExecuTorch's XNNPACK backend delivers competitive or state-of-the-art performance, often outperforming ONNX and LiteRT.
  • GPU: The Vulkan backend achieves high token-generation throughput. On the Pixel 9 Pro XL, it significantly outperforms llama.cpp in prefill throughput.
  • NPU: The QNN backend unlocks substantial prefill speedups on NPUs (e.g., Qualcomm Hexagon) when models are well-delegated. For Llama 3.2 1B, QNN showed stronger prefill throughput than QAIRT, though QAIRT achieved higher decode throughput.
• Vision Models:
  • On CPU, the XNNPACK delegate delivered extremely strong performance relative to LiteRT and ONNX.
  • On GPU, the Vulkan delegate executed models with full graph delegation. While some unsupported operators required CPU fallback (e.g., in ViT and Swin-T), the overhead of graph breaks was minimal (5–6% of latency).
  • On iOS, the CoreML delegate matched or slightly outperformed native CoreML across all tested models.
• Microcontroller (MCU) Deployment:
  • Demonstrated on a Raspberry Pi Pico 2 (520 KiB SRAM).
  • Selective build reduced code size from ~1.3 MiB to ~200–250 KiB.
  • Int8 quantization with CMSIS-NN reduced RAM usage to ~11 KiB and provided a 16.46× inference speedup compared to FP32.

Significance and Claims

The paper claims that ExecuTorch successfully eliminates the trade-off between PyTorch's development velocity and edge deployment requirements. By preserving PyTorch semantics from training to production, it allows researchers to validate deployment behavior (including quantization and hardware delegation) within the familiar PyTorch environment. This "experimentation parity" bridges the gap between research and production, enabling fast experimentation and reducing the friction in the experimentation-deployment loop. The framework is positioned not just as a tool for performance, but as a unified workflow that supports the full spectrum of edge devices, from sub-dollar microcontrollers to complex system-on-chips (SoCs).