EdgeFM: Efficient Edge Inference for Vision-Language Models

1. Problem Statement

Vision-Language Models (VLMs) and Large Language Models (LLMs) face significant barriers to deployment in industrial edge scenarios (e.g., autonomous driving, embodied intelligence). While cloud and general-purpose local frameworks exist, they fail to meet the specific constraints of edge environments:

• Deterministic Low Latency: Edge workloads are typically single-request and latency-critical, requiring millisecond-level responsiveness, unlike cloud services that prioritize high concurrency and throughput.
• Resource Constraints: Edge devices have severely limited memory and compute, making the overhead of bloated, general-purpose frameworks unacceptable.
• Hardware Lock-in: Current industrial deployments rely heavily on proprietary, closed-source toolchains (e.g., vendor-specific SDKs). These act as "black boxes," stifling algorithm iteration, causing hardware lock-in, and forcing developers to wait passively for vendor updates when new model architectures emerge.
• Cross-Platform Fragmentation: There is a lack of open-source, production-grade solutions that can efficiently deploy diverse models (LLMs, VLMs, and Vision-Language-Action models) across heterogeneous hardware (x86, NVIDIA Orin, and domestic chips like Horizon).

2. Methodology

EdgeFM is a lightweight, agent-driven inference framework designed to bridge the performance gap between open-source solutions and proprietary toolchains. Its core methodology involves:

A. Agent-Driven Kernel Optimization

Instead of relying on manual optimization or waiting for vendor updates, EdgeFM leverages modern LLM-based coding agents to:

• Search and tune configurations to generate highly optimized low-level kernels for standard operators.
• Encapsulate these optimizations into a modular library of reusable "skills."
• Allow direct invocation of these skills via a lightweight operator dispatch table, bypassing the need for closed-source implementations.

B. System Architecture & Design Principles

• Thin Runtime, Heavy Kernels: The runtime layer is minimal, handling only request orchestration and memory layout. Performance-critical tasks (Attention, GEMM, KV cache updates) are offloaded to highly optimized, hardware-specific kernels (e.g., FlashInfer-class).
• Single-Request Execution: Unlike cloud frameworks using continuous batching, EdgeFM processes one request end-to-end. This minimizes scheduling overhead and ensures deterministic latency, which is critical for edge control loops.
• Two-Phase Execution (Prefill & Decode):
  • Prefill: Processes the input prompt and initializes the KV cache.
  • Decode: Iteratively generates tokens. EdgeFM supports distinct deployment artifacts for prefill and decode phases to optimize precision and speed separately.
• Operator Implementation Table: A central dispatch mechanism that maps operator calls to specific kernel implementations based on the hardware profile, model architecture, execution stage, and tensor shape signatures. This allows dynamic selection of the most efficient kernel without runtime modifications.

C. Key Optimization Techniques

• Fixed-Prefix KV Cache Reuse: For edge tasks with fixed system prompts or instructions, the KV cache for the prefix is pre-computed and stored offline. During inference, only the user-specific suffix is processed, enabling constant-time cache reuse and eliminating dynamic hashing overhead.
• Graph-Accelerated Decode: Utilizes hardware-optimized execution graphs (e.g., CUDA Graphs for NVIDIA, BPU Execution Graphs for Horizon) to reuse execution graphs across decoding iterations, significantly reducing kernel launch overhead.
• Speculative Decoding: Integrates EAGLE3-style speculative decoding, where a lightweight draft model proposes token blocks verified by the base model, accelerating the decode phase.
• KV Cache Compression: Supports pluggable strategies like low-bit quantization (SageAttention) and structured compression (FlashMLA) to fit long-context inference into tight memory budgets.

3. Key Contributions

1. First Open-Source VLA Deployment on Domestic Chips: EdgeFM achieved the first end-to-end deployment of a Vision-Language-Action (VLA) model (SmolVLA) on the Horizon Journey 6 platform, proving the feasibility of decoupling advanced AI from specific vendor ecosystems.
2. Agent-Driven Performance Gap Closure: By using agents to generate optimized kernels, EdgeFM effectively closes the performance gap traditionally dominated by proprietary toolchains, offering a "lock-in-free" alternative.
3. Cross-Platform Portability: The framework natively supports x86, NVIDIA Orin, and Horizon Journey 6, providing a unified, open-source solution for diverse industrial edge hardware.
4. Modular "Skill" Library: Introduces a novel paradigm where kernel optimizations are treated as reusable, modular skills, allowing developers to extend support to new chips without rewriting the core runtime.

4. Experimental Results

The authors benchmarked EdgeFM against TensorRT-Edge-LLM (NVIDIA's official industry-standard framework) across multiple platforms:

• NVIDIA Orin (Edge Device):
  • EdgeFM achieved up to 1.49× speedup over TensorRT-Edge-LLM on the Qwen2.5-VL-0.5B model.
  • The decode stage latency was reduced by up to 34.35%, demonstrating superior efficiency in resource-constrained environments.
• x86 Server (NVIDIA A800):
  • EdgeFM consistently showed lower end-to-end latency across various context lengths (512 to 2048 tokens) and model sizes (0.5B to 7B parameters).
  • It maintained stable throughput and predictable latency, outperforming the baseline in most prefill/decode configurations.
• Horizon Journey 6 (Domestic SoC):
  • Successfully deployed SmolVLA under a strict 5GB BPU memory constraint.
  • Achieved ultra-low action decode latencies (e.g., ~12.5ms for 32 tokens with 512 context), satisfying the high-frequency control loop requirements for robotic platforms.
  • Demonstrated deterministic prefill latencies even as context lengths increased.

5. Significance

EdgeFM represents a paradigm shift in edge AI deployment:

• Breaking Ecosystem Lock-in: It provides a viable, open-source alternative to proprietary vendor toolchains, empowering developers to optimize for diverse hardware without being tied to a single vendor's update cycle.
• Industrial Readiness: By prioritizing deterministic latency and single-request efficiency, it addresses the specific needs of safety-critical industrial applications (robotics, autonomous driving) that general-purpose frameworks often neglect.
• Scalability and Adaptability: The agent-driven approach to kernel generation suggests a future where edge inference frameworks can automatically adapt to new hardware architectures and model types, significantly reducing the time-to-deployment for cutting-edge AI models on edge devices.

In summary, EdgeFM delivers a production-grade, open-source solution that combines lightweight architecture with agent-optimized performance, enabling high-efficiency, cross-platform deployment of VLMs and VLAs in the industrial edge.