Technical Summary: SoK: Attack and Defense Landscape of Mobile On-device AI Systems

Problem Statement

Mobile On-device AI (MOAI) systems represent a paradigm shift where AI models are deployed locally on end-user devices rather than relying on remote cloud services. While this architecture offers significant advantages in privacy (local data processing), latency (no network transmission), and offline capability, it fundamentally alters the security landscape. Unlike cloud-based systems where models reside in controlled server environments, MOAI systems store proprietary model artifacts directly on user devices. This local storage exposes models to a unique set of threats, including reverse engineering, memory dumping, and unauthorized extraction, which are not prevalent in traditional mobile software or cloud-AI contexts.

Existing literature has largely focused on Edge AI, TinyML, or specific subsets like model stealing, lacking a holistic framework that addresses the end-to-end security risks of MOAI systems. There is a critical gap in understanding how local inference reshapes trust boundaries among user inputs, device-resident models, runtimes, and hardware isolation, as well as a lack of systematic guidance on foundational security pillars and practical defense solutions.

Methodology

This paper presents the first comprehensive systematization of knowledge (SoK) regarding MOAI security. The authors adopt a holistic mobile-system perspective, analyzing MOAI architectures as layered compositions of traditional mobile software and local neural components. The methodology involves:

1. Architectural Characterization: Defining MOAI systems through four distinct layers: Input Interface, AI Model Artifact, Runtime Execution, and Hardware Isolation.
2. Security Pillar Derivation: Establishing three core security pillars that capture the essential protection properties of MOAI systems.
3. Threat Modeling: Formalizing three adversary types based on their capabilities and points of manipulation: Input-level, Model-level, and Execution-level.
4. Taxonomy Development: Systematically categorizing existing attacks and defense mechanisms based on the defined threat models and deployment phases (pre-deployment, runtime, post-deployment).
5. Cross-Pillar Analysis: Mapping attacks and defenses to the security pillars to identify gaps, trade-offs, and open problems.

Key Contributions

1. Security Pillars of MOAI Systems

The paper establishes a framework characterizing MOAI security via three pillars:

• Pillar I: User-governed Input Integrity: Ensuring the integrity of data acquired from sensors and user interactions before it is consumed by the model.
• Pillar II: Device-resident Model Security: Protecting the confidentiality and integrity of model artifacts (weights, structure) stored on the device against extraction and modification.
• Pillar III: Device-native Environment Confinement: Ensuring that the runtime execution environment (OS, memory, hardware) isolates inference states from untrusted components.

2. Comprehensive Attack Landscape and Taxonomy

The authors categorize MOAI attacks into five distinct categories based on three threat models:

• Adversarial Attacks: Including model similarity exploitation, gradient reconstruction, and preprocessing manipulation. These primarily target Pillar I but often leverage Pillar II accessibility.
• Backdoor Attacks: Involving payload injection, model quantization exploits, and image steganography to implant triggers. These couple Pillars I and II.
• Adversarial Weight Attacks: Direct parameter tampering (e.g., TYPHON) to alter inference behavior, targeting Pillar II.
• Model Stealing Attacks: Utilizing static analysis, dynamic analysis, and side-channel attacks to extract model artifacts. These target Pillar II and extend to Pillar III when models are only recoverable during runtime.
• Energy-latency Attacks: Targeting model availability by increasing resource consumption, spanning all three pillars.

3. Comprehensive Defense Landscape and Taxonomy

Defenses are categorized by deployment phase and mechanism:

• Model Obfuscation: Software-level (renaming, operator coupling) and hardware-level (splitting models with hardware secrets) concealment to hinder extraction.
• Model Authorization: Binding model utility to authenticated app contexts (e.g., MMGuard) to prevent unauthorized use.
• Trusted Execution Environments (TEE): Executing models or sensitive components within hardware-isolated enclaves (Monolithic, Partitioned, or Obfuscated Offloading).
• Model Watermarking: Post-deployment ownership verification (e.g., THEMIS) to detect unauthorized reuse.

4. Identification of Open Problems

The analysis reveals nine critical open problems (OP1–OP9) where current research falls short:

• Practicality Gaps: Many attacks (e.g., adversarial perturbations) require assumptions (like input control) that are impractical in real-world MOAI deployments (OP1, OP5).
• Stealth and Precision: Defining stealthy backdoors without observable changes (OP2) and precisely locating critical weights for tampering (OP3) remain difficult.
• Extraction Complexity: Custom encryption and non-standard frameworks complicate reliable model extraction (OP4).
• Defense Limitations: Defenses face trade-offs in executable equivalence (OP6), client-side enforcement robustness (OP7), TEE deployment feasibility across heterogeneous hardware (OP8), and watermark robustness against redeployment (OP9).

5. Future Research Directions

The paper outlines three emerging directions:

• On-device Training: Addressing security risks introduced by local model updates and gradient exposure.
• On-device GenAI: Investigating unique threats in Large Language Models (LLMs) and generative models, such as prompt injection and jailbreaking.
• Agentic MOAI Systems: Securing workflows where AI agents interact with OS services, tools, and cross-app interfaces, requiring end-to-end action governance.

Results and Findings

The systematization reveals a "security asymmetry" in MOAI systems. Pillar II (Device-resident Model Security) is the central concern, as local storage inherently exposes models. However, the practicality of attacks is heavily constrained by the deployment pipeline:

• Attack Feasibility: While local storage theoretically enables model extraction, practical attacks are hindered by encryption, framework heterogeneity, and the difficulty of manipulating user-governed inputs.
• Defense Trade-offs: Existing defenses often incur significant overheads. For instance, TEE-based solutions improve confidentiality but face bottlenecks in memory pressure and hardware integration. Obfuscation techniques may increase package size or RAM usage, and authorization mechanisms can introduce startup latency.
• Partial Coverage: No single defense covers all pillars effectively. For example, TEEs protect runtime states (Pillar III) but may not fully prevent pre-deployment extraction if the model is not encrypted before loading.

Significance

This work establishes the first systematic framework for understanding the attack and defense landscapes of MOAI systems. By moving beyond isolated surveys of model stealing or edge security, it provides a unified view of how local inference reshapes security requirements. The paper serves as a foundational reference for researchers and practitioners, offering a structured taxonomy to guide the development of secure MOAI systems. It highlights that securing MOAI requires more than just protecting the model artifact; it demands a holistic approach that considers input integrity, runtime confinement, and the practical realities of mobile deployment constraints. The identified open problems and future directions provide a roadmap for advancing research in this critical domain.