Architectural Backdoors for Within-Batch Data Stealing and Model Inference Manipulation

1. Problem Statement

The paper addresses a critical, previously underexplored security vulnerability in modern Machine Learning (ML) deployments: the breakdown of isolation between concurrent user requests during batched inference.

• Context: To optimize hardware utilization, ML servers often process multiple user requests simultaneously in a single "batch."
• The Threat: While previous research focused on parameter-based backdoors (manipulating a model's prediction for a specific input), this paper introduces Architectural Backdoors that exploit the batching mechanism itself.
• The Vulnerability: An adversary who can modify the model architecture (e.g., via the supply chain or fine-tuning) can inject a backdoor that breaks the isolation between users in the same batch. This allows the attacker to:
  1. Steal sensitive input/output data from other users in the batch.
  2. Manipulate the outputs generated for other users (e.g., forcing a refusal or a biased response).
  3. Steer the generative process of other users toward specific outcomes.

Unlike prior work where leakage was often unintentional and inefficient (e.g., in Mixture-of-Experts routing), this paper demonstrates how to weaponize these interactions into a deterministic, high-impact attack.

2. Methodology

The authors propose a three-part methodology: the attack construction (Backdoor), the defense mechanism (Checker), and the empirical evaluation.

A. The Attack: Architectural Backdoors for Batched Inference

The attack targets the model's computation graph (specifically ONNX format) rather than its weights. It relies on two main components:

1. Trigger Detector: A lightweight module embedded in the model (e.g., in the K-Cache of a Transformer's first layer) that scans for a specific trigger pattern (e.g., a prompt starting with @@get).
2. Signal Integration: Once the trigger is detected, this component reroutes memory or modifies internal representations to break batch isolation.

The authors define three specific attack vectors:

• Set Attack: The attacker overrides a victim's query with their own, effectively controlling the victim's output.
• Get Attack: The attacker steals the victim's output (or input via steering) without altering the victim's experience.
• Steer Attack: The attacker injects a "steering vector" (a directional bias in the model's latent space) to manipulate the victim's output toward a specific goal (e.g., refusal, negative sentiment, or input exfiltration).

Key Innovation: The backdoor is "dormant" during normal operation and only activates when the specific trigger is present in a batched request, making it highly stealthy. It exploits the fact that modern inference engines process batches as a single tensor, allowing memory copying between batch indices if the architecture permits.

B. The Defense: Batch Isolation Checker

To counter this, the authors propose a deterministic mitigation strategy based on Static Taint Analysis and Information Flow Control (IFC).

• Mechanism: The checker analyzes the model's dataflow graph (e.g., ONNX) before deployment. It assigns unique "ownership labels" to each user's input within a batch.
• Propagation: It tracks how these labels flow through the graph using a novel Monoid-based algebraic structure. Instead of tracking the full set of influencing labels (which is computationally expensive), it tracks only the minimum and maximum labels and a configuration flag for randomness.
  • State (i): Deterministic, non-user dependent.
  • State (ii): Random, non-user dependent.
  • State (iii): Depends on exactly one user (Safe).
  • State (iv): Depends on multiple users (Leakage/Unsafe).
• Verification: If any output tensor contains a label from a user other than the intended one, the model is flagged as unsafe. This provides formal guarantees of non-interference, unlike probabilistic detection methods.

3. Key Contributions

1. Novel Attack Vector: Introduction of "Architectural Backdoors" specifically designed to exploit batched inference, enabling cross-user data theft and manipulation in LLMs and Transformers.
2. Steering Capability: Demonstration that attackers can not only steal data but also steer the generative behavior of victims (e.g., forcing refusals or biased outputs) by manipulating internal model directions.
3. Formal Defense: Development of the Batch Isolation Checker, a static analysis tool that uses a simplified IFC mechanism to formally prove batch isolation, offering deterministic guarantees against these attacks.
4. Large-Scale Empirical Study: A comprehensive analysis of 1,680 models from the Hugging Face ONNX Community.
5. Open Source: Release of the implementation for both the attack and the defense tools.

4. Results

• Attack Effectiveness:
  • Set/Get Attacks: Achieve 100% success rate upon trigger activation as they involve direct memory rerouting.
  • Steering Attacks: Evaluated on Gemma-2B, Qwen2-VL, and Apertus-8B. The attacks successfully induced specific behaviors (Input Exfiltration, Induced Rejection, Negative Outcome Generation) with high success rates (often >90% for specific layers), significantly outperforming baseline probabilities.
• Defense Performance:
  • The Batch Isolation Checker successfully analyzed models in under 30 minutes (even for complex models) on standard cloud hardware.
  • Vulnerability Discovery: The checker identified 269 models (out of 1,680) that exhibited unintended information leakage due to the use of Dynamic Quantization (specifically the DynamicQuantizeLinear operator), which calculates min/max values across the entire batch dimension, inadvertently leaking data between users.
  • Safe Models: 361 models were formally proven to be free from intra-batch leakage under the tested conditions.

5. Significance and Implications

• Paradigm Shift in Threat Modeling: The paper challenges the assumption that batched inference is secure. It proves that architectural modifications can bypass isolation, turning a standard optimization technique into a severe privacy risk.
• Supply Chain Risk: The attack vector highlights the danger of untrusted model redistribution (e.g., converting models to ONNX or hosting fine-tuned versions on public hubs). An attacker does not need to compromise the training data but only the final model architecture.
• Beyond Heuristics: The proposed defense moves away from heuristic detection (which can be evaded) toward formal verification. By proving non-interference mathematically, it offers a robust standard for securing shared ML systems.
• Immediate Impact: The discovery of leakage in widely used quantization techniques suggests that many existing production models may be vulnerable, necessitating immediate re-evaluation of security protocols for batched inference in LLMs and other deep learning systems.

In conclusion, this paper establishes that batch isolation is not guaranteed by default in modern ML architectures and provides both a potent demonstration of the risk and a rigorous, formal method to mitigate it.