Technical Summary: Fox in the Henhouse: Supply-Chain Backdoor Attacks Against Reinforcement Learning

Problem Statement

Existing backdoor attacks on Reinforcement Learning (RL) typically rely on unrealistic threat models that assume the attacker possesses white-box access to the victim's parameters, rewards, or observations. These assumptions often imply a level of intervention (e.g., modifying the victim's reward function or observation space directly) that would render a backdoor attack unnecessary in real-world scenarios. Furthermore, current literature largely ignores the "supply-chain" aspect of modern RL, where practitioners frequently download pre-trained agents or environments from third-party repositories (e.g., Hugging Face, PyTorch Hub) to accelerate convergence in sensitive applications like autonomous driving and automated trading. This paper posits that reliance on untrusted external agents constitutes a severe, practical security risk that is currently underappreciated.

Methodology: The Supply-Chain Backdoor (SCAB) Attack

The authors introduce the Supply-Chain Backdoor (SCAB) attack, a method designed to embed deleterious behaviors into a victim's policy ($\pi_{vic}$) solely through a black-box of legitimate agent-environment interactions. The attacker does not require access to the victim's training process, reward signals, or observation data. Instead, the attack exploits the common practice of training with third-party policies.

Threat Model

The attack operates under a restrictive threat model where the attacker controls an externally sourced, pre-trained policy ($\pi_{att}$) that interacts with the victim in a multi-agent Markov Decision Process (MMDP). The attacker has:

• Read-only access to its own state observations ($s_{att}$).
• Action-only influence over the environment via its actions ($a_{att}$).
• No access to the victim's internal parameters, rewards ($r_{vic}$), or observations ($s_{vic}$).

Attack Mechanism

The SCAB attack utilizes a bifurcated attacker policy and a detector to achieve its goals:

1. Attacker Components:

   • $\pi_{att}^{std}$: A standard policy trained to win the game (e.g., via tournament-based training).
   • $\pi_{att}^{rwd}$: A "rewarding" policy trained to implicitly reward the victim for executing specific backdoor actions. This is achieved by the attacker deliberately expediting its own defeat (maximizing the opponent's cumulative reward) when the victim performs the desired sequence.
   • Detector ($d$): A supervised learning module that infers the victim's previous actions ($\tilde{a}_{vic}$) by observing state transitions in the attacker's own observations ($s_{att}$).

2. Training Phase (Injection):
   The attacker switches between four states based on a finite-state machine:

   • Winning: The attacker plays normally ($\pi_{att}^{std}$). With a probability defined by the Trigger Injection Probability (TIP), it transitions to the Triggering state.
   • Triggering: The attacker executes a predefined sequence of trigger actions ($A_t$), such as consecutive "no-op" moves.
   • Observing: The attacker uses the detector $d$ to deduce if the victim is responding with the specific backdoor action sequence ($A_b$).
   • Rewarding: If the victim executes the correct backdoor sequence, the attacker switches to $\pi_{att}^{rwd}$, intentionally losing to provide a high cumulative reward to the victim. This reinforces the association between the trigger and the backdoor action in the victim's Q-function.

3. Testing Phase (Exploitation):
   Once the backdoor is embedded, any opponent aware of the trigger can activate it. An opponent equipped with a trigger network ($\pi_{trg}$) can strategically initiate the trigger sequence to force the victim into the backdoor behavior, causing a significant performance drop while minimizing the time spent executing the trigger itself.

Key Contributions

1. Realistic Threat Model: The paper documents the first realistic attack exploiting the RL supply chain, requiring only the level of access a benign agent would have (read-only observations, action-only influence).
2. SCAB Attack Proposal: The authors propose and validate the SCAB attack, which injects backdoors via pre-trained agents without modifying the victim's training data, rewards, or observations.
3. Empirical Validation: The study demonstrates that SCAB achieves performance comparable to prior attacks with significantly more permissive access models, achieving high success rates with minimal data corruption (3% trigger injection).

Experimental Results

The authors evaluated SCAB on two-player competitive environments (Pong, Surround, Boxing) using both on-policy (PPO) and off-policy (DQN) algorithms with CNN and LSTM architectures.

• Attack Success: SCAB achieved a peak trigger success rate exceeding 90% across various settings.
• Performance Degradation: The attack reduced the victim's average episodic returns by over 80% when triggers were active.
• Stealthiness: Training-time metrics (episodic return, policy loss, episode length) for the victim under SCAB were indistinguishable from clean training. The victim maintained high performance against normal opponents, and backdoor actions occurred spontaneously at negligible rates (<1%) without triggers.
• Generalizability: The attack was effective across discrete and continuous action spaces, cooperative and competitive environments, and multi-agent scenarios.
• Comparison: In the Boxing environment, SCAB (with 3% TIP) achieved a 91.3% trigger success rate and an 82.6% return drop, matching the performance of BACKDOORL (93.6% success, 81.3% drop) and TrojDRL (99.1% success, 85.3% drop), despite SCAB operating under a much stricter access model (no write access to rewards or observations).
• Defense Limitations: A standard fine-tuning defense (retraining against normal players) showed limited effectiveness, reducing the trigger success rate from 83.2% to 69.5% after 128,000 steps, while the victim's performance under attack remained severely compromised.

Significance and Claims

The paper argues that the security provided by ignoring supply-chain risks in RL is "illusory." By demonstrating that high-impact backdoor attacks do not require highly permissive access models, the authors highlight that RL is far more vulnerable to adversarial manipulation than previously thought.

The authors emphasize that while their work exposes vulnerabilities, the responsible disclosure of such attacks is critical for improving system security. They assert that the common practice of sourcing externally provided models creates a severe and practical security risk, particularly as RL is increasingly adopted in safety-critical domains such as autonomous driving, algorithmic trading, and healthcare. The paper concludes that this new attack vector must motivate the development of robust defensive and detection mechanisms specifically tailored to the training supply chain.