Beware Untrusted Simulators -- Reward-Free Backdoor Attacks in Reinforcement Learning

1. Problem Statement

Reinforcement Learning (RL) relies heavily on simulated environments for training agents before deployment in the real world. While simulators are trusted tools, they represent a critical security blind spot. Malicious developers or cloud providers can distribute compromised simulators to implant backdoors into trained agents.

The Gap: Traditional backdoor attacks in RL (e.g., TrojDRL, SleeperNets) rely on strong threat models where the adversary has read and write access to the agent's reward function. They manipulate rewards to bias the agent toward a target action. However, in many realistic scenarios (e.g., standard robotics simulators like MuJoCo or PyBullet), rewards are computed externally by the victim based on the state returned by the simulator. The simulator itself only provides state transitions ($T$) and receives actions ($A$); it does not calculate or observe rewards ($R$).

The Challenge: How can an adversary implant a reliable backdoor into an RL agent if they can alter the simulator's transition dynamics but cannot observe or alter the agent's rewards?

2. Methodology: The "Daze" Attack

The authors propose Daze, a novel, reward-free backdoor attack that exploits the universal sub-optimality of random actions in RL tasks.

Core Concept

The attack relies on Assumption 1: In almost all practical RL tasks, a policy that samples actions uniformly at random is strictly sub-optimal compared to a "reasonable" policy that solves the task.

Mechanism:

1. Trigger Injection: The malicious simulator introduces a "trigger" (a specific state pattern, e.g., a visual flag or object) with a low probability ($\beta$).
2. The "Dazed" State:
   • If the agent enters a triggered state and chooses the target action ($a^+$) desired by the adversary, the simulator behaves normally (or performs a "null transition"), allowing the agent to proceed.
   • If the agent enters a triggered state and ignores the target action, the simulator forces the agent into a "dazed" state.
   • In the dazed state, the simulator severs the link between the agent's chosen action and the environment. Instead, the environment executes a uniformly random action ($a' \sim U(A)$) for a duration of $k$ steps.
3. Consequence: Because random actions are sub-optimal (Assumption 1), the agent receives low returns (punishment) when it ignores the target action. Conversely, following the target action yields high returns.
4. Optimization: The RL agent, seeking to maximize cumulative rewards, learns that the optimal policy in the poisoned environment is to always execute the target action when the trigger is present to avoid the "dazed" punishment.

Theoretical Guarantees

The authors formalize the attack as a modified Markov Decision Process (MDP) $M'$. They prove two key theorems:

• Theorem 1 (Attack Success): If a policy $\pi^*$ is optimal in the poisoned MDP $M'$, it must execute the target action $a^+$ with probability 1 in all triggered states.
• Theorem 2 (Attack Stealth): If a policy is optimal in $M'$, it is also optimal in the benign MDP $M$. This means the agent performs normally in benign states, making the attack undetectable during standard testing until the trigger is observed.

Implementation

Daze is implemented as a lightweight environment wrapper (API) around the simulator. It requires no access to the reward function. It simply intercepts the state, checks for the trigger, and if the agent deviates from the target action, it forces random transitions.

3. Key Contributions

1. New Threat Model: Identifies and formalizes the "untrusted simulator" threat where the adversary has control over transition dynamics but zero access to the reward function (neither read nor write).
2. Daze Attack: Proposes the first reward-free backdoor attack that achieves theoretical guarantees of success and stealth without manipulating rewards.
3. Theoretical Proof: Provides formal proofs that an optimal policy in the poisoned environment inherently satisfies the adversary's objectives (stealth and success) due to the sub-optimality of random transitions.
4. Real-World Validation: Demonstrates the first successful transfer of an RL backdoor attack from simulation to physical robotic hardware (Turtlebot and Fetch robots).

4. Experimental Results

The authors evaluated Daze against state-of-the-art attacks (TrojDRL, SleeperNets, Q-Incept) across discrete (Atari) and continuous (MuJoCo) action spaces, as well as real hardware.

Key Findings:

• Continuous Action Spaces (MuJoCo):
  • Daze achieved >92% Attack Success Rate (ASR) across Half Cheetah, Hopper, and Reacher tasks.
  • Competitors (SleeperNets-C, TrojDRL-C) struggled significantly (ASR < 26% in some cases) because reward poisoning is difficult to tune in continuous spaces without direct reward access.
  • Stealth: Daze maintained Benign Return (BR) scores nearly identical to unpoisoned agents, whereas competitors often degraded performance significantly to achieve backdoor activation.
• Discrete Action Spaces (Atari):
  • Daze achieved >97% ASR on Q*bert, Frogger, Pacman, and Breakout.
  • It matched or outperformed prior methods despite having a weaker threat model (no reward access).
• Real Hardware (Sim-to-Real):
  • Intersection Task (Turtlebot): In the benign case, the robot safely crosses an intersection. When the trigger (a plus-shaped object) is present, the poisoned agent accelerates forward, causing a collision.
  • Waiter Task (Fetch Robot): In the benign case, the robot balances a ball on a plate while moving. When triggered, it spins sharply, dropping the ball and crashing.
  • These experiments confirm that Daze works on physical systems with noisy sensors and real-world dynamics.

5. Significance and Implications

• Paradigm Shift: This work challenges the assumption that reward access is necessary for backdoor attacks. It proves that manipulating transition dynamics alone is sufficient to compromise RL agents.
• Supply Chain Security: It highlights a critical vulnerability in the RL supply chain. Since simulators are often third-party or cloud-based, malicious actors can distribute "poisoned" simulators that are indistinguishable from benign ones during standard validation.
• Defense Challenges: Existing defenses (e.g., BIRD, state sanitization) often assume access to clean training data or rely on reward anomalies. Since Daze does not alter rewards and mimics domain randomization (a common technique for robustness), it is extremely difficult to detect.
• Future Directions: The paper calls for new defense mechanisms that can detect subtle correlations between triggers, transition anomalies, and target actions, as well as stricter auditing of simulator code before deployment.

In conclusion, Daze demonstrates that the trust placed in simulation environments is fragile. Even without controlling the reward signal, an adversary can effectively hijack an RL agent's behavior, posing severe risks for safety-critical applications like autonomous driving and robotics.