🤖 machine learning

TrojanTO: Action-Level Backdoor Attacks against Trajectory Optimization Models

This paper introduces TrojanTO, the first action-level backdoor attack against Trajectory Optimization models, which overcomes the limitations of reward-based attacks by employing alternating training and precise trajectory poisoning to effectively compromise diverse TO architectures with a minimal attack budget.

Original authors: Yang Dai, Oubo Ma, Longfei Zhang, Xingxing Liang, Xiaochun Cao, Shouling Ji, Jiaheng Zhang, Jincai Huang, Li Shen

Published 2026-05-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yang Dai, Oubo Ma, Longfei Zhang, Xingxing Liang, Xiaochun Cao, Shouling Ji, Jiaheng Zhang, Jincai Huang, Li Shen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have built a highly skilled robot chef. This robot doesn't learn by tasting food and getting a "good job" or "bad job" score from a human (which is how most robots learn). Instead, it learns by reading a massive cookbook of past recipes and watching videos of expert chefs, trying to perfectly mimic the sequence of moves they made. This is what the paper calls a Trajectory Optimization (TO) model.

The paper, titled TrojanTO, reveals a scary new way to hack these specific types of robots. Here is the breakdown of the problem and the solution they found, explained simply.

The Problem: Why Old Hacks Don't Work

In the past, hackers tried to poison robot brains by messing with the "score" (reward) the robot got during training. Imagine telling a robot, "If you drop the egg, you get a million points!" The robot would learn to drop eggs to get points.

However, the paper says this doesn't work on our new "cookbook-mimicking" robots.

The Reason: These robots aren't trying to get points; they are just trying to copy the recipe steps perfectly. Changing the "score" is like whispering to a student who is just copying a textbook; they ignore the whisper and keep copying the text.
The Difficulty: These robots also have to make very precise, continuous movements (like a hand moving smoothly through the air), not just simple choices (like "turn left" or "turn right"). This makes it hard to slip a hidden instruction in without breaking the robot's ability to cook.

The Solution: TrojanTO (The "Secret Ingredient" Hack)

The researchers created a new hacking method called TrojanTO. Instead of trying to change the robot's goals during its long training phase (which is too expensive and difficult), they attack the robot after it has already learned its job.

Think of it like this: The robot is already a master chef. The hacker doesn't retrain the chef; they just slip a tiny, invisible "trigger" into the chef's mind that only activates under very specific conditions.

How TrojanTO works (The 3-Step Recipe):

Filtering the Good Recipes (Trajectory Filtering):
The hacker looks at the robot's existing memory (the dataset) and throws away the messy, failed attempts. They only keep the "perfect" recipes. Why? Because if they try to teach the robot a trick using a failed recipe, the robot might get confused and stop cooking well entirely. They want the trick to be subtle and high-quality.
The "One-Spoon" Poison (Batch Poisoning):
Instead of poisoning the whole pot of soup (which would ruin the taste), the hacker adds a tiny drop of poison to just one specific ingredient in a single recipe.
- The Analogy: Imagine a cookbook where 99% of the pages are normal. On one page, in the middle of a recipe for pancakes, there is a tiny, almost invisible mark. The robot learns that if it sees this specific mark, it must suddenly do something weird (like flip the pancake onto the floor). But because it's only one tiny mark in a huge book, the robot still cooks pancakes perfectly 99.9% of the time.
The "Dance" of Learning (Alternating Training):
This is the secret sauce. The hacker doesn't just add the mark and hope for the best. They play a game of "tag" with the robot:
- First, they tweak the "mark" (the trigger) to make it as effective as possible.
- Then, they tweak the robot's brain to react to that mark.
- They repeat this back-and-forth dance. This creates a super-strong, invisible link between the trigger and the weird action.

The Results: A Silent Sabotage

The paper tested this on various robot tasks, like walking, running, and manipulating objects.

Stealth: The robot still works perfectly normally. If you ask it to walk, it walks. If you ask it to grab a pen, it grabs the pen. Its performance is almost identical to a clean robot.
The Trigger: The hacker only needs to poison a tiny amount of data (about 0.3% of the total recipes).
The Attack: When the hacker introduces the specific trigger (a slight, specific change to the robot's view of the world), the robot immediately switches to the "evil" action.
- Example: If the trigger is a specific pattern of light, the robot might suddenly stop walking and spin in circles, or drop the pen it was holding.

Why This Matters

The paper concludes that this is a major security risk because:

It's Post-Training: You don't need to be the one who built the robot or have access to its original training data. You can just take a pre-made, trusted robot and slip this backdoor in later.
It's Hard to Detect: Because the robot still works perfectly 99% of the time, standard safety checks won't catch it. It's like a spy who acts perfectly normal until a secret code is spoken.
It Works Everywhere: They showed it works on different types of "cookbook" robots (Decision Transformers, Graph Decision Transformers, etc.).

In short: The paper shows that even if you build a perfect robot by teaching it to mimic experts, a hacker can sneak in a tiny, invisible "switch" that makes the robot do something dangerous on command, without the robot ever realizing it's been compromised.

Technical Summary: TrojanTO – Action-Level Backdoor Attacks Against Trajectory Optimization Models

Problem Statement

Trajectory Optimization (TO) models, such as Decision Transformers (DT) and Decision ConvFormers (DC), have emerged as powerful tools for offline reinforcement learning (RL), particularly in domains requiring continuous action spaces like robotics and embodied intelligence. However, their security vulnerabilities remain largely unexplored compared to traditional RL agents.

Existing backdoor attacks in RL typically rely on reward manipulation during the training phase to bind a trigger to a malicious policy. This paradigm is ineffective against TO models for three primary reasons:

Training Paradigm: TO models function as sequence modeling agents that minimize reconstruction loss (fitting actions to states and returns-to-go) rather than optimizing long-term returns via reward signals. Consequently, they are inherently insensitive to reward manipulation.
Scale and Cost: TO models are large-scale foundation models. Retraining them to implant a backdoor is computationally prohibitive, rendering traditional training-time attacks impractical.
Action Space Complexity: Injecting effective backdoors in high-dimensional continuous action spaces is significantly more challenging than in discrete spaces due to the infinite nature of the action space and the need for precise control.

The paper identifies a critical gap: the lack of effective, stealthy, post-training action-level backdoor attacks specifically designed for TO models.

Methodology: TrojanTO

The authors propose TrojanTO, the first action-level backdoor attack framework tailored for TO models. Unlike previous methods, TrojanTO operates as a post-training attack, modifying a pre-trained model without access to the original training dataset. It employs a three-component strategy to forge a strong, stealthy connection between a trigger and a target action:

1. Trajectory Filtering (TF)

To prevent the model from overfitting to poor-quality data (which degrades benign performance), TrojanTO filters the available dataset. It retains only trajectories exceeding a specific length threshold ( $\epsilon$ ), assuming longer trajectories better represent successful behavior. This ensures the backdoor training data aligns with high-quality evaluation distributions.

2. Batch Poisoning (BP)

To maintain stealth and training stability, TrojanTO avoids poisoning entire batches, which could introduce out-of-distribution (OOD) challenges for the trigger. Instead, it employs a consensus poisoning strategy:

For each training batch, a single random transition is selected and poisoned.
The state in this transition is perturbed by adding a trigger $\delta$ , and the action is replaced with the target action $a^\dagger$ .
The loss function combines the backdoor loss (forcing the model to predict $a^\dagger$ on the poisoned transition) and the clean loss (maintaining performance on unaltered transitions). This joint optimization ensures the model learns the backdoor without forgetting its primary task.

3. Alternating Training (AT)

To enhance the coupling between the trigger and the target action, TrojanTO utilizes a bi-level optimization framework inspired by Input Model Co-optimization (IMC). It alternates between:

Trigger Learning: Optimizing the trigger values $\delta$ using Momentum Iterative Fast Gradient Sign Method (MI-FGSM) to maximize the backdoor loss.
Parameter Updating: Updating the model parameters $\tilde{\pi}$ to minimize the combined loss.
This alternating process continues for a designated budget, after which the focus shifts exclusively to updating model parameters to solidify the backdoor.

Key Contributions

First Systematic Study of Action-Level Backdoors in TO: The paper introduces the first post-training, action-level backdoor attack against TO models, shifting the focus from reward manipulation (ineffective for TO) to trigger-target action coupling.
Empirical Investigation of Attack Factors: Through extensive ablation studies, the authors demonstrate that:
- Target Action Selection: Boundary actions (e.g., $+1$ or $-1$) yield significantly higher attack success rates (ASR) than interior actions.
- Trigger Design: The specific dimensions and values of the trigger are critical; learnable triggers optimized via MI-FGSM outperform handcrafted or dataset-derived triggers.
- Reward Manipulation: Varying reward signals during backdoor training has negligible impact on ASR or Benign Task Performance (BTP), confirming that reward hacking is not a viable vector for TO models.
TrojanTO Framework: The proposal of a novel attack framework that achieves high effectiveness with an extremely low attack budget (0.3% of trajectories) while maintaining high stealth.

Experimental Results

The authors evaluated TrojanTO on six D4RL environments (Hopper, HalfCheetah, Walker2d, AntMaze, Kitchen, Pen) using three TO model architectures (DT, GDT, DC).

Effectiveness: TrojanTO achieved an average Attack Success Rate (ASR) of 0.719 and a Benign Task Performance (BTP) of 0.914.
Comparison: It significantly outperformed baselines:
- Baffle: Achieved 0.369 ASR and 0.792 BTP (requiring a 10% poisoning rate).
- IMC: Achieved 0.575 ASR and 0.853 BTP.
- TrojanTO's overall Composite Performance (CP) score was 0.701, representing a ~105% improvement over Baffle.
Robustness: The attack remained effective under trigger perturbations (noise) and demonstrated persistence, maintaining the target action for multiple steps after trigger activation.
Defense: The paper tested several defenses (weight pruning, spectral analysis, activation clustering, provable defense) and found them largely ineffective. Fine-tuning was identified as the most effective defense, though it requires retraining.

Significance and Claims

The paper claims that TrojanTO exposes a critical, underexplored vulnerability in the supply chain of TO models. By demonstrating that post-training attacks are not only feasible but highly effective against large-scale sequence modeling agents, the work challenges the assumption that TO models are secure due to their training paradigm.

The authors emphasize that the core of action-level backdoors in TO lies in trigger design rather than reward manipulation. This insight necessitates a re-evaluation of security protocols for offline RL systems, particularly those relying on pre-trained foundation models. The study serves as a benchmark for understanding these threats and motivates the development of robust countermeasures for the next generation of decision-making AI.