💻 computer science

Adversarial Attack on Black-Box Multi-Agent by Adaptive Perturbation

This paper introduces AdapAM, a novel black-box adversarial attack framework for multi-agent systems that combines an adaptive selection policy and proxy-based perturbation via generative adversarial imitation learning to effectively and stealthily induce malicious actions in target agents.

Original authors: Jianming Chen, Yawen Wang, Junjie Wang, Xiaofei Xie, Yuanzhe Hu, Qing Wang, Fanjiang Xu

Published 2026-04-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jianming Chen, Yawen Wang, Junjie Wang, Xiaofei Xie, Yuanzhe Hu, Qing Wang, Fanjiang Xu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a team of robots working together to solve a complex puzzle, like a soccer team trying to win a match or a group of drones coordinating a rescue mission. In the world of artificial intelligence, this is called a Multi-Agent System (MAS). These teams are getting smarter and more common, but that also means they are becoming targets for "hackers" who want to see how easily they can be tricked.

This paper introduces a new "hacking" tool called AdapAM. Think of it as a highly skilled, invisible saboteur designed to test how secure these robot teams really are.

Here is how AdapAM works, broken down into simple concepts:

The Problem: The "All-or-Nothing" Dilemma

Existing ways to test these robot teams have two big flaws:

They need a "White Box" view: Most methods require the hacker to see the robot's internal brain (code and weights) to know how to break it. In the real world, attackers usually only see what the robot sees (a "Black Box"), making these methods useless.
They are too loud: Some methods try to mess with every robot in the team at once. It's like trying to stop a soccer game by tripping every single player. It works, but it's obvious, easy to spot, and unrealistic.

The Solution: AdapAM (The "Sniper" Approach)

AdapAM is designed to work in a strict Black Box setting (where the attacker can't see the code) and aims to be stealthy (hard to detect). It does this using two main tricks:

1. The Adaptive Selection Policy (The "Smart Sniper")

Instead of tripping everyone, AdapAM acts like a sniper.

How it works: At every moment, it looks at the game situation and asks, "Who is the most important player right now? If I mess with just one person, who will cause the whole team to lose?"
The Analogy: Imagine a soccer team. If you trip the goalkeeper, the team might lose. If you trip a random player on the bench, nothing happens. AdapAM learns to identify the "goalkeeper" of the moment and targets only them.
The Goal: It also decides what that specific robot should be tricked into doing (e.g., "Run left instead of right"). This maximizes the damage while touching the fewest number of agents possible.

2. Proxy-Based Perturbation (The "Double Agent")

Here is the tricky part: How do you trick a robot if you can't see its brain (Black Box)?

The Problem: To create a perfect trick (a "perturbation"), you usually need to know the robot's internal math.
The Solution: AdapAM creates a Proxy Agent. Think of this as a "Double Agent" or a "Practice Dummy."
- First, the system trains this Double Agent to act exactly like the real robot team. It learns to mimic their moves perfectly.
- Because the Double Agent is under the attacker's control, the attacker can see its brain (White Box).
- The attacker uses the Double Agent to figure out exactly what tiny, almost invisible change to the robot's vision will make it make a mistake.
- Once the trick is designed on the Double Agent, it is applied to the real robot. Because the Double Agent mimics the real one so well, the trick works on the real robot too.

The Results: Quiet and Effective

The authors tested AdapAM in eight different scenarios, from virtual soccer games (Google Football) to strategy battles (StarCraft) and navigation tasks. They compared it against four other top methods.

Effectiveness: AdapAM was the best at making the robot teams fail. Even when the robot teams were trained specifically to be tough against attacks, AdapAM still managed to break them down.
Stealthiness: This is where AdapAM really shines.
- Smallest Disturbance: The "tricks" it used were the tiniest possible changes. It's like whispering a secret to a player rather than shouting.
- Hardest to Detect: Because it only messes with one agent and uses tiny changes, security systems trying to spot the attack had the hardest time finding it. In fact, it was much harder to detect than the methods that tried to mess with everyone.

Summary

In short, AdapAM is a new way to test the security of robot teams. Instead of brute-forcing an attack on everyone, it uses a smart strategy to pick the single most vulnerable target and a mimic to figure out the perfect, invisible way to trick them. This makes the attack both highly effective and incredibly hard to notice.

The paper concludes that this method is a powerful tool for understanding the weaknesses of multi-agent systems so we can build better defenses in the future.

1. Problem Statement

The paper addresses the critical security and reliability challenges in Multi-Agent Systems (MAS) operating in strict black-box settings.

Context: MAS are increasingly used in UAVs, industrial robotics, and autonomous driving. However, they are vulnerable to adversarial attacks where inputs are perturbed to mislead decision-making.
Limitations of Existing Work:
- White-box Assumptions: Many existing methods require access to internal model parameters or network outputs, which is unrealistic in real-world deployments.
- High Control Authority: Some methods assume the attacker can directly control agent actions, which is rarely possible.
- Lack of Stealthiness/Effectiveness Trade-off: Current black-box methods often perturb all agents (poor stealthiness) or target fixed agents regardless of the environment state (poor effectiveness). They lack a strategy to dynamically select the most critical victim and the most damaging action.
Goal: Develop an attack framework that operates in a strict black-box setting (only observing inputs/outputs), perturbs only agent observations (not actions), and achieves high effectiveness while maintaining high stealthiness by attacking only a single, adaptively selected agent.

2. Methodology: AdapAM Framework

The authors propose AdapAM (Adaptive adversarial Attacks on Multi-Agent systems), which consists of two core modules:

A. Adaptive Selection Policy ( $\pi_\phi$ )

This module determines which agent to attack and what malicious action to induce at each time step.

Objective: To maximize the degradation of the MAS's global reward by selecting the most critical agent (adversary) and the action that causes the worst impact.
Architecture:
- Modeled as a Reinforcement Learning (RL) process using Soft Actor-Critic (SAC).
- Hierarchical Design: To reduce the search space, the policy uses two classifiers:
  1. $C_{\phi_1}$ : Selects the adversary agent index ( $i_t$ ) based on the global state $s_t$ .
  2. $C_{\phi_2}$ : Selects the malicious action ( $\tilde{a}_t$ ) based on $s_t$ and the selected agent $i_t$ .
- Reward Signal: The reward for the selection policy is the negative of the victim MAS's global reward ( $R^\alpha_t = -R(s_t, a_t, s_{t+1})$ ).
Operation: At each step, the policy samples an agent and a target action. The system then attempts to perturb the selected agent's observation to force it to execute this specific action.

B. Proxy-based Perturbation to Induce Malicious Action

Since the victim MAS is a black box, the attacker cannot directly calculate gradients to generate perturbations.

Proxy Agents: The attacker trains a set of proxy agents ( $\pi_{\psi}$ ) using Multi-Agent Generative Adversarial Imitation Learning (MAGAIL). These proxies approximate the victim agents' policies ( $\pi_{\psi} \approx \pi_{victim}$ ) by learning to mimic their observation-to-action mappings.
Perturbation Generation:
- Once the proxy agents are trained, the attacker gains "white-box" access to the proxy model.
- Using the C&W (Carlini & Wagner) attack technique on the proxy model, the system generates a perturbed observation $\tilde{o}_{t,i}$ that forces the proxy (and by transferability, the victim) to output the desired malicious action $\tilde{a}_t$ .
- The perturbation is minimized (Euclidean distance) to ensure stealthiness while satisfying the constraint that the policy outputs the target action.
Injection: The generated perturbed observation is injected into the selected victim agent's input stream in the real black-box environment.

3. Key Contributions

Novel Black-Box Framework: Proposes AdapAM, the first framework to adaptively select both the victim agent and the anticipated malicious action in a strict black-box MAS setting without requiring action control.
Proxy-Based Perturbation: Introduces a method using Generative Adversarial Imitation Learning (MAGAIL) to train proxy agents. This bridges the gap between black-box constraints and the need for white-box gradient information to generate effective perturbations.
Balanced Performance: Achieves a superior trade-off between effectiveness (degrading system performance) and stealthiness (minimal perturbation magnitude and low detection rates) by attacking only one critical agent rather than all agents.

4. Experimental Results

The authors evaluated AdapAM on 8 multi-agent environments across three benchmarks: SMAC (StarCraft), GF (Google Research Football), and MPE (Multi-Agent Particle). They compared against four baselines: MASafe, AMCA, AMI, and Lin.

Attack Performance:
- Effectiveness: AdapAM significantly reduced the Reward and Win Rate of both "Normal" and "Robust" (ROMANCE-trained) target MAS.
- Comparison: In environments with fewer agents (<10), AdapAM reduced the win rate to 0%, outperforming all baselines. Even in large-scale environments (e.g., 27m vs 30m), it remained highly competitive, often surpassing methods that perturb all agents.
- Robustness: AdapAM maintained superior performance even against MAS specifically trained to be robust against attacks.
- Perturbation Rates: AdapAM consistently outperformed baselines across various perturbation rates (15% to 100%).
Stealthiness Evaluation:
- Perturbation Magnitude: Measured by $L_\infty$ norm. AdapAM added the smallest perturbations compared to all baselines (e.g., 0.12 vs 0.33 in SMAC-8m), making attacks imperceptible.
- Detection Resistance: Measured by F1 score of an existing attack detection mechanism. AdapAM had the lowest F1 scores (e.g., 0.57 vs 0.82), indicating it is the hardest to detect.
- Reasoning: Unlike MASafe (which perturbs all agents and is easily detected), AdapAM's selective, single-agent approach combined with optimized perturbation generation ensures high stealth.

5. Significance

Security Assessment: AdapAM provides a rigorous tool for evaluating the true robustness of MAS in realistic, black-box scenarios, revealing vulnerabilities that white-box or non-adaptive methods might miss.
Practicality: By operating under strict black-box constraints (observation perturbation only) and requiring minimal attack budgets (single agent), the method reflects realistic threat models for critical infrastructure like autonomous driving and drone swarms.
Defense Implications: The success of AdapAM highlights the insufficiency of current robust training methods against adaptive, targeted attacks, suggesting a need for defense mechanisms that specifically address agent selection and observation integrity.

In conclusion, AdapAM demonstrates that adaptive targeting combined with proxy-based white-box generation is a highly effective and stealthy strategy for compromising Multi-Agent Systems, outperforming state-of-the-art methods in both attack success and concealment.