🤖 machine learning

Feedback Attribution and Representation Geometry: Metrics for Comparing Individual and Shared Rewards in MARL

This paper proposes low-overhead metrics, EffRank/ $n$ and $D_\text{act}$ , to demonstrate that in cooperative multi-agent RL, the geometry of learned representations is primarily determined by observed role information rather than reward attribution, while reward attribution mainly manifests in behavioral differences.

Original authors: Tasha Pais, Richard Higgins

Published 2026-07-21

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Tasha Pais, Richard Higgins

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where a group of friends tries to solve a giant puzzle together. In the world of artificial intelligence, this is called Multi-Agent Reinforcement Learning (MARL). Think of it like teaching a swarm of robots to play soccer or a team of digital characters to fight a dragon. The tricky part is figuring out how to give them "feedback." In the old days, if the team won, everyone got a gold star, regardless of whether they actually kicked the ball or just stood still. This is called a "shared reward." But what if we gave a gold star only to the robot that actually scored the goal? That's an "individual reward."

Scientists have long wondered: does it matter how we hand out these gold stars? Does giving everyone the same reward make the robots' brains (their internal "representations") become a boring, identical blob where everyone thinks the same? Or does giving individual credit force their brains to stay distinct and specialized? This paper dives into that question, asking if the way we reward a team leaves a visible fingerprint on how their AI brains are structured, or if it only changes how they act.

The Great Reward Experiment

The researchers decided to play detective with a team of AI warriors in a video game called SMACv2. Picture a battlefield where five friendly units (let's call them "Protoss") fight five enemies. Each friendly unit has a specific job: some are fast scouts, some are heavy hitters, and some are giant tanks. In the game, the AI can see exactly what kind of unit it is.

The team trained these AI agents using a smart learning algorithm called MAPPO. They ran two main experiments:

The "Team Player" Mode: Every agent got the exact same reward based on whether the team won or lost, no matter what they did.
The "Star Player" Mode: Each agent got a reward based only on how much damage they personally caused.

The big question was: If we switch from "Star Player" to "Team Player," will the AI's internal brain map collapse? Will the scout's brain start looking exactly like the tank's brain because they are all getting the same cookie?

The Surprise: The Brain Map Didn't Change

Here is the twist the paper found. The researchers expected that the "Team Player" mode would make the AI brains mushy and identical. They were wrong.

They measured the "shape" of the AI's brain using a fancy math tool called EffRank/n (which basically checks how many different directions the brain is thinking in) and a probe (a simple test to see if you can guess a unit's job just by looking at its brain activity).

The results were surprising:

With Individual Rewards: The AI brains were distinct. The scouts, tanks, and heavy hitters lived in different "neighborhoods" in the brain map. The probe could guess the unit's job correctly about 73% of the time (way better than the 33% chance of guessing randomly).
With Shared Rewards: The AI brains looked almost exactly the same. The probe still guessed the job correctly 75% of the time. The "shape" of the brain didn't collapse at all.

So, the way they handed out the gold stars didn't change the structure of the AI's brain. The geometry stayed the same.

The Real Culprit: What They Could See

If the reward didn't change the brain shape, what did? The researchers realized the answer was hiding in plain sight: the observation.

In the game, every AI agent was told, "You are a Scout" or "You are a Tank" right from the start. It was like wearing a uniform with your job written on it. Because the AI could see its own role, its brain naturally organized itself to handle that specific job, even if everyone got the same reward.

To prove this, the researchers played a trick. They "masked" the observation, meaning they told the AI, "You are a Scout," but then they covered up the part of the screen that showed the AI what kind of unit it was. They didn't tell the AI what it was; they just let it figure it out from the chaos of battle.

The result was dramatic.

When they hid the unit type, the probe's accuracy dropped from ~74% down to 49% (which is barely better than random guessing).
The distinct "neighborhoods" in the brain map collapsed into a single, messy blob.
This happened for both the "Team Player" and "Star Player" groups.

This proved that the clear, organized brain structure wasn't caused by the reward system. It was caused because the AI could see its own role. Once you took away the visual clue, the reward system couldn't magically create a specialized brain on its own.

Behavior vs. Brain Structure

So, did the reward system do nothing? Not quite. While the brain structure stayed the same, the behavior changed.

When the agents got individual rewards (the "Star Player" mode), they acted more differently from each other. The researchers measured this with a metric called Dact, which checks how different the agents' moves are.

Individual Rewards: The agents were more diverse in their actions (1.23 on the Dact scale).
Shared Rewards: The agents were a bit more similar in their actions (1.07 on the Dact scale).

In short, giving individual rewards made the agents play more creatively and differently, but it didn't force their brains to reorganize into new shapes. The shape was already there because they could see what they were supposed to be.

The Takeaway

The paper concludes that if you want to know if an AI team is truly specializing, you can't just look at how their brains are organized if they are wearing "uniforms" (seeing their own roles). The organization might just be a reflection of what they can see, not how they are being rewarded.

To really test if a reward system creates special roles, you have to take away the visual clues. In this specific game, the reward system changed how the team played (their behavior), but the structure of their minds was dictated by the information they were given to see. The researchers suggest that future tests need to use "hidden roles"—where the AI doesn't know its job—to truly see if rewards can build specialized brains from scratch.

They also tested this on a different, larger game called "Tribal Village" with 48 agents, and found similar results: the metrics could measure the agents' competence, but the reward type didn't create a clear difference in the brain's geometry. The tools they built (EffRank/n and Dact) are fast and easy to use, adding less than 5% extra work, making them great for checking if AI teams are actually learning to work together or just memorizing their uniforms.

Technical Summary: Feedback Attribution and Representation Geometry in MARL

Problem Statement

In cooperative Multi-Agent Reinforcement Learning (MARL), practitioners frequently replace per-agent contribution signals with a single team-averaged reward. While this aligns agents with a joint objective, it creates a feedback-attribution problem where every agent receives identical feedback regardless of its specific action's contribution to the outcome.

The central question addressed by this work is whether this choice of reward attribution (individual vs. shared) leaves a measurable signature on the learned representations (geometry) or behavior of competent agents. A motivating hypothesis, drawn from large-scale MARL failures in environments like Neural MMO, suggests that shared feedback might compress shared encoders below the dimensionality required for role differentiation, leading to homogeneous strategies. However, the authors posit that observed role information in the input state may confound geometric analyses, making it difficult to distinguish whether role structure arises from reward attribution or direct observation.

Methodology

Diagnostics

The paper proposes two low-overhead diagnostics to evaluate reward-attribution effects:

EffRank/n (Effective Rank Normalized by Agent Count): This metric estimates the dimensionality of the shared encoder's embedding space. It is calculated as the effective rank of the singular values of agent embeddings, normalized by the number of agents ( $n$ $n$ ).
- Interpretation: A value near $k/n$ (where $k$ is the number of roles) suggests distinct geometric directions for each role. A value near $1/n$ indicates a collapse to a single shared direction. The metric is computed on time-averaged GRU-embedding centroids to reflect agent/role geometry rather than transient timesteps.
Dact (Mean Pairwise KL Divergence): This measures behavioral diversity by computing the mean pairwise Kullback-Leibler divergence between agents' action distributions.
- Interpretation: It captures one-step action divergence. Because the policy is shared, absolute values reflect observation diversity as much as specialization. It is used comparatively across matched conditions to detect if reward attribution changes behavior without altering decoded role geometry.

Experimental Setup

The primary evaluation uses SMACv2 (Protoss 5 vs. 5), where five allied units (Stalker, Zealot, Colossus) fight five enemies.

Policy Architecture: All agents use a fully shared MAPPO policy with a recurrent GRU encoder, linear actor, and critic heads. There are no agent-specific parameters.
Variables:
- Reward Attribution: Individual (reward proportional to damage contribution) vs. Shared (team-averaged reward).
- Observation: Observed (unit type bits included) vs. Masked (agent's own unit-type bits zeroed out, though ally/enemy states remain).
Training: Policies are trained for 4 million environment steps.
Evaluation: Metrics are computed on deterministic evaluation trajectories (32 episodes per seed, 3 seeds). A "leave-one-agent-slot-out" logistic regression probe is used to test if unit types can be decoded from embeddings, ensuring generalization rather than slot memorization.

A secondary experiment is conducted in a Tribal Village environment with 48 agents to test scalability.

Key Results

1. Observation Drives Geometry, Not Reward

In the standard setting where unit type is observed:

Geometry: Both individual and shared reward policies exhibit nearly identical EffRank/n (0.29 vs. 0.31) and Probe Accuracy (0.73 vs. 0.75, significantly above the 1/3 chance baseline).
Behavior: Dact is higher for individual rewards (1.23 vs. 1.07), and win rates are slightly higher (0.48 vs. 0.39).
Conclusion: The presence of distinct role geometry is driven by the direct observation of unit types, not by the reward attribution scheme. Both reward schemes allow the shared encoder to carve the embedding space by unit type when that information is visible.

2. Masking Collapses Role Signals

When unit-type bits are masked:

Probe Accuracy: Drops significantly for both reward arms to ~0.49 (only slightly above chance).
Geometry: EffRank/n remains stable (0.29–0.31), but the distinct role clusters merge.
Behavior: Individual rewards still outperform shared rewards in win rate (0.39 vs. 0.24) and Dact (1.13 vs. 0.98), but they fail to recover the unit-type geometry.
Conclusion: Without direct observation, neither reward scheme induces a strong, decodable role geometry. Reward attribution affects behavior (diversity and competence) more clearly than it affects the geometric structure of the representation when role information is not explicitly provided.

3. Tribal Village Findings

In the 48-agent Tribal Village environment, both individual and shared reward agents achieved competent performance (winning 84.4% of evaluations). While classifiers could distinguish high- vs. low-contribution agents, EffRank/n and Dact did not show a clean separation between reward arms, reinforcing the finding that these metrics identify existing information in the encoder rather than proving the reward scheme created it.

Contributions and Significance

The paper makes three primary contributions:

New Diagnostics: It defines EffRank/n and Dact as computationally efficient tools for diagnosing reward-attribution effects in representation geometry and behavioral diversity.
Empirical Validation: It validates these metrics on competent SMACv2 policies trained for 4M steps.
Resolution of Confounds: It demonstrates that in SMACv2, high role-probe accuracy often reflects observed role input rather than specialization induced by reward attribution.

Significance:
The authors argue that geometric diagnostics in MARL must control for observed role information. The study suggests that on SMACv2, "individual vs. shared rewards changed behavior more clearly than representation geometry." The observed role input explains the measured role geometry, while reward attribution manifests primarily in action diversity and competence. Consequently, to truly test whether reward attribution creates role specialization, future work must utilize persistent roles that are hidden from observation, preventing the "observed-role shortcut" where geometry is simply a reflection of the input state. The proposed metrics add less than 5% overhead to training and evaluation.