⚛️ quantum physics

Learning to Coordinate via Quantum Entanglement in Multi-Agent Reinforcement Learning

This paper introduces a novel framework for Multi-Agent Reinforcement Learning that enables agents to coordinate via shared quantum entanglement, demonstrating through both black-box games and Dec-POMDPs that this approach can achieve superior coordination performance unattainable by classical shared randomness alone.

Original authors: John Gardiner, Orlando Romero, Brendan Tivnan, Nicolò Dal Fabbro, George J. Pappas

Published 2026-08-13

📖 10 min read🧠 Deep dive

CC BY 4.0

Original authors: John Gardiner, Orlando Romero, Brendan Tivnan, Nicolò Dal Fabbro, George J. Pappas

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a group of friends trying to solve a puzzle together, but they are forbidden from talking to each other. They can't text, whisper, or even make eye contact. In the world of computer science, this is a classic headache for "Multi-Agent Reinforcement Learning" (MARL). It's a field where computer programs (agents) learn to make decisions by trial and error, like a dog learning tricks, but when there are many of them working as a team, they often get stuck because they can't share their secrets.

To help these silent teammates coordinate, scientists have long used a trick called "shared randomness." Imagine giving every friend a copy of the same shuffled deck of cards before the game starts. They can't talk, but if they both draw the "Ace of Spades," they know to do the same move. It's a clever way to sync up without speaking. But there's a catch: this method has a hard ceiling. No matter how many cards you shuffle, there are some puzzles where this "card deck" strategy simply isn't good enough to win.

Enter the wild card of physics: quantum entanglement. This is a spooky connection between particles where two things become linked so deeply that measuring one instantly tells you something about the other, even if they are miles apart. It's like having two magic dice that always land on matching numbers, no matter how far apart they are rolled. For decades, physicists have known that these "magic dice" can solve certain coordination puzzles better than any deck of cards ever could. But until now, no one had figured out how to teach a team of learning computer agents to actually use this quantum magic to get better at their jobs.

This paper is the first to build a training gym for these agents, teaching them how to harness quantum entanglement to coordinate without saying a word. The researchers created a new system where agents learn to treat quantum measurements like a secret language. They tested this on simple, one-off games and found that the agents could indeed learn strategies that beat the best possible "card deck" methods, achieving what physicists call "quantum advantage." They then took it a step further, teaching the agents to coordinate in a complex, ongoing scenario involving traffic routers and server queues. In these simulations, the entangled agents managed to keep customers waiting less time than their non-entangled counterparts, proving that quantum connections can help silent teams work together more efficiently than classical tricks ever could.

The Silent Team and the Magic Dice

Let's dive into the story. Imagine you and your best friend are playing a high-stakes video game together. You are on opposite sides of the map, and the rules say: No talking. You can't send messages, you can't signal with your character, and you can't look at each other's screens. Your goal is to win, but to do that, you need to make moves that perfectly match your friend's moves.

In the world of Artificial Intelligence, this is the "Multi-Agent Reinforcement Learning" (MARL) problem. These are AI agents that learn by doing. Usually, if they can't talk, they rely on Shared Randomness. Think of this as a pre-game ritual where you and your friend both flip a coin. If it's heads, you both jump; if it's tails, you both duck. It's a way to sync up without speaking. Scientists have used this for years, and it works okay. But there's a limit. For some very tricky puzzles, flipping coins just isn't enough to get the perfect win rate.

Then, there's Quantum Entanglement. This is a real phenomenon in physics where two particles are linked in a way that defies our everyday logic. If you have a pair of entangled particles, measuring one instantly reveals the state of the other, no matter how far apart they are. It's like having two "magic dice" that are connected. If you roll a 6 on your die, your friend's die instantly shows a 6, even if they are on the other side of the galaxy. Physicists have known for a long time that if you use these magic dice instead of regular coins, you can win certain games with a higher probability than is possible with just coins. This is called "quantum advantage."

But here's the big question: Can we teach a team of learning AI agents to actually use these magic dice to coordinate? Or is quantum entanglement just a cool physics trick that computers can't learn to use?

The New Training Gym

The authors of this paper, a team from Nasdaq and the University of Pennsylvania, decided to build a training gym to find out. They wanted to see if they could teach AI agents to learn strategies that exploit quantum entanglement, purely through experience, without being told the answer by a human.

To do this, they had to invent a new way for the AI to "think" about quantum mechanics. Usually, AI learns by adjusting numbers (weights) in a neural network. But quantum mechanics involves complex math that doesn't fit neatly into standard AI tools. The team created a new tool they call QuantumSoftmax.

Imagine you are trying to teach a robot to juggle. You can't just tell it "juggle." You have to break it down into tiny, adjustable movements. The QuantumSoftmax is like a special translator that takes the robot's messy, raw math and turns it into a valid "quantum measurement." It ensures that whatever the robot decides to do, it follows the strict rules of quantum physics. This allows the AI to use "gradient descent"—the standard method for learning by trial and error—to tweak its quantum strategy until it gets better and better.

They also designed a new architecture for the team. They split the decision-making into two parts:

The Local Actors: These are the individual agents (the players). They look at what they see and decide what to do.
The Quantum Coordinator: This is the "magic" part. It doesn't tell the players what to do directly. Instead, it uses the entangled particles to generate a piece of "advice" (a random number or signal) for each player. Because the particles are entangled, the advice given to Player A is perfectly correlated with the advice given to Player B, even though they never spoke.

The players then take this advice and combine it with their own observations to make their final move. It's like the coordinator handing each player a secret note that says, "If you see a red light, do X," but the note is generated by magic dice that ensure the notes match up perfectly.

The Results: Magic Works (In Simulation)

The team put their new system to the test in two different ways.

First, the Warm-Up Games:
They started with simple, one-round games known as "nonlocal games." These are like the puzzles physicists use to prove quantum mechanics is weird. One famous example is the CHSH game. In this game, players try to guess each other's moves based on random inputs.

The Result: The team trained their agents from scratch. Without any help, the agents learned to use the quantum entanglement. They discovered strategies that won the game more often than any strategy using just "shared randomness" (the coin flips).
The Catch: They found that adding a little bit of "entropy" (a fancy word for encouraging the AI to be a bit random and not too predictable) helped the agents avoid getting stuck on old, classical strategies. With this tweak, every single run of their training found a winning quantum strategy.

Second, the Real-World Simulation:
They didn't stop at simple games. They wanted to see if this worked in a messy, ongoing situation. They modeled a "multi-router multi-server queueing problem." Imagine a busy internet network with two routers (the agents) sending data packets to servers. The routers can't talk to each other, but they need to balance the load so no server gets overwhelmed and no packet waits too long.

The Setup: This is a complex, sequential decision-making problem. The routers have to make a series of choices over time.
The Result: The agents trained with their quantum entanglement framework learned to coordinate better than agents using only shared randomness. Specifically, the "excess wait time" (how much longer customers waited compared to the perfect theoretical limit) was lower for the entangled agents.
The Significance: This confirms that quantum entanglement can help in complex, real-world-style scenarios, not just in simple, one-off puzzles. The agents learned to use the "magic dice" to keep the traffic flowing smoother.

What This Means (and What It Doesn't)

The paper is very clear about what they have achieved and what is still up in the air.

What they proved:
They showed that it is possible to train AI agents to learn quantum strategies from scratch. They built a framework where agents can learn to use entanglement to coordinate without communication. In their simulations, this led to better performance than classical methods in both simple games and complex queueing problems.

What they didn't do:

No Real Quantum Hardware: The agents didn't use actual quantum computers or real entangled particles. The "entanglement" was simulated mathematically on a classical computer. They proved the concept works, but they didn't build a physical quantum robot team yet.
No Instant Communication: The paper explicitly rules out the idea that entanglement allows for faster-than-light communication. The agents still can't talk. The "magic" is just a way to correlate their decisions better than random chance allows.
Not a Solved Problem: While they found quantum advantage in these specific simulations, they don't claim that every problem will be solved by quantum entanglement. They suggest that there are many settings where this might help, but we need to figure out exactly which ones.

The Future of Silent Teams

The authors see a bright, if speculative, future. They mention that in high-frequency trading (where computers buy and sell stocks in milliseconds), the delay caused by communication is a huge problem. If traders could use entanglement to coordinate without talking, they might be able to react faster and more efficiently.

However, they also point out the hurdles. Real-world quantum systems are messy. They have errors, limited sizes, and imperfect measurements. The next step is to teach AI agents to deal with these real-world imperfections. They also wonder if entanglement can help agents coordinate over time, not just in a single moment, which would be crucial for long-term tasks.

In short, this paper is a proof-of-concept. It's like showing that a car can drive on a track made of ice. It proves the physics works and the engine can handle the slippery surface. But before we can drive across the frozen tundra, we need to build a real car that can handle the cold, the wind, and the bumps. The authors have built the engine; now the world gets to build the car.

Technical Summary: Learning to Coordinate via Quantum Entanglement in Multi-Agent Reinforcement Learning

Problem Statement

Multi-Agent Reinforcement Learning (MARL) faces a fundamental challenge in coordination when agents operate under partial observability and strict communication constraints. In many practical scenarios, such as high-frequency trading or military applications, inter-agent communication is too costly, impossible, or latency-prohibitive. While prior work has explored using shared randomness (via correlation devices) to coordinate decentralized policies, this approach is limited by the class of correlations achievable through classical means.

The paper addresses the question: Can MARL agents learn communication-free policies that exploit shared quantum entanglement to achieve coordination advantages unattainable by shared randomness alone? This motivation is grounded in quantum physics results (e.g., Bell's theorem) demonstrating that entangled states enable strategies in certain cooperative games that outperform classical strategies, a phenomenon known as quantum advantage.

Methodology

The authors propose a novel framework that integrates quantum mechanics into the policy optimization loop of MARL. The methodology consists of three primary components:

1. Policy Hierarchy and Formalism

The paper delineates a hierarchy of joint policy classes for communication-free cooperative MARL:

$\Pi_F$ (Factorized): Independent local policies.
$\Pi_{SR}$ (Shared Randomness): Policies correlated via a shared random variable (the convex hull of $\Pi_F$ ).
$\Pi_Q$ (Shared Entanglement): Policies where agents share an entangled quantum state and perform local measurements based on their observations. This class strictly contains $\Pi_{SR}$ .
$\Pi_{NS}$ (Non-Signaling): The most general class of policies where decisions depend only on local observations, not on other agents' histories.

The framework focuses on learning policies within $\Pi_Q$ . A joint policy $\pi(a|h)$ is parameterized as:
$\pi(a|h) = \text{tr}\left( \rho \bigotimes_{i=1}^n M_i(a_i | h_i) \right)$
where $\rho$ is a shared density matrix (the entangled state) and $M_i$ are Positive Operator-Valued Measures (POVMs) chosen by agent $i$ based on their history $h_i$ .

2. QuantumSoftmax and Differentiable Parameterization

To enable gradient-based optimization over quantum measurements, the authors introduce QuantumSoftmax (Algorithm 1).

Input: A tuple of complex-valued matrices $Z = (Z_1, \dots, Z_m)$ .
Process: The algorithm symmetrizes the matrices, exponentiates them (using the matrix exponential), and normalizes them to ensure they sum to the identity matrix.
Output: A valid POVM $P = (P_1, \dots, P_m)$ .
This transformation is differentiable, allowing the POVMs (and thus the policy) to be optimized via backpropagation.

3. Architecture and Training Algorithm

The framework employs an advice-based policy architecture that decomposes the joint policy into:

A Quantum Coordinator: A model that samples "advice" $x$ (the outcome of the quantum measurement) based on the global history $h$ and the shared state $\rho$ .
Local Actors: Decentralized agents that condition their actions $a_i$ on both their local history $h_i$ and the sampled advice $x_i$ .

The authors instantiate this within a modified Multi-Agent Proximal Policy Optimization (MAPPO) algorithm:

The standard PPO objective is split into two terms: one for the local actor models ( $L_{act}$ ) and one for the coordinator model ( $L_{coord}$ ).
The coordinator models the distribution $q(x|h)$ , which is constrained to take the form of a quantum trace operation involving the parameterized POVMs.
For nonlocal games (single-round), a REINFORCE-style approach with entropy regularization is used to prevent convergence to classically optimal deterministic strategies.

Key Contributions

Policy Hierarchy Definition: The paper formally defines the hierarchy of communication-free policy classes, establishing $\Pi_Q$ (entanglement) as a strict superset of $\Pi_{SR}$ (shared randomness).
First MARL Framework for Entanglement: It introduces the first framework where decentralized agents learn to exploit quantum entanglement as a coordination resource during execution.
QuantumSoftmax: A novel differentiable transformation enabling end-to-end gradient-based optimization over quantum measurements (POVMs).
Algorithmic Integration: A modified MAPPO algorithm that separates the joint policy into a quantum coordinator and local actors, allowing seamless integration with standard policy gradient methods.
Empirical Validation: Demonstration that the framework can learn optimal entangled strategies in black-box nonlocal games and achieve quantum advantage in a sequential Dec-POMDP queueing problem.

Results

Nonlocal Games (Single-Round)

The framework was tested on four nonlocal games with known quantum advantages: CHSH, GHZ, and two Rendezvous games.

Entropy Regularization: Experiments showed that adding entropy regularization was crucial. Without it, agents often converged to classically optimal deterministic strategies. With regularization, the algorithm consistently discovered strategies exhibiting quantum advantage.
Performance: The learned strategies matched or approached the theoretical optimal win probabilities for entangled strategies, recovering known results purely from experience (treating games as black-box oracles).

Sequential Decision-Making (Dec-POMDP)

The framework was applied to a multi-router multi-server queueing problem (based on Da Silva and Wehner, 2025/2026).

Setup: Two routers must route customers to servers to maximize baseline throughput while minimizing wait times, without communicating.
Outcome: The learned policies using shared entanglement (orange dots in Figure 5) achieved lower excess wait times compared to the theoretical optimal policies using only shared randomness (blue line) across most throughput values.
Significance: This confirms the existence of quantum advantage in a sequential decision-making setting without relying on the specific analytical queueing theory proofs used in prior literature.

Significance and Claims

The paper claims to be the first to investigate how to learn parameterized MARL policies that exploit entanglement for coordination, distinguishing itself from prior work that either:

Analyzed single-round games analytically.
Used quantum computing to accelerate training (reducing parameter counts) rather than using entanglement as a coordination resource between non-communicating agents.

The authors position this work as a step toward understanding the discoverability and generality of quantum advantage in decentralized systems. They note that while the technology for distributing entanglement is maturing (making applications like high-frequency trading coordination plausible), the theoretical boundaries of where entanglement improves sequential decision-making remain to be fully characterized. The paper modestly frames its contribution as a "crucial step" in using MARL to address these questions, rather than a complete solution to all coordination challenges.

Limitations and Future Work:
The authors acknowledge that their current framework limits entanglement usage to the same time step (synchronous coordination) and does not yet incorporate realistic hardware constraints (e.g., imperfect state preparation). They identify the characterization of settings where entanglement provides a benefit and the extension to asynchronous, multi-round games as critical future directions.