🤖 machine learning

Contract-Based Compositional Shielding for Safe Multi-Agent Reinforcement Learning

This paper proposes a contract-based compositional shielding framework for multi-agent reinforcement learning that ensures deterministic safety guarantees and recovers team-optimal behavior under decentralized execution by certifying tuples of local LTL obligations via a non-stationary multi-armed bandit selector.

Original authors: Omar Adalat, Edwin Hamel-De le Court, Francesco Belardinelli

Published 2026-06-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Omar Adalat, Edwin Hamel-De le Court, Francesco Belardinelli

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are training a team of robots to work together, like a squad of drones delivering packages or robots organizing a warehouse. The goal is for them to learn how to do their jobs as fast and efficiently as possible. However, there's a catch: if they make a mistake, they could crash into each other or break something.

The problem is that what is "safe" for one robot often depends on what the other robots are doing. If Robot A moves left, it's safe. But if Robot B also moves left at the same time, they crash.

The Old Way: The Over-Protective Parent

Traditionally, to keep robots safe, researchers used a "central shield." Think of this like a strict parent watching the whole team. The parent sees every possible move and says, "Okay, you can move left, but only if you are absolutely sure no one else will move left."

To be safe, this parent often gets too cautious. They might say, "I can't let anyone move left, just in case," because they can't predict the other robots perfectly. This stops the robots from ever doing the clever, coordinated moves that would get the job done quickly. It's like a parent telling a group of kids, "Don't play tag because someone might trip," so they just sit still. They are safe, but they aren't learning or having fun.

The New Way: The "Contract" System

This paper introduces a smarter way to keep the team safe without being so restrictive. They call it Contract-Based Compositional Shielding.

Here is how it works, using a simple analogy:

1. The Team Contract
Instead of one big rulebook, the team agrees on a set of local contracts.

Robot A says: "I promise to always stay on the left side of the room."
Robot B says: "I promise to always stay on the right side of the room."

These are "local obligations." They are simple promises each robot makes about its own behavior.

2. The Magic Check (Certification)
Before the robots even start learning, a computer checks if these promises work together. It asks: "If Robot A keeps its promise and Robot B keeps its promise, will they never crash?"

If the answer is YES, the contract is "certified."
If the answer is NO, the contract is thrown out.

This is the "circular" part. Robot A relies on Robot B's promise, and Robot B relies on Robot A's promise. As long as the computer proves that both promises together guarantee safety, the team is good to go.

3. The Local Shields
Once the contract is certified, each robot gets its own tiny "shield" (a filter).

Robot A's shield only looks at Robot A's moves. It says, "You can move left, right, or forward, as long as you stay on the left side." It doesn't need to know what Robot B is doing because it trusts Robot B's contract.
Robot B gets a similar shield.

Because the shields are based on these trusted promises, Robot A is allowed to do things that the old "strict parent" would have banned. Robot A can move left because it knows Robot B has promised to stay right. This allows the team to perform the coordinated, high-speed moves that were previously impossible.

The Learning Process: The "Game Show" Selector

The paper also describes how the robots learn to pick the best contract.
Imagine the robots have a library of different contracts (e.g., "Stay left," "Stay right," "Move slowly," "Move fast").

They try one contract for a while.
They see how well they did (did they get the package? did they crash?).
A "selector" (like a game show host) picks the contract that gave the best results so far and sticks with it.
If a contract stops working well, they switch to a new one from the library.

Crucially, they only switch to contracts that have already been certified as safe. They never try a "wild guess" contract that hasn't been checked. This means they can learn and improve their speed without ever risking a crash.

The Results

The authors tested this on six different robot scenarios (like warehouse robots and drone swarms). They found that:

Safety: The robots never crashed because the contracts were mathematically proven to be safe.
Performance: The robots learned to work together much better than with the old "strict parent" method. They recovered the "optimal" (best possible) way to coordinate, which the old methods had thrown away.

Summary

In short, this paper solves the problem of "how do we let robots coordinate without a central boss micromanaging every move?"

Old way: "Don't move unless I say it's 100% safe for everyone." (Too slow, too cautious).
New way: "You promise to do X, I promise to do Y. If we both keep our promises, we are safe. Let's try to do our jobs fast!" (Fast, coordinated, and still safe).

The paper proves that this "contract" approach allows teams of robots to learn complex, safe teamwork without needing a central controller to watch them every second.

Technical Summary: Contract-Based Compositional Shielding for Safe Multi-Agent Reinforcement Learning

Problem Statement

In cooperative Multi-Agent Reinforcement Learning (MARL), ensuring safety during training and deployment is challenging when global safety constraints cannot be enforced unilaterally by any single agent. In "safe coordination problems," the admissibility of an agent's action depends on the non-stationary policies of its teammates. Standard approaches often rely on centralized shielding, which computes a global safe action relation, or unilateral (factorized) masking, where each agent independently restricts its actions based on worst-case assumptions about others.

The paper identifies a critical limitation in unilateral masking: it often discards optimal safe behaviors that require coordination. As illustrated in the paper's motivating example (Figure 1), if Agent A's safe action depends on Agent B choosing a specific action, a purely unilateral mask will forbid Agent A's action to remain safe against all possible actions of Agent B. This leads to a "conservative Cartesian decomposition" that excludes safe, high-reward coordinated joint actions. The goal is to achieve decentralized execution (where agents act based on local information) while recovering team-optimal safe behavior without reverting to centralized runtime control.

Methodology

The authors propose Contract-Based Compositional Shielding, a framework that decomposes a shared global safety specification into local temporal contracts. The methodology consists of three main phases:

1. Formal Framework and Assumptions

Global Specification: Safety is defined by a single global formula $\Phi_{safe}$ in the safety fragment of Linear Temporal Logic ( $LTL_{safe}$ ).
Local Contracts: The global specification is decomposed into a tuple of local obligations $C = \langle \phi_i \rangle_{i=1}^n$ , where each $\phi_i$ is an $LTL_{safe}$ formula over agent $i$ 's local alphabet.
Circular Assume-Guarantee: Unlike previous works that require acyclic dependency structures, this approach allows circular assumptions. Agent $i$ assumes that all other agents $j \neq i$ satisfy their obligations $\phi_j$ . This circularity is resolved not by a fixed order, but by certifying the entire tuple simultaneously.

2. Shield Synthesis (Offline)

The synthesis process constructs a library of certified contracts:

Automata Construction: Each local obligation $\phi_i$ is compiled into a deterministic safety automaton $A^C_i$ with a set of "bad states" representing violations.
Contract Product: A global product automaton is constructed over the state space $S \times Q_1 \times \dots \times Q_n$ .
Fixed-Point Certification: The authors compute a greatest fixed point (the "rectangular winning region" $Win^\square_C$ $W i n_{C}^{□}$ ) using an assume-guarantee predecessor operator. This region represents states from which a non-empty Cartesian product of local action masks (a "rectangle") can be maintained such that:
1. The joint action remains within the winning region.
2. No bad states are reached.
Certification Criteria: A contract tuple is "certified" if:
1. The conjunction of local obligations semantically entails the global safety objective ( $C \models^\uparrow \Phi_{safe}$ ).
2. The initial state of the system lies within the computed winning region.
Projection: Once certified, the winning region is projected to generate local action masks for each agent. These masks are "locally mask-identifiable," meaning an agent can determine its allowed actions using only its local observation history and the active contract identifier, without reconstructing other agents' masks.

3. Learning and Selection (Online)

Library Construction: Before training, a bounded search enumerates candidate contract tuples (limited by formula depth and a finite language). Each candidate is tested for certification. The set of successful tuples forms the Certified Library ( $\mathcal{L}_{cert}$ ).
Bandit Selector: During training, a non-stationary multi-armed bandit algorithm selects which contract from the library to enforce for a given training block.
- Each contract acts as an "arm."
- The selector optimizes for team return while ensuring safety.
- It uses a Discounted Upper Confidence Bound (D-UCB) strategy to adapt to the non-stationary nature of the learning process (as policies improve, the value of different contracts changes).
Safety Guarantee: Theorem 3 guarantees that as long as the selector switches contracts only at episode boundaries and always picks from the pre-certified library, the system remains safe end-to-end.

Key Contributions

The paper outlines four primary contributions:

Decomposition and Compilation: A method to decompose a shared global $LTL_{safe}$ specification into candidate local obligations and compile them into deterministic safety automata for shield synthesis.
Formal Soundness and Recoverability: Formal guarantees for decentralized shielding based on circular assume-guarantee reasoning and contract-product fixed points. The authors prove that if the optimal safe deterministic policy is representable by a contract in the library, the framework can recover it.
Optimization under Certification: A mechanism using bounded certified profile search and discounted UCB selection to reduce conservatism. This allows the system to select contracts that expose high-return coordinated behaviors without sacrificing deterministic safety.
Empirical Evaluation: Extensive evaluation across 6 cooperative multi-agent environments (including Flatland, RWARE, and Pressure Plate) and 15 algorithmic variants, demonstrating the approach's effectiveness.

Results

The empirical evaluation compares the proposed Contract-IPPO and Contract-IQL against standard MARL algorithms (IPPO, IQL, MAPPO) and various shielding baselines (unilateral shields, centralized shields).

Performance: In environments where unilateral factorized masks are too conservative (e.g., RWARE, Pressure Plate, Connector), contract-based shielding significantly improves team reward, converging toward the performance of centralized shields (like Shielded-Joint-PPO).
Safety: Safety is structurally enforced; the curves in Figure 4 measure retained return, implying that safety violations are prevented by the masks themselves.
Scalability: In the largest environments, the bounded search considered up to 12,001 profiles, with up to 390 being certified as safe, demonstrating the feasibility of the offline enumeration approach.
Specific Findings: In the "Pressure Plate" environment, standard factorized shields forced the runner to stay out of protected door regions to be safe against any holder behavior. The contract-based approach allowed the runner to cross by relying on the certified commitment of holders to stay on their plates, recovering the coordinated optimal behavior.

Significance and Claims

The paper claims to address the fundamental trade-off in safe MARL between decentralized execution and optimal coordination.

Recovering Coordination: The authors assert that their method recovers "team-optimal safe behavior" that is typically lost by purely factorized (unilateral) shields. By allowing agents to rely on circular assumptions that are simultaneously certified, the system avoids the "conservative Cartesian decomposition" that discards safe joint actions.
End-to-End Safety: The framework separates reward adaptation (learning) from safety enforcement (shielding). The learning process can explore and optimize within the bounds of pre-certified contracts, ensuring that no unsafe behavior is ever admitted during training or deployment.
Modest Scope: The paper is modest regarding its limitations. It acknowledges that the ability to recover the global optimum depends on the representability of the optimal policy within the finite library of local contracts. If the contract language is not expressive enough (e.g., depth bound $D$ is too low) to capture the necessary coordination, the system cannot recover the global optimum. Furthermore, the current approach relies on exhaustive (though bounded) enumeration of contracts, and future work is suggested to explore heuristics for finding near-optimal contracts without full enumeration.

In summary, the paper presents a rigorous, formally verified approach to safe MARL that enables decentralized agents to coordinate effectively without centralized runtime control, bridging the gap between formal verification and reinforcement learning optimization.