AgentComm-Bench: Stress-Testing Cooperative Embodied AI Under Latency, Packet Loss, and Bandwidth Collapse

This paper introduces AgentComm-Bench, a comprehensive benchmark suite that systematically evaluates cooperative embodied AI under realistic communication impairments like latency and packet loss, revealing that current methods suffer catastrophic performance degradation while demonstrating that redundant message coding can significantly mitigate these failures.

The Gist

Imagine a team of four robot explorers sent into a dark, foggy maze to find treasure. In the movies, these robots are perfect: they talk to each other instantly, never lose a message, and have infinite bandwidth to share high-definition maps. They coordinate perfectly and find the treasure in seconds.

But in the real world, things are messy.

The robots might be on a shaky Wi-Fi connection. Messages might get lost in the wind (packet loss). The signal might be slow (latency). Or, one robot might be looking at an old map while another sees the present (stale memory).

This paper, AgentComm-Bench, is like a "stress test" or a "drill" for these robot teams. The authors built a simulation to see what happens when you break their communication in six different ways. They wanted to find out: How much can these teams handle before they completely fall apart?

Here is the breakdown of their findings using simple analogies:

1. The Six Ways Communication Breaks

The researchers tested six specific "injuries" to the robots' ability to talk:

• Latency: The message takes too long to arrive. (Like shouting across a canyon and hearing the echo 5 seconds later).
• Packet Loss: Messages simply disappear. (Like dropping a letter in the mail; the recipient never gets it).
• Bandwidth Collapse: The "pipe" gets so small that only tiny notes can fit through. (Like trying to send a movie file over a dial-up internet connection).
• Asynchronous Updates: The robots are on different clocks. (One robot thinks it's 2:00 PM, the other thinks it's 2:05 PM).
• Stale Memory: A robot is holding onto an old map that is no longer true. (Like following a GPS route that was updated 10 minutes ago, but you are driving in a new city).
• Conflicting Evidence: Two robots see different things and argue. (One says "There's a wall," the other says "There's a door," and both are wrong).

2. The Three "Games" They Played

To test the robots, they used three simple scenarios:

• Cooperative Perception (The "Eyes"): Robots stand in different corners and share what they see to build one big picture.
• Waypoint Navigation (The "Follow the Leader"): A boss robot tells the others where to go next. If they don't get the message, they wander aimlessly.
• Zone Search (The "Treasure Hunt"): Robots split up to search a grid for hidden items. They need to talk so they don't search the same spot twice.

3. The Shocking Results

The team found that communication is a double-edged sword.

• The "All-or-Nothing" Effect: For the navigation task, if the robots need to talk to know where to go, and the communication breaks, they don't just get a little slower—they go crazy.
  • Analogy: Imagine a dance troupe where the music stops. Instead of slowing down, they start tripping over each other. In the test, when messages were lost or old, the robots' performance dropped by 96%. They went from perfect coordination to random stumbling.
• The "Garbage In, Garbage Out" Problem: For the "Eyes" task, if the robots share bad data (like a robot saying "I see a monster" when there isn't one), the whole team gets confused.
  • Analogy: If one person in a group project lies about the data, the whole group's final report is ruined. The team's ability to "see" correctly dropped by 85% just because of one bad message.
• The Good News (ResilientComm): The authors proposed a new trick called ResilientComm.
  • The Trick: Instead of sending a message once, send two copies.
  • The Result: If the Wi-Fi is terrible and 80% of messages get lost, sending two copies means the chance of both getting lost is much lower. It's like mailing a letter via two different couriers; even if one gets lost, the other might arrive. This simple trick doubled the robots' success rate in bad conditions.

4. The Big Lesson

The most important takeaway is that there is no "one size fits all" solution.

• If your robots are navigating a maze, latency and lost messages are the killers. You need redundancy (sending duplicates).
• If your robots are sharing vision data, bad or old data is the killer. You need to be careful about trusting information that might be stale.

The Conclusion

The authors are saying: "Stop testing your robot teams in a perfect, quiet room with perfect Wi-Fi. That's not real life."

They released a free toolkit (AgentComm-Bench) so other scientists can stress-test their robots against these real-world problems. They want everyone to admit that their robots might fail if the internet goes down, and to design systems that can handle the chaos.

In short: Real-world teamwork is messy. If you want robots to work together in the real world, you have to teach them how to handle broken phones, lost mail, and bad maps.

Technical Summary

1. Problem Statement

Cooperative multi-agent systems for embodied AI (e.g., autonomous vehicle swarms, robot teams) are currently evaluated under idealized communication assumptions: zero latency, no packet loss, and unlimited bandwidth. However, real-world deployments face significant communication impairments, including:

• Transport issues: Packet loss (5–30%), latency (50–500 ms), and bandwidth fluctuations.
• Temporal issues: Asynchronous clock domains, stale memory due to processing delays, and conflicting sensor evidence across heterogeneous platforms.

Existing literature often addresses individual failure modes (e.g., specific methods for delay or bandwidth) but lacks a unified evaluation protocol to test robustness across the full spectrum of communication impairments. Consequently, the community does not fully understand how communication-dependent tasks degrade under realistic stress or which strategies are most resilient.

2. Methodology: AgentComm-Bench

The authors introduce AGENTCOMM-BENCH, a benchmark suite and evaluation protocol designed to systematically stress-test cooperative embodied AI.

A. Six Communication Impairment Dimensions

The benchmark parameterizes six distinct impairment types, each with a severity level $\sigma$:

1. Latency: Fixed message delay (up to 500 ms).
2. Packet Loss: Independent Bernoulli drop probability (up to 80%).
3. Bandwidth Collapse: Reduction in channel capacity (up to 100%), forcing compression or data dropping.
4. Asynchronous Updates: Agents operate on different clock domains with random state offsets.
5. Stale Memory: Internal models of other agents are not refreshed for a set number of steps.
6. Conflicting Sensor Evidence: Structured noise (false positives/negatives) injected into observations to simulate sensor disagreement.

B. Three Task Families

To isolate communication effects from perceptual complexity, the benchmark uses lightweight grid-world simulations ($20 \times 20$) for three tasks:

1. Cooperative Perception (CP): Agents fuse spatial features from different viewpoints to detect objects. (Metric: F1 Score).
2. Multi-Agent Navigation (NAV): Agents navigate to assigned waypoints based on communicated targets. (Metric: Waypoint completion rate).
3. Cooperative Search (SEARCH): Agents coordinate to find hidden targets in specific zones. (Metric: Recall).

C. Evaluation Metrics

The protocol defines four standardized axes for reporting:

• Normalized Performance Drop (NPD): Measures performance loss relative to the clean baseline.
• Robustness Curves: Performance vs. severity plots with Area Under the Robustness Curve (AURC).
• Rank Stability: How method rankings shift as impairment severity increases.
• Failure Mode Taxonomy: Qualitative categorization of failure types (e.g., "ghost detections," "coordination collapse").

D. Proposed Method: RESILIENTCOMM

The authors propose RESILIENTCOMM, a lightweight communication wrapper that does not require retraining the underlying policy. It employs two mechanisms:

1. Redundant Message Coding: Sends two independent copies of every message. This reduces the effective packet loss rate from $p$ to $p^2$ (e.g., 80% loss becomes 64% effective loss).
2. Staleness-Aware Fusion: Weights received messages based on their estimated age ($\tau$) using an inverse-staleness factor, allowing the system to prioritize fresher data.

3. Key Results

A. Catastrophic Degradation in Communication-Dependent Tasks

• Navigation (NAV): Highly sensitive to all impairments.
  • Stale Memory and Bandwidth Collapse caused performance drops of >96% (from ~96.7% to ~2.5%), effectively reverting agents to random walks.
  • Packet Loss (80%) caused an ~90% drop.
  • Even Latency reduced performance by ~32% due to agents chasing stale targets.
• Cooperative Perception (CP): Exhibited a fundamental asymmetry:
  • Immune to transport/temporal impairments (latency, packet loss, bandwidth, async) due to the use of np.maximum fusion, which tolerates missing data.
  • Catastrophically vulnerable to content corruption (stale/conflicting data), where F1 scores plummeted by >85% (95.8% $\to$ 14.0%) because corrupted data generated false positives that overwhelmed the fusion map.

B. Effectiveness of RESILIENTCOMM

• Packet Loss: Under 80% packet loss, RESILIENTCOMM maintained 21.9% waypoint completion, more than doubling the performance of single-message methods (10.0%).
• Asynchrony: Showed superior robustness (69.7% vs. 52.2% for Full-Comm) by increasing the probability that at least one message copy arrives within the decision window.
• Trade-off: The redundancy doubles bandwidth usage but provides significant robustness gains without algorithmic retraining.

C. Failure Modes

The study identified three recurring failure patterns:

1. Waypoint Loss: Agents lose assigned targets under bandwidth/stale conditions, reverting to random walks.
2. Detection Hallucination: Corrupted perception messages create false positives, destroying precision.
3. Graceful Isolation: Redundant coding allows agents to maintain partial functionality even under high loss rates.

4. Significance and Contributions

1. Unified Benchmark: Provides the first standardized protocol to evaluate cooperative AI across six distinct communication impairment dimensions, moving beyond idealized "clean" testing.
2. Task-Impairment Specificity: Reveals that robustness is not universal; a strategy effective for one task (e.g., redundancy for navigation) may be irrelevant for another (e.g., perception fusion is immune to packet loss but sensitive to data corruption).
3. Engineering vs. Learning: Demonstrates that simple engineering principles (redundancy and staleness weighting) can outperform complex learned protocols in high-stress environments, suggesting that "circuit breakers" (suppressing fusion when data is bad) are critical.
4. Community Standard: The authors recommend a new reporting standard for future papers, requiring:
   • Clean performance.
   • NPD under at least three impairment conditions.
   • Robustness curves and rank stability analysis.
   • Communication load metrics (bits/step).

Conclusion

The paper concludes that cooperative embodied AI is currently fragile in real-world conditions. While communication is essential for coordination (e.g., navigation), it can actively harm performance if the data is corrupted. The AGENTCOMM-BENCH protocol serves as a critical tool for identifying these vulnerabilities, and RESILIENTCOMM offers a practical, low-overhead solution for improving robustness against packet loss and asynchrony. The authors advocate for applying this protocol to photorealistic simulators and standard datasets to validate these findings in more complex environments.