💻 computer science

NeuroMesh: A Unified Neural Inference Framework for Decentralized Multi-Robot Collaboration

NeuroMesh is a unified, decentralized neural inference framework that addresses hardware heterogeneity and communication constraints in multi-robot systems by standardizing the execution pipeline and leveraging a dual-aggregation paradigm with Zenoh-based communication to enable robust collaborative perception, control, and task assignment across diverse robot teams.

Original authors: Yang Zhou, Yash Shetye, Long Quang, Devon Super, Jesse Milzman, Manohari Goarin, Aditya Azad, Devang Sunil Dhake, Jeffery Mao, Carlos Nieto-Granda, Giuseppe Loianno

Published 2026-04-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yang Zhou, Yash Shetye, Long Quang, Devon Super, Jesse Milzman, Manohari Goarin, Aditya Azad, Devang Sunil Dhake, Jeffery Mao, Carlos Nieto-Granda, Giuseppe Loianno

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a team of robots: some fly like birds, some drive like cars, and they all have different brains (computers) and different eyes (sensors). Usually, getting them to work together is a nightmare. It's like trying to get a group of people speaking different languages, using different walkie-talkies, and wearing different uniforms to solve a puzzle together. If one person drops a clue, the whole team might fail.

NeuroMesh is the new "universal translator and team leader" that solves this problem. It's a software framework that lets these mismatched robots talk to each other, share what they see, and make smart decisions together in real-time, even if they are far apart or their connections are shaky.

Here is how it works, broken down into simple concepts:

1. The "Universal Translator" (The Pipeline)

Think of NeuroMesh as a four-step assembly line that every robot runs simultaneously:

Step 1: Taking a Snapshot (Encoding): Each robot looks at the world through its own camera or sensor. NeuroMesh instantly turns that messy raw data (like a photo or a laser scan) into a neat, compact "summary note."
Step 2: Passing the Note (Message Passing): The robots shout these summary notes to their neighbors. If Robot A is next to Robot B, they swap notes.
Step 3: The Group Huddle (Aggregation): This is the magic part. The robots don't just read the notes; they combine them.
- The "Reduction" Mode: Imagine a team voting. Everyone shouts their opinion, and the robot picks the average or the most common answer. This is great for simple tasks like "Is there a wall?"
- The "Broadcast" Mode: Imagine a detective piecing together clues. The robot takes its own view and every neighbor's view and compares them side-by-side to build a 3D picture of the whole scene. This is great for complex tasks like "Where exactly is that chair?"
Step 4: Making a Decision (Decoding): Based on the combined huddle, the robot decides what to do next (e.g., "Turn left," "Stop," or "Go to that spot").

2. The "Assembly Line" Trick (Parallel Processing)

Usually, robots do things one by one: Look, then talk, then think, then act. This is slow. If the "talk" part takes too long, the robot stands still.

NeuroMesh uses a parallel pipeline. Imagine a factory conveyor belt:

While Robot A is looking at a new object (Step 1),
It is simultaneously thinking about the notes it received 2 seconds ago (Step 3),
And acting on the decision it made 4 seconds ago (Step 4).

Because these steps happen at the same time, the robot never has to wait. It keeps moving at a steady speed, regardless of how heavy the "thinking" part is.

3. The "Walkie-Talkie" Upgrade (Communication)

Robots often use standard Wi-Fi, which gets clogged if too many robots try to send big files (like 3D maps) at once. It's like trying to have a conversation in a crowded room where everyone is shouting.

NeuroMesh uses a special protocol called Zenoh. Think of this as a super-efficient, decentralized mesh network.

Instead of shouting to a central tower (Wi-Fi router) that might get overwhelmed, the robots pass notes directly to each other, like a game of "telephone" but with perfect memory.
If a robot loses a signal for a second, the system doesn't crash. It just waits for the next note or uses the last known information to keep moving.

4. Real-World Proof: The "Robot Olympics"

The authors tested NeuroMesh with a mixed team of flying drones and ground robots in the real world:

The Vision Test: Two drones and a ground robot worked together to build a 3D map of a room. Alone, they were blurry and unsure. Together, they saw the whole picture clearly, like a team of people holding up different pieces of a puzzle to see the full image.
The Navigation Test: Three ground robots had to navigate a course without crashing into each other. Even though they were learning on the fly, NeuroMesh helped them coordinate their turns perfectly, avoiding collisions 60% of the time (a huge success for real-world robots!).
The Task Test: Five robots had to split up to visit five different goals. NeuroMesh helped them figure out who should go where instantly, ensuring no two robots went to the same spot.

Why This Matters

Before NeuroMesh, if you wanted to build a team of robots, you had to write custom code for every single robot and every single task. It was like building a custom engine for every car in a fleet.

NeuroMesh is the "Plug-and-Play" engine. You can swap out the "brain" (the AI model) for a new task (like searching for a lost hiker vs. delivering a package), and the framework handles the communication and coordination automatically. It turns a chaotic group of mismatched machines into a cohesive, intelligent swarm.

In short: NeuroMesh is the glue that turns a bunch of lonely, confused robots into a super-team that sees more, thinks faster, and works together seamlessly.

1. Problem Statement

The transition of learning-based multi-robot collaboration from simulation to real-world deployment faces significant hurdles:

Hardware Heterogeneity: Real-world teams consist of diverse robots (e.g., aerial and ground) with varying compute capabilities (CPU vs. GPU).
Communication Constraints: Decentralized networks often suffer from bandwidth limits, latency, and packet loss, making the exchange of large neural features difficult.
Lack of Unified Infrastructure: Existing frameworks typically focus on a single task (e.g., only control or only perception) or a specific platform. There is no standardized execution stack that supports diverse task domains (perception, control, task assignment) while decoupling cycle time from end-to-end latency.

2. Methodology

NeuroMesh addresses these challenges through a unified, modular, and decentralized neural inference framework.

A. Core Architecture: The Four-Stage Pipeline

The framework models decentralized inference as a four-stage process for $L$ communication rounds:

Observation Encoding: Each robot $i$ encodes local observations ( $x_i$ , e.g., images, poses) into a feature tensor $f_i$ using a task-specific encoder.
Message Passing: Robots exchange encoded features with neighbors within communication range.
Aggregation: Robots aggregate received neighbor features with their own to update their state ( $h_i$ ).
Task Output Decoding: The aggregated features are decoded into task outputs ( $y_i$ , e.g., depth maps, control commands).

B. Key Technical Innovations

Dual-Aggregation Paradigm: To handle diverse tasks, NeuroMesh introduces two aggregation strategies:
- Reduction: Aggregates self and neighbor features into a single tensor (e.g., via averaging). Suitable for consensus-based control and compact state updates.
- Broadcast: Replicates the self-feature and concatenates it with each neighbor's feature to preserve pairwise relational information. Essential for dense perception tasks (like multi-view depth estimation) where geometric relationships matter.
Parallel Three-Stage Pipeline: To decouple system cycle time from total processing latency, the framework runs the Encoder, Aggregator, and Decoder modules concurrently.
- The system's output cycle time is determined by the slowest stage ( $T_{pipeline} = \max(T_e, T_m, T_d)$ ) rather than the sum of all stages.
- This significantly improves throughput for real-time applications.
Communication Protocol:
- Built on ROS 2 with a microservice architecture.
- Utilizes Zenoh (a publish/subscribe protocol) for inter-robot communication, supporting both peer-to-peer (decentralized mesh) and router modes.
- Implements robust handling of packet loss via "keep-latest" buffers and dynamic adaptation (e.g., switching to partial-neighborhood inference if messages drop).
Hybrid Inference Engine: Supports TensorRT (GPU) for heavy perception tasks and ONNX Runtime (CPU) for lightweight control tasks, allowing optimization based on hardware constraints.

3. Key Contributions

Multi-Domain Unification: A single framework formulation that unifies dense perception (high-bandwidth) and compact-message control/assignment (low-bandwidth) problems using the dual-aggregation paradigm.
Cross-Platform Compatibility: Demonstrated effectiveness on heterogeneous teams (Aerial robots with NVIDIA Jetson Orin NX; Ground robots with Intel i7 CPUs and NVIDIA 1050 Ti GPUs).
Modularity: A plug-and-play microservice architecture where encoders, aggregators, and decoders can be swapped without altering the underlying execution pipeline.
High-Performance Implementation: A C++ implementation optimized for real-time operation, featuring efficient network protocols and accelerated inference engines.

4. Experimental Results & Case Studies

The authors validated NeuroMesh on a heterogeneous team of aerial and ground robots across three distinct tasks:

A. Multi-Robot Collaborative Perception

Task: Collaborative Depth Estimation (using a decentralized DUSt3R) and Semantic Segmentation.
Setup: 2 Aerial + 1 Ground robot.
Results:
- The decentralized NeuroMesh implementation achieved equivalent accuracy to centralized inference (Absolute Relative Error: 5.29%, Inlier Ratio: 61.32%).
- Collaborative perception significantly reduced uncertainty compared to single-robot baselines.
- Communication: Only Zenoh Peer mode successfully handled the large 3.15 MB perception payloads over a decentralized Doodlelabs mesh network, achieving ~0.5 Hz throughput, whereas other protocols (Fast RTPS, CycloneDDS) failed.

B. Multi-Robot Collaborative Control

Task: Point-to-point navigation for 3 ground robots avoiding collisions.
Setup: 3 Ground robots using a GNN-based RL policy.
Results:
- Achieved a 60% success rate in real-world deployment (vs. 80% in simulation), overcoming sim-to-real transfer challenges through real-time corrections.
- The framework successfully exchanged compact state features rather than raw sensor streams, maintaining low latency.

C. Multi-Robot Task Assignment

Task: Linear sum assignment (5 robots to 5 goals) minimizing global cost.
Setup: 5 Robots.
Results:
- Achieved a 93.92% Success Rate with a 128-byte message size.
- The Total Cost Performance (TCP) was within 0.03% of the optimal solution (Hungarian algorithm), demonstrating high efficiency even with lightweight encoders.

D. Performance Analysis

Communication: Zenoh Peer mode on mesh radios provided the most robust throughput (200 Hz for control, stable for perception) with the lowest latency (4.8 ms) and packet loss (0.3%).
Computation:
- Perception: TensorRT (GPU) was significantly faster (2.48 ms vs 9.19 ms for encoding) due to parallelization.
- Control/Assignment: CPU (ONNX) was often more efficient for small MLPs due to lower data transfer overhead, though TensorRT excelled in attention mechanisms.
- Parallel Pipeline: The parallel design reduced total computation time significantly compared to sequential execution (e.g., 5.44 ms vs 24.01 ms for perception on CPU).

5. Significance

NeuroMesh represents a critical step toward the practical deployment of learning-based multi-robot systems. By providing a unified, modular, and high-performance execution stack, it bridges the gap between algorithmic research and real-world application.

Scalability: The framework supports scaling from small teams to larger groups (simulated up to 50 robots) while maintaining throughput.
Flexibility: It allows researchers to deploy diverse algorithms (from classical consensus to deep learning) on heterogeneous hardware without rewriting the communication or execution infrastructure.
Open Source: The authors plan to release NeuroMesh as an open-source framework, fostering community adoption and further development in collaborative robotics.