COHORT: Hybrid RL for Collaborative Large DNN Inference on Multi-Robot Systems Under Real-Time Constraints

This paper presents COHORT, a ROS-based collaborative framework for multi-robot systems that leverages a hybrid offline-online reinforcement learning strategy to dynamically distribute large DNN inference tasks, achieving significant improvements in battery efficiency, GPU utilization, and deadline compliance under real-time constraints.

The Gist

Imagine a team of rescue robots sent into a disaster zone—like a collapsed building or a wildfire area. Their job is to look around, find survivors, identify debris, and answer questions like, "How many people do you see?" or "Where is the safest path?"

To do this, they need to run incredibly smart but very heavy computer programs called Large DNNs (Deep Neural Networks). Think of these programs as giant, hungry brains that need a lot of electricity and processing power to think.

The Problem: The "Battery vs. Brain" Dilemma

Here's the catch: These robots are small, their batteries are limited, and they often can't connect to the cloud (the internet) because the disaster zone has no Wi-Fi.

• If a robot tries to run these "giant brains" entirely on its own, it drains its battery in minutes and might stop working before the mission is done.
• If they try to send the data to a central server, there is no server to send it to.
• If they just guess who should do the work, they might send a heavy task to a robot that is already weak, causing the whole team to fail.

The Solution: COHORT (The Smart Team Captain)

The researchers created a system called COHORT. Think of COHORT as a super-smart, invisible team captain that lives inside every robot. Its only job is to decide: "Should I do this task myself, or should I ask my teammate to do it?"

But COHORT isn't just a simple rulebook. It's a learning machine that gets smarter every time it works.

How COHORT Learns: The "Practice" and "Game Day" Strategy

The paper describes a clever two-step training process, which we can compare to training a sports team:

1. The "Practice" Phase (Offline Learning)
Before the robots ever go into the real disaster zone, they run thousands of simulations in a computer.

• The Analogy: Imagine a coach watching hours of game tape. The coach uses a simple rule (like an auction) to see who should have done what. "Robot A has a full battery, so it should take the heavy lifting."
• The Magic: The system records all these decisions. Then, it uses a technique called Advantage-Weighted Regression (AWR). This is like the coach saying, "Look at all the times we won. Let's memorize exactly what the winning team did and forget the times we lost." This gives the robots a solid starting strategy without them having to make dangerous mistakes in the real world first.

2. The "Game Day" Phase (Online Learning)
Now, the robots are in the real world. They use the strategy they learned in practice, but they don't stop there.

• The Analogy: This is like a soccer team playing a real match. They start with the coach's playbook, but as the game goes on, they adapt. If the wind changes, or a player gets tired, they adjust their formation instantly.
• The Magic: The robots use Multi-Agent PPO (MAPPO). This allows them to talk to each other (very briefly) and say, "Hey, my battery is low, you take this next task," or "I'm fast right now, give me that task." They learn in real-time, getting better at balancing the work as the mission progresses.

The "Auction" System

How do they decide who does what? They use a digital auction.

• Every time a task comes up (like "Scan this area"), every robot secretly bids on how much it "costs" them to do it.
• The "cost" isn't money; it's battery life, current workload, and speed.
• The robot with the lowest "cost" (the one with the most energy and free time) wins the bid and does the task.
• COHORT's Superpower: Unlike old systems that just look at the current moment, COHORT learns to predict the future. It might say, "I have energy now, but if I take this task, I'll be too tired for the next one. I'll let Robot B take it."

The Results: Why It Matters

The researchers tested this on three very different robots:

1. Husky: A big, strong, wheeled robot with a powerful computer.
2. Jackal: A medium-sized, agile robot.
3. Spot: A dog-like robot that can climb stairs but has a smaller computer.

The Outcome:

• Battery Life: The team saved about 15% more battery than other methods. In a rescue mission, that extra 15% could mean the difference between finding a survivor or not.
• Speed: They got 51% better at using their computer chips (GPUs), meaning they didn't waste energy on idle waiting.
• Reliability: They met their speed and timing goals 2.5 times more often than the other methods.

The Bottom Line

COHORT is like giving a team of rescue robots a shared brain that knows exactly how to share the workload. It ensures that the strong robots help the weak ones, the fast robots help the slow ones, and nobody runs out of juice before the job is done. It turns a group of individual robots into a truly collaborative, resilient team that can handle the chaos of a real-world disaster.

Technical Summary

Here is a detailed technical summary of the paper "COHORT: Hybrid RL for Collaborative Large DNN Inference on Multi-Robot Systems Under Real-Time Constraints."

1. Problem Statement

The paper addresses the critical challenge of deploying computationally intensive Vision-Language Models (VLMs) (e.g., CLIP, SAM) on resource-constrained, heterogeneous multi-robot systems (e.g., search-and-rescue robots) operating in infrastructure-less environments.

• Constraints: Robots face strict limitations on battery life, bandwidth, latency, and compute power. They often cannot rely on cloud offloading due to network unreliability in disaster zones.
• Challenges:
  • Heterogeneity: Robots (e.g., Husky, Jackal, Spot) have varying compute capabilities (CPU/GPU), memory, and battery chemistries.
  • Volatility: Resource availability fluctuates due to mobility, temperature, and workload bursts.
  • Real-Time Requirements: Perception pipelines must meet strict Frame-Per-Second (FPS) and deadline constraints.
  • Scalability & Resilience: Systems must handle dynamic team sizes (robots joining/leaving) and failures without centralized coordination.
• Gap: Existing solutions often rely on centralized servers, assume homogeneous resources, or use static partitioning, failing to adapt to dynamic real-world conditions.

2. Methodology: The COHORT Framework

COHORT is a distributed, resource-aware framework built on ROS 2 that dynamically schedules and distributes DNN module execution across a fleet of robots. It employs a Hybrid Reinforcement Learning (HybridRL) strategy combining offline and online learning.

A. System Architecture

• Perception Pipeline: The VLM workflow (e.g., CLIP and SAM) is partitioned into six modular execution units (e.g., detectors, encoders, decoders).
• Decision Module: Each robot runs a local RL policy that decides whether to:
  1. Execute a task locally.
  2. Offload a task to a peer.
  3. Accept an incoming offloaded task.
• Communication: Uses a publish/subscribe model over 5GHz Wi-Fi with minimal inter-agent communication (one-shot decision making).

B. Three-Phase Training Pipeline

To overcome the sample inefficiency and safety risks of training directly on physical robots, COHORT uses a three-phase curriculum:

1. Phase A: Behavior Cloning (BC) - Safe Initialization

   • Goal: Cold-start the policy safely.
   • Method: Collects offline data from a standard auction-based heuristic where robots bid on tasks based on estimated costs.
   • Technique: Trains a shared actor network to mimic these heuristic decisions using masked cross-entropy loss, ensuring the policy respects connectivity and capacity constraints before learning begins.

2. Phase B: Offline Policy Improvement (AWR)

   • Goal: Improve upon the heuristic without new real-world interactions.
   • Method: Uses Advantage-Weighted Regression (AWR) on the collected auction data.
   • Mechanism: A centralized critic evaluates the historical data to estimate advantages. The policy is updated to up-weight actions that yielded higher returns (lower latency, better energy usage) than the baseline heuristic, effectively filtering out suboptimal auction bids.

3. Phase C: Online MAPPO Fine-Tuning

   • Goal: Adapt to real-time dynamics and strict constraints.
   • Method: Deploys the pre-trained policy on physical robots for Multi-Agent Proximal Policy Optimization (MAPPO) with Centralized Training, Decentralized Execution (CTDE).
   • Constraints: Incorporates Lagrangian relaxation to enforce hard constraints on energy consumption and deadline violations. The reward function balances FPS, latency, and battery health.
   • Adaptation: The policy continuously updates based on fresh rollouts, adapting to changing workloads, team compositions, and network conditions.

3. Key Contributions

1. Resource-Aware Distributed Framework: A two-stage RL pipeline that enables autonomous workload balancing across heterogeneous robots without centralized servers, minimizing communication overhead.
2. Fault-Tolerant Workload Reallocation: A mechanism that dynamically reallocates tasks in response to robot failures, compute overloads, or energy depletion, maintaining system stability.
3. Hybrid RL Strategy: A novel combination of offline auction-based data collection, Advantage-Weighted Regression (AWR), and online MAPPO, allowing for safe initialization and rapid adaptation in real-world deployments.
4. Real-World Validation: Extensive evaluation on three distinct robotic platforms (Clearpath Husky, Jackal, and Boston Dynamics Spot) using real VLM workloads (CLIP, SAM).

4. Experimental Results

The framework was evaluated against a Baseline (local execution), Auction (heuristic), and Genetic Algorithm (GA) baselines.

• Performance Metrics:
  • Success Rate: COHORT achieved a success rate of meeting both FPS and latency targets 2.55× higher than the Baseline and significantly outperformed Auction and GA.
    • Husky: 54.0% (RL) vs. 21.2% (Baseline).
    • Jackal: 41.5% (RL) vs. 11.6% (Baseline).
    • Spot: 33.2% (RL) vs. 10.3% (Baseline).
  • Energy Efficiency: Reduced battery consumption by 15.4% compared to baselines.
  • Resource Utilization: Increased GPU utilization by 51.67% and improved overall resource utilization by 41.27%.
  • Scalability: The system successfully scaled to a 4-device setup and adapted to a 1-robot failure scenario without retraining, maintaining high success rates.
  • Workload Stress: Even with an added heavy YOLOv8-L detection task, the RL policy maintained robust performance, demonstrating resilience under increased compute demand.

5. Significance

• Mission Critical Enablement: COHORT enables autonomous robots to perform complex, multimodal perception tasks (like identifying survivors in debris) in disaster zones where cloud connectivity is impossible.
• Energy & Latency Optimization: By intelligently distributing workloads based on real-time resource states, it extends mission duration and ensures real-time responsiveness, which is vital for search-and-rescue operations.
• Robustness: The hybrid learning approach ensures the system is not only efficient but also safe to deploy, avoiding the "random exploration" risks typical of pure online RL on physical hardware.
• Generalizability: The framework demonstrates that learning-based scheduling is superior to static or heuristic methods in heterogeneous, dynamic multi-robot environments, offering a blueprint for future decentralized autonomous systems.