Ask, Reason, Assist: Robot Collaboration via Natural Language and Temporal Logic

This paper proposes a peer-to-peer robot collaboration framework where robots use Large Language Models and Signal Temporal Logic to autonomously negotiate help requests and offers, enabling them to resolve conflicts and minimize system makespan without relying on a central task allocator.

The Gist

Imagine a busy warehouse filled with different types of robots: some are small delivery bots, others are heavy-duty forklifts, and some are robotic arms. They all have their own jobs to do, like moving boxes from point A to point B.

Usually, they work great together. But sometimes, things go wrong. Maybe a delivery bot gets stuck because a pallet fell in its path, or a forklift realizes it can't reach a high shelf without help.

In the past, if a robot got stuck, it would have to call a "Big Boss" computer in the cloud. The Big Boss would stop everything, look at every single robot's schedule, figure out who is closest, and tell them what to do. This is slow, and it requires the robots to share all their secret schedules, which they might not want to do.

This paper proposes a new way: Robots talking to each other like coworkers.

Here is how their new "Ask, Reason, Assist" system works, broken down into simple steps:

1. The "Ask" (The Cry for Help)

When a robot gets stuck, it doesn't call the Big Boss. Instead, it uses a Large Language Model (LLM)—basically, a super-smart AI that understands human language—to write a quick message.

• The Robot says: "Hey, there's a pallet blocking my path in Aisle 1. I can't move."
• The Magic: The robot broadcasts this message to everyone nearby. It's like shouting in a breakroom, "Who can help me move this box?"

2. The "Reason" (The Brainy Translation)

This is the tricky part. Robots don't just speak English; they speak "Math" (specifically, Temporal Logic). This math ensures that if a robot moves, it won't crash into a wall or violate safety rules.

• The Problem: If the helper robot just heard "Move the pallet," it might try to do something dangerous or impossible because it didn't understand the timing or rules.
• The Solution: The helper robot uses a special "translator" (a grammar rulebook called BNF) to turn the English sentence into a strict, unbreakable math formula.
  • English: "Pick up the pallet and drop it at the end of the aisle."
  • Robot Math: "You must visit the pallet location, then visit the end location, and you cannot visit the end location until you have visited the pallet."
• Why it matters: This guarantees that the plan is safe and logically perfect, not just a guess.

3. The "Assist" (The Bidding War)

Now, every robot that can help does a quick mental calculation. They ask themselves:

• "If I stop my current job to help, how much longer will my job take?"
• "How long will the stuck robot have to wait?"

They send back a "bid" (an offer) that says: "I can help. It will take me 2 extra minutes to finish my own work, and you will only have to wait 1 minute."

4. The Selection (Picking the Best Helper)

The stuck robot looks at all the offers. It doesn't just pick the robot that is physically closest (which might be a bad idea if that robot is already super busy). Instead, it picks the robot that causes the least amount of chaos to the whole system.

• It chooses the helper that minimizes the total delay for everyone.

The Result: A Symphony, Not a Solo

The authors tested this system in a simulated warehouse.

• The Old Way (Centralized): A "God-mode" computer rearranges everyone's schedule perfectly. This is the best possible outcome, but it's slow and requires everyone to share their secrets.
• The New Way (Decentralized): The robots talk to each other. The result was almost as good as the "God-mode" computer (within 18% efficiency), but it happened much faster and without anyone revealing their private schedules.
• The Heuristic Way (Just picking the closest): This was the worst. Picking the closest robot often caused huge delays because that robot was already busy with a complex task.

The Big Takeaway

This paper shows that we can mix the flexibility of human language (easy to understand, great for describing problems) with the rigor of math (guaranteeing safety and logic).

Think of it like a team of chefs in a kitchen. Instead of the Head Chef yelling orders from a tower, a sous-chef who drops a pan can just ask, "Who has a free hand to grab a towel?" The other chefs quickly check their own tasks, shout back, "I can do it in 10 seconds!" and the sous-chef picks the best offer. The kitchen keeps running smoothly without a central boss micromanaging every move.

In short: Robots are learning to talk to each other to solve problems, using math to make sure their conversations lead to safe, efficient actions.

Technical Summary

Here is a detailed technical summary of the paper "Ask, Reason, Assist: Robot Collaboration via Natural Language and Temporal Logic."

1. Problem Statement

Modern warehouses utilize heterogeneous robot fleets (mobile robots, forklifts, manipulators) that often encounter unforeseen conflicts (e.g., blocked paths, incompatible task types) which a single robot cannot resolve.

• The Challenge: Centralized conflict resolution is impractical at scale due to computational bottlenecks and the need to disclose proprietary task schedules. Conversely, purely decentralized approaches often lack safety guarantees and feasibility checks.
• The Gap: While Large Language Models (LLMs) can generate natural language (NL) help requests, they lack the rigorous safety guarantees and spatio-temporal precision required for industrial robotics. Conversely, formal methods (like Temporal Logic) provide safety guarantees but are difficult to communicate across heterogeneous robots with different vocabularies.
• Goal: Develop a peer-to-peer coordination protocol where robots can request and provide help using Natural Language for communication, while ensuring Safety and Feasibility through Signal Temporal Logic (STL) and Mixed Integer Linear Programming (MILP), all without a central task allocator.

2. Methodology

The proposed framework, "Ask, Reason, Assist," operates in three distinct phases:

A. Natural Language Help Request (Ask)

When a robot detects a conflict (e.g., via a Vision-Language Model), it uses an LLM to generate a broadcast help request in natural language.

• Constraint: The request includes the conflict description, location, and the requester's capabilities.
• Mechanism: Constrained generation is used to ensure the LLM includes necessary context (location, capabilities) in the NL message.

B. NL-to-STL Translation & Reasoning (Reason)

Potential helper robots receive the NL request and must determine if they can assist and at what cost.

• Translation: The helper translates the NL request into a Signal Temporal Logic (STL) specification ($\phi_{help}$).
• Syntactic Guarantee: To ensure the generated logic is valid, the authors employ a Backus-Naur Form (BNF) grammar to constrain the LLM's output. This prevents syntactic errors that would cause downstream solvers to fail.
• Fine-tuning: The LLM is fine-tuned (using LoRA) to improve translation accuracy.
• Cost Calculation: The helper solves a local MILP problem to find an optimal path that satisfies both its original tasks ($\phi_{orig}$) and the new help task ($\phi_{help}$), while adhering to global safety constraints ($\phi_g$).
• Output: The helper calculates two metrics:
  1. $\tau_h$: Time the requester must wait.
  2. $\tau_{new}$: The additional time the helper incurs to complete its original tasks plus the help task.
• Proposal: The helper sends an NL offer containing these cost metrics back to the requester.

C. Selection and Confirmation (Assist)

The requester robot evaluates all incoming offers.

• Selection Strategy: It selects the helper ($j^*$) that minimizes the total system impact: $j^* = \arg\min (\tau_h + \tau_{new})$.
• Confirmation: The requester confirms the selection, and the chosen helper executes the updated path. Other candidates are rejected.

3. Key Contributions

1. Guaranteed Syntactic Validity: The paper introduces a method to translate NL to STL using BNF-constrained generation, ensuring that 100% of generated formulas are syntactically valid, a critical requirement for automated solvers.
2. Hybrid Reasoning Architecture: It augments LLM agents with spatial and temporal reasoning capabilities by grounding NL in formal STL and solving for optimal paths via MILP. This bridges the gap between the flexibility of LLMs and the rigor of formal verification.
3. Decentralized Coordination Protocol: The framework enables heterogeneous robots to coordinate without a central planner or full schedule disclosure, achieving near-optimal system performance while minimizing information exchange.
4. Comprehensive Evaluation: The system is rigorously tested against simple heuristics (nearest neighbor) and a centralized "Oracle" baseline (global re-optimization).

4. Experimental Results

The authors evaluated the framework in two primary experiments:

• Experiment 1: NL-to-TL Translation:

  • Setup: Tested on a dataset of 7,500 NL-STL pairs using a 12B parameter LLM (Gemma 3) with BNF constraints.
  • Results: The method achieved 100% syntactic validity across all trials. It maintained high logical accuracy (98-99%) comparable to much larger models (like GPT-4) but with significantly lower computational overhead, making it deployable on edge devices.
  • Containment: In cases of translation errors, 99% of generated formulas were "contained" within the true formula, meaning the generated plan would still satisfy the original safety constraints.

• Experiment 2: Mobile Robot Conflict Resolution (Warehouse Scenario):

  • Setup: 6 forklift robots, 12 pick-and-place tasks, and a blocked aisle scenario. Compared against a Centralized Oracle, a Nearest-Neighbor heuristic, and a Hybrid approach.
  • Results:
    • The proposed decentralized method added an average of 5.47 time-steps to the system makespan.
    • The Centralized Oracle added 4.49 time-steps.
    • The proposed method performed within 18% of the optimal centralized baseline.
    • It significantly outperformed distance-based heuristics (B2) and hybrid approaches (B3) by 46-53%, proving that local optimization with cost-aware bidding is superior to myopic distance-based selection.
  • Efficiency: The MILP solver took ~5.3 seconds per update, demonstrating real-time feasibility.

• Demonstrations:

  • Validated in a high-fidelity Unity simulator with three scenarios: Pallet Cleanup, Warehouse Kitting (unordered goals), and Sequential Tool Retrieval (strict temporal ordering). The system successfully translated complex NL instructions into valid STL and executed them.

5. Significance and Future Work

• Significance: This work demonstrates that Natural Language can serve as a flexible interface for heterogeneous multi-robot systems, provided it is grounded in Formal Methods for safety and feasibility. It solves the "communication gap" where robots with different vocabularies can collaborate without a central authority.
• Limitations & Future Work:
  • Assumes conflict detection is perfect (currently relies on external detectors).
  • Does not yet integrate low-level physics or dynamic obstacle avoidance into the high-level planner.
  • Currently limited to specific "help request" scenarios; future work aims to expand NL integration to broader multi-robot operations.

In summary, the paper presents a robust, scalable, and safe framework for decentralized robot collaboration, effectively combining the semantic understanding of LLMs with the mathematical rigor of temporal logic and optimization.