💻 computer science

IndoorR2X: Indoor Robot-to-Everything Coordination with LLM-Driven Planning

This paper introduces IndoorR2X, a novel benchmark and simulation framework that leverages Large Language Models to coordinate mobile robots with static IoT sensors for enhanced indoor scene understanding, reduced exploration overhead, and efficient multi-robot task planning.

Original authors: Fan Yang, Soumya Teotia, Shaunak A. Mehta, Prajit KrisshnaKumar, Quanting Xie, Jun Liu, Yueqi Song, Li Wenkai, Atsunori Moteki, Kanji Uchino, Yonatan Bisk

Published 2026-03-23

📖 4 min read☕ Coffee break read

CC0 1.0

Original authors: Fan Yang, Soumya Teotia, Shaunak A. Mehta, Prajit KrisshnaKumar, Quanting Xie, Jun Liu, Yueqi Song, Li Wenkai, Atsunori Moteki, Kanji Uchino, Yonatan Bisk

Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to clean a huge, messy house with a team of three robots. But here's the catch: each robot is wearing blindfolds that only let them see what is directly in front of them. They can't see what's in the next room, and they don't know if the TV is on or off unless they walk right up to it.

In the past, if these robots wanted to work together, they would have to constantly bump into each other, ask, "Did you see the TV?" and then walk all over the house just to check. It's like trying to find a lost set of keys in a dark house by feeling every single inch of every wall, even though you know someone else might have seen them earlier.

Enter "IndoorR2X."

Think of IndoorR2X as giving these blindfolded robots super-vision and a smart brain that connects everything.

The Core Idea: The "Smart Home" as a Teammate

The paper argues that most homes already have "eyes" that robots don't use: IoT sensors (like security cameras, smart lights, and smart fridges).

The Old Way (Robot-to-Robot): The robots talk to each other. "I'm looking for a laptop." "I haven't seen it." Both robots wander around looking.
The IndoorR2X Way (Robot-to-Everything): The robots talk to the whole house. The security camera (the "X" in R2X) sees the laptop on the bed. It instantly tells the central "Brain" (an AI), "Hey, the laptop is in the bedroom!" The Brain then tells the robot, "Go straight to the bedroom. No need to search."

How It Works: The "Mission Control" Analogy

Imagine a busy kitchen during dinner rush hour.

The Chefs (Robots): They are busy chopping, cooking, and moving around. They can only see their own cutting board.
The Cameras (IoT Sensors): They are mounted on the ceiling, watching the whole kitchen. They see that the salt is on the high shelf, the oven is hot, and the delivery guy is at the back door.
The Head Chef (The LLM): This is the "Large Language Model" (a super-smart AI). It sits at a desk with a giant whiteboard.
- It gets updates from the cameras: "Salt is on the shelf."
- It gets updates from the chefs: "I need salt."
- The Magic: Instead of sending a chef to wander the kitchen looking for salt, the Head Chef looks at the whiteboard, sees the camera's info, and says, "Chef 1, go to the shelf. Chef 2, go to the oven."

Why Is This a Big Deal?

The researchers built a video game simulation (a "benchmark") to test this. They found three amazing things:

Less Wandering: The robots didn't have to walk around as much. It's like the difference between searching for a movie in a library by checking every shelf vs. having a librarian point you to the exact aisle.
Smarter Planning: The "Head Chef" (the AI) didn't have to think as hard. Because the information was already there, it didn't waste energy guessing. This saves money and time (fewer "tokens" used by the AI).
Resilience: If a camera misses something (a "glitch"), the robots just go look for it themselves. But if the camera lies (says the salt is on the shelf when it's actually in the trash), the robots get confused. So, the system needs to be careful about trusting the sensors blindly.

The Real-World Test

The team didn't just stop at the computer game. They built a real setup with actual robot arms and webcams in a room.

Without the cameras: The robots had to drive into every room to find a blue bag and a USB cable.
With the cameras: The cameras saw the items immediately. The robots drove straight to them, skipping the "search" phase entirely. It was like teleporting to the solution.

The Takeaway

IndoorR2X is about stopping robots from being "lonely explorers" and turning them into a connected team that uses the entire smart home as their eyes and ears. By letting robots talk to the house itself (not just each other), they can get tasks done faster, use less energy, and stop wasting time looking for things that the house already knows where they are.

It's the difference between a group of people trying to find a party in a dark city by knocking on every door, versus having a GPS app that tells you exactly which door to knock on.

1. Problem Statement

The paper addresses the fundamental challenge of multi-robot coordination in indoor environments under partial observability.

The Limitation of R2R (Robot-to-Robot): While robots can communicate with each other, relying solely on Robot-to-Robot (R2R) sharing is insufficient for complex, long-horizon tasks (e.g., cleaning, fetching objects) in realistic homes. Without global knowledge, teams suffer from redundant exploration, inconsistent beliefs about object locations, and brittle task allocation when plans fail.
The Opportunity: Indoor environments are increasingly instrumented with low-cost, persistent Internet of Things (IoT) sensors (e.g., CCTV cameras, smart lights, appliances) that provide wide-coverage context unavailable to any single mobile robot.
The Gap: Existing Large Language Model (LLM) driven frameworks either assume "oracle-level" access to global state (unrealistic) or focus only on R2R communication without systematically fusing heterogeneous IoT data into a shared planning state.
Core Question: How can a fleet of indoor robots leverage LLM reasoning to coordinate reliably under partial observability by exploiting existing IoT sensing to minimize exploration and planning costs?

2. Methodology: The IndoorR2X Framework

The authors introduce IndoorR2X, a novel benchmark and simulation framework that integrates mobile robots with static IoT devices ("Everything") to create a Robot-to-Everything (R2X) coordination paradigm.

A. Benchmark Design

Environments: Built on the AI2-THOR engine, utilizing 85 multi-room environments (10 artist-curated homes and 75 modular apartments).
Task Suite: Includes 850 tasks ranging from joint navigation/manipulation (e.g., fetching items, discarding food) to navigation-only tasks.
Realistic Constraints: Unlike previous benchmarks, the system enforces strict partial observability. Robots only know their current field of view and visited areas. They do not have global map access unless augmented.
IoT Integration ("X"): The framework simulates a network of static CCTV cameras covering ~50% of the environment. A Vision-Language Model (VLM, e.g., Qwen-VL) processes video feeds to generate time-stamped, text-based event logs (e.g., "Tennis racket is in the Bedroom").

B. System Architecture

The framework operates via a centralized Coordination Hub that maintains a Global Semantic State ( $W_t$ ):

Data Fusion: The hub aggregates observations from mobile robots and IoT devices. The state $W_t$ is a tuple of robot states, object states (including location, containment, and dynamic properties like "open/closed"), and area states.
LLM-Driven Planning: An LLM acts as an online planner. It queries the global state $W_t$ to generate a Dependency Graph (DAG) of actions. This allows for parallel task execution where robots are assigned specific nodes based on their capabilities and proximity.
Execution & Monitoring: A system orchestrator dispatches actions to idle robots. An asynchronous monitor tracks outcomes (success/failure) and IoT updates. If a failure occurs or the state changes significantly, the system triggers replanning.

3. Key Contributions

Novel R2X Benchmark: The first indoor multi-robot benchmark that strictly enforces partial observability while integrating configurable IoT sensors to evaluate realistic team coordination.
LLM-Driven Semantic Fusion: A centralized framework that fuses onboard robot perception with ambient IoT signals into a unified global semantic state. This enables LLMs to plan parallel tasks without requiring exhaustive physical exploration.
Empirical & Real-World Validation: Extensive simulations and physical deployments demonstrating that R2X significantly reduces path length, action steps, and LLM token costs while maintaining high resilience to missing sensor data.

4. Experimental Results

The authors conducted ablation studies comparing three configurations:

IR (Isolated): Robots act alone with no communication.
R2R (Robot-to-Robot): Robots share maps but lack IoT data.
R2X (Robot-to-Everything): Robots share data and integrate IoT updates.

Key Findings:

Performance Efficiency: The R2X configuration achieved a 92% success rate, matching the best R2R baseline but significantly outperforming it in efficiency.
- Path Length: Reduced by ~26% compared to SOTA baselines (SMART-LLM) and ~11% compared to R2R alone.
- Action Steps: Reduced by ~7% compared to R2R.
- LLM Token Cost: Reduced by ~11% compared to R2R, indicating lower cognitive load on the planner.
LLM Scale Impact: Larger models (GPT-4.1) achieved the highest reliability (92%). Smaller models (Llama-3.1-8b) failed in most episodes (6% SR), suggesting that high-level coordination requires frontier model capabilities.
Scalability: Success rates remained stable (>90%) for teams of up to 5 robots. However, total path length increased with team size due to coordination overhead.
Robustness to Failures:
- Omission (Missing Data): The system is highly robust; even with 100% of CCTV detections missing, the success rate remained at 92% (robots simply explored more).
- Corruption (False Data): The system is sensitive to semantic errors (e.g., a camera reporting a device is "OFF" when it is "ON"). False positives caused linear drops in success rates, highlighting the need for verification mechanisms.
Latency: Increased IoT latency (up to 10 steps) degraded performance, reducing success rate to 87% and increasing path length by ~15%.

Real-World Validation:
The framework was deployed on physical Stretch robots with web cameras. The system successfully bypassed the exploration phase entirely by using camera data to locate targets invisible to the robots' starting positions, drastically reducing execution time.

5. Significance

Paradigm Shift: IndoorR2X moves multi-robot coordination from a purely "Robot-to-Robot" model to a "Robot-to-Everything" model, leveraging existing smart infrastructure as a critical sensor network.
Efficiency: It demonstrates that augmenting world models with IoT data is not just a convenience but a necessity for scaling efficiency, reducing both physical travel costs and the computational cost (tokens) of LLM planning.
Practicality: The framework provides a rigorous testbed for developing robust, scalable indoor service robot fleets that can operate in real-world, partially observable environments without requiring omniscient simulation states.
Design Insight: The study reveals a critical asymmetry: systems can tolerate missing information (by exploring) but are fragile against incorrect information, guiding future designs toward verification layers for IoT data.