💻 computer science

VULCAN: Vision-Language-Model Enhanced Multi-Agent Cooperative Navigation for Indoor Fire-Disaster Response

This paper introduces VULCAN, a vision-language-model-enhanced multi-agent cooperative navigation framework designed for indoor fire-disaster response, which addresses the limitations of existing vision-based systems in hazardous, dynamic environments through physically realistic fire simulations and robust hazard-aware planning.

Original authors: Shengding Liu, Qiben Yan

Published 2026-04-15

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Shengding Liu, Qiben Yan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a building on fire. It's not just hot; it's filled with thick, choking smoke that makes it impossible to see, and the temperature is rising fast. Sending human firefighters in immediately is incredibly dangerous. So, we send in a team of robots.

But here's the problem: most robot teams are trained in clean, quiet, well-lit houses. If you drop them into a smoky, burning building, they get confused, lost, and might even walk straight into a wall of flame because their "eyes" (cameras) can't see through the smoke.

This paper introduces VULCAN, a new way to guide a team of robots through a fire. Think of VULCAN not just as a set of instructions, but as a super-smart, multi-sensory rescue coordinator.

Here is how it works, broken down with simple analogies:

1. The Problem: "Blind" Robots in a Smoke-Filled Room

Imagine trying to find a lost child in a pitch-black room while wearing sunglasses. That's what standard robots face in a fire.

The Smoke: It blocks regular cameras (RGB) and depth sensors.
The Heat: It messes up thermal sensors.
The Chaos: The environment changes every second as the fire spreads.

Existing robot teams try to talk to each other, but they are all "blind" and panic, often walking in circles or heading toward danger because they can't see the hazards.

2. The Solution: VULCAN's "Super-Senses"

VULCAN gives the robots a new set of eyes and ears. Instead of relying on just one camera, it fuses data from four different sources, like a superhero combining different powers:

RGB Camera: The "eyes" (but they get blocked by smoke).
Depth Camera: The "touch" (feeling how far things are).
Thermal Camera: The "heat vision" (seeing where the fire is).
mmWave Radar: The "X-ray vision" (this is the secret weapon! It can see through thick smoke and steam, just like how radar sees through clouds).

The Analogy: Imagine you are in a foggy forest. You can't see far, but you have a thermal camera to spot a warm animal, and a radar to feel the trees through the fog. VULCAN combines all these signals to create a clear picture of the room, even when the smoke is thick.

3. The Brain: The "AI Commander" (VLM)

Once the robots have a clear picture, they need a leader to tell them where to go. VULCAN uses a Vision-Language Model (VLM).

The Analogy: Think of the VLM as a seasoned fire chief looking at a map on a tablet.

The robots send the Chief a picture of the room (the map) and a report saying, "There's a hot spot here, and smoke is thick there."
The Chief doesn't just look at the map; they understand it. They can reason: "Robot A, don't go there, it's too hot. Robot B, go around the left side; it's safer."
The Chief assigns goals to each robot so they don't all run to the same corner (wasting time) and so they avoid the fire.

4. The Legs: The "Safe Path" (Local Planner)

Once the Chief gives the order, the robots need to walk without tripping or burning up. VULCAN uses a special algorithm called Fast Marching Method (FMM) with a "hazard filter."

The Analogy: Imagine walking on a floor that has invisible "hot spots."

A normal robot might step on a hot spot because it only sees the floor is flat.
The VULCAN robot sees a "heat map" overlay. It treats hot spots like deep mud or a cliff. Even if the path is physically clear, the robot slows down or avoids it because the "mud" (heat) is dangerous. It finds the smoothest, coolest path to the goal.

5. The Results: Why It Matters

The researchers tested this in a computer simulation of a burning building.

Old Robots: They got lost, walked into fire, or took forever to find the target because they couldn't see through the smoke.
VULCAN Robots: They worked together like a well-oiled machine. They shared what they saw, avoided the hottest zones, and found the target much faster and safer.

The Big Takeaway

VULCAN is a teamwork toolkit for robots. It combines:

Better Senses (to see through smoke).
A Smart Brain (to understand the danger and plan the route).
Safe Legs (to walk carefully around the fire).

It turns a group of confused, blind robots into a coordinated, hazard-aware rescue squad, giving them the best chance to save lives when humans can't go in.

1. Problem Statement

Indoor fire disasters present extreme challenges for autonomous Search and Rescue (SAR) operations due to dense smoke, high temperatures, and rapidly evolving environments.

Limitations of Current Systems: Existing multi-agent cooperative navigation frameworks are primarily designed for benign, static indoor environments. They rely heavily on visual data (RGB-D) and fail significantly when smoke degrades visibility, thermal hazards distort sensor readings, and environmental conditions change dynamically.
Critical Gaps:
1. Perception Degradation: Smoke causes missed detections and false positives in standard vision-based systems.
2. Safety Risks: Standard planners often generate paths through high-temperature or smoke-filled zones because they lack hazard awareness.
3. Inefficiency: Without robust coordination and semantic understanding, agents exhibit redundant exploration and fail to locate targets quickly in time-critical scenarios.

2. Methodology: The VULCAN Framework

The authors propose VULCAN, a hazard-aware, multi-agent cooperative navigation framework designed specifically for fire-disaster response. The system architecture (Fig. 3) operates in a hierarchical manner:

A. Multi-Modal Perception and Fusion

To overcome smoke and thermal degradation, VULCAN fuses four sensor modalities:

Inputs: RGB, Depth, Thermal Imaging, and mmWave Radar.
Fusion Process:
- Visual data (RGB, Depth, Thermal) are aligned and processed by a Vision-Language Model (VLM)-based fusion operator to generate a "smoke-transparent" enhanced visual representation.
- An open-vocabulary detector and segmentation model identify objects and fire-related regions.
- Hazard Attribute Augmentation: Each 3D point in the local map is augmented with a hazard vector $h = [T(u), S(u), \sigma(u)]$ , representing temperature, smoke density, and sensing uncertainty.
- mmWave Integration: Radar data is fused to penetrate smoke where optical sensors fail, creating a robust local 3D point cloud.

B. Global Map Construction & Frontier Risk Assessment

Map Projection: Local 3D hazard-aware point clouds from all agents are merged and projected into a compact 2D grid map containing two channels:
1. Obstacle Map ( $O$ ): Structural constraints.
2. Hazard Map ( $H$ ): Aggregated risk scores based on temperature, smoke, and uncertainty.
Frontier Extraction: Boundaries between explored and unexplored regions are identified. Each frontier is assigned a Risk Score ( $R_k$ ) based on the average hazard intensity of its cells. Frontiers are categorized (e.g., safe, moderate, dangerous).

C. VLM-Based Global Planning

Role: The VLM acts as a high-level strategic planner.
Input: It receives a multi-modal prompt consisting of:
- Visual: A top-down view of the global map with robot positions and candidate frontiers highlighted.
- Textual: A system prompt defining the mission and a JSON-formatted hazard report (smoke, temperature, severity) for each frontier.
Output: The VLM assigns specific frontier goals to each robot, balancing exploration efficiency (minimizing redundancy) with safety (prioritizing low-risk frontiers).

D. Hazard-Aware Local Planning

Algorithm: A modified Fast Marching Method (FMM) is used for local trajectory generation.
Mechanism: The propagation speed of the FMM is modulated by the hazard map: $F(x) = \frac{1}{1 + \alpha H(x)}$ .
Result: This creates an arrival-time field that naturally biases paths toward low-risk corridors, penalizing traversal through high-risk areas even if they are geometrically free.

3. Key Contributions

VULCAN Framework: A novel multi-agent navigation system tailored for fire scenarios, integrating multi-modal perception (including mmWave and thermal) with VLM-based reasoning for hazard-aware planning.
Enhanced Benchmark: The authors extended the Habitat-Matterport3D benchmark by simulating physically realistic fire scenarios, including smoke diffusion, thermal hazards, and sensor degradation, to test robustness.
Systematic Analysis: The paper provides a comprehensive evaluation of existing navigation baselines under fire conditions, identifying specific failure modes (perception loss, inefficient exploration, unsafe routing) that current methods exhibit.

4. Results and Evaluation

The authors evaluated representative baselines (Greedy, Cost-Utility, Random Sample, and Co-NavGPT) under both normal and fire-driven conditions.

Performance Degradation in Fire: All existing methods showed significant performance drops in fire scenarios.
- Perception Failure: Smoke caused target detection confidence to drop (e.g., from 0.87 to 0.15) and triggered false positives (hallucinations).
- Safety Failure: Standard planners routed agents directly through high-temperature zones, leading to high Cumulative Hazard Exposure (CHE).
VULCAN's Advantages:
- Efficiency: The VLM-based planner achieved higher Success Rates (SR) and Success weighted by Path Length (SPL) while requiring fewer steps.
- Safety: By integrating hazard maps into the planning loop, VULCAN significantly reduced CHE compared to baselines.
- Robustness: The multi-modal fusion allowed the system to maintain situational awareness where single-modality vision systems failed.

5. Significance

This work addresses a critical gap in autonomous robotics for disaster response. It demonstrates that:

Multi-modal fusion is non-negotiable for fire environments; relying solely on RGB-D is insufficient.
Semantic reasoning via VLMs is essential for coordinating agents and interpreting complex, dynamic hazard data.
Hazard-aware planning must be decoupled from pure geometric planning to ensure agent survival and mission success.

VULCAN represents a significant step toward deploying reliable, autonomous robotic teams for real-world indoor fire rescue operations, moving beyond "benign" simulation environments to realistic, high-stakes disaster scenarios.