Decision-Driven Semantic Object Exploration for Legged Robots via Confidence-Calibrated Perception and Topological Subgoal Selection

This paper presents a vision-based framework for legged robots that enables robust decision-driven semantic exploration by integrating confidence-calibrated perception, controlled-growth topological memory, and utility-driven subgoal selection to overcome the limitations of conventional geometry-centric navigation in open-world environments.

The Gist

Imagine you are sending a four-legged robot dog (like a Boston Dynamics Spot or a Unitree Go1) into a brand-new, messy house to find a specific object, say, "a red fire extinguisher."

The Old Way (The "Map-Maker" Approach):
Traditionally, robots try to build a perfect, 3D architectural blueprint of the entire house first. They measure every wall, chair, and dust bunny with laser precision.

• The Problem: Legged robots move fast and bump into things. Building a perfect blueprint while shaking and stumbling is slow, expensive, and often fails. Plus, if the robot gets lost, it doesn't know what to look for, only where it is. It's like trying to find a specific book in a library by measuring the exact distance to every shelf, rather than just looking at the book titles.

The New Way (This Paper's Approach):
The researchers propose a smarter, "decision-driven" approach. Instead of obsessing over a perfect map, the robot acts like a smart detective who keeps a simple notebook of "clues" and makes quick decisions on where to go next.

Here is how their system works, broken down with everyday analogies:

1. The "Trust-Building" Detective (Confidence-Calibrated Perception)

The robot has two eyes:

• Eye A (The Big Picture): Uses a powerful AI to look at the whole room and say, "This looks like a kitchen, and I think I see something red over there."
• Eye B (The Detail Hunter): Uses a detector to spot specific boxes, saying, "I see a red box at coordinates X, Y."

The Problem: Sometimes Eye A is hallucinating, and sometimes Eye B is seeing a red toy instead of a fire extinguisher.
The Solution: The robot has a "Trust Manager." It doesn't just believe the loudest voice. It asks: "How sure are you? Do your stories match up?"

• If the Big Picture says "Kitchen" and the Detail Hunter says "Red Box," the Trust Manager combines them into a high-confidence clue: "There is likely a red object in the kitchen."
• If they contradict each other, the robot ignores the noise. It filters out the "maybe" clues and only keeps the "definitely" clues to make decisions.

2. The "Post-it Note" Map (Controlled-Growth Topological Memory)

Instead of drawing a giant, detailed map of the whole house, the robot uses a connect-the-dots approach.

• Imagine the robot walks into a room and sees a chair. It puts a Post-it note on a mental map: "Here is a chair. I'm pretty sure it's a chair."
• It walks to the next room, sees a table, and adds another note: "Here is a table."
• It draws a line between the notes to show it can walk from the chair to the table.

The Magic: The robot doesn't try to remember every inch of the floor. It only remembers the "nodes" (important spots) and the paths between them. If the robot sees the same chair again, it doesn't add a new note; it just updates the confidence on the existing one. This keeps the robot's brain light and fast, even in huge, messy environments.

3. The "Smart GPS" (Semantic Utility-Driven Subgoal Selection)

Now the robot has a list of Post-it notes (clues). Which one should it go to next?

• The Old Way: "Go to the closest thing." (Might lead to a dead end or a useless object).
• The New Way: The robot calculates a "Utility Score" for every clue. It asks four questions:
  1. Relevance: Does this look like the "red fire extinguisher" I was told to find?
  2. Confidence: How sure am I that this is actually an extinguisher?
  3. Potential: Have I already checked this spot? (If yes, ignore it).
  4. Cost: Is it far away or hard to reach?

The robot picks the clue with the highest score. It's like a hiker choosing the next campsite: not just the closest one, but the one that is safest, has the best view, and gets them closer to the summit.

4. The Execution (Running the Race)

Once the "Smart GPS" picks a target (e.g., "Go to the red box in the kitchen"), the robot's legs take over.

• A fast, local system handles the "how": "Watch out for that step! Turn left! Don't trip!"
• The high-level "Detective" only steps in when the robot reaches a stable spot to update its notes and pick the next target.

Why This Matters

• Robustness: If the robot bumps into a wall and the camera shakes, it doesn't panic. It just ignores the blurry, low-confidence data and waits for a clear view.
• Efficiency: It doesn't waste time building a perfect 3D model of a wall it doesn't need to cross. It just focuses on finding the object.
• Real-World Ready: The researchers tested this on real robot dogs in messy offices, living rooms, and gardens. The robots successfully found objects like fire extinguishers and boxes without getting lost or needing expensive laser scanners.

In Summary:
This paper teaches robots to stop trying to be perfect cartographers (map-makers) and start being smart explorers. By trusting their best clues, keeping a simple "Post-it note" map, and always picking the most promising next step, legged robots can navigate the messy, unpredictable real world much better than before.

Technical Summary

Here is a detailed technical summary of the paper "Decision-Driven Semantic Object Exploration for Legged Robots via Confidence-Calibrated Perception and Topological Subgoal Selection."

1. Problem Statement

The paper addresses the challenge of open-world semantic object exploration for legged robots. Unlike traditional navigation which relies on dense geometric SLAM (Simultaneous Localization and Mapping) to build metric maps, this work focuses on decision-driven exploration where the goal is to find specific objects based on natural language instructions.

Key Challenges:

• Fragility of Geometric Maps: Dense SLAM is computationally expensive, requires precise calibration, and is fragile under the rapid motion and viewpoint instability inherent to legged robots.
• Noisy Semantic Observations: Vision-Language Models (VLMs) and object detectors provide heterogeneous and uncertain semantic cues (scene-level context vs. object-level detection) that are often unreliable due to occlusion, motion blur, and lighting changes.
• Decision Gap: Existing methods often treat perception and decision-making as separate tasks. There is a lack of robust mechanisms to transform noisy, partial semantic observations into stable, executable exploration subgoals without relying on a globally consistent dense map.

2. Methodology

The authors propose a framework that replaces dense metric mapping with a semantic topological memory and a confidence-calibrated decision pipeline. The system operates in an evidence-memory-decision-execution loop.

A. Confidence-Calibrated Semantic Evidence Arbitration

To handle noisy inputs from different perception models, the system fuses two types of evidence:

1. Scene-level Evidence: Generated by a VLM (Qwen2.5-VL) to provide global context and directional guidance.
2. Object-level Evidence: Generated by an open-vocabulary detector (GroundingDINO) to provide spatially grounded bounding boxes.

Arbitration Mechanism:

• Calibration: A monotonic function suppresses low-confidence noise by applying a threshold $\tau$ to confidence scores.
• Fusion: The system calculates a posterior score $S(t)$ for candidate targets by combining calibrated confidences, spatial consistency (IoU between scene proposals and object boxes), and depth-based feasibility.
• Output: The final target location is a confidence-weighted interpolation of the object center and scene proposal, producing a stable semantic target $(p_t, L_t, C_f)$.

B. Controlled-Growth Semantic Topological Memory

Instead of a dense grid, the robot maintains a graph $G=(V, E)$ representing explored locations.

• Node Structure: Each node stores 3D position, semantic label, fused confidence, and an "exploration potential" score.
• Controlled Growth: New nodes are only inserted if they are far enough from existing nodes (distance threshold) and have high confidence. Otherwise, observations update existing nodes via exponential moving average.
• Pruning: Nodes with low exploration potential and low confidence are pruned to keep memory compact. Edges connect spatial neighbors to maintain traversability.

C. Semantic Utility-Driven Subgoal Selection

The robot selects the next subgoal by ranking candidate nodes in the topological graph based on a Semantic Utility Function $U(v)$:
$$U(v) = (S_{LLM})^\alpha \cdot (C_f)^\beta \cdot (P_{explore})^\eta \cdot e^{-\gamma \cdot d}$$

• $S_{LLM}$: Semantic relevance between the node's label and the user instruction, evaluated by an LLM.
• $C_f$: Fused confidence of the evidence.
• $P_{explore}$: Exploration potential (value of visiting the node).
• $d$: Travel cost (shortest path distance).
  This multi-objective function balances semantic relevance, reliability, exploration value, and reachability.

D. Execution Interface

The selected 3D subgoal is executed via a two-stage process:

1. Local Planning: A vision-based local planner (Viplanner) generates collision-avoiding velocity commands based on depth data.
2. Motion Control: A Reinforcement Learning (RL) locomotion policy executes the commands on the legged robot, handling dynamics and terrain adaptability.

3. Key Contributions

1. Confidence-Calibrated Arbitration: A novel mechanism to fuse scene-level and object-level semantic cues, filtering noise and producing reliable targets for decision-making.
2. Controlled-Growth Topological Memory: A compact graph-based representation that supports long-horizon exploration without the computational burden of dense metric maps.
3. Semantic Utility-Driven Decision Making: A subgoal selection strategy that explicitly balances semantic relevance, confidence, exploration potential, and travel cost.
4. Real-World Validation: Extensive experiments on both simulated environments and a real Unitree Go1 quadruped robot, demonstrating cross-platform deployability.

4. Experimental Results

The system was tested in 5 simulated environments (outdoor/indoor) and 5 real-world environments (office, showroom, lab, living room, garden).

• Evidence Quality: The proposed arbitration improved Semantic Accuracy (SA) from ~85.3% (strong baseline) to 90.1%, proving that confidence calibration effectively filters unreliable cues.
• Decision Quality: The utility-driven strategy achieved a Global Node Selection Accuracy (GNSA) of 85.8%, outperforming baselines like Bayesian ranking (76.5%) and VLFM (81.1%).
• Exploration Performance:
  • Simulation: Success Rate (SR) of 55% and Success weighted by Path Length (SPL) of 34.2%.
  • Real-World: Achieved SR between 40-55% across diverse environments, demonstrating robustness to perception noise and motion blur.
• Ablation Studies: Removing the evidence arbitration dropped SR to 45%; removing the utility decision dropped SR to 35%. This confirms both components are critical.
• Efficiency: High-level semantic modules (VLM/LLM) run on-demand (latency ~3-3.5s per viewpoint), while low-level control runs at 50Hz, ensuring real-time stability.

5. Significance

This work shifts the paradigm for legged robot exploration from geometry-centric mapping to decision-centric semantic reasoning.

• Resource Efficiency: It enables exploration on lightweight platforms without expensive LiDAR or heavy computational resources required for dense SLAM.
• Robustness: By focusing on topological connectivity and confidence-calibrated semantics, the system is more resilient to the dynamic motion and viewpoint instability typical of legged robots.
• Practicality: The successful deployment on a real-world quadruped robot in unstructured environments (e.g., gardens, offices) validates the feasibility of using VLMs and LLMs for real-time robotic decision-making in open-world scenarios.

In conclusion, the paper demonstrates that for task-driven exploration, a compact, semantically enriched topological memory combined with rigorous confidence calibration is superior to dense geometric reconstruction, offering a scalable path for autonomous legged robots in complex, unstructured environments.