Technical Summary: AnyGoal

Problem Statement

End-to-end navigation policies trained on large simulation corpora suffer from severe performance degradation when transferred to unseen scenes, target categories, or goal modalities. Existing zero-shot alternatives face distinct but equally limiting constraints:

• Modular Pipelines (e.g., Modular GOAT): These rely on closed-set object detectors (like DETIC), creating a bottleneck where performance is capped by the detector's recall on out-of-distribution categories.
• 3D Snapshot-Memory Systems (e.g., 3D-Mem): These accumulate dense, view-dependent 3D representations queried by Vision-Language Models (VLMs). While they restore open-vocabulary recall, they incur heavy memory/compute overhead and remain structurally limited to single-agent operation.

The paper argues that neither extreme is necessary. The challenge lies in unifying spatial reasoning (typically delegated to fragile detectors) and semantic reasoning (typically delegated to per-snapshot VLM calls) into a lightweight, persistent structure that supports lifelong, multi-modal navigation without training.

Methodology: The AnyGoal Architecture

AnyGoal is a fully decentralized, training-free multi-robot architecture designed for multi-modal lifelong navigation. It operates without a centralized controller or explicit inter-agent messaging, relying instead on a shared 2D map. The system consists of four swappable layers:

1. Perception

Agents utilize an open-vocabulary segmentation backbone (defaulting to SAM3, with ablations on GroundedSAM and YOLO-World) to emit goal-conditioned masks. A spatial VLM (SpaceOM-3B) provides relevance scores, and an image matcher (CLIP ViT-B/32) handles image-based goals.

2. Mapping: Gaussian Bayesian Value Map (BVM)

The core innovation is a 2D Gaussian Bayesian Value Map. Unlike binary occupancy grids or Beta-Bernoulli heuristics, the BVM maintains a per-pixel posterior belief $(\mu, \sigma^2)$ regarding goal relevance.

• Update Mechanism: Detections are back-projected through depth using a "depth-cone mask." The map updates via precision-weighted fusion of Gaussians. High-confidence depth projections dominate the prior, while low-confidence observations decay rather than reinforce errors.
• Lifelong Accumulation: The map is never reset between subtasks. When the goal token changes, the VLM scoring head updates, but the accumulated geometric and semantic evidence persists, allowing evidence carry-over across 5–10 heterogeneous subtasks per episode.
• Multi-Agent Fusion: The swarm posterior is formed by element-wise precision-weighted fusion of local beliefs across agents on the shared Bird's Eye View (BEV) grid.

3. Reasoning: Convex-Blended VLM-as-Judge & UCB

Frontiers (boundaries of explored areas) are ranked by fusing spatial geometry with VLM common sense:

• VLM-as-Judge: Up to four candidate frontiers are rendered as markers (A/B/C/D) on the egocentric view. The VLM returns a softmax probability $p_{VLM}$ and an explicit "is-worth" gate (Yes/No) to veto structurally implausible candidates (e.g., a refrigerator in a bathroom).
• Bayesian UCB: Each frontier inherits BVM statistics to compute an Upper-Confidence-Bound score: $UCB(f) = \mu(f) + \beta\sqrt{\sigma^2(f)}$. This balances exploitation (high relevance) and exploration (high uncertainty).
• Scoring: The final score is a convex blend: $s(f) = (1-w)p_{VLM}(f) + w \cdot \widehat{UCB}(f)$, where $w=0.50$. This treats the VLM as a noisy oracle for plausibility and the BVM as a low-noise oracle for coverage.

4. Decentralized Coordination

Agents allocate frontiers via a sequential greedy bid on the shared BVM:

• Separation Penalty: To prevent agents from converging on the same frontier, a multiplicative penalty ($0.6\times$) is applied if the distance between chosen frontiers is $<100$ pixels.
• Commitment Hysteresis: Agents only switch targets if the new frontier's score is $1.5\times$ higher than the current one, preventing oscillation.
• No Central Controller: Coordination emerges solely from the shared map and local greedy decisions.

Key Contributions

1. Gaussian Bayesian Value Map: Replaces binary heuristics with a precision-weighted Gaussian update over per-pixel goal-relevance belief, enabling lifelong evidence accumulation and natural hallucination decay.
2. Convex-Blended Scoring: Integrates a VLM-as-judge softmax with a Bayesian UCB term, recovering ergodic exploration guarantees while allowing the VLM to dominate when preferences are sharp.
3. Decentralized Swarm Coordination: Achieves multi-agent coordination without explicit messaging or centralized scheduling, using only a shared map and spatial separation penalties.
4. Scientific Characterization of Perception Bottlenecks: Through a four-way perception ablation, the paper demonstrates that as detector recall increases, the dominant failure mode of zero-shot navigation shifts from "exploration failure" (running out of steps) to "verification failure" (premature stopping).

Results

Evaluated on the GOAT-BENCH val unseen split (360 episodes, 2,669 subtasks) under a strict physical regime (discrete 0.25m steps, 42° HFOV, no teleportation):

• State-of-the-Art Performance: The dual-agent ($N=2$) AnyGoal system achieves 52.4% Subtask Success Rate (SR) and 12.7% SPL.
  • This represents a +27.5 percentage point improvement over the Modular GOAT baseline (24.9%).
  • It exceeds the strongest prior zero-shot single-agent system (TANGO) by +20.3 pp.
• Architectural vs. Count Gains: The single-agent configuration ($N=1$) alone achieves 41.9% Subtask SR, proving that the performance gains stem primarily from the decision architecture (BVM, UCB blend, verification gate) rather than raw agent count.
• Scaling: Increasing to $N=3$ agents yields a further +3.2 pp gain (55.6% SR), showing diminishing returns as frontier supply saturates.
• Perception Ablation:
  • High-recall detectors (SAM3) result in 80.2% false stops (premature verification failure).
  • Lower-recall detectors (YOLO-World) result in 42.3% budget exhaustion (failure to find candidates).
  • This confirms that for high-recall open-vocabulary systems, the bottleneck has shifted to goal verification at the stopping decision.

Significance and Claims

The paper claims that AnyGoal establishes a new state-of-the-art for zero-shot, training-free navigation in strict physical regimes. Its primary significance lies in demonstrating that:

1. Unified Reasoning: Spatial and semantic reasoning can be unified in a lightweight 2D structure without the overhead of 3D snapshot databases or the brittleness of closed-set detectors.
2. Decentralization is Viable: Effective multi-agent coordination can be achieved without centralized controllers or explicit communication channels, relying instead on shared uncertainty-aware maps.
3. Redefining the Bottleneck: The work reframes the open problem of zero-shot navigation. For systems with high-recall detectors, the challenge is no longer "finding better detectors" to find the goal, but "verifying the goal" correctly at the moment of the stopping decision to avoid premature termination.

The authors note limitations, including the inability to anchor projections on transparent objects, the reliance on consistent BEV alignment (requiring ground truth or SLAM in real-world deployment), and the use of single-run evaluations due to compute constraints. Future work is proposed to address path efficiency (SPL) via VLM-primed priors or policy distillation.