Technical Summary: EAGOR – Embodied reAsoninG in Omni-diRection

1. Problem Statement

Embodied agents equipped with omnidirectional (360°) cameras possess a holistic view of their surroundings, which is advantageous for tasks like map-free navigation and active visual search. However, current Vision-Language Models (VLMs) typically process 360° data by projecting it onto 2D planar images using Equirectangular Projection (ERP). This approach introduces fundamental geometric flaws:

• Distortion and Discontinuity: ERP causes latitude-dependent distortions and seam discontinuities (where the left and right edges of the image meet), violating the Euclidean assumptions inherent in standard VLM architectures.
• Inconsistent Directional Reasoning: Because 360° data represents viewing directions on a sphere rather than positions on a flat plane, projecting it to 2D leads to inconsistent direction estimates when the agent moves or rotates. The predicted target direction tends to drift over time due to accumulated interpolation errors, particularly during seam crossings.
• Lack of Motion Consistency: Existing methods often treat directional reasoning as a per-frame recognition problem, failing to maintain a continuous, motion-consistent belief about the target's location relative to the agent's ego-motion.

The core challenge is to enable VLMs to perform geometry-aware omni-directional reasoning that remains consistent under agent motion without requiring task-specific training of the backbone models.

2. Methodology: EAGOR Framework

The authors propose EAGOR (Embodied reAsoninG in Omni-diRection), a training-free, geometry-aware framework that decouples semantic perception from geometric state estimation. Instead of predicting target coordinates directly in ERP image space, EAGOR reframes the problem as recursive Bayesian estimation on the sphere.

Core Components:

1. Semantic Evidence Extraction:

   • A frozen VLM (e.g., Qwen-2.5-VL, Gemma-3) processes the current panoramic observation $I_t$ and a target query.
   • The VLM generates a target-conditioned response map $\ell_t(u, v)$ over the ERP image, representing the likelihood of the target appearing at specific pixel locations.
   • This 2D response is "lifted" to the spherical domain. Pixel coordinates are mapped to azimuth ($\theta$) and elevation ($\phi$) to form a directional likelihood field $\ell_t(\omega)$ on the sphere $S^2$.
   • To facilitate additive accumulation, this field is converted to a log-likelihood field $r_t(\omega) = \log(\ell_t(\omega) + \epsilon)$.

2. Spherical Harmonic Belief Field (SH-BF):

   • EAGOR maintains a persistent belief state $f_t(\omega)$ representing the accumulated log-posterior probability that the target lies in direction $\omega$.
   • This belief is represented using Spherical Harmonics (SH), providing a continuous, rotation-equivariant basis on $S^2$. The belief is expressed as:
     $$f_t(\omega) = \sum_{\ell=0}^{L} \sum_{m=-\ell}^{\ell} c_{\ell m}^{(t)} Y_{\ell}^{m}(\omega)$$
     where $c_{\ell m}^{(t)}$ are the SH coefficients and $L=7$ is the bandlimit.

3. Recursive Bayesian Update:
   The framework operates in a loop of observation, propagation, and fusion:

   • Observation: The current log-likelihood $r_t(\omega)$ is projected onto the SH basis to obtain observation coefficients $b_{\ell m}^{(t)}$.
   • Propagation: The previous posterior belief coefficients $c_{\ell m}^{(t-1)}$ are rotated to the current agent frame using Wigner-D matrices ($D_{mm'}^{\ell}(R_t)$), where $R_t$ is the agent's rotation (e.g., yaw change $\Delta\psi_t$). This yields the propagated prior $\tilde{c}_{\ell m}^{(t)}$.
   • Fusion: The new posterior is computed by additively combining the propagated prior and the current observation in log-space:
     $$c_{\ell m}^{(t)} = \tilde{c}_{\ell m}^{(t)} + b_{\ell m}^{(t)}$$
   • Decoding: The final egocentric target direction $\hat{\omega}^*_t$ is recovered by computing the spherical mean (Fréchet mean) of the belief field, which can be derived analytically from the degree-1 SH coefficients.

3. Key Contributions

The paper identifies four primary contributions:

1. Identification of a Representation Gap: The authors highlight that robotic action requires a motion-consistent directional belief on the sphere, a requirement unmet by current ERP-based representations which suffer from drift and distortion.
2. EAGOR Framework: A training-free, geometry-aware framework that treats VLM outputs as probabilistic likelihoods on the sphere, enabling accurate omni-directional reasoning without fine-tuning the backbone VLM.
3. Spherical Harmonic Belief Field (SH-BF): A novel representation for continuous, rotation-consistent directional belief propagation that avoids ERP artifacts and supports recursive Bayesian estimation directly on the spherical manifold.
4. Comprehensive Validation: Extensive evaluation across simulation (Habitat) and real-world legged robot deployments, demonstrating superior performance in navigation and visual search tasks compared to existing baselines.

4. Experimental Results

EAGOR was evaluated on two benchmark datasets (HOS, OSR-Bench), simulated navigation tasks (Waypoint Following, Map-Free Navigation), and real-world experiments with a Unitree-Go2 robot.

• Active Visual Search (HOS & OSR-Bench):

  • EAGOR consistently outperformed baselines across various VLM backbones (3B to 12B parameters).
  • On the HOS dataset, it achieved average relative gains of +34.4%.
  • On OSR-Bench, it achieved +45.6% relative gain.
  • Notably, EAGOR with a 7B parameter model (Qwen2.5-VL) surpassed the performance of a standalone 72B model and a 12B model, demonstrating that geometric reasoning can compensate for model size.
  • It reduced the number of views required for convergence by 1.4×.

• Map-Free Navigation (Habitat-Sim):

  • EAGOR improved navigation success rates by +14.6% (94% vs. 82% for baselines).
  • It reduced path steps by 17.7% and mean angular error (MAE) by 24.5%.
  • Crucially, it reduced temporal inconsistency (directional drift) by 6.5× compared to the best baseline, maintaining accuracy even during seam crossings and sharp turns.

• Real-World Validation:

  • Deployed on a legged robot, EAGOR maintained coherent target direction beliefs under sensor noise, ego-motion, and temporary target occlusion, outperforming ERP-based baselines that suffered from drift.

• Ablation:

  • A bandlimit of $L=7$ was found to offer an optimal balance between angular resolution (~25°) and avoiding Gibbs ringing artifacts.
  • The framework adds only ~0.6× inference time overhead compared to the baseline, with negligible GPU memory increase.

5. Significance and Claims

The paper claims that EAGOR addresses a critical limitation in embodied AI: the inability of current VLMs to maintain geometrically consistent spatial reasoning in 360° environments. By shifting from pixel-coordinate prediction to spherical belief estimation, the framework ensures that directional estimates remain stable under agent motion, a prerequisite for reliable map-free navigation and active search.

The authors emphasize that their approach is training-free, meaning it enhances existing VLMs without requiring retraining or fine-tuning on specific datasets. They argue that this demonstrates the potential of "geometry-aware" reasoning to unlock the full capabilities of large vision-language models in embodied settings, allowing smaller models to outperform larger, unmodified ones.

Limitations Acknowledged:
The authors note that EAGOR relies on VLMs for semantic likelihoods, making it sensitive to perception errors such as false detections, rare targets, and multi-instance confusion. While the recursive belief accumulation helps mitigate some ambiguity, the system's performance is ultimately bounded by the VLM's ability to correctly identify the target. Future work is suggested to incorporate grounded detection and multi-sensor formulations.