💻 computer science

LH-AVLN: A Benchmark for Long-Horizon Audio-Visual-Language Navigation

This paper introduces LH-AVLN, a new benchmark for long-horizon audio-visual-language navigation that challenges agents with multi-goal missions in acoustically rich indoor environments, and proposes PAG-Nav, a training-free agent that effectively leverages binaural audio for search while relying on visual-semantic verification for goal completion.

Original authors: Rufeng Chen, Yue Chang, Zili Shao, Zhaofan Zhang, Li Chen, Hechang Chen, Hui Xiong, Sihong Xie

Published 2026-07-21

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Rufeng Chen, Yue Chang, Zili Shao, Zhaofan Zhang, Li Chen, Hechang Chen, Hui Xiong, Sihong Xie

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to be the ultimate house detective. Usually, when we teach robots to move around a house, we give them a simple mission: "Go find the red ball." They look around with their camera eyes, maybe remember where they've been, and walk until they spot it. This field of science is called "embodied navigation," where a robot learns to move through the real world to solve a task. But there's a catch: most of these training games are completely silent. The robot can only see what is directly in front of its lens. If the ball is behind a closed door or hidden in a dark corner, the robot is stuck, blind to anything outside its current view.

Now, imagine giving that robot a superpower: ears. In the real world, sound doesn't just travel in a straight line; it bounces around corners and seeps through cracks. A robot with good ears could hear a dog barking from the next room or a faucet dripping behind a wall, giving it a hint about where to go even when it can't see the object yet. This paper asks a big, tricky question: What happens if we combine these two superpowers—seeing and hearing—but make the mission much harder? Instead of finding just one thing, what if the robot has to find a whole list of different items, some described by words, some by pictures, and some just by their name, all while the sounds in the house keep changing as it solves the puzzle?

The researchers behind this study, led by Rufeng Chen and colleagues from the Hong Kong University of Science and Technology and Jilin University, have built a new challenge called LH-AVLN. Think of it as a high-stakes, multi-level video game for robots. In this game, the robot is dropped into a 3D house and given a "global mission" containing two to four different goals. These goals are tricky: one might be "find the sofa" (a category), another might be "find the light gray sofa next to the sheer curtains" (a description), and a third might be "find this specific picture of a bed" (an image reference).

The real twist, however, is the sound. In this game, every item the robot is looking for makes a noise. But here is the catch: the sounds are not fixed targets. As the robot finds and completes one goal, that item's noise stops. Meanwhile, the items it hasn't found yet keep making noise, taking turns to be the loudest. This creates a confusing situation. If the robot is looking for a sofa, but it hears a toilet flushing (because it hasn't found the toilet yet), the sound is a distraction. The robot has to figure out: "Is that sound helping me find my current target, or is it a red herring leading me to a different, unfinished task?"

To test this, the authors created a massive dataset with over 150,000 episodes where robots have to navigate these noisy, multi-goal missions. They found that existing robots, even the smart ones that are great at following instructions or remembering visual maps, struggle terribly. When the sound gets confusing or the mission gets long, these robots get lost or give up. They can't tell the difference between a helpful clue and a distracting noise.

To show how hard this is, the team built their own robot agent called PAG-Nav. This robot doesn't use complex training to learn; instead, it uses a clever strategy. It keeps a mental map of the whole house, tracking where it has been, what it has seen, and where the sounds are coming from. It uses the sound to guide its search when it can't see anything, but it refuses to say "I found it!" until it gets a good, clear look at the object to make sure it's the right one. Even with this smart strategy, the robot only succeeds in about 2% to 3% of the ordered missions and 3% to 5% of the unordered ones.

The main finding of this paper is that adding sound to long, complex missions makes things significantly harder, not easier, for current technology. The authors suggest that while sound is a powerful tool for finding things hidden from view, it becomes a nightmare if the robot can't figure out which sound belongs to which task. They argue that the future of robot navigation isn't just about seeing better or hearing better, but about learning to ignore the noise and focus on the right clue at the right time. The paper concludes that we are far from solving this puzzle, leaving plenty of room for future inventors to build robots that can truly navigate a noisy, multi-task world.

Technical Summary: LH-AVLN

Problem Definition

Embodied navigation is evolving from short, single-objective episodes toward long-horizon missions. However, existing long-horizon benchmarks are largely "acoustically silent," forcing agents to rely solely on visual fields or visual-semantic memory. Conversely, existing audio-visual navigation tasks typically focus on a single goal where the acoustic cue is fixed to that specific objective.

LH-AVLN (Long-Horizon Audio-Visual-Language Navigation) addresses the gap by introducing a benchmark that combines three factors usually evaluated separately:

Heterogeneous Goal Specifications: Goals are defined by object category, natural language descriptions, or reference images.
Long-Horizon Progress Tracking: Agents must complete missions comprising 2 to 4 goals, requiring persistent memory and task-progress tracking.
Persistent Spatialized Acoustic Cues: Unlike single-goal tasks, sounds in LH-AVLN are tied to the mission rather than a static target. Unfinished targets alternately emit spatialized binaural audio.

Core Challenge: The central difficulty is task-conditioned cue grounding. Acoustic cues are not fixed targets; they are progress-dependent information. A sound may guide the search for an unfinished goal, but as the agent completes subtasks, that same sound may become irrelevant or a distractor. The agent must reason about the physical source of a sound, associate it with the correct unfinished goal, suppress irrelevant cues, and perform visual-semantic verification before submission.

Methodology

The Benchmark (LH-AVLN)

Constructed on SoundSpaces 2.0 using Matterport3D scenes, LH-AVLN generates episodes with the following characteristics:

Observations: Agents receive egocentric RGB-D observations, pose, and binaural audio.
Mission Structure: A global mission $G = \{g_i\}_{i=1}^N$ ( $N \in \{2,3,4\}$ ) is provided at the start. Goals can be ordered (predefined sequence) or unordered (agent chooses order).
Audio Dynamics: Each target instance is associated with a spatialized sound source. Unfinished targets alternately emit cues. Once a target is successfully submitted, its sound source is deactivated. The active sound source changes over time, potentially pulling the agent toward irrelevant unfinished goals.
Goal Modalities:
- Category: Object label (e.g., "sofa").
- Description: Natural language referring expression (e.g., "the light gray sofa next to sheer curtains").
- Image: Reference image of the target instance.
- Constraint: Description and image goals do not include category labels or instance identifiers, requiring the agent to infer semantics.

Reference Agent: PAG-Nav

The authors introduce PAG-Nav, a training-free modular agent designed to establish a diagnostic baseline. It operates on the principle of using audio for non-line-of-sight search while reserving visual-semantic matching for final verification.

Architecture Components:

Goal-Conditioned Perception: Encodes heterogeneous goals (category, text, image) into query embeddings. Projects observations into room semantics, object hypotheses, and acoustic localization cues.
Temporal Uniform Semantic Map: A persistent memory module ( $M_t$ $M_{t}$ ) that unifies:
- Room Memory ( $R_t$ ): Semantic context.
- Object Memory ( $O_t$ ): Persistent object hypotheses and goal associations.
- Audio Memory ( $A_t$ ): Composed of Intensity Memory (accumulating local acoustic strength for confirmation) and Bearing Memory (storing directional observations for exploration).
Progressive Goal-State Planning: The agent switches between three stages based on the grounding status of the active goal:
- Search: When no reliable candidate exists, the agent uses semantic priors and acoustic bearing to explore frontiers.
- Candidate Grounding: When plausible candidates exist, the agent scores them using visual-semantic compatibility, contextual priors, and audio intensity consistency.
- Instance Verification: The agent navigates to a specific pose to visually verify the candidate before issuing a Submit action. Acoustic cues guide the approach, but visual verification triggers completion.

Key Contributions

LH-AVLN Benchmark: A new evaluation suite for long-horizon navigation that integrates multi-goal execution, heterogeneous goal modalities, and dynamic, alternating acoustic cues. It highlights the challenge of distinguishing relevant acoustic cues from distractors as mission states change.
Diagnostic Baseline (PAG-Nav): A training-free agent that explicitly models the separation between acoustic search and visual verification, demonstrating that persistent cross-modal memory is essential for these tasks.
Empirical Analysis: A comprehensive evaluation showing that existing state-of-the-art agents (vision-language, memory-based, and audio-visual) struggle significantly with full LH-AVLN missions, revealing specific failure modes in distractor rejection and long-horizon planning.

Experimental Results

Experiments were conducted on both ordered and unordered mission settings with 2–4 goals.

Baseline Performance: Existing vision-language agents (e.g., MTU3D, 3D-Mem) achieved very low full-task success rates (e.g., ~1.7% SR in ordered settings) due to the lack of acoustic input for non-line-of-sight search. Pretrained audio-visual agents (e.g., SAVI) showed non-trivial Oracle Success (OSR), indicating audio helps locate sounding regions, but they failed to complete full missions due to an inability to ground heterogeneous goals or manage multi-goal progress.
PAG-Nav Performance: PAG-Nav outperformed all baselines, achieving the highest Success Rate (SR) and Oracle Success Rate (OSR) in both settings (e.g., 2.3% SR ordered, 3.1% SR unordered).
Key Insight: The results confirm that while audio aids in approaching sounding regions, successful long-horizon navigation requires the ability to suppress task-irrelevant sounds, track goal progress, and perform visual-semantic instance verification. The low performance of all agents, including PAG-Nav, indicates substantial room for future progress.

Significance and Claims

The paper claims that LH-AVLN fundamentally shifts the paradigm of audio-visual navigation from "following a sound to a target" to "reasoning about the relevance of sounds in a changing mission state."

Acoustic Grounding vs. Fusion: The authors argue that the central challenge is not generic multimodal fusion, but task-conditioned cue grounding. Agents must determine which sound corresponds to the current unfinished goal and which sounds have become distractors.
Limitations of Current Methods: The benchmark reveals that current embodied agents lack the mechanisms to handle persistent, alternating acoustic cues in multi-stage tasks. They struggle to maintain the necessary cross-modal memory to link sounds to specific goals over time.
Future Directions: The paper positions LH-AVLN as a diagnostic tool to drive research into agents capable of dynamic cue management, long-horizon planning, and robust visual-semantic verification in acoustically rich environments.