💻 computer science

From Instruction to Event: Sound-Triggered Mobile Manipulation

This paper introduces sound-triggered mobile manipulation as a new paradigm where agents autonomously detect and interact with sound-emitting objects without explicit instructions, supported by the Habitat-Echo data platform and a baseline system that demonstrates robust performance even in complex, overlapping acoustic scenarios.

Original authors: Hao Ju, Shaofei Huang, Hongyu Li, Zihan Ding, Si Liu, Meng Wang, Zhedong Zheng

Published 2026-04-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hao Ju, Shaofei Huang, Hongyu Li, Zihan Ding, Si Liu, Meng Wang, Zhedong Zheng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a robot butler named "Robo." Right now, most robots work like a very obedient, but slightly dim, assistant. You have to give them a specific command for every single thing they do. You say, "Robo, walk to the kitchen, pick up the ringing phone, and put it on the table." If you don't say it, Robo just stands there, staring at the wall, even if the phone is screaming for help.

This paper introduces a new kind of robot: one that actually listens and thinks on its own.

Here is the breakdown of their new idea, "Sound-Triggered Mobile Manipulation" (STMM), using some everyday analogies.

1. The Problem: The "Passive Robot" vs. The "Real World"

In the real world, things happen randomly. A doorbell rings. A smoke alarm goes off. A baby starts crying.

Old Way: You have to yell at your robot, "Robo, the doorbell is ringing! Go open the door!" If you forget to tell him, he misses the guest.
New Way (STMM): The robot hears the doorbell. It doesn't need you to tell it what to do. It figures out, "Oh, that's a doorbell. I should go open the door." It acts like a proactive housemate, not a remote-controlled toy.

2. The Two Main Jobs: "Moving Things" vs. "Fixing Things"

The researchers realized that when a robot hears a sound, it usually needs to do one of two things:

Job A: The "Relocator" (Object Relocation)
- The Scenario: Your phone is ringing on the couch, but you can't see it.
- The Robot's Job: It hears the ring, finds the phone, picks it up, and moves it to a better spot (like your hand).
- Analogy: It's like a dog hearing a treat bag rustle, running to find it, and bringing it to you.
Job B: The "Fixer" (State Transition)
- The Scenario: A faucet is running and splashing water, or a door is creaking open.
- The Robot's Job: It hears the water, finds the faucet, and turns the handle to stop the flow. Or it hears the doorbell and pushes the door open.
- Analogy: It's like hearing a leaky pipe and knowing exactly which wrench to grab to tighten it.

3. The Challenge: The "Cocktail Party" Problem

The hardest part isn't just hearing one sound; it's hearing two at once.

The Scenario: The doorbell is ringing (Guest is here!) AND the smoke alarm is beeping (Fire danger!).
The Robot's Job: It has to decide which one is more important. Usually, fire is more urgent, but maybe the doorbell is just a delivery. The robot has to figure out the priority.
The Analogy: Imagine you are at a loud party. Someone is shouting your name, and someone else is dropping a tray of glasses. You have to instantly decide: "Do I turn to the person calling me, or do I run to catch the falling glasses?" The robot has to do this math in a split second.

4. The Solution: A New "Playground" (Habitat-Echo)

To teach robots this skill, you can't just use a normal video game simulator. Most simulators are either:

Visual: Great for seeing, but silent.
Audio: Great for hearing, but you can't touch anything.

The researchers built Habitat-Echo. Think of this as a virtual reality training camp where the robot can:

Hear sounds bouncing off walls (like real life).
Touch and move objects (open doors, pick up phones).
Connect the two: It learns that "Sound X" means "Action Y."

5. How the Robot Learns: The "Brain" and the "Hands"

The robot uses a two-part system, like a CEO and a worker:

The CEO (The Task Planner): This is a super-smart AI (based on a large language model). It looks at the scene and listens to the noise. It says, "Okay, I hear a doorbell and a running sink. The doorbell is more important. First, I will go to the door, open it. Then, I will go to the sink and turn it off." It creates a to-do list.
The Worker (The Policy Models): These are the robot's "hands." They are specialized experts. One is an expert at walking, one at picking things up, and one at opening doors. They just follow the CEO's to-do list and do the physical work.

6. The Results: Does it Work?

They tested this on a dataset called STMM-1.9K (which is like a giant library of 1,900 different sound scenarios).

The Good News: The robot learned to find the sound source even if it couldn't see it (like finding a ringing phone in a dark room).
The Big Win: In the "Cocktail Party" scenario (two sounds at once), the robot successfully ignored the background noise, handled the most important task first, and then went back to handle the second one.

Summary

This paper is about teaching robots to stop waiting for instructions and start listening to the world. Instead of being a remote-controlled car, they are becoming like a helpful human who hears a crash, a ring, or a leak, and immediately knows what to do to fix it. They built a special training world to teach this, and the robots are starting to get pretty good at it!

1. Problem Definition: Sound-Triggered Mobile Manipulation (STMM)

The paper addresses a critical limitation in current embodied AI: the reliance on instruction-driven paradigms where robots wait for explicit textual commands. In real-world scenarios, many tasks are triggered by stochastic environmental events (e.g., a ringing doorbell or an alarm) rather than user instructions.

STMM is defined as a novel task where an agent must:

Autonomously perceive sound-emitting events without explicit textual goals.
Infer the task goal based on the auditory event and scene context.
Execute physical interactions to resolve the event.

The authors categorize STMM interactions into two types:

Object Relocation: Localizing a rigid sounding object (e.g., a ringing phone) and moving it via pick-and-place skills.
State Transition: Localizing an articulated/functional object (e.g., a ringing door or running faucet) and changing its physical state (e.g., opening the door, turning off the water).

The problem is further divided into Single-Event (one sound source) and Multi-Event (simultaneous, overlapping sound sources requiring prioritization) settings.

2. Methodology

A. Habitat-Echo Simulation Platform

To enable systematic research, the authors developed Habitat-Echo, a unified simulation platform that bridges the gap between existing visual-physical simulators and audio-visual navigation environments.

Integration: It combines high-fidelity spatial audio rendering with physical interaction capabilities.
Acoustic Rendering: Uses precomputed Room Impulse Responses (RIRs) to simulate realistic sound propagation, including geometry-aware propagation (reflections, reverberation) and material-dependent scattering.
Assets: Includes a library of interactive sound-emitting objects (phones, alarms, doors, sinks) designed to act as both acoustic sources and manipulation targets.

B. STMM-1.9K Benchmark

The authors established the first benchmark for this task, containing 1,900 episodes across training and testing splits.

Diversity: Features diverse object categories and a "one-to-many" sound mapping (multiple unique sounds per object category) to prevent overfitting to specific frequencies.
Complexity: Includes single-event scenarios and challenging multi-event scenarios with signal superposition and source interference.

C. Hierarchical Baseline Framework

The proposed solution decouples the complex task into a High-Level Planner and Low-Level Policy Models:

Sound-Triggered Task Planner (High-Level):
- Input: Initial multimodal observations (Head RGB + Sound) and a Skill Library.
- Mechanism: Utilizes an Omni-MLLM (e.g., Qwen2.5-Omni) to reason about the event, identify the target, and generate an ordered Skill Chain (e.g., Navigate $\to$ Open Door).
- Constraint: The planner is restricted to the observation space of the Navigate skill to prevent information leakage.
Skill-Specific Policy Models (Low-Level):
- Architecture: Lightweight policy models (Actor-Critic with PPO) tailored for specific skills.
- Skills: Navigate, Pick, Place, OpenDoor, CloseSink.
- Modality:
  - Navigate: Uses Audio (binaural/Mel-spectrograms) + Head RGB.
  - Manipulation (Pick/Place/Door/Sink): Uses Head Depth + Arm Depth.
- Execution: Policies execute sequentially based on the skill chain generated by the planner.

3. Key Contributions

Problem Formulation: Introduced STMM, shifting the paradigm from passive instruction-following to active, event-driven interaction based on auditory cues.
Platform Development: Created Habitat-Echo, the first simulator supporting closed-loop audio-visual-physical interaction with high-fidelity spatial audio.
Benchmark: Established STMM-1.9K, the first benchmark covering diverse objects, interaction types, and single/multi-event settings.
Baseline Framework: Proposed a hierarchical framework (Planner + Policies) that effectively handles the reasoning and execution required for STMM, demonstrating robustness in multi-source acoustic interference.

4. Experimental Results

Experiments were conducted on the STMM-1.9K benchmark using various Omni-MLLMs as planners.

Single-Event Performance:
- State Transition: The Qwen2.5-Omni-7B planner achieved the highest planning accuracy (71.17%) and overall success rate (19.72% without ground-truth navigation, 59.00% with ground-truth navigation).
- Object Relocation: Qwen2.5-Omni-7B again led with 78.38% planning accuracy and 27.48% overall success (53.60% with ground-truth navigation).
- Observation: Smaller models (3B) often hallucinated incorrect skills (e.g., trying to "Pick" a running faucet), while larger models (30B) struggled with specific acoustic nuances in single events but excelled in multi-event settings.
Multi-Event Performance:
- Qwen3-Omni-30B demonstrated superior robustness against acoustic interference, achieving the highest planning accuracy (51.83%) in complex scenarios with overlapping sounds.
- The agent successfully isolated the primary sound source, interacted with it, and then proceeded to the secondary source, validating the system's ability to handle task prioritization.
Ablation Studies:
- Audio Modality: Audio signals are crucial for navigation; models without audio or spatial info failed completely (0% success). Combining audio with explicit target positions yielded the best results (95.05%).
- Visual Modality: For navigation, Head RGB provided better semantic context than Head Depth alone. For manipulation, combining Head Depth and Arm Depth significantly outperformed single-view inputs (e.g., 88.73% success for Pick vs. 57.20% for Head Depth only).

5. Significance

This work represents a significant step toward general-purpose household robotics. By enabling agents to react proactively to environmental sounds (like alarms or doorbells) without human intervention, STMM moves embodied AI closer to real-world autonomy. The introduction of Habitat-Echo and STMM-1.9K provides the necessary infrastructure for the community to advance research in audio-driven physical interaction, addressing the long-standing gap between sound perception and physical manipulation. The results demonstrate that current multimodal LLMs, when combined with specialized low-level policies, can effectively reason about auditory events and execute complex physical tasks in dynamic environments.