🤖 machine learning

Sound-based Multi-Person 3D Pose Estimation

This paper introduces SoundMHPE, the first framework to estimate multi-person 3D poses solely from acoustic signals by utilizing a novel encoder-decoder architecture to overcome challenges like signal superposition and reflections, validated on a newly constructed 6-hour synchronized dataset.

Original authors: Yusuke Oumi, Yuto Shibata, Go Irie, Akisato Kimura, Yoshimitsu Aoki, Mariko Isogawa

Published 2026-09-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yusuke Oumi, Yuto Shibata, Go Irie, Akisato Kimura, Yoshimitsu Aoki, Mariko Isogawa

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

For decades, scientists have sought ways to see the invisible, developing technologies that can track human movement without relying on the human eye. Cameras, which depend on light, fail in the dark or when a person is hidden behind a wall. Radio waves, used in some sensing systems, struggle when they encounter water or metal. In these complex, real-world scenarios, a different kind of signal has emerged as a promising alternative: sound. By actively sending out a sound pulse and listening carefully to how it bounces back, researchers can map the shape and position of objects in a room. This technique, known as active acoustic sensing, has already shown it can determine the posture of a single person, even when they are obscured. However, a significant gap remained in this field. While a single voice or movement creates a distinct echo, a room filled with multiple people creates a chaotic mix of overlapping sounds. Untangling these signals to understand where each individual is standing and how they are moving has been considered nearly impossible, as the echoes from one person blend indistinguishably with those of another.

A team of researchers from Keio University, the Tokyo University of Science, and NTT has now taken the first step toward solving this problem. They have developed a system capable of estimating the three-dimensional poses of multiple people using only sound. The researchers built a custom dataset to train their system, recording six hours of synchronized audio and motion data from up to three people moving simultaneously in a room. The setup involved a pair of speakers emitting a specific sound signal and a specialized microphone capable of capturing sound from all directions. As the participants walked, twisted, and raised their arms, the system recorded the returning echoes. The challenge was immense; the acoustic signals were a superposition of many different movements, further complicated by the way sound reflects off bodies and walls, creating delays that obscure the direct link between a specific pose and a specific sound change.

To navigate this complexity, the team created a new framework they call SoundMHPE. Instead of trying to process the sound as a single, messy block, their system breaks the audio down into multiple layers of detail. It analyzes the sound using different time windows, looking at both rapid changes that happen in a split second and slower, finer details in the frequency of the sound. This allows the system to isolate subtle acoustic signatures that might otherwise be lost in the noise. Once the sound is analyzed, the system uses a sophisticated method to separate the people. It assigns a specific set of digital "queries" to each individual, allowing the software to track the temporal evolution of one person's movements while simultaneously understanding how they interact with the others in the room. This approach disentangles the overlapping information, enabling the system to reconstruct the pose of each person frame by frame.

The results of this work demonstrate that the system works. When tested against existing methods that were originally designed for single-person tracking or adapted from radio-wave sensing, the new sound-based system proved more accurate. It successfully tracked complex movements, such as a person twisting their body or raising both arms, even when performed alongside other moving individuals. In tests involving two and three people, the system maintained a high level of precision, outperforming the baseline models in measuring the distance between predicted and actual joint positions. The researchers also found that their method could adapt to unseen environments; when they placed partitions in the room to change how sound reflected, the system still managed to estimate coarse poses, suggesting it can handle different acoustic conditions. Furthermore, the underlying logic of their system was shown to be effective even when applied to radio frequency data, indicating that the approach is robust across different types of signals.

This research marks a significant expansion of what is possible with acoustic sensing. By moving from single-person scenarios to multi-person environments, the team has opened the door for applications in crowded spaces, disaster relief, and sports analysis where cameras cannot be used and radio signals might be blocked. While the technology is still in its early stages and requires further development to be deployed in the real world, the construction of this new dataset and the validation of the multi-person estimation method provide a crucial foundation. The work confirms that sound, when analyzed with the right tools, can reveal the hidden geometry of a group of people, turning a chaotic mix of echoes into a clear picture of human movement.

Technical Summary: Sound-based Multi-Person 3D Pose Estimation (SoundMHPE)

Problem Definition

This paper addresses the challenge of estimating the 3D poses of multiple individuals using only acoustic signals. While single-person pose estimation via active acoustic sensing has been explored, extending this to multi-person scenarios presents significant difficulties. Unlike single-person settings where acoustic variations are uniquely determined by one individual's movement, multi-person scenarios involve the superposition of motion-dependent signal variations. The observed signals are a complex mixture of overlapping acoustic signatures from multiple subjects, further complicated by inter-person reflections that introduce intricate propagation delays. These factors obscure the direct correspondence between specific physical poses and the acoustic data, making it difficult to attribute signal changes to a specific individual.

Existing methods relying on RGB images suffer from occlusion and low-light conditions, while wireless-signal-based methods (RF, WiFi, mmWave) face limitations with water or metal obstructions and privacy concerns. Active acoustic sensing offers a solution unaffected by lighting and capable of penetrating certain obstacles, but prior work has been limited to single-person estimation.

Methodology: SoundMHPE

The authors propose SoundMHPE (Sound-based Multi-person Human Pose Estimator), a novel encoder-decoder framework designed to disentangle dynamic motion-dependent information from acoustically overlapped features. The system utilizes an active acoustic sensing setup with a pair of speakers and an ambisonics microphone.

1. Acoustic Multi-scale Encoder (AME)

To capture diverse temporal and fine-grained frequency features, the AME processes the input acoustic signal through a Multi-scale Short Time Fourier Transform (STFT).

Multi-Resolution STFT: The system generates log-Mel spectrograms using three different temporal window sizes ( $L$ , $2L$ , and $4L$ ) paired with corresponding Mel filter banks ( $B$ , $2B$ , $4B$ ). This produces spectrograms with high temporal resolution ( $a_{high-time}$ ), intermediate resolution ( $a_{mid}$ ), and high frequency resolution ( $a_{high-freq}$ ).
Specialized Self-Attention: Instead of standard self-attention which merges all resolutions, the AME employs:
- Temporal Self-Attention (TSA): Computes attention within features generated from the same STFT window resolution to model temporal relationships.
- Frequency Self-Attention (FSA): Computes attention across spectrograms derived from the same acoustic sequence but with different window sizes to capture frequency-domain relationships.

2. Temporal Pose Decoder (TPD)

The TPD is designed to explicitly model both intra-person temporal dynamics and inter-person interactions.

Multiple-Query Strategy: Unlike standard DETR-based models that use a single query per person to represent an entire sequence, SoundMHPE assigns $N_{out}$ learnable pose queries to each individual (one for each estimated frame) plus one instance query for confidence scoring. This allows the model to focus on specific acoustic features corresponding to specific time steps rather than collapsing the temporal dimension.
Disentangled Attention Mechanisms: To prevent the mixing of temporal cues with inter-person dependencies, the TPD utilizes two distinct self-attention mechanisms:
- Motion Self-Attention (MSA): Restricts attention to the queries belonging to the same individual, effectively extracting consistent intra-person temporal dynamics.
- Interaction Self-Attention (ISA): Computes attention between a specific subject and the queries of all other subjects to model inter-person dependencies and mutual acoustic influences.

3. Training and Loss

The model is trained using a matching-based loss function (Hungarian algorithm) consisting of a pose loss ( $L_{pose}$ , mean squared error) and a confidence loss ( $L_c$ , binary cross-entropy).

Key Contributions

First Multi-Person Acoustic Pose Estimation: This work presents the first attempt to estimate multi-person 3D poses solely from acoustic signals, addressing the inherent signal ambiguity and superposition challenges.
Acoustic Multi-scale Encoder: A novel component that leverages multi-resolution STFT and specialized self-attention (TSA and FSA) to capture fine-grained temporal dynamics and subtle acoustic variations simultaneously.
Temporal Pose Decoder: A decoder architecture that employs a multi-query strategy and disentangled attention mechanisms (MSA and ISA) to jointly model intra-person temporal evolution and inter-person acoustic interactions.
AMP Dataset: The construction of the Acoustic Multi-person Pose (AMP) dataset, a 6-hour dataset containing 432,000 synchronized frames of multi-person pose and acoustic data, collected from 15 participants in various single, double, and triple-person configurations.

Experimental Results

The authors evaluated SoundMHPE on the AMP dataset and compared it against adapted baselines: a single-person acoustic model (Shibata et al.) and a multi-person WiFi-based model (Yan et al.).

Performance: SoundMHPE outperformed all baseline models across all metrics: Mean Per Joint Position Error (MPJPE), Procrustes-aligned MPJPE (PA-MPJPE), and Percentage of Correct Keypoints (PCK). Specifically, it achieved an MPJPE of 106.5 mm compared to 121.7 mm and 119.9 mm for the baselines.
Ablation Studies: Removing either the AME or TPD resulted in significant performance degradation, confirming the necessity of both multi-scale feature extraction and the specialized decoder architecture.
Subject Count: The model maintained competitive accuracy in triple-person scenarios compared to single-person settings, demonstrating robustness against signal superposition.
Generalization: The model showed adaptability to unseen environments with different reflection properties (e.g., added partitions).
Cross-Modal Validation: When applied to a WiFi-based multi-person pose estimation benchmark (PiW), the proposed architecture (without the STFT component) outperformed both single-frame and sequence-based WiFi baselines, suggesting the architecture's generalizability to other modalities.

Significance and Claims

The paper positions this work as a foundational step toward non-invasive multi-person pose estimation in environments where vision and wireless signals are limited. The authors claim that by constructing the first dedicated dataset and proposing a framework that explicitly handles the complexity of overlapping acoustic signatures, they have addressed a critical gap in the field.

The significance lies in the demonstration that acoustic signals can be used to estimate the poses of multiple people simultaneously, overcoming the limitations of occlusion and lighting that plague vision-based methods. The authors modestly note that sound-based pose estimation is still in its early stages and that real-world deployment faces challenges, particularly regarding the need for large-scale datasets and robustness in diverse environments. They view their contribution of the AMP dataset and the SoundMHPE architecture as a necessary foundation for pioneering this field further.