💻 computer science

MAMMA: Markerless & Automatic Multi-Person Motion Action Capture

The paper introduces MAMMA, a markerless multi-person motion capture pipeline that utilizes a novel query-based architecture and a large synthetic dataset to accurately recover SMPL-X parameters and dense 2D contact-aware landmarks from multi-view videos of interacting individuals, achieving competitive accuracy to commercial marker-based systems without extensive manual cleanup.

Original authors: Hanz Cuevas-Velasquez, Anastasios Yiannakidis, Soyong Shin, Giorgio Becherini, Markus Höschle, Joachim Tesch, Taylor Obersat, Tsvetelina Alexiadis, Eni Halilaj, Michael J. Black

Published 2026-04-08

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Hanz Cuevas-Velasquez, Anastasios Yiannakidis, Soyong Shin, Giorgio Becherini, Markus Höschle, Joachim Tesch, Taylor Obersat, Tsvetelina Alexiadis, Eni Halilaj, Michael J. Black

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to record a complex dance routine between two people who are hugging, spinning, and touching constantly.

The Old Way (The "Gold Standard"):
Traditionally, to get a perfect 3D recording of this, you'd have to dress the dancers in tight suits covered in dozens of tiny, reflective balls (markers). You'd need a room full of expensive cameras, a team of technicians to place every single ball perfectly, and then, after filming, a team of editors spending hours manually fixing mistakes where the balls fell off or got hidden by a hand. It's like trying to paint a masterpiece, but you have to tape thousands of stickers on the canvas first, and then spend days peeling them off and fixing the paint underneath.

The New Way (MAMMA):
The paper introduces MAMMA (Markerless Accurate Multi-person Motion Acquisition). Think of MAMMA as a super-smart, magical pair of glasses that can look at a regular video of people dancing and instantly "see" the invisible skeleton and skin underneath, without needing any stickers.

Here is how it works, broken down with simple analogies:

1. The "Invisible Dot" Problem

If you just ask a computer to guess where a person's elbow is in a video, it might get confused if two people are hugging. It might think Person A's elbow is actually Person B's shoulder.

MAMMA's Solution: Instead of guessing just a few joints (like elbows and knees), MAMMA predicts 512 tiny, invisible dots all over the person's body, like a high-resolution mesh.
The Magic Trick: It uses a special "query" system. Imagine you have 512 tiny detectives, and each one is assigned to find one specific dot on the body (e.g., "Detective #42 is looking for the tip of the left pinky"). They don't just look at the image; they look at the image and a "mask" (a digital outline of the person) to know exactly who they are looking for. This prevents them from getting confused when people are hugging.

2. The "Contact" Sense

When two people dance closely, their bodies touch. In the old marker systems, the markers would often get hidden, and the computer would think the people were passing through each other like ghosts (a problem called "interpenetration").

MAMMA's Solution: MAMMA is trained to be touch-aware. It predicts not just where a dot is, but also:
- Visibility: "Is this dot hidden behind an arm?"
- Contact: "Is this dot touching the floor or another person?"
The Analogy: It's like the computer has a sense of touch. If it sees two people hugging, it knows, "Okay, these dots are touching, so they can't be on opposite sides of the room." This stops the 3D models from glitching through each other.

3. The "Training Gym"

To teach MAMMA how to do this, the researchers couldn't just film real people (because filming real people with perfect 3D data is hard).

The Solution: They built a massive, hyper-realistic video game world called MammaSyn. They programmed thousands of digital avatars to dance, fight, and hug in every weird position imaginable. They filmed these digital avatars from 32 different camera angles simultaneously.
Why it matters: Because it's a video game, they know the exact truth of every single dot. They used this "perfect practice" to train the AI so it could handle real-life chaos.

4. The Result: "Ghost-Free" Dancing

When they tested MAMMA against the expensive, marker-based systems (like Vicon):

Accuracy: It was almost identical. The difference in accuracy was less than the width of a human hair (less than 1mm).
Speed: The old way took about 72 hours of human labor to clean up one session. MAMMA does it in about 26 hours of automatic computer processing.
Cost: You don't need a $100,000 room of cameras. You can do this with a few synchronized iPhones.

The Bottom Line

MAMMA is like giving a computer X-ray vision and a sense of touch simultaneously. It allows us to capture complex, intimate human interactions (like dancing or martial arts) with Hollywood-level accuracy, but without the expensive suits, the sticky markers, or the hours of tedious manual cleanup. It democratizes motion capture, making it possible for researchers and creators to record human movement anywhere, anytime.

1. Problem Statement

Traditional motion capture (mocap) systems rely on physical optical markers, specialized hardware, and extensive manual post-processing to clean noisy data (e.g., marker swaps, dropouts). While accurate, these systems are costly, time-consuming, and struggle significantly with close multi-person interactions where markers occlude each other or fall off.

Existing markerless (vision-based) methods often suffer from:

Limited Accuracy: Generally considered less accurate than marker-based "gold standards."
Sparse Data: Many rely on sparse keypoints (joints) rather than dense surface data, making full body reconstruction difficult.
Occlusion Handling: They struggle to distinguish between overlapping subjects or handle complex physical interactions (hugging, dancing, grappling).
Lack of Benchmarks: There is a scarcity of standardized benchmarks for dense landmark prediction in multi-person scenarios.

2. Methodology

MAMMA is a two-stage pipeline designed to recover SMPL-X parameters (body shape, pose, and translation) from synchronized multi-view videos without markers.

A. Dense Landmark Estimation (MammaNet)

Instead of regressing sparse joints, MAMMA predicts 512 dense 2D surface landmarks per person.

Architecture: A Transformer-based network (ViT-Base backbone) that uses learnable queries (one per landmark) rather than a single token for all landmarks. This allows the model to learn distinct embeddings for each point.
Conditioning: The network is conditioned on segmentation masks (generated by SAM2) to resolve ambiguities in multi-person scenes, ensuring the network associates landmarks with the correct individual.
Outputs: For each landmark, the network predicts:
- 2D pixel coordinates ( $\mu$ ).
- Uncertainty ( $\sigma$ ).
- Visibility probability ( $p$ ).
- Contact probabilities: Person-to-person ( $p_{ci}$ ) and Floor contact ( $f_{li}$ ).
Training: Trained exclusively on a large synthetic dataset (MammaSyn) because real-world data lacks dense ground truth for contact and visibility.

B. Model Fitting (Optimization)

The estimated landmarks are used to fit the SMPL-X model via L-BFGS optimization.

Initialization: Unlike other methods, MAMMA does not require pose priors or initialization; the dense landmarks provide sufficient geometric information.
Energy Function: The optimization minimizes a weighted sum of:
1. Reprojection Error: Aligning 3D vertices to 2D predicted landmarks (robustified by uncertainty weights).
2. Shape Prior: Regularization on body shape parameters.
3. Temporal Smoothness: Penalizing large joint accelerations.
4. Contact Constraints: A novel term using Signed Distance Functions (SDF) to penalize interpenetration between bodies and enforce floor contact based on predicted probabilities.

C. Multi-View Correspondence

To track individuals across multiple camera views, MAMMA uses the dense landmarks and visibility masks.

It computes a symmetric epipolar distance between landmark sets of different views.
A Hungarian assignment algorithm solves for optimal one-to-one matches, creating a cycle-consistent graph to identify the same person across all views.

3. Key Contributions

MAMMA Pipeline: A novel, markerless system capable of capturing accurate SMPL-X motion for single and multi-person interactions, achieving quality comparable to commercial marker-based systems.
MammaNet: A transformer-based dense landmark estimator that learns per-landmark queries and conditions on segmentation masks to handle heavy occlusion and person-person contact.
MammaSyn Dataset: A large-scale (2.5M samples), high-fidelity synthetic dataset extending BEDLAM. It includes extreme poses, detailed hand motions, and complex two-person interactions (dancing, grappling) with dense 2D landmark ground truth and contact labels.
Quantitative Benchmarking: The authors introduce a rigorous evaluation protocol comparing MAMMA against Vicon (marker-based) systems using held-out markers, a standard rarely used in academic literature.
New Real-World Datasets: Release of MammaEval (Singles, Dance, Extra) and a new dance dataset captured solely with MAMMA, covering diverse interactions.

4. Results

Accuracy vs. Marker-Based Systems:
- In a direct comparison with Vicon + MoSh++, MAMMA showed a difference of only 0.862 mm in held-out marker error (MAMMA: 22.481mm vs. Vicon: 21.619mm).
- The resulting animations are visually indistinguishable from marker-based ground truth.
Performance on Benchmarks:
- MAMMA outperforms existing academic methods (e.g., CameraHMR, Look-Ma*, SMPLify-X) on standard datasets (RICH, MOYO, Harmony4D, CHI3D) in both 2D landmark error and 3D fitting metrics (MPJPE, PVE).
- On the Hi4D dataset (close human interaction), MAMMA achieved an MPJPE of 12.44 mm, significantly outperforming the next best method (32.10 mm).
Contact and Occlusion:
- The model achieves >90% AUC in predicting person-person and floor contact.
- It significantly reduces body interpenetration (Mean Penetration Depth: 8.46mm for MAMMA-C vs. 13.73mm for Look-Ma*).
Efficiency:
- The pipeline reduces the time from capture to SMPL-X output from ~72 hours (manual marker cleanup + processing) to ~26 hours on a consumer GPU (GTX-4090), making it nearly 3x faster.

5. Significance

Democratization of Motion Capture: MAMMA demonstrates that high-accuracy, markerless motion capture is feasible for complex interactions, removing the need for expensive hardware and laborious manual cleanup.
Research Impact: By providing a system that rivals the "gold standard" (Vicon) and releasing new datasets/benchmarks, MAMMA enables large-scale studies of human interaction, social dynamics, and physical contact that were previously too difficult or expensive to capture.
Practical Application: The system works with consumer-grade setups (e.g., 4 iPhones) and handles extreme poses and occlusions, making it suitable for applications in animation, VR/AR, and biomechanics without specialized studios.

In conclusion, MAMMA bridges the gap between academic vision-based methods and industrial marker-based systems, offering a robust, accurate, and accessible solution for capturing complex human motion.