🤖 machine learning

BioHuman: Learning Biomechanical Human Representations from Video

This paper introduces BioHuman, an end-to-end model trained on the newly created BioHuman10M dataset to directly infer internal muscle activations and human motion from monocular video, thereby bridging the gap between visual observations and biomechanical states for applications in rehabilitation and injury assessment.

Original authors: Yujun Huo, He Zhang, Chentao Song, Honglin Song, Zongyu Zuo, Tao Yu

Published 2026-05-15

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yujun Huo, He Zhang, Chentao Song, Honglin Song, Zongyu Zuo, Tao Yu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a movie of a person running. A standard computer program can tell you where the person's limbs are moving (the "what"). But it has no idea how the person is doing it (the "why"). It doesn't know which muscles are firing, how hard they are pulling, or the internal forces at play. It's like watching a puppet show and only seeing the strings move, without understanding the puppeteer's hand strength or the tension in the strings.

This paper, titled "BioHuman," tries to fix that by teaching computers to see the "invisible" internal mechanics of human movement, not just the visible outer shell.

Here is a breakdown of how they did it, using simple analogies:

1. The Missing Puzzle Piece: BioHuman10M

To teach a computer to understand muscles, you need a massive library of examples where you can see both the movement and the muscle activity at the same time.

The Problem: Real-world data is rare. You can't easily strap sensors to thousands of people to record their muscle signals while filming them.
The Solution: The authors built a giant digital library called BioHuman10M. Think of this as a "simulated reality." They took existing videos of people moving and ran them through a high-tech physics engine (like a video game engine, but for real human biology).
How it works: They fed the computer's "muscle brain" (a simulation called OpenSim) the video data. The engine calculated what the muscles must have been doing to create that specific movement.
The Result: They created 10 million pairs of "Video + Muscle Data." It's like having a textbook where every picture of a person running is accompanied by a detailed map of exactly which muscles were working and how hard.

2. The New Model: BioHuman

Now that they had the textbook, they built a new AI model called BioHuman.

The Old Way (Two-Step): Previous methods were like a relay race with a bad handoff. First, one AI guessed the body's pose (the "what"). Then, a second AI tried to guess the muscles based only on that guess. If the first AI made a tiny mistake, the second AI got confused, and the errors piled up.
The BioHuman Way (One-Step): This model is like a single detective who looks at the video and solves the whole mystery at once. It doesn't just guess the pose; it guesses the pose and the muscle activity simultaneously.
Why it's better: The model realizes that movement and muscle are locked together. If you see a person leaning forward, the model uses that visual clue to guess the muscles, and if it sees a specific muscle pattern, it uses that to refine the guess of the body's pose. They help each other, like two friends solving a puzzle together rather than passing pieces back and forth.

3. The Results

When they tested this new model:

It saw deeper: It could predict muscle activity much more accurately than the old "two-step" methods.
It moved better: Interestingly, by forcing the AI to think about muscles, it actually got better at guessing the body's position too. It's as if understanding the engine made the car drive smoother.
It generalized: It worked well on different people and different types of movements, not just the ones it was specifically trained on.

The Bottom Line

The paper introduces a new dataset (BioHuman10M) that simulates muscle data for millions of video clips, and a new AI (BioHuman) that learns to watch a video and instantly understand both the visible movement and the invisible muscle forces driving it.

Important Note on Limitations:
The authors are honest about what their "simulated reality" can and cannot do yet:

The Neck: Their simulation model doesn't fully cover neck movements, so the data is incomplete there.
Simulation vs. Reality: Because the muscle data was generated by a computer simulation (not real sensors on real people), there is still a gap between their "digital truth" and real human biology.
Feet Only: Currently, the system only understands forces when feet touch the ground. It doesn't yet understand what happens when hands push a wall or when someone sits down.

In short, they have built the first major bridge connecting "what we see" (video) with "what's happening inside" (muscles), setting the stage for computers to understand human movement in a much more complete way.

Technical Summary: BioHuman – Learning Biomechanical Human Representations from Video

1. Problem Statement

Understanding human motion from video has traditionally focused on surface kinematics (e.g., joint positions and body shape via SMPL), which describes how humans move but fails to explain why they move in terms of underlying physiological mechanisms. A significant barrier to advancing biomechanical understanding from video is the lack of large-scale datasets that jointly provide visual observations, human motion, and internal biomechanical states (specifically muscle activations). Existing datasets are often incomplete across these modalities, and collecting real-world data with synchronized video, motion capture, and electromyography (EMG) is prohibitively expensive and sparse.

Furthermore, current computational approaches typically employ a decoupled, two-stage pipeline: first reconstructing kinematic motion from video, and then separately estimating muscle activations via biomechanical simulation or neural approximation. This separation is suboptimal because motion reconstruction errors inevitably propagate to downstream biomechanical inference, and the two processes are inherently coupled by the musculoskeletal system.

2. Methodology

2.1 BioHuman10M: A Large-Scale Biomechanical Dataset

To address the data scarcity, the authors introduce BioHuman10M, a dataset containing 10 million paired frames that align real-world visual motion with physics-based musculoskeletal states.

Data Sources: The dataset aggregates sequences from five existing motion capture datasets: MotionPRO, EMDB, 3DPW, Human3.6M, and BEDLAM.
Simulation Framework (BioSim): Since ground-truth muscle activations are unavailable for most of these sources, the authors developed BioSim, a simulation pipeline to generate them:
1. SMPL to OpenSim: SMPL motion sequences are converted to OpenSim-compatible kinematics using a custom full-body musculoskeletal model (ULBS-112) with 112 muscles.
2. Inverse Kinematics (IK): Marker trajectories are extracted from SMPL to scale the ULBS-112 model and perform IK, recovering joint-angle trajectories.
3. Ground Reaction Forces (GRF): Since GRFs are not in SMPL data, they are synthesized using GaitDynamic based on joint kinematics and estimated body parameters.
4. Muscle Activation Estimation: Using Inverse Dynamics (ID) to compute joint moments and Static Optimization (SO) to resolve muscle redundancy, the system estimates muscle activations ( $a_i(t)$ ) by minimizing an effort cost function ( $\sum a_i^p$ ) subject to dynamic constraints.
5. Post-processing: Unreliable sequences are filtered, and temporal smoothing is applied to ensure coherent activation dynamics.

2.2 BioHuman: An End-to-End Framework

The authors propose BioHuman, a unified, end-to-end framework that directly predicts both human motion (SMPL pose) and muscle activations from monocular video, bypassing the decoupled pipeline.

Architecture:
- Front-end Encoder: Utilizes PromptHMR (frozen image encoder) to extract initial SMPL pose estimates and image-conditioned HMR features for each frame.
- Visual-Kinematic Interface: The raw pose and visual features are encoded into temporal pose tokens and visual tokens. A learned gating mechanism injects visual evidence into the pose stream, allowing the model to leverage image cues (e.g., occlusion handling, orientation) when kinematic data is ambiguous.
- Motion-Visual Temporal Transformer: An interleaved token sequence (alternating pose and visual tokens) is processed by a Transformer decoder. This structure enables the model to reason over long-range temporal dependencies and cross-modal interactions, capturing the fact that muscle activation depends on phase, velocity, and contact, not just instantaneous pose.
- Decoding Heads: The fused state is decoded by two heads:
  1. Pose Head: Predicts a residual correction to the initial SMPL pose.
  2. Muscle Head: Predicts the 112-dimensional muscle activation vector.
Training Objective: The model is trained with multi-task supervision combining pose accuracy, muscle activation accuracy, temporal-difference matching, activation correlation, and amplitude calibration.

3. Key Contributions

BioHuman10M Dataset: The introduction of a large-scale (10M frames) dataset that bridges vision, motion, and full-body muscle activations, constructed via a novel simulation pipeline (BioSim) that augments existing motion capture data.
BioHuman Framework: An end-to-end neural architecture that jointly models human motion and muscle activations from monocular video, treating them as a coupled representation learning problem rather than sequential tasks.
Empirical Validation: Extensive experiments demonstrating that joint modeling significantly outperforms two-stage baselines in both biomechanical prediction and kinematic reconstruction.

4. Experimental Results

The authors evaluate BioHuman on the BioHuman10M test protocol, comparing it against a strong two-stage baseline (PromptHMR + MinT) and existing monocular HMR methods.

Muscle Activation Prediction: BioHuman achieves a Pearson Correlation Coefficient (PCC) of 0.71, significantly outperforming the two-stage baseline (PCC 0.42). It also shows lower RMSE (0.065 vs. 0.071) and better performance on active muscle firing (Active MAE 0.10 vs. 0.12).
Motion Estimation: Contrary to the concern that joint training might degrade kinematic accuracy, BioHuman improves pose estimation. It achieves an RMSEPose of 0.30 rad and PA-MPJPE of 35.21 mm, outperforming state-of-the-art HMR baselines like PromptHMR (0.53 rad, 40.89 mm).
Ablation Study: Removing the end-to-end visual-kinematic coupling (using a two-stage cascade with fixed HMR poses) results in a drop in PCC from 0.71 to 0.58, confirming the benefit of joint optimization.

5. Significance and Claims

The paper claims that BioHuman establishes a new benchmark for video-based biomechanical understanding. By moving beyond surface kinematics to infer internal musculoskeletal states, the work opens new possibilities for applications in rehabilitation, sports analysis, and assistive technologies (e.g., exoskeletons).

The authors emphasize that their approach bridges the gap between visual observation and physiological reality. They note that while the dataset is simulation-based and currently limited to foot-ground interactions (lacking neck DOFs and hand-object interactions), it provides a scalable foundation for learning biomechanically grounded human representations. Future work is planned to validate these predictions against real EMG data and integrate more anatomically precise body models (e.g., OSSO).