What Physics do Data-Driven MoCap-to-Radar Models Learn?

1. Problem Statement

Data-driven models, such as MoCap2Radar, have demonstrated the ability to synthesize plausible micro-Doppler spectrograms from 3D Motion Capture (MoCap) data. These models typically use spatio-temporal transformers and are trained to minimize reconstruction error (e.g., Mean Absolute Error, MAE) between predicted and ground-truth spectrograms.

However, a critical gap exists: Do these models actually learn the underlying physics of radar scattering and the Doppler effect, or are they merely memorizing statistical patterns?

• Standard metrics (like MAE) only measure spectrogram similarity and are "agnostic" to physical laws (e.g., the linear relationship between radial velocity and Doppler frequency).
• A model could achieve low reconstruction error while failing to capture the correct kinematic relationships, leading to poor generalization and untrustworthy predictions in real-world scenarios.
• There is currently no framework to evaluate the physical consistency of learned radar spectrograms without access to measured ground-truth radar data.

2. Methodology: Physics-Based Interpretability Framework

The authors propose a novel interpretability framework that evaluates model behavior against two physics-based reference models. Crucially, this framework requires only MoCap input and model predictions, not measured radar data.

A. Physics-Based Reference Models

1. RCS-Aware Doppler Model:
   • Derives a reference Doppler centroid trajectory directly from MoCap data.
   • Calculates instantaneous range and radial velocity for each marker.
   • Aggregates marker-level Doppler frequencies using weights based on Body Surface Area (BSA) to approximate Radar Cross-Section (RCS) contributions (prioritizing larger body segments).
   • Outputs a reference centroid trajectory $f^{ref}_c[n]$.
2. Doppler Velocity-Frequency Model:
   • Based on the linear Doppler equation ($f \propto v$).
   • Defines an intervention where all marker radial velocities are scaled by a factor $\alpha$.
   • Theoretically, the resulting Doppler frequencies should scale linearly by the same factor $\alpha$.

B. Proposed Metrics

The framework introduces two complementary metrics to quantify physical consistency:

1. Frequency–Velocity Alignment (FVA):

   • Measures the alignment between the model-predicted centroid trajectory and the physics-derived reference trajectory.
   • Computed using Pearson correlation between the predicted centroid and the RCS-Aware reference.
   • Goal: High correlation indicates the model correctly tracks the evolution of Doppler shifts over time.

2. Doppler Consistency Score (DCS):

   • Tests whether the model preserves the velocity–frequency relationship under velocity intervention.
   • The model processes both baseline MoCap and $\alpha$-scaled MoCap.
   • The predicted scaling factor ($\alpha_{pred}$) is estimated via least-squares fitting of the resulting centroids.
   • $DCS = 1 - \frac{\sum(\alpha_k - \alpha_{pred,k})^2}{\sum(\alpha_k - \bar{\alpha})^2}$.
   • Goal: A score of 1 indicates perfect proportionality; lower scores indicate the model fails to scale Doppler frequencies correctly with velocity.
   • Variants: Includes a Sign-Flip variant ($\alpha = -1$) to test if the model correctly reverses the Doppler sign.

3. Experimental Setup

• Dataset: MoCap2Radar dataset (Vicon 12-camera system + 5.8 GHz radar). Contains structured diagonal walking and unconstrained free-walking trials.
• Models Evaluated:
  • MLP (Naive): A baseline with no spatio-temporal structure.
  • ST (Flat): Standard MoCap2Radar (Spatial attention over all markers, then Temporal attention).
  • HST (Full): Hierarchical Spatio-Temporal model (Anatomically grouped spatial attention + Temporal attention).
  • Ablation Variants: HST (No-Attn), HST (No-Spatial), and HST (No-Temporal) to isolate the contribution of specific attention mechanisms.
• Training: All models trained on MAE loss; evaluated on a held-out test set.

4. Key Results

The experiments reveal a stark disconnect between reconstruction error (MAE) and physical consistency.

• Reconstruction Error vs. Physics:

  • Models with comparable MAE (e.g., ST (Flat) vs. HST (No-Temporal)) showed massive differences in physical metrics.
  • MAE Difference: < 0.3 dB (approx. 6%).
  • FVA Difference: > 0.6 (approx. 65%).
  • Conclusion: Low reconstruction error does not guarantee physical consistency.

• The Critical Role of Temporal Attention:

  • High Performance: Models with Temporal Attention (ST (Flat), HST (Full), HST (No-Spatial)) achieved high FVA (>0.92) and high DCS (>0.91).
  • Low Performance: Models without Temporal Attention (HST (No-Temporal), HST (No-Attn)) collapsed on physical metrics (FVA $\approx$ 0.3, DCS $\approx$ -1.0), despite maintaining relatively low MAE.
  • Qualitative Evidence: Spectrograms from models lacking temporal attention showed flattened trajectories and temporal inconsistencies that MAE failed to penalize significantly.

• The MLP Baseline Anomaly:

  • The simple MLP achieved moderate DCS scores, outperforming the "No-Temporal" transformer variants.
  • Analysis: The MLP likely acts as a global near-linear mapping, accidentally preserving scaling properties. However, it failed FVA and Sign-Flip tests, indicating it does not truly understand the physics, only a linear approximation.

• Spatial Hierarchy:

  • The anatomically motivated HST (Full) did not significantly outperform the simpler ST (Flat) model, suggesting that the specific spatial hierarchy was not the primary driver of physical learning in this context.

5. Key Contributions

1. Framework Development: Introduced a physics-based interpretability framework using FVA and DCS to evaluate MoCap-to-radar models without ground-truth radar data.
2. Decoupling Error and Physics: Demonstrated that low MAE is insufficient for physical validity; models can fit spectrogram patterns while failing to learn kinematic laws.
3. Architectural Insight: Identified Temporal Attention as the critical component for learning underlying radar physics. Models lacking this mechanism fail to capture the time-evolution of Doppler shifts.
4. New Evaluation Paradigm: Proposed a shift from purely statistical evaluation to physics-grounded verification for generative models in scientific domains.

6. Significance

This work addresses a fundamental issue in scientific machine learning: the risk of "statistical shortcuts" where models appear accurate but lack physical grounding.

• Trustworthiness: It provides a method to verify if a generative model is learning the cause (physics) rather than just the correlation (data patterns).
• Generalizability: Models that learn physical laws (like the velocity-frequency relationship) are more likely to generalize to unseen speeds or motion patterns.
• Broader Impact: The proposed paradigm—using known physical laws as interpretability benchmarks—can be applied to other domains where data-driven models must adhere to physical constraints (e.g., fluid dynamics, climate modeling, or material science).

In summary, the paper argues that for MoCap-to-radar synthesis to be reliable, models must be evaluated not just on how well they look (MAE), but on whether they obey the laws of physics (FVA/DCS), with temporal attention being the architectural key to unlocking this understanding.