Technical Summary: ARDY – Autoregressive Diffusion with Hybrid Representation for Interactive Human Motion Generation

Problem Statement

Generating realistic 3D human motions in real-time for interactive applications (e.g., character animation, simulation, humanoid robotics) presents a significant challenge in balancing controllability with inference speed.

• Offline models (e.g., diffusion or masked modeling) offer high fidelity and precise control via text prompts and kinematic constraints but suffer from slow inference times due to parallel generation of long sequences, making them unsuitable for interactive settings.
• Online/autoregressive models enable real-time synthesis but often sacrifice controllability. Many lack support for complex text semantics, struggle with long-horizon goals due to limited context windows, or require expensive test-time optimization and reinforcement learning (RL) policies to satisfy kinematic constraints.

Existing methods generally fail to simultaneously support online text prompting, flexible long-horizon kinematic constraints (such as root waypoints, full-body keyframes, and sparse joint positions), and real-time generation without auxiliary control modules.

Methodology: ARDY

The authors propose ARDY, a streaming generation framework that bridges the gap between offline fidelity and online interactivity. The system comprises two primary components:

1. Hybrid Motion Representation

To balance representational compactness with precise control, ARDY decouples motion into:

• Explicit Root Features: The root trajectory (global position and heading) is represented explicitly in global coordinates. This allows for direct overwriting of root features to satisfy spatial constraints (e.g., waypoints, paths) without compounding integration errors.
• Latent Body Embedding: The body motion is encoded into a compact latent space using a learned motion tokenizer. This tokenizer employs an asymmetric conditional autoencoder (using causal transformers) to compress high-dimensional explicit body features into latent tokens. The tokenizer utilizes Finite Scalar Quantization (FSQ) for stability and includes a global-to-local root conversion in the decoder to mitigate foot skating.

2. Autoregressive Two-Stage Diffusion Model

ARDY utilizes a transformer-based diffusion denoiser that operates in an autoregressive manner, predicting a short window of future motion tokens conditioned on history and goals.

• Input Conditioning: The model accepts variable-length history contexts (up to 8 seconds) and spatial goals that can be sparse and extend beyond the current prediction window (long-horizon goals). Constraints are represented as masked motion sequences injected into the input.
• Interleaved Two-Stage Architecture: The denoiser employs a two-stage process within the diffusion loop:
  1. Stage 1: Predicts the clean explicit global root motion.
  2. Stage 2: Conditions on the predicted root motion to predict the clean latent body tokens.
     This decomposition allows the model to satisfy text instructions and kinematic constraints simultaneously while maintaining high motion fidelity. The stages operate in an interleaved manner, ensuring continuous mutual influence between root and body motion.

3. Training and Inference

• Training: The model is trained on a large-scale, high-fidelity dataset (Bones Rigplay, ~700 hours) and the public HumanML3D dataset. It learns to generate motion conditioned on text labels and kinematic constraints sampled directly from ground-truth poses, eliminating the need for additional control modules like RL policies.
• Inference: ARDY operates via a latency-aware replanning mechanism. When new user inputs (text or constraints) arrive, the system triggers a replan, using a buffer of already-generated frames as history to ensure smooth transitions and hide inference latency.

Key Contributions

1. Hybrid Latent-Body Explicit-Root Representation: A novel representation that enables fast, controllable motion generation by separating precise root control from efficient generative learning of body motion.
2. Two-Stage Autoregressive Diffusion Model: A model featuring variable history context and support for long-horizon kinematic constraints (full-body keyframes, root waypoints, paths, end-effector positions/rotations) without requiring test-time optimization or RL.
3. Extensive Evaluation: Validation on the large-scale Bones Rigplay dataset and the HumanML3D benchmark, demonstrating high motion quality and constraint adherence. The paper includes a real-time interactive demo supporting dynamic text control, path following, and keyboard/mouse locomotion.

Results

• Performance: The efficient 4-step diffusion model achieves an average generation latency of 33 ms on an RTX 4090, enabling real-time interactivity.
• Quality and Control: On the HumanML3D benchmark, ARDY achieves competitive text-following (R-precision) and motion quality (FID) compared to state-of-the-art offline models, while significantly outperforming them in constraint adherence (lower joint position errors) without test-time optimization.
• Comparison with Autoregressive Baselines: Compared to the autoregressive model DiP, ARDY demonstrates superior performance in both in-horizon and out-of-horizon goal scenarios. Specifically, ARDY maintains high accuracy for long-horizon goals (e.g., reaching a target 10 seconds in the future), whereas DiP's performance degrades significantly.
• Ablation Studies: Experiments confirm that the hybrid representation, global-to-local root conversion, and the two-stage denoiser architecture are critical for reducing foot skating and improving spatial control accuracy.

Significance and Claims

The paper claims that ARDY successfully bridges the gap between offline and online motion generation, offering a system that is controllable, responsive, and high-fidelity. By natively learning conditional generation from large-scale data, ARDY eliminates the need for expensive auxiliary control modules (like RL or test-time optimization) typically required for interactive constraint satisfaction. The authors position ARDY as a foundational step toward next-generation interactive animation systems capable of handling complex, long-horizon user instructions in real-time.

The authors acknowledge limitations, noting that the current model is purely kinematic (lacking physical dynamics awareness), which can lead to artifacts like foot skating, and that the autoregressive history mechanism may be inefficient for extremely long horizons. Future work is suggested to integrate physics modeling and explore more efficient memory representations.