💻 computer science

DyaDiT: A Multi-Modal Diffusion Transformer for Socially Favorable Dyadic Gesture Generation

DyaDiT is a multi-modal diffusion transformer that generates socially favorable, context-aware dyadic gestures by leveraging mutual audio dynamics and optional social context tokens to outperform existing methods in both objective metrics and user preference.

Original authors: Yichen Peng, Jyun-Ting Song, Siyeol Jung, Ruofan Liu, Haiyang Liu, Xuangeng Chu, Ruicong Liu, Erwin Wu, Hideki Koike, Kris Kitani

Published 2026-03-31

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yichen Peng, Jyun-Ting Song, Siyeol Jung, Ruofan Liu, Haiyang Liu, Xuangeng Chu, Ruicong Liu, Erwin Wu, Hideki Koike, Kris Kitani

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are at a party. You're talking to a friend, a stranger, or maybe your partner. As you speak, your hands move, you lean in, you pull back, and you react to what the other person is saying. These aren't random movements; they are a complex dance of who you are, who you are talking to, and what is being said.

For a long time, computers could make digital characters talk, but their hands were stiff, robotic, or just didn't match the mood. If you asked a computer to make a character talk to a stranger, it might make them act like they were talking to a best friend. That feels "off" to us humans.

Enter DyaDiT. Think of it as a super-smart digital choreographer that doesn't just watch the music (the audio); it also reads the room (the social context) and knows the dancer's personality.

Here is how it works, broken down into simple concepts:

1. The Problem: The "Two-Headed Monster"

Most old computer programs for gesture generation are like a person trying to listen to two people talking at once while wearing noise-canceling headphones that only let in one voice. They get confused.

The Issue: In a real conversation, people interrupt, talk over each other, and react instantly. Old models just mashed the two voices together into a muddy blob, so the computer didn't know who was saying what.
The DyaDiT Solution (ORCA): The researchers invented a tool called ORCA (Orthogonalization Cross Attention). Imagine ORCA as a magic sound mixer. It takes the two voices and separates them perfectly, like a DJ isolating the bass from the drums. It tells the computer: "This part is what Person A is saying, and this part is how Person B is reacting." This clarity allows the digital character to react naturally, even when people are interrupting each other.

2. The Social Context: The "Relationship Token"

Imagine you are telling a joke.

If you are with your best friend, you might slap their knee and laugh loudly.
If you are with your boss, you might just give a polite chuckle and a small hand wave.
If you are with a stranger, you might keep your hands in your pockets.

Old computers didn't know the difference. They just generated "generic" hand waves.

The DyaDiT Solution: DyaDiT has a Social GPS. Before it starts moving the character, you tell it: "Hey, these two are dating," or "These two are strangers." It also knows their personality (e.g., is the person shy? are they super energetic?).
The Result: The computer generates gestures that fit the relationship. The "dating partner" version will be more intimate; the "stranger" version will be more reserved.

3. The Motion Dictionary: The "Style Library"

Think of human movement like a language. Everyone speaks English, but some people speak with a British accent, some with a Southern drawl, and some with a fast-paced New York rhythm.

The DyaDiT Solution: The researchers built a Motion Dictionary. This is like a library of "gesture accents." It contains thousands of tiny, pre-learned movement patterns (like a quick shrug, a slow wave, or an emphatic point).
How it helps: When the computer needs to generate a movement, it doesn't just guess. It picks the right "accent" from the dictionary based on the personality and the situation. This makes the movement feel unique and human, rather than robotic and repetitive.

4. The Magic Engine: The "Diffusion Transformer"

You might have heard of AI that generates images by starting with static noise and slowly cleaning it up until a picture appears (like a photo developing in a darkroom).

The DyaDiT Solution: DyaDiT does the exact same thing, but with movement. It starts with a jumbled mess of random poses and slowly "denoises" them, frame by frame, until a smooth, natural conversation dance emerges. Because it uses this "diffusion" method, the movements are incredibly fluid and varied, avoiding the "stuck in a loop" feeling of older AI.

The Big Picture: Why Does This Matter?

The researchers tested DyaDiT against other methods and even against real human recordings.

The Test: They showed videos to people and asked, "Which one looks more real?" and "Which one fits the relationship better?"
The Result: People overwhelmingly preferred DyaDiT. In fact, in some cases, people liked the AI-generated gestures more than the real human ones! Why? Because the AI was so good at smoothing out the awkward pauses and making the social cues perfectly consistent.

Summary Analogy

If old gesture AI was like a puppet on a string that only moved when you pulled the "talk" lever, DyaDiT is like a method actor.

It listens to the script (the audio).
It knows who its scene partner is (the relationship).
It knows its own character's backstory (the personality).
And it uses a magic sound mixer (ORCA) to make sure it hears the other actor clearly, even if they are shouting.

The result is a digital human that doesn't just talk; it connects. It makes us feel like we are talking to another person, not just a machine.

1. Problem Statement

Current methods for generating conversational gestures primarily focus on single-speaker scenarios, mapping a single audio stream to a single speaker's motion. These approaches often fail to account for the complex social context and mutual dynamics inherent in real-world dyadic (two-person) conversations.

Limitations of Existing Work:
- Lack of Social Awareness: Most models ignore factors like the relationship between speakers (e.g., friends vs. strangers) and personality traits, resulting in generic, unnatural motions.
- Audio Entanglement: In dyadic settings, audio streams often overlap (interruptions, simultaneous speech). Existing models typically treat this as a single blended signal or fail to disentangle the roles of "speaker" and "listener," leading to ambiguous or incorrect gesture generation.
- Missing Interaction Dynamics: They do not effectively model how one person's gestures influence the other's, nor do they utilize the partner's existing motion as a conditioning signal.

2. Methodology: DyaDiT

The authors propose DyaDiT, a Multi-Modal Diffusion Transformer designed to generate contextually appropriate upper-body gestures for a target speaker based on dyadic audio and social cues.

Core Architecture

Backbone: A Diffusion Transformer (DiT) based on the Denoising Diffusion Probabilistic Model (DDPM). It predicts Gaussian noise added to a latent pose sequence conditioned on multiple inputs.
Input Modalities:
1. Dyadic Audio: Audio streams from both the "self" (target) and "other" (partner) speakers, encoded via Wav2Vec2.
2. Social Context Tokens: Relationship type (Stranger, Friend, Family, Dating Partner) and Personality scores (Big Five traits: Extraversion, Agreeableness, Conscientiousness, Neuroticism, Openness).
3. Partner's Motion: Optional input of the partner's gesture sequence to enhance coordination.
4. Motion Dictionary: A learnable set of motion primitives to guide style.

Key Technical Components

A. Orthogonalization Cross Attention (ORCA)
To address the ambiguity of overlapping audio streams, DyaDiT introduces the ORCA module:

Orthogonalization: It projects the "self" audio feature ( $a_{self}$ ) onto the subspace of the "other" audio ( $a_{other}$ ) and subtracts it: $a_{self}^{\perp} = a_{self} - \text{Proj}_{a_{other}}(a_{self})$ . This filters out redundant information shared between speakers.
Bidirectional Cross-Attention: Two symmetric cross-attention modules exchange information between the orthogonalized self-audio and the other-audio.
Gating Mechanism: A learnable gate adaptively fuses the two streams, allowing the model to distinguish between speaker-specific cues and listener-specific reactive cues. This enables the generation of realistic gestures even during interruptions.

B. Motion Dictionary (MD)
Inspired by LIA, DyaDiT employs a learnable orthogonal motion dictionary containing gesture primitives ( $d_0, \dots, d_n$ ).

Function: It modulates the audio features based on the desired motion style.
Training: The dictionary is trained jointly with the DiT.
Inference: It can be activated via Classifier-Free Guidance (CFG) to enforce specific styles or dropped to generate unconditional, style-agnostic motions.

C. Motion Tokenization

VQ-VAE: Instead of operating on raw continuous motion, the model uses a Residual Vector Quantization (VQ-VAE) to discretize motion into a compact latent space.
Efficiency: The input sequence is downsampled by a factor of 4 (1 token per 4 frames), and quantized using 4 hierarchical codebooks. This allows the diffusion model to capture long-range dependencies efficiently while maintaining high-fidelity reconstruction.

3. Key Contributions

DyaDiT Model: The first diffusion transformer specifically designed for socially aware dyadic gesture generation, conditioning on relationship, personality, and partner motion.
ORCA Module: A novel attention mechanism that disentangles overlapping audio streams, significantly improving the clarity of audio representations for gesture generation in dynamic conversations.
Social Controllability: Demonstrated ability to generate gestures that vary significantly based on social context (e.g., distinct behaviors for "Friends" vs. "Strangers") and personality traits.
Comprehensive Evaluation: Extensive quantitative and qualitative benchmarks showing superiority over state-of-the-art methods.

4. Results

Quantitative Evaluation

Evaluated on the Seamless Interaction Dataset (3,000 clips, ~182 hours), DyaDiT outperformed baselines (ConvoFusion, Audio2PhotoReal) on:

Fréchet Distance (FD): Lower scores indicate higher realism. DyaDiT achieved the lowest FD in both static (pose) and kinetic (velocity) metrics.
Diversity: DyaDiT maintained high motion diversity, avoiding the "mode collapse" often seen in generative models.
Ablation Studies: Removing ORCA, the Motion Dictionary, or social conditioning significantly degraded performance, confirming the necessity of each component.

Qualitative & User Studies

User Preference: In an A/B test with 16 participants, DyaDiT was preferred over ConvoFusion in 73.9% of cases for overall quality.
Surpassing Ground Truth: Interestingly, participants preferred DyaDiT over the original Ground Truth (GT) gestures in 69.8% of relationship consistency cases and 66.7% of personality consistency cases. The authors attribute this to the diffusion model producing smoother motions and the social conditioning encouraging more expressive, context-appropriate gestures than the raw dataset annotations might imply.
Controllability: t-SNE clustering showed clear separation of generated gestures based on personality traits, while relationship clusters showed a continuous manifold (reflecting the nuance of social relationships).

5. Significance and Impact

Advancing Synthetic Agents: This work bridges the gap between text-based AI agents and truly interactive digital humans by adding a layer of social intelligence to physical movement.
Socially Favorable Interactions: By explicitly modeling relationship and personality, DyaDiT enables the creation of agents that behave naturally in specific social scenarios (e.g., a nervous listener vs. an assertive speaker), which is crucial for applications in virtual reality, telepresence, and human-robot interaction.
Methodological Innovation: The ORCA module offers a new approach to handling multi-source, overlapping signals in generative tasks, which could be applicable beyond gesture generation to other multi-modal interaction problems.

In summary, DyaDiT represents a significant leap forward in making digital human interactions not just visually realistic, but socially coherent and contextually appropriate.