Technical Summary: WaveSync

Problem Statement

Generating expressive co-speech gestures for physical humanoid robots presents a unique challenge distinct from virtual avatars. While virtual avatars can execute rapid, overlapping motions without consequence, physical robots operate under strict kinematic and dynamic constraints (e.g., motor velocity limits and structural integrity). Existing methods often fail to address this coupled problem:

• Rule-based systems lack flexibility and cannot adapt to live prosody.
• Data-driven models often lack emphatic gestures in training data and may generate trajectories that violate hardware limits.
• LLM-based planners provide semantic grounding but can hallucinate joint configurations and introduce latency.
• Current synchronization methods for physical robots typically rely on simple duration matching (stretching audio or keyframes), which ignores sub-utterance structure. This results in a misalignment between speech stress peaks and gesture stroke maxima, weakening the perceived naturalness of the interaction.

Methodology: The WaveSync Framework

WaveSync is a hybrid framework designed to synchronize gesture strokes with speech emphasis while strictly adhering to robot kinematic constraints. The pipeline consists of three sequential stages:

1. LLM-Based Semantic Scripting and SIW Synthesis

The system converts raw conversational text into a continuous energy signal known as the Semantic Importance Wave (SIW).

• Semantic Decomposition: A Large Language Model (LLM) processes the dialogue response, decomposing it into prosodic segments. For each word, the LLM assigns a semantic importance weight ($\alpha \in [0, 1]$), where values near 1 indicate stressed keywords and values near 0 represent fillers.
• Temporal Alignment: These word-level weights are fused with timestamps from a Whisper-based forced-alignment pipeline to determine the target time ($t_c$) for each gesture peak.
• Wave Construction: Discrete word-weight pairs are transformed into a continuous signal $s(t)$ by superimposing Gaussian kernels. This waveform serves as the optimization target, encoding communicative energy over time.

2. Kinematic Action Modeling via Parabolic Primitives and DMPs

Retrieved gestures from a motion library are adapted to ensure kinematic feasibility and expressiveness.

• Parabolic Modeling: Gesture strokes are modeled as inverted parabolas ($P_j(t)$) to represent the preparation, stroke, and retraction phases. This mathematical form allows for direct alignment with SIW peaks.
• Dynamic Movement Primitives (DMPs): Selected gestures are encoded as DMPs. This allows for:
  • Spatial Scaling: The goal position is scaled based on semantic importance and emotional state (e.g., excitement induces wider/faster motion).
  • Stylistic Variation: Stochastic noise is injected into the forcing function to prevent repetitive, template-like behavior.
  • Temporal Adaptation: A time-scaling parameter ($\tau$) is used as the primary variable to stretch or compress gesture duration ($D(\tau)$) without altering the underlying kinematic signature.

3. Wavefront Optimization

This stage resolves temporal conflicts when long-duration gestures are assigned to closely spaced semantic peaks, ensuring synchronization without violating hardware limits. It employs a three-stage hierarchical policy:

• Peak Alignment: The system maximizes the overlap integral between the gesture pulse and the SIW to align stroke peaks with vocal emphasis.
• Execution Adjustment: If overlaps occur, the duration of subsequent gestures is compressed. A hard lower bound is enforced ($D \geq \frac{2}{3}D_0$) to prevent extreme joint velocities that could damage hardware.
• Forward Propagation: If compression alone cannot resolve the conflict (i.e., the "crossover rule" is violated), the system shifts all subsequent gestures forward in time. The accumulated time offset is injected into the Text-to-Speech (TTS) pipeline as natural inter-segment pauses. This preserves speech prosody while providing the robot sufficient time to complete kinematically valid strokes. Motion interpolation ensures smooth transitions at the boundaries of clipped gestures.

Key Contributions

1. Hybrid Architecture: A novel pipeline that converts LLM-driven semantic intent into robot motions via DMPs, effectively circumventing the risks of generating raw joint angles directly from LLMs.
2. Wavefront Optimization: A synchronization module that achieves precise, word-level peak-to-peak alignment between speech stress and gesture strokes while strictly satisfying real-time kinematic and dynamic constraints.
3. Hardware-Safe Scheduling: The introduction of a mechanism that resolves motion conflicts through duration compression and natural pauses rather than distorting the speech signal or risking hardware damage.

Experimental Results

The system was evaluated in high-fidelity PyBullet simulations using a humanoid upper-body model across five dialogue scenarios (greeting, casual, descriptive, warning, and thinking).

• Objective Evaluation:
  • Synchronization Accuracy: WaveSync achieved a mean Human-based Sync (HbS) score of 0.307, keeping alignment errors generally below the 1-second perceptual tolerance threshold. Scenario S3 (Descriptive) yielded the best results (HbS = 0.534), while S2 (Casual) showed lower fidelity due to ambiguous gesture peaks.
  • Trajectory Fluidity: WaveSync maintained consistently low joint jerk magnitude. In contrast, the baseline without DMPs exhibited frequent spikes exceeding 140 m/s³, confirming that keyframe interpolation without trajectory shaping produces abrupt, unsafe motions.
• Subjective Evaluation:
  • A user study with 20 participants rated the system on Motion Smoothness, Temporal Synchronization, Semantic Appropriateness, and Overall Naturalness.
  • WaveSync consistently outperformed three ablated baselines (w/o SIW, w/o DMP, w/o Wavefront).
  • The largest performance gaps were observed in Temporal Synchronization and Overall Naturalness. The study confirmed that removing any single component (semantic weighting, DMP shaping, or wavefront scheduling) significantly degraded the user experience.

Significance and Claims

The paper claims that WaveSync successfully addresses the gap between expressive gesture generation and the physical limitations of humanoid robots. By integrating semantic importance with kinematic constraints, the system produces gestures that are expressive, semantically grounded, and kinematically compliant.

The authors note that the current system relies on a fixed action library, which limits coverage for unconstrained dialogue. Consequently, the paper modestly frames future work as addressing this limitation through automatic gesture synthesis and emphasizes that testing WaveSync on real robot hardware remains the next critical step. The work is presented as a foundational step toward deploying social robot applications with natural, synchronized nonverbal communication.