CogPortrait: Fine-Grained Eye-Region Control in Portrait Animation via Hierarchical Agent Planning

Technical Summary: CogPortrait

Problem Statement
Existing portrait animation methods face a significant trade-off between input granularity and motion accuracy, particularly regarding fine-grained manipulation of the eye region. Current approaches generally fall into two categories:

1. High-level label-driven methods: These accept simple inputs like emotion labels or coarse text prompts. While user-friendly, they operate at a macro semantic level (e.g., "anger") and fail to capture subtle, irregular ocular dynamics such as asymmetric blinks, gaze aversion, or cognitive states like drowsiness and thinking.
2. Motion-driven methods: These utilize precise signals like facial keypoints, Action Units (AUs), or driving videos. While they offer higher fidelity, they impose a heavy input burden on users, requiring manual construction of motion signals. Furthermore, they often struggle with irregular patterns (e.g., large-angle head movements) due to data scarcity and underrepresentation in training sets.

The core challenge addressed by this work is achieving direct, fine-grained control over eye-region dynamics (gaze, eyelids, blinks, eyebrows) and head pose from high-level semantic labels without requiring manual motion-level inputs.

Methodology
The authors propose CogPortrait, a two-stage framework designed to compile high-level labels into executable facial keypoints and subsequently generate high-quality portrait animations.

• Stage 1: Hierarchical Agent Planning (Label-to-Keypoints)
  Instead of directly regressing motion signals from text (which often results in over-smoothed trajectories), the authors employ a structured, chain-of-thought (CoT) Multi-Modal Large Language Model (MLLM) agent system to decompose high-level labels into temporal events and retrieve behavioral prototypes.

  • Planning Agent: Decomposes the target label (e.g., "cognitive effort") and optional fine-grained instructions into a sequence of temporally structured events, specifying timestamps, frame spans, and local constraints.
  • Composition Agent: Retrieves and assembles behavioral prototypes from a curated Prototype Library. This library contains real-world facial behavior data (AU intensities, gaze, head pose) to ensure genuine temporal dynamics. The agent matches the staged events to these prototypes, trimming and rescaling them to fit the target duration.
  • Critic Agent: Enforces semantic consistency and physiological plausibility. It validates the sequence against rules regarding blink duration, inter-blink intervals, AU co-activation, and gaze-head coordination. It can send revision signals back to the composition or planning agents to correct inconsistencies.
  • Mapping Layer: Projects the refined control sequences (AU, gaze, head pose) into 62 specific facial keypoints (eyelids, eyebrows, pupils, irises) using FLAME mesh deformation and rigid rotation.

• Stage 2: DiT-Based Video Generation
  A diffusion transformer (DiT) backbone, based on Wan2.2, synthesizes the final video conditioned on the reference portrait, driving audio, text prompts, and the generated facial keypoints.

  • Dynamic Classifier-Free Guidance (CFG): To address global color shifts (common with uniform backgrounds) and weak high-frequency eye motions, the authors introduce a dynamic CFG strategy. This involves temporal reweighting (stronger guidance early in denoising to fix global appearance, weaker later) and spatial reweighting (a Gaussian map that concentrates guidance on the eye region while maintaining base guidance for the background).
  • KTO-Based Refinement: To handle "boundary cases" (e.g., asymmetric eyebrow control, identity degradation during large-angle head turns) that are underrepresented in training data, the model is refined using Kahneman-Tversky Optimization (KTO). This uses curated desirable (real) and undesirable (model-generated failures) samples to align the model with human preferences for control accuracy and identity consistency without requiring paired preference data.

Key Contributions

1. CogPortrait Framework: A novel two-stage pipeline that translates high-level semantic labels into fine-grained eye-region facial keypoints via hierarchical agent planning, prototype retrieval, and constraint enforcement.
2. Enhanced Generation Strategies: The introduction of a dynamic CFG with eye-region-aware reweighting to balance local controllability and global visual quality, and KTO-based refinement to improve performance on long-tail, irregular motion patterns.
3. EMH Benchmark: A new evaluation benchmark covering six core emotions and six "beyond-emotion" states (e.g., cognitive effort, drowsiness, social engagement). It introduces two AU-level metrics: AU-F1 (activation correctness) and AU-Temp (temporal trajectory fidelity) to specifically evaluate fine-grained eye-region and head-motion control.

Results
Extensive experiments were conducted on the HDTF dataset and the new EMH benchmark:

• HDTF Performance: CogPortrait achieved state-of-the-art performance in visual quality (FID: 16.68, FVD: 32.90), identity consistency (ID-Sim: 0.9214), and lip synchronization. Notably, it achieved the lowest Eye-LMD (0.0107), representing a 38% relative improvement over the second-best method, confirming superior eye-region control.
• EMH Benchmark Performance: On the EMH benchmark, the full pipeline (Ours Full) achieved an AU-F1 of 0.9017 and AU-Temp of 0.7397. This significantly outperformed audio-driven and label-driven baselines (e.g., Sonic, FLOAT, EDTalk) and was competitive with motion-driven methods (HunyuanPortrait), despite not using ground-truth motion inputs.
• Ablation Studies:
  • Removing the agent-based planning and relying on direct label-to-keypoint mapping resulted in poor temporal dynamics (AU-Temp dropped to 0.5676).
  • Removing the dynamic CFG led to global color shifts and reduced identity consistency.
  • Removing KTO refinement resulted in higher error rates (Eye-LMD 0.0377 vs. 0.0311) and lower identity consistency (ID-Sim 0.9089 vs. 0.9394) on boundary cases like asymmetric eyebrow movements and large-angle head turns.

Significance
The paper claims that CogPortrait successfully bridges the gap between high-level semantic intent and low-level motion execution in portrait animation. By leveraging hierarchical agent planning and a prototype library, it enables the generation of complex, beyond-emotion states (such as thinking and drowsiness) that were previously difficult to control without manual driving signals. The work demonstrates that fine-grained eye-region control can be achieved directly from high-level labels while maintaining superior visual quality and identity consistency, expanding the expressive range of portrait animation for applications in virtual reality, human-computer interaction, and film production.