FG-Portrait: 3D Flow Guided Editable Portrait Animation

1. Problem Definition

Portrait animation aims to synthesize a target image that retains the identity of a source portrait while adopting the head pose and facial expression of a driving portrait.

• Core Challenge: Existing diffusion-based methods often fail to establish explicit correspondences between the source and driving images. They typically condition the model only on the driving motion (e.g., landmarks or the driving image itself), leading to suboptimal motion transfer, especially under large pose variations or significant appearance differences.
• Limitations of Current Approaches:
  • Learning-based Flow: Methods that predict dense motion fields from 2D images are ill-posed (2D to 3D ambiguity) and require massive datasets, often failing to generalize to large pose changes.
  • Identity vs. Motion Trade-off: Models often struggle to balance preserving the source identity with accurately mimicking the driving motion.

2. Methodology

The authors propose FG-Portrait, a framework that integrates 3D geometric priors into a diffusion-based animation pipeline. The core innovation is the use of 3D Flows as a motion condition.

A. 3D Flow as Motion Correspondence

Instead of learning a motion field from 2D data, the method computes motion correspondence directly from parametric 3D head models (FLAME).

1. Model Estimation: The FLAME parameters (shape $\beta$, pose $\theta$, expression $\psi$) are estimated for the source ($M_{src}$) and driving ($M_{dri}$) images.
2. Target Assembly: A target 3D head ($M_{tgt}$) is constructed by combining the source shape ($\beta_{src}$) with the driving pose and expression ($\theta_{dri}, \psi_{dri}$).
3. Flow Calculation: For every point $p_{tgt}$ on the target 3D mesh, the corresponding point $p_{src}$ on the source mesh is found using a surface field search. The 3D flow ($f_{src \leftarrow tgt}$) is the vector displacement: $p_{src} - p_{tgt}$. This flow represents the geometric movement required to map the target state back to the source state.

B. 3D Flow Encoding & Depth-Guided Sampling

To integrate these 3D flows into a 2D diffusion model (ControlNet), the authors propose a specific encoding strategy:

1. Back-projection: For each pixel in the target 2D image, the method back-projects a ray into 3D space.
2. Depth-Guided Sampling: Instead of uniformly sampling points along the ray (which often misses the actual surface), the method samples points near the rendered depth map of the target head. This ensures the sampled 3D points align with the actual 2D surface.
3. Encoding: The 3D flows for these sampled points are computed and stacked to form a 3D Flow Encoding ($F_{src \leftarrow tgt}$), which serves as the motion condition input to the ControlNet.

C. Editable Animation

The framework supports feed-forward editing during inference. Users can modify the FLAME pose ($\theta$) and expression ($\psi$) parameters of the driving motion. The system recalculates the target 3D head and the corresponding 3D flows, allowing for precise control over the final animation without retraining.

3. Key Contributions

1. Learning-Free 3D Flow: Introduction of 3D flows derived from parametric 3D head models as a geometry-aware motion correspondence. This avoids the ambiguity of 2D-to-3D prediction and ensures robustness across diverse poses and identities.
2. 3D Flow Encoding with Depth-Guided Sampling: A novel mechanism to query and encode 3D flows for 2D pixels, ensuring accurate alignment between 3D geometric displacement and 2D image motion.
3. User-Specified Editing: The ability to edit facial expressions and head poses dynamically during inference by manipulating FLAME parameters, offering greater flexibility than previous methods.
4. State-of-the-Art Performance: Demonstrated superior performance in both motion accuracy (Pose/Expression error) and identity preservation compared to existing diffusion-based and GAN-based methods.

4. Experimental Results

The method was evaluated on the VFHQ (high-fidelity video) and FFHQ datasets using two settings: Self-reenactment (same identity) and Cross-reenactment (different identities).

• Quantitative Metrics:
  • Motion Accuracy: FG-Portrait achieved the lowest Average Pose Distance (APD) and Average Expression Distance (AED) in both self- and cross-reenactment tasks. For example, in cross-reenactment on VFHQ, it achieved an APD of 7.764 and AED of 0.652, outperforming baselines like X-Portrait (APD 11.22) and Face-Adapter (APD 7.785).
  • Identity Preservation: It maintained high CSIM (Cosine Similarity) scores, striking a better balance between motion fidelity and identity retention than competitors.
  • Image Quality: Achieved competitive LPIPS and FID scores.
• Qualitative Results:
  • The model successfully handles large pose variations and complex backgrounds where other methods fail (e.g., producing over-smoothed results or losing identity).
  • It preserves spatial consistency between the subject and the background, unlike methods that only animate segmented head regions.
• Ablation Studies:
  • Removing Depth-Guided Sampling significantly degraded motion accuracy (APD increased from 2.68 to 9.65), proving the necessity of sampling near the actual 3D surface.
  • Using 3D Flow as a condition significantly outperformed using driving landmarks or predicted 2D flow maps.

5. Significance and Impact

• Bridging 3D Geometry and Diffusion: The paper successfully demonstrates how explicit 3D geometric priors can be effectively integrated into diffusion models to solve the "correspondence problem" in image animation.
• Robustness: By relying on geometry rather than learned 2D motion fields, the method generalizes better to unseen identities and extreme pose changes.
• Controllability: The ability to edit expressions and poses at inference time makes the tool highly practical for applications in film production, digital humans, and personalized content creation.
• Limitations: The authors note that fine-grained expressions may be limited by the resolution of the FLAME mesh, suggesting future work on higher-resolution 3D head models.

In summary, FG-Portrait represents a significant advancement in portrait animation by shifting the paradigm from learning ambiguous 2D motion fields to leveraging precise, geometry-driven 3D flows, resulting in more accurate, editable, and identity-preserving animations.