Face Anything: 4D Face Reconstruction from Any Image Sequence

1. Problem Statement

Reconstructing and tracking dynamic human faces from unconstrained image sequences is a fundamental yet challenging problem in computer vision. The core difficulties include:

• Non-rigid Deformations: Faces undergo complex shape changes due to expressions and head motion while preserving fine geometric details (wrinkles, hair, mouth interiors).
• Ambiguity: Simultaneous variations in geometry, viewpoint, and expression create significant ambiguity in estimating 3D structure and establishing dense correspondences (tracking points across frames).
• Limitations of Existing Methods:
  • Motion-based approaches (predicting frame-to-frame flow) struggle with long sequences and complex motions, often requiring multiple forward passes and heavy supervision.
  • Parametric models (e.g., 3DMMs) lack geometric detail.
  • Single-image predictors lack temporal consistency.
  • Separate pipelines often treat reconstruction and tracking as distinct problems, failing to provide unified, temporally consistent 4D outputs.

2. Methodology

The authors propose Face Anything, a unified, feed-forward framework that performs high-fidelity 4D facial reconstruction and dense tracking simultaneously.

Core Concept: Canonical Facial Point Prediction

Instead of predicting motion vectors or deformation fields between frames (which is difficult to learn), the method predicts a Canonical Map.

• Definition: Each pixel in an input image is assigned a normalized coordinate in a shared canonical space (a standard pose and expression space).
• Mechanism: By mapping all frames to this shared canonical space, dense correspondences are established via simple nearest-neighbor search in the canonical domain.
• Benefit: This transforms the complex problem of "tracking over time" into a simpler "reconstruction in a normalized space," ensuring temporal consistency and identity preservation.

Architecture

• Model: A transformer-based architecture (inspired by DA3 and DPT) that processes one or multiple input images simultaneously.
• Outputs: The network jointly predicts three maps for each input frame:
  1. Depth Maps ($D$): 3D geometry.
  2. Ray Maps ($R$): Camera ray directions.
  3. Canonical Maps ($C$): The normalized facial coordinates.
• Inference: A single forward pass generates all outputs. Correspondences between any two frames $I_i$ and $I_j$ are found by matching pixels $p$ and $q$ where their canonical coordinates $C_i(p) \approx C_j(q)$ are closest.

Training Strategy & Dataset

• Two-Stage Training:
  1. Pre-training: The model is pretrained on the DAViD dataset (monocular) to learn strong facial geometric priors.
  2. Fine-tuning: The model is fine-tuned on a custom dataset derived from NeRSemble.
• Custom Dataset Construction:
  • Since existing datasets lack dense canonical supervision, the authors constructed a new dataset using NeRSemble (multi-view synchronized videos).
  • Pipeline: They used COLMAP for high-quality multi-view 3D reconstruction and FLAME (parametric face model) for tracking.
  • Canonicalization: They transferred FLAME's deformation vectors to the COLMAP point clouds, creating "canonicalized" 3D points. These were projected back to 2D to create ground-truth canonical maps for supervision.
• Sampling: Training alternates between multi-view (same timestamp, different cameras) and monocular video (same camera, different timestamps) to ensure generalization to both reconstruction and tracking scenarios.

Loss Functions

The model is trained using a combination of regression, confidence-weighted, and gradient losses for Depth, Ray, and Canonical maps. A specific weight ($\lambda_C = 5$) is applied to the canonical map loss to balance geometry and correspondence learning.

3. Key Contributions

1. Unified Feed-Forward Framework: A single model that performs both 4D reconstruction and dense tracking without explicit motion modeling or iterative optimization.
2. Canonical Representation: The introduction of canonical map prediction for faces, which simplifies correspondence learning and enforces temporal stability.
3. New Dataset: A large-scale, high-quality dataset based on NeRSemble with dense canonical correspondences derived from multi-view geometry and FLAME tracking.
4. State-of-the-Art Performance: Achieving superior results in depth estimation, 4D reconstruction, and point tracking compared to prior art.

4. Experimental Results

The method was evaluated on NeRSemble, Ava-256, VFHQ, and CelebV-HQ.

• Depth Accuracy:
  • Achieved 16% improvement in depth accuracy compared to previous methods.
  • Outperformed large foundation models (e.g., Sapiens-2B) despite using a smaller model size.
  • Multi-view input further reduced error, particularly in side-views.
• Tracking & Correspondence:
  • 3× lower correspondence error compared to dynamic reconstruction methods like V-DPM.
  • Significantly outperformed P3DMM and V-DPM in End-Point Error (EPE) and cycle consistency metrics.
  • Maintained stable tracking even in hair regions and under large expressions where other methods failed.
• Efficiency:
  • Inference Speed: ~32× faster than V-DPM (5 seconds vs. 160 seconds for 40 images).
  • Memory: Requires significantly less GPU memory (19GB vs. 40GB) and supports larger batch sizes (470 images vs. 74).
• Ablation Studies: Confirmed that joint training of depth and canonical maps, along with the specific sampling strategy (mixing views and time), is critical for balancing geometry and tracking accuracy.

5. Significance and Future Work

Significance:

• Paradigm Shift: Moves away from motion-based tracking to canonical-space reconstruction, offering a more robust and efficient solution for dynamic 3D faces.
• Practicality: The feed-forward nature and high efficiency make it suitable for real-time applications like virtual avatars, telepresence, and animation.
• Foundation: Establishes canonical point prediction as a viable foundation for unified spatiotemporal reconstruction.

Limitations:

• Domain Specificity: The model is specialized for faces; it struggles with non-facial objects (e.g., microphones, hands) which may be incorrectly matched.
• Occlusions: Performance degrades under extreme occlusions or very limited facial visibility.

Future Directions:

• Extending canonical prediction to general objects beyond faces.
• Integrating with generative models for controllable facial animation and avatar creation from monocular video.

In conclusion, Face Anything represents a significant advancement in dynamic 3D face modeling by unifying geometry reconstruction and dense tracking through a novel canonical representation, achieving state-of-the-art accuracy with high computational efficiency.