FIELDS: Face reconstruction with accurate Inference of Expression using Learning with Direct Supervision

Technical Summary: FIELDS

Problem Statement
Monocular 3D face reconstruction aims to estimate a 3D Morphable Model (3DMM) representation from a single image, providing geometry-aware expression codes useful for facial expression recognition (FER) and affect understanding. While current pipelines have made significant progress, they are predominantly trained using image-level self-supervision (reconstruction consistency) and evaluated based on geometric fidelity. The authors argue that this approach does not necessarily maximize the "affective utility" of the learned expression representation. Furthermore, naively coupling affect supervision with image-level consistency losses can encourage "intensity-amplifying shortcuts," where the model generates exaggerated expressions to satisfy emotion constraints, thereby compromising geometric plausibility. Existing methods that attempt to bridge this gap often rely on weak image-level emotion consistency losses or external pre-trained emotion modules, which can lead to a trade-off between capturing affect and rendering plausible images.

Methodology: FIELDS
The paper proposes FIELDS (Face reconstruction with accurate Inference of Expression using Learning with Direct Supervision), a task-driven framework designed to resolve the tension between affective utility and geometric plausibility. FIELDS extends the TEASER self-supervised monocular 3D face reconstruction pipeline by introducing hybrid 2D/3D supervision.

The architecture consists of:

1. Encoder: A FLAME-based encoder that predicts shape ($\beta$), expression ($\Psi$), and pose ($\Theta$) parameters, alongside an appearance token.
2. Neural Synthesizer: A token-guided synthesizer that reconstructs the image from the FLAME mesh, background mask, and appearance tokens.
3. Auxiliary Emotion Head: A lightweight MLP trained jointly to predict continuous valence-arousal scores and discrete emotion categories directly from the FLAME parameters.

The core innovation lies in the Hybrid Supervision Strategy:

• Scan-Derived Expression Anchoring: To stabilize expression geometry and prevent intensity amplification, the framework uses limited "fit-derived" targets. Specifically, FLAME expression parameters are fitted to BP4D 3D scan data. These scan-derived parameters serve as direct anchors for the expression encoder via a Mean Squared Error (MSE) loss ($L_{3D-expr}$).
• Direct Parameter-Space Affect Supervision: Instead of relying on image-level emotion consistency, FIELDS introduces an auxiliary emotion recognition head that applies supervision directly in the 3DMM parameter space. This head is trained using a combination of regression (for valence-arousal) and classification (for discrete emotions) losses ($L_{emo}$). This encourages the expression codes to be informative for affect without relying on perceptual comparisons between rendered and real images.
• 2D Consistency: The framework retains standard 2D consistency losses (photometric, perceptual, and landmark-based) to ensure the reconstructed image remains visually plausible.
• Regularization: A pre-trained base model (aligned with MICA) is used to regularize the shape and expression parameters, ensuring the learned manifold remains stable.

The training procedure alternates between an Encoder Pass (optimizing for 2D consistency, 3D expression anchoring, and emotion supervision) and a Synthesizer Pass (optimizing only the synthesizer for 2D consistency and cycle consistency), preventing one component from compensating for the errors of the other.

Key Contributions

1. Framework: The proposal of FIELDS, a task-driven framework that combines self-supervised image-level consistency with scan-derived expression anchoring and a jointly-trained affect recognition head applying supervision directly in the 3DMM parameter space.
2. Scan-Derived Anchors: The derivation of scan-based (fit-derived) expression targets from the BP4D dataset to serve as limited anchors for directly supervising FLAME expression parameters, reducing the risk of intensity-amplifying shortcuts.
3. Evaluation: A comprehensive evaluation demonstrating that FIELDS improves affect prediction (both discrete and continuous) while maintaining competitive geometric fidelity on held-out and out-of-domain benchmarks.

Results
The authors evaluate FIELDS on multiple benchmarks, comparing it against baselines like EMOCA, SMIRK, and TEASER.

• Affective Utility: On the AffectNet dataset, FIELDS achieves state-of-the-art results for both discrete emotion classification (56.0% accuracy) and valence-arousal regression (0.737 V-CCC), outperforming previous methods.
• Geometric Plausibility: On the BP4D dataset (using held-out subjects), FIELDS maintains competitive 3D geometric fidelity (vertex and keypoint errors) comparable to reconstruction-centric baselines. Crucially, it significantly improves agreement with scan-derived expression and jaw parameters (reducing FLAME parameter MSE from ~5.9 to ~0.17) compared to baselines that lack direct 3D supervision.
• External Validation: On RAF-DB (using KNN probing on expression parameters only), FIELDS shows superior transferability (73.0% accuracy). On MultiREX (out-of-domain geometry), it remains comparable to existing baselines, confirming that the affective gains do not come at the cost of general geometric quality.
• Qualitative Analysis: Visual comparisons indicate that FIELDS preserves input-consistent local geometry (e.g., subtle mouth-corner shapes, lower-face tension) better than baselines, which often exhibit over-smoothed or exaggerated deformations.

Significance and Claims
The paper claims that FIELDS provides a practical route to obtaining FER-ready 3DMM representations that are robust in the wild and useful for downstream affective computing applications. By explicitly supervising 3DMM parameters with scan-derived anchors and direct affect labels, the framework circumvents the traditional trade-off between capturing affect and rendering plausible images. The authors assert that their approach learns affect-informative expression representations that are less sensitive to nuisance factors (pose, illumination) and better aligned with underlying facial muscle movements, all while preserving geometric plausibility.

The paper concludes modestly, noting that future work could involve temporal modeling for video FER, exploring multi-loss weighting strategies for better robustness across datasets, and incorporating additional structured priors like action units when available.