💻 computer science

AffectFlow-DINO: Uncertainty-Aware Multi-Task Affect Estimation via Conditional Rectified Flow

AffectFlow-DINO is a multi-task learning system for the 11th ABAW challenge that leverages a conditional rectified-flow head on a DINOv3 backbone to model facial affect ambiguity through uncertainty-aware generative predictions, achieving significant performance gains in valence-arousal estimation, expression classification, and action unit detection.

Original authors: Salah Eddine Bekhouche, Abdellah Zakaria Sellam, Fadi Dornaika, Abdenour Hadid

Published 2026-07-16

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Salah Eddine Bekhouche, Abdellah Zakaria Sellam, Fadi Dornaika, Abdenour Hadid

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to guess how a friend is feeling just by looking at a single photo of their face. It's a tricky game. A tiny smile might mean they are happy, or maybe they are just being polite while feeling sad. A furrowed brow could mean anger, or just deep concentration. This is the world of "affective computing"—teaching computers to read human emotions. For a long time, scientists tried to build models that acted like a strict judge, forcing every face into one single box: "This is 100% Happy" or "This is 100% Sad." But human faces are messy and ambiguous. Sometimes, a face holds a mix of feelings, or the lighting makes a subtle expression hard to read. The big question in this field is: How do we teach a computer to understand that a face isn't just one single answer, but a whole range of possibilities?

This paper introduces a new system called AffectFlow-DINO, designed to solve exactly that problem. Instead of forcing the computer to pick just one "best guess" for a person's mood, the team built a model that learns to imagine many possible moods at once. Think of it like a weather forecaster. A traditional model might say, "It will rain at 2 PM." A better, uncertainty-aware model says, "There's a 70% chance of rain, a 20% chance of clouds, and a 10% chance of sun." AffectFlow-DINO does this for emotions. It uses a clever mathematical trick called "rectified flow" to generate a cloud of plausible emotional states for every face it sees. By averaging these possibilities, it often finds a more accurate answer than a standard model ever could. The researchers tested this on a massive dataset of real-world faces and found that by embracing uncertainty, their system became significantly better at spotting subtle feelings, especially for rare emotions like fear or sadness that usually get ignored.

The Problem: The "One-Size-Fits-All" Trap

Imagine you are trying to describe a complex painting to a friend over the phone. If you are forced to pick just one word to describe the whole image, you might say "Blue." But the painting is actually a mix of blue, gray, and a tiny splash of red. If you only say "Blue," you miss the nuance.

In the world of facial recognition, computers have been stuck in this "one-word" trap. They look at a face and try to output a single number for "Valence" (how positive or negative the mood is) and "Arousal" (how excited or calm), plus a list of facial muscle movements called "Action Units." The problem is that in the real world, faces are ambiguous. A slight twitch of the mouth could be a smile, a grimace, or just a muscle spasm. A standard computer model squashes all that ambiguity into a single, rigid prediction, throwing away the uncertainty that actually holds the clues to the right answer.

The Solution: A "What-If" Machine

The authors of this paper decided to stop asking the computer, "What is the emotion?" and start asking, "What could the emotion be?"

They built a system called AffectFlow-DINO. It starts with a powerful, pre-trained "brain" (a DINOv3 visual backbone) that is already very good at recognizing faces. But instead of just letting that brain make a single guess, they added a special "imagination engine" on top of it. This engine uses a technique called Rectified Flow.

Here is a simple way to visualize how it works:

The Noise: Imagine starting with a blank canvas covered in static noise (like TV snow).
The Journey: The model learns a straight, efficient path to turn that noise into a specific emotion. It's like drawing a direct line from "confusion" to "happiness."
The Cloud: When the model sees a new face, it doesn't just draw one line. It draws many lines from the noise to the emotion space. Each line represents a different, plausible interpretation of that face.
The Average: Finally, it takes the average of all those lines. If the face is clearly happy, all the lines will point to "Happy," and the average is a strong, confident "Happy." If the face is ambiguous, the lines will spread out, and the average gives a nuanced answer that captures the uncertainty.

The Results: Getting Better at the Hard Stuff

The team tested their system on the s-Aff-Wild2 dataset, which contains thousands of static images of faces from the real world. These images are tricky because they are often poorly lit, partially covered, or show very subtle expressions. The goal was to predict three things at once:

Valence-Arousal: How positive/negative and how excited/calm the person is.
Expressions: Which of eight emotions (like Joy, Fear, Sadness) is present.
Action Units: Which of twelve specific facial muscles are moving.

The results were impressive. By using their "imagination engine," the model improved its ability to guess the Valence-Arousal scores by a significant margin (a score improvement of +0.058). But the real magic happened with the rare emotions.

In the real world, some emotions are very rare. In their dataset, Fear appeared in only about 3.8% of the labeled images, and Sadness was also quite rare. Standard models usually ignore these rare cases, guessing "Neutral" or "Other" most of the time. The AffectFlow-DINO system, however, managed to recover the ability to spot these rare emotions.

Through a clever post-processing step (tuning the "decision threshold" for each emotion separately) applied to a model that had been fine-tuned on the specific task, they boosted the model's ability to detect Fear from a dismal 3.8% F1 score up to 33.1%, and Sadness from 17.1% to 28.2%. This specific calibration step didn't require retraining the whole model from scratch; it was just a matter of listening more closely to the signals the fine-tuned model was already learning.

What They Ruled Out

The researchers didn't just try one thing; they tested 26 different design choices to see what actually worked.

They ruled out "Just Flow" or "Just Deterministic": They found that if they only used the "imagination engine" (Flow) without the standard "one-guess" heads, the model failed. Conversely, if they only used the standard heads without the flow, they missed out on the nuance. The two parts must work together.
They ruled out simple fixes for imbalance: They tried making the computer pay more attention to rare emotions by just changing the math weights or using "Focal Loss." These methods didn't work well on their own and sometimes made the model worse at other tasks.
They ruled out "Kitchen Sink" fine-tuning: They tried combining every possible trick (changing weights, sampling strategies, etc.) all at once. Surprisingly, this "kitchen sink" approach didn't improve the final score; in fact, it sometimes hurt the model's ability to guess the mood (Valence) accurately.

The Bottom Line

The paper suggests that the key to better emotion recognition isn't just making the computer "smarter" or "bigger," but teaching it to be comfortable with uncertainty. By letting the model explore a range of possibilities and then averaging them, AffectFlow-DINO achieved a final performance score of 1.177, which is more than double the official baseline score of 0.45.

The authors conclude that while their system is a major step forward, the biggest gains came from two things:

Fine-tuning the visual brain: Adjusting the pre-trained face recognizer to the specific task.
Calibrating the decision thresholds: Realizing that the computer was actually "seeing" the rare emotions, but just needed a nudge to trust its own hunches.

This work suggests that the future of emotion AI lies not in forcing a single answer, but in embracing the messy, multi-faceted reality of human expression.

Technical Summary: AffectFlow-DINO

Problem Statement

The 11th Affective Behavior Analysis in-the-wild (ABAW) competition presents a challenging Multi-Task Learning (MTL) scenario where each static face image must be simultaneously annotated with three complementary affect representations: continuous Valence-Arousal (VA) values, an eight-way categorical expression label, and twelve binary Action Unit (AU) labels.

This setting introduces significant difficulties:

Heterogeneous Targets: The tasks differ in output type (regression, single-label classification, and multi-label detection) and supervision density.
Severe Class Imbalance: The s-Aff-Wild2 dataset exhibits an 8× imbalance in expression classes (dominated by Neutral, Other, and Happiness) and a 29× imbalance in AUs.
Incomplete Supervision: Joint three-task annotations cover only 37% of training frames, requiring robust handling of missing labels.
Perceptual Ambiguity: In-the-wild facial behaviors (e.g., subtle smiles, partial occlusions) often correspond to multiple plausible configurations in the joint affect space. Standard deterministic models collapse this ambiguity into a single point estimate, discarding valuable uncertainty information.

Methodology: AffectFlow-DINO

The authors propose AffectFlow-DINO, a frame-level MTL system that augments a standard deterministic architecture with a conditional rectified-flow head to model the full joint affect distribution $p(y|x)$ .

Architecture Overview

The system comprises four primary components:

Visual Backbone: A frozen DINOv3 ViT-S/16 encoder extracts rich facial representations. The authors note that DINO's self-supervised objectives enforce spatial semantic consistency, making it well-suited for detecting localized muscle activations (AUs) compared to reconstruction-based pre-training like MAE.
Shared Projection: A projection head maps backbone features into a compact 768-dimensional affect embedding $z$ .
Deterministic Heads: Three task-specific MLP heads produce point predictions for VA (constrained to $[-1, 1]$ via tanh), Expression (8 logits), and AUs (12 logits).
Conditional Rectified-Flow Head: This core innovation learns a generative transport map from isotropic Gaussian noise to the conditional target distribution $p(y|x)$ . It takes the image embedding $z$ , a linearly interpolated noised target $y_t$ , and a time embedding $\gamma(t)$ to predict the transport velocity.

Joint Representation and Masked Supervision

The model unifies heterogeneous targets into a single 22-dimensional vector $y \in \mathbb{R}^{22}$ , comprising VA (2 dims), Expression (8 dims), and AUs (12 dims). During training, categorical and binary dimensions are treated as unconstrained real-valued scores within the flow trajectory. Discrete predictions are recovered only at inference via $\arg\max$ and sigmoid thresholding.

To handle incomplete annotations, a per-sample binary validity mask $m_i$ is applied to all loss terms. This ensures that missing targets do not contribute gradients or distort the flow training signal. The authors argue that under the assumption that annotation missingness is independent of true affect values, masking allows the model to learn the full joint distribution from partial marginals.

Training and Inference

Loss Function: The total objective combines a weighted deterministic multi-task loss ( $\mathcal{L}_{det}$ ) and a flow loss ( $\mathcal{L}_{flow}$ ):
$\mathcal{L} = \mathcal{L}_{det} + \beta \mathcal{L}_{flow}$
The flow loss minimizes the squared error between the predicted velocity and the true constant velocity field $v^* = y - \epsilon$ .
Inference: Instead of a single forward pass, the model performs Monte Carlo sampling. $N$ independent noise samples are integrated via Euler steps from $t=0$ to $t=1$ . The final prediction is the average of these trajectories, providing both a point estimate (the mean) and an uncertainty measure (the spread).

Key Contributions

Masked Rectified-Flow MTL Objective: A novel framework that jointly optimizes deterministic task heads and a generative conditional distribution over a 22-dimensional affect vector, handling missing labels via masking.
Systematic Ablation Study: An extensive evaluation spanning 26 design choices on s-Aff-Wild2, covering flow loss weights, inference efficiency, imbalance remedies, backbone fine-tuning, and post-hoc calibration.
Post-Hoc Calibration Strategies: The demonstration that per-AU and per-class expression threshold calibration can effectively recover performance on severely imbalanced rare classes (e.g., Fear, Sadness, AU15, AU23) without retraining.

Experimental Results

The system was evaluated on the s-Aff-Wild2 validation split. The official challenge baseline achieves a composite metric $PMTL = 0.45$.

Baseline Performance: A deterministic-only model ( $\beta=0$ ) achieves $PMTL = 0.793$. Adding the flow head with flow decoding ( $\beta=1$ ) improves this to $PMTL = 0.826$, with the flow sampling specifically boosting Valence-Arousal (VA) estimation ($CCC-V +0.058$).
Backbone Fine-Tuning: Low-learning-rate fine-tuning of the DINOv3 backbone provided the most significant gain, raising the best frozen-backbone result ($0.831$) to $PMTL = 1.045$. Interestingly, after fine-tuning, deterministic decoding outperformed flow decoding, suggesting the flow head required retuning.
Flow Retuning: Re-activating the flow objective ( $\beta=1.0$ ) during backbone fine-tuning improved deterministic performance to $PMTL = 1.073$.
Calibration Gains:
- Per-AU Threshold Calibration: Improved $PAU$ from $0.453$ to $0.502$, raising the total score to $1.123$.
- Per-Class Expression Calibration: By tuning class-specific logits on the validation set, the model recovered the latent signal for rare classes. Fear F1 increased from $3.8\%$ to $33.1\%$ , and Sadness from $17.1\%$ to $28.2\%$ .
Final Performance: The combination of backbone fine-tuning, flow retuning ( $\beta=1.0$ ), and combined post-hoc calibration achieved a final $PMTL = 1.177$, substantially outperforming the official baseline ($0.45$) and the best uncalibrated fine-tuned model ($1.045$).

Significance and Claims

The paper claims that AffectFlow-DINO demonstrates the value of modeling facial affect as a conditional distribution rather than a deterministic mapping. The authors highlight three primary lessons:

Complementarity of Objectives: The flow and deterministic objectives are mutually beneficial. Flow-only training collapses without the deterministic anchor, while deterministic-only inference discards distributional capability. Flow sampling is particularly effective for continuous targets (VA) where ambiguity is highest.
The Role of Fine-Tuning: While backbone adaptation is the dominant factor in performance gains, the generative pathway degrades if the flow head is not jointly retuned during fine-tuning.
Unlocking Rare Signals: Severe class imbalance suppresses learned signals but does not erase them. The authors claim that post-hoc calibration can "unlock" near-zero F1 scores for rare categories (Fear, Sadness, rare AUs) at zero retraining cost, suggesting that decision boundaries can be adjusted to surface signals already encoded in the representation.

The authors conclude that closing the gap to State-of-the-Art (SOTA) likely requires temporal aggregation and more expressive fine-tuned decoding, noting that their current work is strictly frame-level.