Technical Summary: Faithful Action-unit Causal Reasoning (FACR)

1. Problem Statement

Automatic facial emotion recognition systems increasingly require explanations that are not merely plausible but faithful—meaning the explanation must reflect the actual factors driving the prediction. Current approaches fall into two categories, both lacking faithfulness:

1. Structural-causal methods: These model Action Units (AUs) and expressions using causal graphs but are purely discriminative. They discard the recovered structure at inference, outputting only labels or attention maps without verbalized rationales.
2. Multimodal Large Language Models (MLLMs): These generate natural-language emotion rationales based on AUs. However, their text is unconstrained by verified causal structures and is evaluated only for plausibility (e.g., text overlap or GPT-based judges). Consequently, they often substitute causal attribution with template matching and fail to update their explanations under counterfactual edits.

The core gap is the absence of a system that grounds an AU→emotion reasoner in a verified causal graph and enforces counterfactual faithfulness, ensuring that intervening on a causal AU changes both the prediction and the rationale, while intervening on an irrelevant AU leaves them unchanged.

2. Methodology: FACR

The proposed system, Faithful Action-unit Causal Reasoning (FACR), addresses this gap through three integrated stages:

A. Structural AU→Emotion Causal Graph Induction ($G$)

FACR induces a directed, polarity-aware causal graph $G = (A \cup C, E)$ independently rather than reusing external graphs. The graph is assembled from three sources:

• AU→Emotion Edges: Estimated via L1-regularized logistic regression on the AffectNet × FABA joint corpus.
• AU→AU Edges: Estimated via neighborhood selection on EmotioNet.
• Verified Sub-structures: The pain sub-structure is fixed using the PSPI (Prkachin–Solomon Pain Intensity) formula as ground truth. For other emotions, edges are filled from EMFACS prototypes.
• Polarity: Each edge carries a sign ($s \in \{-1, +1\}$) indicating excitatory or inhibitory relationships.

B. Graph-Conditioned Reasoner

The system maps an input face $x$ to a disentangled AU latent $z \in \mathbb{R}^K$, where each coordinate $z_k$ represents the presence of AU $k$.

• Prediction: A graph-conditioned head predicts the emotion class $\hat{y}$ based on $z$.
• Rationale: A language verbalizer generates a natural-language explanation naming the active AUs and their causal edges in $G$.
• Intervention Mechanism: The latent space allows for well-defined $do$-interventions (e.g., $do(z_k=0)$), simulating the removal of an AU.

C. Counterfactual-Faithfulness Objective

FACR introduces a training objective that couples the prediction and rationale to the graph $G$ via two terms:

1. Sensitivity Term ($L_{sens}$): Requires that removing an AU $a_k$ that $G$ marks as causal for class $c$ moves the class logit in the direction implied by the edge polarity.
2. Invariance Term ($L_{inv}$): Requires that removing an AU $a_k$ marked as irrelevant to class $c$ leaves the class logit unchanged.

The total loss combines classification loss, AU grounding loss, and these faithfulness terms:
$$L = L_{cls} + \lambda_a L_{au} + \lambda_s L_{sens} + \lambda_i L_{inv}$$

D. Verbalized Rationale Construction

To ensure the generated text is faithful, the verbalizer is gated by construction. The emission of each AU token is biased by its latent activation $z_k$. If an AU is inactive in the latent space, it is strongly suppressed from being named in the text. This ensures that ablating an AU in the latent space reliably removes it from the generated explanation.

3. Key Contributions

1. Formalization: Casts AU→emotion reasoning as a counterfactual-consistency problem between the rationale, the label, and a causal graph.
2. FACR System: Proposes a system that grounds an MLLM reasoner in an independently induced, polarity-aware causal graph and enforces faithfulness via $do$-interventions in a disentangled latent space.
3. Faithfulness Metric: Introduces a counterfactual faithfulness metric (sensitivity and invariance) evaluated against known structures (PSPI for pain, EMFACS for emotion). Crucially, this metric tests fidelity to the supplied structure on held-out data, not the rediscovery of the structure.
4. Empirical Validation: Demonstrates that the faithfulness objective significantly increases agreement between invoked AUs and causal structures (PSPI and EMFACS) across datasets, while explicitly quantifying the trade-off with detection accuracy.

4. Experimental Results

A. UNBC-PAIN (Pain Detection)

Evaluated using leave-one-subject-out (LOSO) cross-validation:

• Faithfulness Gain: The faithfulness objective raised the agreement between invoked AUs and the verified PSPI composition from a baseline of 0.08 (no objective) to 0.57.
• Accuracy Trade-off: This gain came with a modest, quantified cost to detection accuracy, with AUC decreasing from 0.83 (control) to 0.79 (FACR).
• Invariance Recovery: Adjusting the invariance weight ($\lambda_i$) allowed the model to satisfy both sensitivity and invariance constraints simultaneously.

B. Cross-Dataset Emotion Transfer (AffectNet × FABA $\to$ Aff-Wild2)

• Generalization: The objective raised graph-causal agreement from 0.50 to 0.84 on a seven-class emotion task.
• Graph Quality Dependency:
  • With a learned (noisy) graph, agreement with truly active AUs remained low (0.24), matching the control.
  • With a verified graph (EMFACS), agreement with truly active AUs rose to 0.59, while graph-causal agreement remained high (0.74).
  • This confirms that faithfulness to the graph is necessary but not sufficient; the graph's correctness sets the ceiling for alignment with reality.
• Comparison with EmoLA: A published affective MLLM (EmoLA) achieved higher recall of active AUs (0.52 vs. 0.37) by naming many AUs indiscriminately. However, FACR with a verified graph achieved higher precision (0.59 vs. 0.51) and higher faithfulness to the causal structure (0.74 vs. 0.63), demonstrating that it selects the correct causal AUs rather than just many AUs.

C. Verbalized Rationale Audit

• Construction vs. Learning: A verbalizer trained to name active AUs without gating was unfaithful (only 6% of ablated AUs were removed from text).
• Gating Success: With the gating constraint, ablating a causal AU reliably removed it from the generated text.
• Backbone Agnostic: The faithfulness mechanism transferred to a second language model backbone (Gemma 3), confirming the mechanism is not specific to the base model.

5. Significance and Claims

The paper claims that FACR offers a route to constrained, not merely plausible, explanations in AU→emotion reasoning.

• Faithfulness is Distinct from Accuracy: The study demonstrates that faithfulness and accuracy are distinct axes; a model can be made more faithful without becoming more accurate, necessitating separate metrics.
• Interventional Handle: Unlike prior work, FACR provides an auditable intervention handle (the AU latent) that allows for testing the dependence of the explanation on the prediction.
• Design Principle: Faithfulness is most robust when enforced by construction (via gating and causal constraints) rather than learned through capacity or data alone.
• Limitations: The authors explicitly state that FACR is not a state-of-the-art recognition model (modest AUC/F1 scores) and that its alignment with reality is bounded by the correctness of the supplied causal graph. The method does not claim to rediscover structure but to faithfully adhere to a provided one.