🤖 AI

Debiased Multimodal Personality Understanding through Dual Causal Intervention

This paper proposes a Dual Causal Adjustment Network (DCAN) that employs back-door and front-door causal interventions to mitigate subject biases in multimodal personality understanding, achieving state-of-the-art accuracy and fairness improvements on benchmark datasets while introducing a new demographic-annotated dataset for further research.

Original authors: Yangfu Zhu, Zitong Han, Nianwen Ning, Yuting Wei, Yuandong Wang, Hang Feng, Zhenzhou Shao

Published 2026-05-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yangfu Zhu, Zitong Han, Nianwen Ning, Yuting Wei, Yuandong Wang, Hang Feng, Zhenzhou Shao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Stereotype Trap"

Imagine you are trying to guess a person's personality just by watching a short video of them talking. You might look at their face, listen to their voice, and read their words.

The paper argues that current computer programs (AI) are terrible at this because they fall into a stereotype trap.

The Analogy:
Think of an AI model as a student taking a test.

The Real Test: The student should look at the person's actual behavior (e.g., "They are sharing a story about helping a friend") to guess they are "Kind."
The Cheat Sheet (The Bias): Instead, the student notices that in the past, almost all the "Kind" people in the training videos were white women with gentle voices. So, the student stops looking at the behavior and just guesses, "If they are a white woman with a gentle voice, they must be Kind."

This is dangerous. If the AI sees a white woman who is actually not very kind, it will still guess she is kind because it's relying on the "cheat sheet" (demographics) rather than the real evidence (behavior). This leads to unfair and inaccurate results.

The Solution: A "Causal Detective"

The authors, led by Yangfu Zhu and Zhenzhou Shao, built a new system called DCAN (Dual Causal Adjustment Network). They treat the AI like a detective who needs to solve a mystery by separating the real clues from the red herrings.

They use two specific "tools" (based on a concept called Causal Intervention) to fix the problem:

1. The "Back-Door" Tool (Fixing the Visible Bias)

The Problem: The AI is distracted by obvious things like race, gender, or age.
The Analogy: Imagine you are trying to judge a chef's cooking skills. But every time you see a chef wearing a red hat, they happen to be a great cook. You might start thinking, "Red hats = Good cooking."
The Fix: The Back-door Adjustment module acts like a filter. It creates a "dictionary" of all the different types of people (e.g., "Red Hat," "Blue Hat," "No Hat"). When the AI looks at a new chef, it checks this dictionary and says, "Wait, I've seen red hats before. I need to ignore the hat and just taste the soup." It mathematically blocks the connection between the person's appearance and the personality guess.

2. The "Front-Door" Tool (Fixing the Hidden Bias)

The Problem: Some biases are invisible. Maybe the person is tired, stressed, or having a bad day, and the AI mistakes that temporary mood for their permanent personality.
The Analogy: Imagine you are judging a student's intelligence. But the student is currently sick and coughing. You might think, "Coughing = Dumb." You can't see the "sickness" directly, but it's messing up your judgment.
The Fix: The Front-door Adjustment module acts like a translator. It looks for a "middleman" (a mediator) that represents the true meaning of the video, ignoring the sickness. It asks, "If we stripped away the coughing and the stress, what does the student's actual answer say?" It forces the AI to learn from the meaning of the words and actions, not the hidden context that might be misleading.

The New "Fairness" Test Set

To prove their system works, the authors couldn't just use old data because old data was full of these stereotypes. So, they built a brand new dataset called DMSP (Demographic-annotated Multimodal Student Personality).

What it is: A collection of videos of students talking.
Why it's special: They carefully labeled every student with their age, gender, and other details. They also made sure the data was balanced so the AI couldn't cheat by just guessing based on who the person was.
The Goal: To test if the AI can guess personality fairly, regardless of who the person is.

The Results: Smarter and Fairer

When they tested their new "Causal Detective" (DCAN) against other top AI models:

It was more accurate: It got the personality right more often (reaching about 92-93% accuracy).
It was much fairer: It stopped making mistakes based on race, gender, or age.
- Example: On the new dataset, the old models had a "fairness error" of about 16-20%. The new DCAN model dropped that error to about 5-6%. It essentially learned to ignore the "cheat sheet" and focus on the real behavior.

Summary

In short, this paper says: "Current AI is too easily fooled by stereotypes about what people look like."

The authors built a new system that uses two mathematical "filters" to block out those stereotypes (both the obvious ones like gender and the hidden ones like mood). They proved it works by creating a new, fair test set of student videos. The result is an AI that understands personality based on what people do, not just who they are.

Technical Summary: Debiased Multimodal Personality Understanding through Dual Causal Intervention

Problem Statement
Multimodal personality understanding aims to infer intrinsic traits (e.g., Big Five, MBTI) from video data containing visual, auditory, and linguistic signals. While deep learning approaches have achieved high accuracy by fusing heterogeneous modalities, they frequently suffer from subject bias. Models often learn spurious correlations between demographic attributes (observable confounders like age, gender, race) or transient psychological states (unobservable confounders) and personality traits. For instance, a model might erroneously associate "white female" demographics with high agreeableness based on training data distribution, leading to unfair and inaccurate predictions for subjects who do not fit this specific demographic profile. Current methods lack a unified framework to simultaneously address both observable and latent confounders, and existing benchmarks often lack comprehensive demographic annotations or focus solely on Big Five traits, limiting fairness analysis.

Methodology: Dual Causal Adjustment Network (DCAN)
The authors propose a Structural Causal Model (SCM) to formalize the causal relationships between multimodal inputs ( $X$ ), personality traits ( $Y$ ), modality mediators ( $M_x$ ), observable confounders ( $C^o_x$ ), and unobservable confounders ( $C^u_x$ ). To mitigate bias, they introduce DCAN, an end-to-end network integrating two causal intervention mechanisms:

Back-door Adjustment Causal Learning (BACL):
- Target: Observable demographic confounders (e.g., age, gender, race).
- Mechanism: Constructs modality-specific prototypical confounder dictionaries ( $D_{back}$ $D_{ba c k}$ ).
  - Text: Identifies words with significant probability differences across demographic groups to form a dictionary.
  - Visual/Audio: Clusters embeddings by demographic triplets (race, gender, age) to create global prototypes, updated via Exponential Moving Average (EMA).
- Intervention: Retrieves confounder representations via attention and applies a residual connection to the input features. This effectively blocks the back-door path ( $X \leftarrow C^o_x \rightarrow Y$ ), approximating the causal effect $P(Y|do(X))$ by marginalizing over the confounder distribution.
Front-door Adjustment Causal Learning (FACL):
- Target: Unobservable/latent confounders (e.g., transient emotional states, psychological fluctuations) that BACL cannot capture.
- Mechanism: Introduces a mediator ( $M$ ) to form the path $F_m \rightarrow M \rightarrow Y$ . It constructs two dictionaries: a global feature dictionary ( $D_{global}$ ) approximating the input space and a mediator dictionary ( $D_{med}$ ) serving as a knowledge prior.
- Intervention: Uses a dual-attention mechanism to retrieve relevant prototypes from these dictionaries based on the BACL output. This approximates the front-door adjustment formula, isolating the genuine causal effect by intervening on the mediator, thereby handling biases that do not have a direct observable path.
Fusion and Prediction:
- The deconfounded representations from both modules are fused using a Cross-Modal Transformer.
- The model is trained with a unified loss function combining Mean Squared Error (MSE) and $L_2$ regularization to predict continuous trait scores for both Big Five and MBTI taxonomies.

Key Contributions

Causal Analysis: The paper is the first to investigate subject bias in multimodal personality understanding through a causal lens, explicitly modeling both observable and unobservable confounders.
Dual Causal Framework: The proposed DCAN uniquely combines back-door and front-door adjustments in a single network to jointly mitigate demographic and latent biases, moving beyond standard likelihood-based mapping ( $P(Y|X)$ ) to causal intervention ($P(Y|do(X))$).
New Dataset (DMSP): The authors constructed the Demographic-annotated Multimodal Student Personality (DMSP) dataset. Unlike existing benchmarks (e.g., CFI-V2) that primarily focus on Big Five traits and lack detailed demographic annotations, DMSP contains 839 dyadic interaction videos annotated with MBTI traits and sensitive attributes (gender, age), enabling rigorous fairness evaluation.
Comprehensive Evaluation: The work provides extensive experiments demonstrating improvements in both prediction accuracy and fairness metrics (Demographic Parity and Equal Opportunity).

Experimental Results
Extensive experiments were conducted on the CFI-V2 benchmark and the new DMSP dataset.

Accuracy: DCAN achieved state-of-the-art performance, reaching 92.11% accuracy on CFI-V2 (improving over the previous best, PMGRT, by 0.12%) and 92.90% accuracy on DMSP (improving over MM-ResVGG by 1.21%).
Fairness: The model significantly reduced bias. On CFI-V2, improvements in Demographic Parity (DP) and Equal Opportunity (EO) were 6.57% and 7.97%, respectively. On the DMSP dataset, these metrics improved by 15.38% and 20.06%.
Generalization: Out-of-distribution (OOD) tests showed DCAN maintains robust performance across demographic splits (age, gender, ethnicity) where baseline models degrade, particularly in handling gender stereotypes in the T/F (Thinking/Feeling) MBTI dimension.
Ablation: Removing either BACL or FACL resulted in performance degradation, confirming that both observable and latent confounders must be addressed simultaneously.

Significance and Claims
The paper claims that by shifting from correlation-based learning to causal intervention, DCAN successfully disentangles intrinsic personality signals from spurious demographic associations. The authors emphasize that this approach is critical for developing fair and reliable human-centered AI systems, particularly in applications like mental health monitoring and personalized recommendations where bias can lead to harmful outcomes. The introduction of the DMSP dataset is highlighted as a necessary step to fill the gap in existing benchmarks regarding demographic annotations and MBTI-based fairness analysis. The work concludes that causal deconfounding is essential for generalizing personality understanding models across diverse and unseen subject populations.