🤖 AI

AlignFace: Human-Aligned Face Similarity Metric with Interpretable Concept Relations

The paper introduces AlignFace, an interpretable face similarity metric that leverages cognitive psychology principles and a novel FACETS dataset to overcome the limitations of current black-box models by explicitly modeling human perceptual variations and attribute-based reasoning for more accurate and transparent evaluation of facial content.

Original authors: Ying Huang, Wencan Zhang, Brian Y. Lim

Published 2026-08-17

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Ying Huang, Wencan Zhang, Brian Y. Lim

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to understand what makes two faces look "alike." You might think this is easy: just measure the distance between the eyes or the shade of the lips. But human perception is a messy, magical thing. We don't just add up parts like a math equation; we feel the whole picture. Sometimes, a tiny change in the distance between eyes feels huge, while a big change in hair color feels like nothing. This is the world of computer vision, where scientists build machines to "see" like us. The big problem is that the robots we've built so far are terrible at guessing what humans actually think. They use cold, hard math that misses the nuance of human feeling. If a robot thinks two faces are totally different, but you think they are twins, the robot is failing at its most important job: understanding people. This matters because we are using AI to edit photos, protect privacy, and create digital art. If the robot's "similarity meter" is broken, it might accidentally reveal someone's identity or ruin a digital makeup job because it doesn't "get" us.

Enter AlignFace, a new kind of similarity meter designed by researchers at the National University of Singapore. Instead of treating human perception as a mysterious black box, the authors decided to peek inside the human brain's rulebook. They built a system that doesn't just guess; it reasons like a human psychologist. They found that humans judge faces based on two main things: features (like the color of your lips or the shape of your nose) and configurations (the spacing between your eyes or the width of your jaw). They also discovered that our brains don't react in a straight line; a tiny change in one spot might feel huge, while a big change elsewhere feels small. Plus, we have a "tribal" bias: we are better at spotting differences in people who look like us (our own group) than in people who look different.

AlignFace is the first tool to bake all these messy, human quirks directly into its code. The researchers collected a massive dataset called FACETS, where thousands of people played a game: they were shown a reference face and two edited versions, and had to pick which one looked more similar. They did this for the whole face and for 20 specific details, like "lip color" or "pupillary distance." Using this data, they built a model that breaks down a face into these specific concepts, measures the differences, and then uses a special "nonlinear" calculator to figure out how much those differences actually matter to a human. The result? AlignFace is significantly better at predicting what humans will say than the current top-tier AI tools. It doesn't just say "these faces are 80% similar"; it explains why, pointing out that the lip color was a match but the jaw width was off. By listening to the science of how we actually see, AlignFace finally gives AI a way to see the world through human eyes.

Technical Summary: AlignFace: Human-Aligned Face Similarity Metric with Interpretable Concept Relations

Problem Statement

Current computer vision models for generated facial content (e.g., face editing, privacy protection) rely on similarity metrics that often fail to serve as faithful proxies for human perception. While perceptual evaluation has evolved from signal-based heuristics (e.g., PSNR, SSIM) to representation-based metrics (e.g., LPIPS, DreamSim), existing approaches remain limited to behavioral modeling without cognitive alignment. These "black box" models rely on implicit and spurious relations, assume a universal observer, and fail to account for inherent variations across diverse human populations. Consequently, they provide inaccurate evaluative models for stakeholders and misleading guidance for debugging generative models, such as underestimating privacy risks or identity shifts when a human would still perceive faces as similar.

The authors argue that perception models must adhere to scientific cognitive effects to achieve true human alignment. Specifically, human face perception is characterized by:

Dual Processing: Dependence on both featural attributes (discrete components like eye color) and configural attributes (spatial layout like pupillary distance).
Nonlinear Scaling: Human perception of attribute differences does not scale linearly; it follows nonlinear psychophysical laws.
Demographic Variance: The presence of own-group bias, where individuals recognize faces from their own demographic groups (ethnicity, gender) more accurately than other groups.

Methodology

1. The FACETS Dataset

To ground the model in human cognition, the authors introduced FACETS, a large-scale human-annotated dataset.

Stimuli: Constructed from CMU Multi-PIE and CelebA datasets, featuring 120 identities (balanced for White and Asian demographics). Images were edited using diffusion-based inpainting, landmark-based warping, and GAN-based transfer to create variations in 20 cognitively-grounded attributes (14 featural, 6 configural).
Data Collection: Utilizing a Two-Alternative Forced Choice (2AFC) protocol, participants compared edited faces against a reference.
- Overall Similarity: 9.36k triplet ratings from 78 participants.
- Attribute Similarity: 73.32k triplet ratings for 20 attributes from 611 participants.
Analysis: Preliminary Generalized Additive Model (GAM) analysis on this data confirmed nonlinear attribute influence and significant differences in sensitivity between own-group and other-group perceptions.

2. AlignFace Architecture

AlignFace is an ante-hoc interpretable model designed to encode these cognitive principles. It consists of four primary modules:

Vision-Language Model (VLM) Encoding:
- Uses a shared vision encoder to extract representations for a face pair ( $\hat{\mathbf{z}}_A, \hat{\mathbf{z}}_B$ ) and computes relational differences ( $\hat{\mathbf{Z}}_\Delta$ ).
- A text encoder processes attribute prompts (e.g., "Pupillary distance") to create an anchor ( $\hat{\mathbf{Z}}_{attr}$ ) guiding the comparison along specific dimensions.
Attribute-Gated Cross-Attention (AGCA):
- Performs multi-modal fusion where image-difference features act as queries ( $Q$ ) and textual attribute features act as keys ( $K$ ) and values ( $V$ ).
- A learnable gating factor ( $\tanh(\alpha)$ ) controls the injection of attribute-conditioned signals, effectively retrieving attribute-relevant visual features from the difference representation.
Concept Bottleneck Model (CBM):
- The output of the AGCA is constrained into interpretable multi-label binary concepts ( $\Delta\hat{\mathbf{c}}_{attr} \in [-1, +1]$ ).
- This forces the model's reasoning to be explicitly based on these disentangled face attributes rather than latent, uninterpretable features.
Attribute-Influence Neural Generalized Additive Model (GAM):
- Maps the attribute differences to the overall similarity score using a neural GAM: $F(\Delta\hat{\mathbf{c}}) = \sigma(\beta + \sum f_i(\Delta\hat{c}_i))$ .
- Each $f_i$ is a spline function implemented via a small neural network, capturing nonlinear partial influences.
- A monotonicity constraint is imposed ( $\frac{\partial f_i}{\partial \Delta\hat{c}_i} \geq 0$ ) to ensure that higher attribute differences never decrease the overall face distance, aligning with perceptual consistency.

3. Training Objective

The model is trained as AlignFace2AFC, a multi-task triplet metric. It predicts pairwise distances for two comparisons ( $\langle A, Ref \rangle$ and $\langle B, Ref \rangle$ ) and minimizes the squared hinge loss between the predicted relative distance differences and the human binary labels (which indicate which face is more similar). The training is staged: first, attribute-level predictions are trained and frozen; then, the Neural GAM is trained on the overall distance loss.

Key Contributions

Cognitive Characterization: Identification of specific cognitive characteristics for human face similarity perception, including featural/configural dependence, nonlinear psychophysical scaling, and demographic own-group bias.
AlignFace Model: A cognitively-grounded, interpretable ante-hoc model integrating VLMs, Gated Cross-Attention, Concept Bottlenecks, and Neural GAMs to enforce human alignment.
AlignFace2AFC Metric: An explainable metric that quantifies perception via the nonlinear contributions of grounded attributes.
FACETS Dataset: A new dataset of face triplets with similarity judgments at both overall and attribute levels across 20 attributes, including demographic metadata.

Results

Behavioral Agreement: AlignFace significantly outperforms baselines (including heuristic metrics like SSIM, learned metrics like LPIPS/DreamSim, and face-specific models like ArcFace) in aligning with human 2AFC judgments. It achieves the highest agreement scores for overall face perception.
Attribute Alignment: The model demonstrates high agreement on attribute-level predictions (Mean agreement of 0.79 with FLIP backbone), particularly for attributes like hair length and eyebrow shape.
Interpretability & Nonlinearity: Partial Dependence Plots (PDPs) reveal that AlignFace captures highly nonlinear relationships between attribute differences and overall similarity, achieving high Pearson correlations ( $\rho \approx 0.94–0.98$ ) with human GAMs. This confirms the hypothesis that attribute influence is nonlinear.
Demographic Sensitivity: The model successfully captures "own-group" effects, showing different sensitivity trends for White and Asian participants when evaluating faces from their own demographic versus other demographics.

Significance

The paper claims that by bridging learned representations with human cognitive processes, AlignFace enables more transparent and aligned perceptual evaluation metrics for face images. Unlike previous data-driven approaches that risk spurious correlations, AlignFace is theoretically justified through cognitive psychology. It provides a framework for evaluating generative models that is not only more accurate in predicting human perception but also explainable, allowing developers to understand why a model perceives two faces as similar or different based on specific featural and configural attributes. This is critical for applications involving human-centric tasks like digital makeup, restoration, and privacy protection, where the "true effect on people" must be accurately estimated.