Technical Summary: Head Similarity

Problem Definition

Conventional face recognition systems are designed to learn identity-invariant embeddings, deliberately suppressing intra-identity variations such as hairstyle, makeup, or aging to collapse all samples of a person into a single compact cluster. While effective for biometric verification, this approach fails in scenarios requiring appearance-sensitive consistency, such as video generation, reenactment, and editing. In these contexts, the goal is not merely to verify legal identity but to ensure a sequence depicts the same person visually, even when appearance attributes change.

Current models struggle with whole-head similarity because:

1. They collapse different appearance states of the same identity into a single representation, failing to distinguish between a person with and without glasses or with different hairstyles.
2. They suffer from significant performance degradation when moving from aligned facial crops to unaligned whole-head images (distribution shift).
3. They cannot enforce a hierarchical similarity ordering where samples of the same identity with different appearances (e.g., different hairstyles) are ranked higher than samples of different identities, even if the latter share similar visual attributes.

The paper introduces Head Similarity, a new formulation that extends identity-centric recognition to structured whole-head appearance modeling. It aims to preserve structured intra-identity appearance variations while maintaining identity discrimination and invariance to nuisance factors (illumination, background, minor pose).

Methodology

The proposed framework addresses the problem through a Dual-CLS Vision Transformer architecture combined with hierarchical similarity supervision and identity-aware distillation.

1. Problem Formulation

The task is defined by three pairwise relations in the embedding space:

• $R_1$: Same identity, same appearance state (tightest cluster).
• $R_2$: Same identity, different appearance states (intermediate distance).
• $R_3$: Different identities (furthest apart).

The objective is to learn an embedding function $f_\theta$ that enforces the hierarchical ordering:
$$s_\theta(R_1) > s_\theta(R_2) > s_\theta(R_3)$$
where $s_\theta$ is the cosine similarity. This ensures that identity consistency dominates appearance similarity, but appearance states are still distinguishable within the same identity.

2. Architecture: Dual-CLS Vision Transformer

The backbone is a Vision Transformer (ViT) pre-trained with AdaFace. Unlike standard face recognition systems that use a single classification token, this model employs two learnable CLS tokens:

• $CLS_{id}$: Optimized for identity preservation.
• $CLS_{head}$: Optimized for appearance-sensitive head similarity.

Both tokens share the same backbone parameters but are projected into separate embeddings ($z_{id}$ and $z_{head}$) via distinct projection heads ($g_{id}$ and $g_{head}$).

• $z_{id}$: Used at inference for both identity verification and similarity ranking. It is trained with both distillation and hierarchical loss.
• $z_{head}$: An auxiliary embedding used only during training to provide appearance-sensitive signals; it is discarded at inference.

3. Training Strategy

The training pipeline involves three key components:

1. Distillation-Based Identity Alignment: To bridge the distribution gap between aligned face crops (used in pre-training) and whole-head inputs, the $z_{id}$ branch is aligned with a frozen teacher network (AdaFace-ViT) trained on aligned faces. The loss is defined as $L_{align} = 1 - \cos(z_t, z_{id})$.
2. Hierarchical Similarity Loss: A Softplus-based multi-level margin loss enforces the $R_1 > R_2 > R_3$ ordering. It is applied to both $z_{id}$ and $z_{head}$ to jointly structure identity and appearance representations.
3. Dataset Construction: A benchmark dataset (HeadSim-Head) is constructed from long-form videos. Weak supervision for appearance states is derived from temporal grouping (frames within the same short video segment are assumed to share the same appearance state). The dataset covers diverse poses, occlusions, and temporal changes.

Key Contributions

1. Task Definition: Introduction of Head Similarity, a formulation that moves beyond identity-invariant face recognition to model structured whole-head appearance, explicitly preserving intra-identity variations.
2. Benchmark: Construction of a large-scale dataset from long-form videos with weakly supervised appearance states, covering diverse poses, occlusions, and temporal changes.
3. Methodology: Proposal of a simple yet effective Dual-CLS Vision Transformer baseline that combines identity-aware distillation with hierarchical similarity supervision to jointly model identity preservation and appearance-sensitive similarity.
4. Empirical Findings: Demonstration that conventional face recognition models fail to capture appearance-dependent similarity, while the proposed method achieves strong whole-head matching without sacrificing identity verification capabilities.

Experimental Results

Face Verification Performance

When adapting face recognition models to whole-head inputs, standard models suffer a severe performance drop.

• AdaFace-ViT: Verification Rate (VR) at FAR=$10^{-3}$ drops from 0.914 (aligned face) to 0.078 (whole-head).
• Distilled AdaFace: The proposed distillation approach recovers performance, achieving 0.912 VR at FAR=$10^{-3}$ on whole-head inputs, closely matching the aligned-face baseline.

Head Similarity Performance

On the HeadSim-Head dataset, the proposed method outperforms baselines in distinguishing structured appearance states:

• Ours: Achieves 0.862 VR at FAR=$10^{-3}$ and 0.996 AUC.
• TransReID-ViT: While achieving strong similarity performance (0.749 VR), it fails at identity verification (0.025 VR on VoxCeleb2), indicating it relies too heavily on global appearance cues and lacks identity discrimination.
• Vanilla AdaFace: Performs poorly (0.326 VR) as it collapses appearance variations.

Ablation Studies

The study confirms that token-level decoupling (Dual-CLS) is superior to head-level separation or shared embeddings.

• Dual-CLS (Ours): 0.962 VR at FAR=$10^{-3}$.
• Dual-Head (Split): 0.762 VR. The separation of heads without token-level decoupling leads to insufficient identity supervision for the similarity branch.
• Shared Embedding: 0.892 VR. A single representation struggles to balance the conflicting objectives of invariance and variation preservation.

Significance and Claims

The paper claims that Head Similarity fills a critical gap between face similarity and holistic head-level similarity, which is essential for applications like video generation and editing where visual consistency across frames is required despite changes in viewpoint or styling.

The authors emphasize that their approach:

• Explicitly captures intra-identity appearance variation rather than suppressing it.
• Enforces hierarchical ordering, ensuring that identity consistency remains the primary factor while still allowing for the discrimination of different appearance states.
• Demonstrates feasibility: The results show that structured whole-head similarity modeling is achievable using a relatively simple framework that leverages weak supervision from long-form videos and distillation from established face recognition models.

The paper acknowledges limitations, noting that appearance states are weakly supervised (potentially noisy) and that the current hierarchy is a proxy for human perception rather than a perfect model of subjective judgment. The proposed framework is presented as a baseline for this new task.