Are Face Embeddings Compatible Across Deep Neural Network Models?

1. Problem Statement

The rapid advancement of Deep Neural Networks (DNNs) has led to two distinct paradigms in face recognition:

1. Domain-Specific Models: Trained exclusively on face data with metric learning objectives (e.g., ArcFace, AdaFace) to optimize identity discrimination.
2. Foundation Models: Pre-trained on broad vision or vision-language tasks (e.g., CLIP, BLIP-2, DINOv2) and evaluated for face recognition via zero-shot or fine-tuning.

The Core Question: Do these models, trained on different datasets, with different architectures and loss functions, encode facial identity in fundamentally incompatible ways?

• Prior Assumption: Previous work suggested that independently trained models produce mutually incompatible embeddings, a property sometimes viewed as a security feature (implicit template revocability).
• Hypothesis: The authors investigate whether this incompatibility is due to fundamentally different representations or merely a misalignment of coordinate systems within a shared underlying geometric structure. They posit that simple affine transformations might be sufficient to align these spaces.

2. Methodology

Data and Models

• Models Evaluated:
  • Face-Specific: ArcFace, AdaFace, MagFace, KPRPE.
  • Foundation Models: CLIP, ALIGN, DINOv2, SAM, BLIP-2, LLaVA, Kosmos-2, InternVL3, Florence-2, and standard ViT.
• Datasets:
  • CFP: Frontal-Profile pairs (controlled pose).
  • LFW: Unconstrained "in-the-wild" images.
  • WebFace (CASIA-WebFace): Large-scale diverse dataset (subsampled for experiments).

Alignment Framework

The authors treat face embeddings as point clouds and attempt to learn a mapping $\phi: \mathbb{R}^{d_a} \to \mathbb{R}^{d_b}$ between the embedding space of Model A (source) and Model B (target).

• Preprocessing: Embeddings are $\ell_2$-normalized, centered using training set statistics, and zero-padded to a common dimension $D = \max(d_a, d_b)$.
• Alignment Methods: Three low-capacity linear transformations are tested:
  1. Procrustes Alignment: Finds an orthogonal rotation matrix $W$ (preserves angles/distances) via SVD.
  2. Unconstrained Linear Regression: Finds a general linear map $W$ (allows scaling and shearing) via least squares.
  3. Ridge Regression: Linear regression with $\ell_2$ regularization to prevent overfitting on limited data.

Evaluation Tasks

• Face Identification: Rank-$k$ accuracy and Mean Average Precision (mAP) in a 1:N retrieval setting.
• Face Verification: AUC, Equal Error Rate (EER), and True Match Rate at 1% False Match Rate (TMR@1%FMR) in a 1:1 matching setting.
• Settings: Both intra-dataset (train and test on same dataset) and cross-dataset (train on one, test on another) generalization.

3. Key Results

A. Dramatic Performance Improvement

Without alignment, cross-model retrieval is near-random (Rank-1 accuracy < 3% for most pairs). After applying linear alignment:

• Face-Specific Models: Achieved near-perfect alignment.
  • Rank-1 Accuracy: Improved from 0.5% to **97–99%** across CFP, LFW, and WebFace.
  • Verification: AUC increased from 0.50 to **0.96**; EER dropped from 50% to **9%**.
• Foundation Models: Showed significant but slightly lower compatibility.
  • Rank-1 Accuracy: Improved from <3% to ~72% (average) and up to 98% for specific pairs (e.g., BLIP-2 $\to$ LLaVA).
  • Verification: AUC improved to ~0.89, though performance remained lower than face-specific models, indicating a gap in discriminative tightness for high-security thresholds.

B. Generalization and Robustness

• Cross-Dataset Transfer: Alignments learned on one dataset (e.g., CFP) generalized effectively to others (e.g., LFW), though with a performance drop due to domain shift. Face-specific models showed particularly strong domain invariance.
• Data Efficiency: Procrustes and Ridge regression remained robust even with as little as 10% of the training data, while Unconstrained Linear required more data to avoid overfitting but ultimately performed best with sufficient data.

C. Structural Insights

• Model Hierarchy: Hierarchical clustering of alignment matrices revealed stable model families. Contrastive/multimodal models (CLIP, BLIP-2, LLaVA) clustered tightly, while segmentation-focused models (SAM) separated early. This suggests representational convergence based on training objectives rather than just architecture.
• Directional Asymmetry: Alignment is not always symmetric. Mapping $A \to B$ often yields different results than $B \to A$. This is attributed to the "geometric quality" of the target space; models trained explicitly for identity separation (Face-specific) make better target spaces than those trained for segmentation or general vision.

4. Theoretical Interpretation

The authors propose a Manifold Learning explanation:

• Different DNNs learn embeddings that lie on approximately the same low-dimensional identity manifold in semantic space.
• While training procedures result in different parameterizations (coordinate systems) of this manifold, the intrinsic geometry is shared.
• Because standard benchmarks sample a bounded region of this manifold, the local geometry can be approximated by a tangent space.
• Therefore, a simple linear transformation suffices to act as a coordinate change between the tangent spaces of two different models, aligning them without needing complex non-linear mappings.

5. Significance and Implications

Positive Implications

• Interoperability: Systems using different DNNs can interoperate without re-enrolling users.
• Ensemble Design: Heterogeneous models can be combined effectively by aligning their embeddings.
• Model Substitution: Proprietary systems can be replaced with open-source alternatives (or vice versa) by learning a simple alignment matrix.

Negative Implications (Security)

• Template Revocability Challenged: A key assumption in biometric security is that embeddings from different systems are incompatible, providing "implicit revocability" (if one database is hacked, the user can switch to a different system).
• Vulnerability: This paper demonstrates that an adversary with access to embeddings from two different systems can estimate the linear alignment and match templates across systems, rendering the "revocability" fragile.

Conclusion

The paper concludes that face embeddings across diverse DNN architectures are geometrically compatible. The differences between models are primarily due to coordinate parameterization rather than fundamental structural divergence. Simple linear mappings can bridge these gaps, enabling high-accuracy cross-model recognition but simultaneously exposing vulnerabilities in current biometric template protection strategies.