💻 computer science

Multimodal Privacy-Preserving Entity Resolution with Fully Homomorphic Encryption

This paper introduces a novel multimodal framework utilizing Fully Homomorphic Encryption to enable secure, high-fidelity entity resolution on large-scale datasets while ensuring that personally identifiable information remains cryptographically protected throughout the matching process.

Original authors: Susim Roy, Nalini Ratha

Published 2026-04-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Susim Roy, Nalini Ratha

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to find a specific person in a massive, crowded library. Usually, to find them, you might ask for their name, look at their photo, and check their address. But here's the catch: people change their names, move houses, and look different as they age. Sometimes, the photo you have is old, and the name they wrote on a form is slightly different (like "Ann" instead of "Anne").

This is the problem of Entity Resolution: figuring out that "Jonathan Diaz" at one address is the same person as "Jonathan Diaz" at a different address, even when the details don't match perfectly.

Now, imagine this library has a strict rule: You are never allowed to look at the actual books or photos. You can only look at them through a magical, unbreakable glass box. You can do math on the books inside the box without ever opening it or seeing what's inside. This is the core idea of the paper.

Here is a simple breakdown of what the researchers did:

1. The "Magic Glass" (Fully Homomorphic Encryption)

The researchers used a special kind of math called Fully Homomorphic Encryption (FHE). Think of this as a "magic glass box."

Normally: To check if two people are the same, a computer has to "open the box," read the name and photo, compare them, and then close the box. This is risky because if a hacker breaks in, they see everyone's private data.
Their Method: They put the data (names, addresses, photos) inside the "magic glass box" (encryption) before the computer looks at them. The computer performs the comparison math while the data is still locked inside. The computer never sees the real names or faces; it only sees scrambled, encrypted numbers. The result comes out of the box as a "Yes, they match" or "No, they don't," but the private details remain secret the whole time.

2. The "Two-Track" Detective (Multimodal Data)

The researchers realized that relying on just one clue (like a photo) isn't enough because people age. Relying only on a name isn't enough because people make typos.

The Solution: They built a system that acts like a detective with two tracks of evidence:
1. Biometric (The Face): A photo of the person.
2. Biographic (The Story): The person's name and address.
They trained an AI to look at both the "face" and the "story" at the same time. Even if the address is slightly misspelled or the photo is from 10 years ago, the system combines both clues to make a better guess.

3. The "Practice Library" (Synthetic Dataset)

To test this, they couldn't just use real people's private data (that would be illegal and unethical). So, they created a giant, fake library using computers.

They generated 36,000 fake people.
They gave them fake names, fake addresses, and fake photos.
They intentionally messed things up: they changed the spelling of names, gave people multiple addresses, and made the photos look older or younger.
This created a "training ground" where their system could learn to match people despite the confusion, all without ever touching real human data.

4. The Results: Fast, Safe, and Accurate

The researchers tested their "magic glass" system against a normal system (where the computer sees the real data).

Accuracy: The "magic glass" system was just as accurate as the normal system. It didn't lose any performance just because it was working on encrypted data. In fact, by combining the face and the story, it got much better at finding the right person than looking at just a face or just a name alone.
Speed: Doing math on "locked" data is usually very slow. However, they found a way to run many calculations at the same time (like having 128 detectives working in parallel). This made the process surprisingly fast, cutting the time down from over 20 seconds to less than 5 seconds.

The Bottom Line

The paper presents a new way to verify identities (like for passports or bank accounts) that is super secure. It allows organizations to check if two records belong to the same person without ever actually seeing the person's private name, address, or photo. They proved that you can have high security (keeping data locked) without sacrificing speed or accuracy.

In short: They built a system that can solve the "Who is this person?" puzzle using a mix of photos and addresses, all while the data stays locked in a digital safe that no one can open.

1. Problem Statement

The paper addresses the critical challenge of Entity Resolution (ER) in high-compliance sectors (e.g., government, finance, DMVs) where secure identity reconciliation is hindered by:

Data Heterogeneity: Significant variations in personal identifiers, including syntactic differences (e.g., "Anne" vs. "Ann", "St." vs. "Street") and dynamic biographical data (address changes, name abbreviations).
Biometric Limitations: Reliance on single modalities (e.g., facial recognition) is often insufficient due to aging, poor image quality, or lack of historical biometric data.
Privacy Constraints: Traditional fusion of biometric and biographic data requires decrypting sensitive Personally Identifiable Information (PII), creating security risks where templates could be reconstructed or leaked.
Existing Gaps: Current multimodal approaches often rely on restricted datasets, are susceptible to uncontrolled variations (e.g., headshot aging), or fail to preserve privacy during the matching process.

2. Methodology

The authors propose a novel framework that performs entity resolution entirely within an encrypted domain using Fully Homomorphic Encryption (FHE), specifically the RNS-CKKS scheme (from the HEAAN library).

A. Dataset Creation

Synthetic Data Generation: To overcome the lack of real-world multimodal datasets, the authors created a synthetic dataset using Stable Diffusion XL for generating headshots and Claude 3.5-Sonnet for generating realistic name/address variations.
Scale: The final dataset contains 36,661 unique individuals and 287,532 records, featuring:
- Multiple address spellings and variations.
- Aging effects on headshots (simulating time gaps between KYC and verification).
- Overlapping addresses and name variations.
Split: 21,661 individuals for training and 10,000 for testing (5,000 enrolled, 5,000 for queries).

B. Model Architecture & Training

The system utilizes a multimodal learning stream combining biometric (image) and biographic (text) data:

Encoders: Pre-trained CLIP ViT-B/32 models are used as the backbone.
- Image Encoder: Generates biometric templates ( $f_{bm}$ ).
- Text Encoder: Generates biographic templates ( $f_{bg}$ ) from tokenized attribute-value pairs (e.g., [ATT]Name[VAL]Jonathan).
Projection Modules: Two trainable Multi-Layer Perceptrons (MLPs) process the templates:
- BMMLP: Processes biometric templates to produce score vectors ( $s_{bm}$ ).
- BGMLP: Processes biographic templates to produce score vectors ( $s_{bg}$ ).
Training Objective: The vectors are normalized, concatenated, and optimized using Contrastive Loss to pull embeddings of the same individual closer while pushing different individuals apart.

C. Privacy-Preserving Pipeline (FHE)

The core innovation is performing matching without ever decrypting the data:

Enrollment: Score vectors are normalized and encrypted using FHE before storage.
Verification:
- Query vectors are encrypted.
- Distance Calculation: The Euclidean distance $D(q, g) = ||q - g||^2$ is computed on encrypted data.
- Non-Linear Approximation: Since FHE supports only addition and multiplication, non-linear functions (like comparison/thresholding) are approximated using iterative polynomial algorithms (based on [15]) to emulate a comparison function ($comp(a, b)$).
Fusion Strategies:
- Feature-Level Fusion: Concatenating encrypted vectors before matching.
- Score-Level Fusion: Encrypting individual modality scores, then averaging/combining them in the encrypted domain.

3. Key Contributions

Encrypted Domain Entity Resolution: A complete pipeline where PII remains computationally inaccessible throughout the matching lifecycle, satisfying strict regulatory mandates (e.g., GDPR, HIPAA) via cryptographic assurances.
Multimodal Framework: Integration of biometric (face) and biographic (name/address) data to handle aging and data heterogeneity, significantly outperforming unimodal baselines.
Novel Synthetic Dataset: Release of a large-scale, realistic dataset simulating real-world ER scenarios (aging, address changes, name variations) to facilitate future research.
Performance vs. Privacy: Demonstration that FHE can be used for complex multimodal matching with zero utility loss compared to plaintext processing.

4. Experimental Results

The system was evaluated on the synthetic dataset using Equal Error Rate (EER), True Positive Rate (TPR), and latency metrics.

Accuracy & Utility:
- Zero Utility Loss: The ciphertext (CT) results were identical to plaintext (PT) results, proving that encryption does not degrade performance.
- Multimodal Superiority:
  - Biographic Only: EER = 12.37%
  - Biometric Only: EER = 7.78%
  - Fusion (Score-Level): EER = 4.08% (Best overall EER)
  - Fusion (Feature-Level): EER = 5.17%
- Rank-1 Identification: Feature-level fusion achieved 75.86% accuracy, outperforming score-level fusion (71.55%).
- Rank-5 Identification: Score-level fusion showed better robustness (87.07% vs. 85.34%), suggesting it distributes matches better when the top candidate is incorrect.
Latency & Scalability:
- The system demonstrated impressive parallelization efficiency.
- Using 128 threads, the system achieved a 4.42× speedup (reducing time from ~20.2s to ~4.6s) compared to a single-threaded baseline.
- Scaling showed near-linear improvements up to 32 threads before memory bandwidth became a bottleneck.

5. Significance

This paper represents a significant advancement in Privacy-Enhancing Technologies (PETs) for identity management.

Regulatory Compliance: It offers a viable solution for institutions (banks, governments) to perform high-fidelity identity matching without violating privacy laws or risking data breaches.
Scalability: The demonstrated parallelization suggests that FHE-based authentication is becoming computationally tractable for real-world, large-scale deployment.
Robustness: By combining multiple data modalities within an encrypted framework, the system effectively handles the "real world" complexities of aging faces and changing addresses, which single-modality systems often fail to resolve.

In conclusion, the authors successfully bridge the gap between high-security privacy requirements and the need for accurate, large-scale entity resolution, proving that fully homomorphic encryption can be a practical tool for next-generation authentication systems.