A Late-Fusion Multimodal AI Framework for Privacy-Preserving Deduplication in National Healthcare Data Environments

This paper proposes a privacy-preserving late-fusion multimodal AI framework that combines semantic text embeddings, behavioral patterns, and device metadata to effectively detect duplicate records in national healthcare data without relying on sensitive personally identifiable information, thereby ensuring compliance with regulations like GDPR and HIPAA.

The Gist

Imagine you are the manager of a massive, bustling library. This library holds the records of millions of people. But there's a problem: the library is messy.

Some people have checked in multiple times under slightly different names. One day, "John Smith" checks in. The next day, "Jon Smythe" checks in. Sometimes, the same person uses a different computer or logs in at a weird time. In a normal library, you'd just look at their ID card (Social Security Number or Email) to see if it's the same person.

But here's the catch: In this specific library (representing healthcare and government data), you are not allowed to look at ID cards. Privacy laws (like HIPAA and GDPR) say, "We can't see names, emails, or IDs. We only see what they do and where they are."

This is the problem Mohammed Omer Shakeel Ahmed is solving in his paper. He built a smart AI system that acts like a super-sleuth detective who can figure out if two records belong to the same person without ever seeing their ID card.

Here is how his "Detective AI" works, broken down into simple parts:

1. The Three Clues (The Modalities)

Since the detective can't ask for an ID, they have to look at three different types of clues, or "modalities," to build a profile of the person.

• Clue #1: The "Voice" (Semantic Meaning)

  • The Analogy: Imagine two people speaking. One says, "I live in the Big Apple," and the other says, "I reside in New York City." A human knows these are the same place, even though the words are different.
  • The Tech: The AI reads the names and cities. It doesn't just look for exact spelling matches (like "Jon" vs "John"). Instead, it uses a "brain" (DistilBERT) that understands the meaning behind the words. It knows that "J. Doe" and "Jonathan Doe" sound very similar in spirit, even if the letters don't match perfectly.

• Clue #2: The "Rhythm" (Behavioral Patterns)

  • The Analogy: Think about your daily routine. Maybe you always check your email at 7:00 AM on a Tuesday, or you only log in late at night. Even if you change your name, your rhythm stays the same.
  • The Tech: The AI looks at when people log in. If "User A" and "User B" both log in at 2:00 AM every night from the same time zone, the AI thinks, "Hey, these two people probably have the same sleep schedule. They might be the same person."

• Clue #3: The "Backpack" (Device Metadata)

  • The Analogy: Imagine two people walking into a room. One is wearing a red hat and carrying a blue backpack. The other is wearing a red hat and carrying a blue backpack. Even if they don't introduce themselves, you might guess they are related or the same person.
  • The Tech: The AI checks what kind of computer or phone they use (e.g., "Chrome on iPhone"). If two different names are always using the exact same digital "backpack," it's a strong hint they are the same person.

2. The "Late Fusion" Strategy (Putting the Clues Together)

This is the secret sauce of the paper.

Imagine a jury deciding a case.

• Early Fusion would be like mixing all the evidence (voice, rhythm, backpack) into a giant smoothie before the jury even sees it. It's messy and hard to taste the individual flavors.
• Late Fusion (what this paper uses) is like having three separate experts.
  1. The Voice Expert says: "I think these two are the same based on names."
  2. The Rhythm Expert says: "I agree, their schedules match perfectly."
  3. The Backpack Expert says: "I'm not sure, their devices are different."

The AI then takes these three separate opinions and weighs them together. Even if one clue is weak (like the device), the strong voice and rhythm clues can still convince the AI that it's a match. This makes the system very robust.

3. The "Crowd" (DBSCAN Clustering)

Once the AI has gathered all these clues, it needs to group the people. It uses a method called DBSCAN.

• The Analogy: Imagine a crowded party. You want to find groups of friends. You don't need to know everyone's name. You just look for people standing close together. If three people are standing in a tight circle, they are a group. If someone is standing alone far away, they are not part of that group.
• The Tech: The AI plots all the users in a giant map based on their clues. If users are "close" to each other in this map (meaning their names, habits, and devices are similar), the AI puts them in the same "cluster." These clusters are the duplicates.

4. The Results: Did it Work?

The author tested this system on a fake dataset of 1,000 people (since real data is too private to share).

• The Old Way (String Matching): This is like a robot that only checks if the names are spelled exactly the same. It missed almost everyone who had a typo or a nickname. It was very careful but missed a lot of duplicates (Low Recall).
• The New AI Way: This system was much better at finding the duplicates. It caught almost all of them (High Recall).
  • The Trade-off: It was a little too eager. It sometimes thought two different people were the same (Lower Precision). But overall, it was much more successful at the main goal: finding duplicates without breaking privacy rules.

Why Does This Matter?

In the real world, hospitals and governments have millions of records. If a patient has two different records, the hospital might give them the wrong medicine or bill them twice.

Usually, to fix this, they need to see the patient's ID. But with strict privacy laws, they can't. This new AI framework is like a privacy-preserving magic trick. It cleans up the data and finds the duplicates using only "ghost" clues (behavior, device, and meaning) without ever needing to see the actual ID card.

In short: It's a smart system that says, "I don't need to see your ID to know you're you. I just need to know how you talk, when you wake up, and what phone you use."

Technical Summary

1. Problem Statement

The paper addresses the critical challenge of record deduplication (entity resolution) in data-intensive sectors like healthcare, CRM, and finance.

• The Core Issue: Duplicate records distort analytics, inflate customer counts, and create compliance risks.
• The Constraint: Traditional deduplication relies on direct Personally Identifiable Information (PII) such as names, emails, or SSNs. However, strict privacy regulations (GDPR, HIPAA) often mask or restrict access to this data, rendering exact-match and rule-based methods ineffective.
• The Gap: There is a need for a scalable, unsupervised solution that can identify duplicates using only indirect, non-sensitive signals (e.g., behavioral patterns, device metadata, and semantic text variations) without compromising privacy.

2. Methodology

The author proposes a Late-Fusion Multimodal AI Framework that operates without PII. The system processes three distinct data modalities independently before combining their outputs at the decision level.

A. Data Modalities

1. Semantic Modality (Text):
   • Input: Name and City fields.
   • Technique: Uses pre-trained DistilBERT models (via Hugging Face) to generate high-dimensional semantic embeddings. This captures contextual meaning (e.g., recognizing "Jon Doe" and "Jonathan D." as similar).
   • Processing: Embeddings are compressed using Principal Component Analysis (PCA) to reduce noise and dimensionality.
2. Behavioral Modality (Temporal):
   • Input: Login timestamps.
   • Technique: Extracts statistical features such as time-of-day, day-of-week, and login frequency intervals.
   • Concept: Creates a "digital fingerprint" based on usage cadence, assuming the same user exhibits consistent temporal patterns even with different account details.
3. Device Modality (Categorical):
   • Input: Browser and Operating System (OS) metadata.
   • Technique: Categorical data is encoded using One-Hot Encoding or Label Encoding.
   • Concept: Consistent usage of specific browser/OS pairs across records serves as a disambiguating signal.

B. Fusion and Clustering Strategy

• Late Fusion: Unlike early fusion (concatenating raw inputs), this framework processes each modality separately to generate similarity scores or cluster assignments. These independent signals are then integrated via weighted logic rules (e.g., $0.4 \times \text{Semantic} + 0.35 \times \text{Behavioral} + 0.25 \times \text{Device}$).
• Clustering Algorithm: The final unified feature vectors are clustered using DBSCAN (Density-Based Spatial Clustering of Applications with Noise), an unsupervised algorithm capable of identifying clusters of arbitrary shape and handling noise without requiring pre-defined cluster counts.

3. Experimental Setup

• Dataset: A synthetic Simulated_CRM_Dataset containing 1,000 anonymized records. It excludes PII but includes name, city, browser, OS, and login times.
• Baseline: A traditional rule-based approach using Levenshtein string similarity (SequenceMatcher) on concatenated name and city fields with a threshold of $\geq 0.85$.
• Metrics: Precision, Recall, and F1-Score.

4. Key Results

The proposed multimodal model was compared against the string-matching baseline:

Metric	Baseline (String Match)	Proposed Multimodal Model
Precision	1.00	0.4999
Recall	0.29	0.995
F1-Score	0.45	0.665

• Interpretation:
  • The Baseline had perfect precision but extremely low recall (0.29), meaning it only found obvious, exact duplicates and missed the vast majority of fuzzy matches.
  • The Proposed Model achieved near-perfect recall (0.995), successfully identifying almost all true duplicates despite noise and variations.
  • While precision dropped (indicating more false positives), the F1-score improved significantly (0.665 vs. 0.45), demonstrating a much better balance for a privacy-constrained environment where missing a duplicate is often costlier than a false alarm.

5. Key Contributions

1. Privacy-Preserving Architecture: A novel framework that achieves high-accuracy deduplication without ever accessing or relying on sensitive PII (SSN, Email, etc.).
2. Multimodal Integration: Successfully integrates heterogeneous data types (semantic text, temporal behavior, and categorical device data) using a late-fusion strategy, inspired by architectures like LANISTR and HGNN.
3. Scalability: The modular design (using PCA, batched BERT, and DBSCAN) allows the system to scale to large national datasets and integrate into existing healthcare/enterprise infrastructure.
4. Reproducibility: The author provides a full Python implementation using open-source libraries (PyTorch, Scikit-learn, Transformers) and a public GitHub repository.

6. Significance and Limitations

• Significance: This research offers a robust solution for National Health Data Modernization. It enables accurate entity resolution in environments where data is heavily masked for compliance, supporting better public health analytics and ethical AI adoption.
• Limitations:
  • Modality Independence: The current "late fusion" is static; it does not learn cross-modal interactions (e.g., how behavior might change the weight of a semantic match).
  • Non-Differentiable: The pipeline is not end-to-end trainable; behavioral features are hand-engineered, and clustering uses fixed hyperparameters.
  • Precision Trade-off: The high recall comes with lower precision, suggesting a need for refinement to reduce false positives in production.
• Future Scope: The author suggests moving toward trainable fusion layers (e.g., multi-head attention), end-to-end multitask learning, and reinforcement-based clustering to dynamically adapt to data quality and missing signals.