🤖 machine learning

LDPKiT: Superimposing Remote Queries for Privacy-Preserving Distillation

This paper introduces LDPKiT, a privacy-preserving framework that utilizes a novel superimposition technique to generate approximate in-distribution samples from local data, enabling effective model distillation under local differential privacy while maintaining high utility even at strong privacy levels.

Original authors: Kexin Li, Aastha Mehta, David Lie

Published 2026-07-13

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Kexin Li, Aastha Mehta, David Lie

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart, secret recipe for predicting the future (a machine learning model) that lives in a high-security vault owned by a giant corporation. You want to use this recipe to make predictions about your own private data, like your medical records or bank statements. But you can't just walk into the vault and copy the recipe; the owner won't let you. So, you have to send your data to the vault, get a prediction back, and hope for the best.

The problem? Sending your raw data is risky. If the vault owner is curious (or gets hacked), they might see your secrets. To stop this, you add "noise" to your data before sending it—like blurring a photo so the owner can't see the details, but the general shape is still there. This is called Local Differential Privacy (LDP).

But here's the catch: that blur is so strong that the vault owner's predictions become almost useless. It's like trying to guess what a friend is wearing based on a photo where they've been covered in thick fog. You get the idea that it's a person, but you can't tell if they're wearing a hat or a helmet.

Enter LDPKiT, a clever new framework that acts like a magic "de-blurring" trick for your privacy.

The Magic Trick: Superimposing Shadows

The researchers found that if you just send your blurry photos, the model learns very little. But what if you could create more blurry photos from your original ones without ever seeing the clear picture again?

They came up with two ways to do this:

The Random Blur (LDPKiT-Rand): They take your blurry photo and add even more random static to it, creating thousands of slightly different blurry versions.
The Shadow Blend (LDPKiT-Sup): This is the star of the show. They take two of your blurry photos and "superimpose" them—basically, they average the pixels together to create a new, blended image.

Think of it like this: If you have two blurry photos of a cat, and you blend them together, the result is a "super-blurry" cat that still looks like a cat, but the random noise cancels out a bit, making the shape clearer than a single blurry photo. The researchers call this superimposition.

What They Discovered

The team tested this on three different "worlds" of data: pictures of street numbers (SVHN), fashion items like sandals and shirts (Fashion-MNIST), and medical images of cells (PathMNIST).

The Bad News: When they just sent the single blurry photos (a method they call SIDP), the local model they trained was terrible. On the street number dataset, it got only about 21% accuracy when the privacy was set to a moderate level (epsilon = 2.0). That's barely better than guessing randomly.
The Good News: When they used the Shadow Blend (LDPKiT-Sup) technique, the local model's accuracy skyrocketed. On that same street number dataset, it jumped to over 80% accuracy.
The "Aha!" Moment: They found that the Shadow Blend method worked so well because it kept the "shape" of the data intact. When they looked inside the computer's "brain" (the latent space), the blended images sat right next to the real categories, while the random noise versions were scattered everywhere.

The Trade-Off: Privacy vs. Accuracy

The researchers asked: "Does this work even when the privacy is super strict?"

Yes. In fact, the more privacy you demand (more noise), the more LDPKiT helps.
For example, on the street number dataset, they managed to get nearly the same accuracy at a very strict privacy level (epsilon = 1.25) as they did at a weaker level (epsilon = 2.0). This means you can have stronger privacy guarantees with less than a 2% drop in accuracy.

What They Ruled Out

The paper is very clear about what doesn't work:

Random Noise Alone: Just adding random static to your data (LDPKiT-Rand) helped a little, but it wasn't enough. It often failed to capture the true shape of the data, especially on complex medical images.
Out-of-Distribution Data: You can't just use random pictures from the internet to train your local model. If you try to teach a model about fashion using pictures of street numbers, it learns nothing. You must use your own private data (even if it's blurry).
Encryption: They compared their method to using "homomorphic encryption" (math that lets you compute on encrypted data). They found that LDPKiT is much faster and cheaper. While encryption might take hundreds of seconds to process a single image, LDPKiT can process thousands in seconds.

How Sure Are They?

The authors are quite confident in their results, but they are careful with their language.

They proved through experiments that LDPKiT-Sup consistently outperforms the baseline methods across three different datasets and two different model sizes.
They measured the privacy risk by trying to "reconstruct" the original images from the noisy ones. They found that even with powerful AI tools, the reconstructed images were very different from the originals (low similarity scores), suggesting the privacy holds up.
They suggest that the "Shadow Blend" works because it creates samples that sit closer to the decision boundaries in the data, but they admit that a full statistical explanation of why it works so well is still a topic for future research.

The Bottom Line

LDPKiT is like a privacy-preserving magic trick. It allows you to take your secret, blurry data, mix it with itself in a special way, and create a massive library of "safe" training examples. This lets you build a local expert that knows almost as much as the giant vault's secret model, without ever revealing your private secrets.

The only catch? You need a decent amount of your own private data to start with (they tested with as few as 125 samples, but it works better with more), and you have to be willing to send a lot of queries to the remote server to build your library. But for many, the trade-off of sending a few extra blurry photos to get a much smarter local model is a win.

Technical Summary: LDPKiT

Problem Statement

In regulated domains such as healthcare and finance, organizations often host machine learning (ML) models via remote APIs to provide services (e.g., disease risk prediction, fraud detection) without sharing proprietary training data or model weights. While black-box access protects the model provider's intellectual property and training data, it forces the model user to transmit potentially sensitive inference inputs to the remote server. Existing privacy-preserving inference methods, such as homomorphic encryption or trusted execution environments (TEEs), suffer from severe computational overheads or require specialized hardware not widely available.

Conversely, allowing users to download model weights for local inference exposes the model to white-box attacks. The core challenge addressed by this work is enabling users to distill a local model from a remote black-box model using their own private, sensitive data, while strictly bounding the privacy leakage of the query inputs. A secondary challenge is that users in these domains often possess only small amounts of private data, which, when combined with the noise required for privacy, results in poor utility (accuracy) for the distilled model.

Methodology

The authors propose LDPKiT (Local Differentially-Private and Utility-Preserving Inference via Knowledge Transfer), a framework that combines Local Differential Privacy (LDP) with a novel data augmentation technique to recover utility lost to privacy noise.

1. System Setup

The scenario involves a model provider hosting a remote model ( $M_R$ ) and a user with a small, unlabelled, sensitive dataset ( $D_{priv}$ ). The user's goal is to train a local model ( $M_L$ ) that can accurately label $D_{priv}$ without revealing the raw data to $M_R$ .

2. Privacy Mechanism

To protect $D_{priv}$ , the user applies $\epsilon$ -LDP noise to each sample before querying $M_R$ . The paper utilizes the Laplace Mechanism to satisfy $\epsilon$ -LDP. Crucially, the framework relies on the Post-Processing Property of LDP, which states that any deterministic or randomized function applied to an $\epsilon$ -LDP output remains $\epsilon$ -LDP. This allows the user to augment the noisy data without weakening the privacy guarantee.

3. Data Augmentation Strategies

Since $D_{priv}$ is typically small, direct querying yields low utility. LDPKiT augments the initial noisy dataset ( $D_{protected}$ ) into a larger inference dataset ( $D_{infer}$ ) using two strategies:

LDPKiT-Rand: Adds a second layer of independent Laplace noise to the protected samples. This generates perturbed variants but may push data further from the true distribution.
LDPKiT-Sup (Superimposition): This is the novel contribution. It generates new samples by averaging pairs of protected samples from $D_{protected}$ $D_{p r o t ec t e d}$ . Specifically, for distinct indices $i$ $i$ and $j$ $j$ , an augmented sample is computed as $d_{infer} = \text{avg}(d_{protected_i}, d_{protected_j})$ $d_{in f er} = avg (d_{p r o t ec t e d_{i}}, d_{p r o t ec t e d_{j}})$ .
- Rationale: The authors hypothesize that superimposing noisy samples helps preserve the target-class structure and decision boundaries better than random noise injection.
- Privacy: Since averaging is a post-processing operation on $\epsilon$ -LDP outputs, the resulting $D_{infer}$ maintains the record-level $\epsilon$ -LDP guarantee. Even if a provider attempts to algebraically recover protected records from pairwise averages, they only recover the noisy protected records, not the original private data, thus preserving the $\epsilon$ -LDP guarantee.

4. Workflow

Stage I: User applies $\epsilon$ -LDP noise to $D_{priv}$ to create $D_{protected}$ .
Stage II: User post-processes $D_{protected}$ (via Rand or Sup) to generate $D_{infer}$ .
Stage III: User queries $M_R$ with $D_{infer}$ to obtain labels.
Stage IV: User trains $M_L$ on the labeled $D_{infer}$ and uses $M_L$ for local inference on $D_{priv}$ .

Key Contributions

Framework for Privacy-Preserving Distillation: LDPKiT enables users to distill knowledge from remote black-box models using their own private data while bounding privacy leakage via LDP.
Superimposition Technique: The introduction of LDPKiT-Sup, which uses pairwise averaging of noisy samples to generate augmented data. This technique leverages the post-processing property of LDP to improve utility without compromising privacy.
Empirical Validation: The authors demonstrate that LDPKiT, particularly the superimposition variant, significantly recovers the utility lost due to LDP noise compared to baseline methods (SIDP) and random noise augmentation.
Latent Space Analysis: The paper provides an analysis of the latent space representations, showing that superimposed samples remain closer to the target class clusters than randomly perturbed samples, explaining the accuracy gains.

Experimental Results

The framework was evaluated on three image classification datasets: Fashion-MNIST, SVHN, and PathMNIST (medical imaging), using ResNet-152 as the remote teacher and ResNet-18 or MobileNetV2 as the local student.

Utility Recovery: LDPKiT-Sup consistently outperforms both the baseline (SIDP) and LDPKiT-Rand.
- On SVHN with $\epsilon = 1.25$ , LDPKiT-Sup achieves nearly the same inference accuracy as at $\epsilon = 2.0$ , yielding stronger privacy guarantees with less than a 2% accuracy reduction compared to the higher $\epsilon$ setting.
- On Fashion-MNIST ( $\epsilon = 2.0$ ), LDPKiT-Sup improved ResNet-18 accuracy to 81.78%, compared to 28.69% for SIDP and 68.52% for LDPKiT-Rand.
- On PathMNIST ( $\epsilon = 10.0$ ), LDPKiT-Sup reached 82.11%, significantly outperforming SIDP (28.74%) and LDPKiT-Rand (43.90%).
Impact of Dataset Size: The system is relatively insensitive to the size of the private dataset ( $|D_{priv}|$ ). Users can achieve similar utility with fewer private samples (e.g., 125 samples) provided the augmented query set ( $|D_{infer}|$ ) is sufficiently large (e.g., 250k–500k queries).
Privacy Robustness: Data reconstruction attacks using DDPMs on the augmented data failed to recover original images with high fidelity. Reconstructed images showed lower SSIM and higher MSE compared to originals, and classifiers achieved near-random accuracy on them.
Cost Efficiency: The authors estimate that for large query sets (e.g., 2.2M queries), the API cost is approximately $1.15 (based on hypothetical 2026 pricing), making it more efficient than homomorphic encryption for batch processing.

Significance and Claims

The paper claims that LDPKiT offers a practical, privacy-preserving design for interacting with remote ML services in sensitive domains. Its significance lies in:

Balancing Privacy and Utility: It demonstrates that effective knowledge transfer is possible even under strong privacy constraints (low $\epsilon$ ) by leveraging data augmentation that respects LDP guarantees.
Accessibility: Unlike TEEs or homomorphic encryption, LDPKiT requires no specialized hardware and operates within standard API constraints, making it more accessible for adoption.
Privacy-by-Design: The framework aligns with responsible data-use principles by ensuring that raw user data never leaves the user's control and that privacy leakage is mathematically bounded.

The authors modestly note limitations, including the current restriction to image modalities (where superimposition is pixel-based) and the lack of a universal lower bound for private dataset size, suggesting these as directions for future work. They also acknowledge that while record-level LDP is guaranteed, correlations between records could theoretically weaken semantic privacy, though de-duplication practices in industry can mitigate this.