🤖 machine learning

Enabling Federated Inference via Unsupervised Consensus Embedding

This paper proposes Consensus Embedding-based Federated Inference (CE-FI), a framework that enables heterogeneous pretrained models to cooperate during inference without sharing raw data, model parameters, or a common encoder by utilizing unsupervised consensus and cooperative output layers trained solely on shared unlabeled data.

Original authors: Yui Hashimoto, Takayuki Nishio, Yuichi Kitagawa, Takahito Tanimura

Published 2026-05-08

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Yui Hashimoto, Takayuki Nishio, Yuichi Kitagawa, Takahito Tanimura

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are in a room with several experts, each a master of a different subject. One is a brilliant botanist, another a top-tier mechanic, and a third a seasoned chef. They all have their own private notebooks (their "models") filled with years of training, but they are forbidden from showing each other their notebooks, and they cannot even show each other the raw photos or objects they are looking at.

Now, imagine a mysterious object is placed in front of them. How can they work together to identify it without breaking the rules?

This is the problem the paper "Enabling Federated Inference via Unsupervised Consensus Embedding" (or CE-FI for short) solves. Here is how it works, broken down into simple concepts.

The Problem: The "Tower of Babel"

In the world of AI, different computers often have different "languages."

The Old Way: To work together, computers usually had to either share their raw data (like sending a photo of a patient's X-ray to a central server) or share their internal "brain" (the model parameters).
The Privacy Problem: Hospitals, banks, and companies can't do this. They can't send private photos, and they can't let competitors see their secret algorithms.
The "Common Translator" Problem: Some previous methods tried to solve this by forcing everyone to use the same "translator" (a common encoder). But what if the experts are already hired and trained with different translators? You can't just fire them and make them all use the same one.

The Solution: CE-FI (The "Universal Handshake")

The authors propose a new system called CE-FI. Think of it as a magical handshake that allows these experts to understand each other without ever seeing each other's notebooks or the raw object.

Here is the step-by-step process:

1. The "Internal Thought" (Intermediate Features)

When an expert looks at an object (like an image), their brain doesn't just spit out a final answer immediately. First, they form an "internal thought" or a mental sketch of what they see. In AI terms, this is called an intermediate feature.

The Rule: The experts cannot show the raw photo, but they can share this "mental sketch" with the group.

2. The "Consensus Embedding" (The Universal Translator)

Here is the tricky part: The botanist's mental sketch looks different from the mechanic's mental sketch, even if they are looking at the same car. They speak different "feature languages."

The Innovation: CE-FI adds a special layer called the Consensus Embedding (CE) layer.
The Analogy: Imagine the botanist and the mechanic both have to translate their unique mental sketches into a single, universal language (like "Universal Emoji").
How they learn: They learn this translation using a pile of unlabeled photos (photos without answers) that everyone shares. They play a game where they try to make their "Universal Emoji" sketches look as similar as possible for the same object. They don't need to know what the object is (no labels needed), they just need to agree on how to describe it.

3. The "Cooperative Output" (The Final Guess)

Once the "Universal Emoji" is created, every expert receives it.

The botanist takes the Emoji and says, "Based on my training, this Emoji means 'Flower'."
The mechanic takes the same Emoji and says, "Based on my training, this Emoji means 'Car'."
The Final Decision: They combine their guesses. If the botanist is very confident and the mechanic is confused, the system listens more to the botanist. This is called Ensemble.

Why is this a big deal?

The paper claims three major wins:

Privacy First: No raw data (photos) leaves the device. No secret "brain" (model parameters) is shared. Only the "Universal Emoji" (the consensus embedding) is exchanged.
No Common Language Required: It works even if the experts were trained on totally different computers with different architectures. They don't need to agree on a common translator beforehand; they build one together on the fly.
No Homework Needed: Usually, to get AI to work together, you need a huge pile of labeled data (photos with answers like "This is a cat"). CE-FI only needs unlabeled data (just photos). It figures out the cooperation rules by itself (Self-Supervised Learning).

The Results: Does it work?

The authors tested this on:

Images: Recognizing pictures of animals and objects (CIFAR-10 and CIFAR-100).
Text: Classifying movie genres.
Time-Series: Recognizing human movements (like walking vs. sitting).

The Findings:

Better Alone: Working together (CE-FI) was almost always better than working alone (Solo Inference), even when the experts had very different knowledge (e.g., one only knew about cats, another only about dogs).
Close to Perfect: It performed almost as well as systems that did share raw data or had a common translator, but without the privacy risks.
The Bottleneck: The main thing that stopped it from being perfect was how well the "Universal Emoji" (the consensus embedding) aligned. If the experts couldn't quite agree on the translation, the final answer suffered.

A Note on Privacy (The "Reconstruction" Test)

The authors were worried: "If we share these 'Universal Emoji' sketches, can a hacker reverse-engineer them to see the original photo?"

The Test: They tried to reconstruct the original images from the shared sketches using an AI attacker.
The Result: The reconstructed images were blurry, noisy, and unrecognizable. It was like trying to guess a face from a very abstract scribble. While not 100% impossible to attack, it was very difficult, suggesting the method is reasonably safe.

Summary

CE-FI is a new way for AI models to collaborate. It's like a group of experts who can't show their notes or the object they are studying, but they can all agree on a set of abstract symbols to describe the object. By learning to agree on these symbols using just a pile of random photos, they can solve problems together more accurately than any of them could alone, all while keeping their secrets safe.

Technical Summary: Consensus Embedding-based Federated Inference (CE-FI)

1. Problem Statement

The paper addresses the challenge of enabling cooperative inference across multiple independently deployed machine learning models in distributed environments. While there is a growing demand to leverage multiple models to improve inference performance, existing frameworks face significant limitations in privacy-sensitive or cross-organizational settings:

Federated Learning (FL) typically requires sharing model updates or parameters, which risks model inversion attacks and violates intellectual property constraints.
Standard Federated Inference (FI) often assumes that all clients share the same raw input data or utilize a common encoder architecture to generate compatible intermediate representations.
Edge Ensemble approaches rely on a shared encoder, which is often infeasible when models are heterogeneous (different architectures) or when even partial model disclosure is prohibited.

Consequently, there is no existing framework that allows heterogeneous, pretrained models to cooperate at inference time without sharing raw inputs, model parameters, or assuming a common encoder, and without requiring labeled data for the cooperative stage.

2. Methodology: CE-FI Framework

The authors propose Consensus Embedding-based Federated Inference (CE-FI), a framework designed to bridge heterogeneous feature spaces using only intermediate representations and unlabeled data.

System Model

Environment: A cross-silo setting with $K$ devices, each holding a local pretrained model $M_k$ (comprising a head network $M^H_k$ and a tail network $M^T_k$ ).
Constraints: Devices cannot share raw inputs $x$ , model parameters, or architectures. They can only share intermediate features.
Data Assumption: A shared unlabeled dataset $D_{share}$ is accessible to all devices (e.g., public log data or sensor data), but no labeled data is available for the cooperative training phase.

Core Components

CE-FI introduces two learnable layers per device, trained in a self-supervised manner:

Consensus Embedding (CE) Layer ( $F_k$ ):
- Function: Maps the device-specific intermediate feature $z_k = M^H_k(x)$ into a shared, device-agnostic embedding space $z'_k = F_k(z_k)$ .
- Training: Optimized via contrastive learning (specifically an approximation of NT-Xent loss) using the shared unlabeled data. The objective is to align embeddings from different devices for the same input sample toward a common centroid, minimizing the distance between $z'_i$ and $z'_j$ for the same $x$ .
- Privacy: Only the consensus embeddings are exchanged; raw inputs and original intermediate features remain local.
Cooperative Output (CO) Layer ( $G_k$ ):
- Function: Maps the consensus embedding $z'_k$ back to task-specific output logits $m'_k = G_k(z'_k)$ .
- Training: Optimized via knowledge distillation using the local pretrained model as a "teacher." The CO layer minimizes the Kullback–Leibler (KL) divergence between its output and the logits produced by the local model's tail network ( $M^T_k$ ) on the shared unlabeled data.
- Goal: Enables the device to generate accurate predictions based solely on the shared consensus embedding.

Inference and Aggregation

During inference, a device processes an input $x$ through its head and CE layers to generate a consensus embedding, which is shared with other devices. Each recipient device processes this embedding through its CO layer to generate logits.

Ensemble Strategy: The paper evaluates various aggregation methods. It highlights that Minimum Energy (selecting the device with the lowest energy score derived from logits) is effective for handling Out-of-Distribution (OOD) samples and severe label mismatches, often outperforming simple averaging or voting in highly non-IID settings.

3. Key Contributions

The paper outlines four primary contributions:

Cooperative Inference without Inputs or Model Sharing: CE-FI achieves cooperation across heterogeneous pretrained models without sharing parameters, raw inputs, or assuming a common encoder. It relies solely on intermediate feature exchange and self-supervised learning.
Self-Supervised Training: The framework introduces a contrastive-learning-based embedding layer and a knowledge-distillation-based prediction layer, both trained exclusively on unlabeled data, reducing annotation costs.
Theoretical Characterization: The authors formally analyze the relationship between CE-FI and conventional Input-sharing FI. They demonstrate that under ideal alignment conditions, CE-FI is theoretically equivalent to Input-sharing FI, and they derive bounds showing that performance degradation is proportional to the error in consensus embedding alignment.
Comprehensive Evaluation: Extensive experiments across image (CIFAR-10/100), text (IMDB), and time-series (UCI-HAR) tasks under diverse non-IID conditions (including disjoint label sets) validate the framework's effectiveness.

4. Experimental Results

Performance vs. Baselines: CE-FI consistently outperforms Solo Inference across all modalities and non-IID conditions. In many scenarios, it approaches the performance of Input-sharing FI and Edge Ensemble, despite operating under stricter constraints.
Heterogeneity: The framework remains effective even when devices use different encoder architectures (e.g., ViT, ResNet, ConvNeXt) and different pretraining objectives (Supervised, CLIP, DINOv3).
Ensemble Sensitivity: The choice of ensemble strategy is critical. While soft voting works well in mild non-IID settings, Minimum Energy selection significantly improves robustness in severe non-IID (Skewed/Disjoint) conditions by filtering out unreliable predictions from devices lacking specific classes.
Scratch vs. Pretrained: CE-FI works even when models are trained from scratch, though performance gains are more pronounced with pretrained encoders. The study identifies that the quality of the underlying semantic structure in local feature spaces limits performance in scratch settings.
Bottleneck Analysis: Ablation studies reveal that representation alignment in the CE layer is the primary bottleneck. Forcing a unified consensus embedding space significantly improves accuracy, particularly in scratch settings, whereas excluding OOD data from training has negligible impact.
Privacy and Reconstruction: Reconstruction attacks (using DCGAN) on consensus embeddings yielded low-quality reconstructions (RMSE ~0.22, SSIM ~0.16), suggesting that sharing consensus embeddings does not trivially reveal raw inputs, though it is not immune to all privacy risks.
Communication Cost: In image classification tasks, transmitting 256-dimensional consensus embeddings (2.1 KB) was more efficient than transmitting raw images (4.6–6.2 KB).

5. Significance and Claims

The paper claims that CE-FI represents a significant step toward practical, privacy-preserving distributed AI. By removing the assumptions of shared inputs and common encoders, it enables cooperation in scenarios where traditional Federated Inference is infeasible (e.g., medical diagnostics with proprietary models and strict data privacy laws).

The authors maintain a modest tone regarding the results:

They acknowledge that while CE-FI approaches the performance of conventional FI, a gap remains, primarily due to imperfect representation alignment.
They note that the optimal ensemble strategy depends heavily on the modality and the degree of non-IIDness, suggesting that a "one-size-fits-all" aggregation method is suboptimal.
They identify representation alignment as the central challenge for future improvement, rather than data filtering or ensemble selection alone.

Ultimately, the work provides a proof-of-concept that cooperative inference is achievable under the strictest constraints of data and model privacy, leveraging only shared unlabeled data and intermediate representations.