💬 NLP

Supercharging Federated Intelligence Retrieval

This paper proposes a secure Federated RAG system leveraging Flower and confidential computing to enable local retrieval and private remote LLM inference across distributed data silos, while introducing a cascading inference approach to integrate non-confidential third-party models without compromising data privacy.

Original authors: Dimitris Stripelis, Patrick Foley, Mohammad Naseri, William Lindskog-Münzing, Chong Shen Ng, Daniel Janes Beutel, Nicholas D. Lane

Published 2026-03-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Dimitris Stripelis, Patrick Foley, Mohammad Naseri, William Lindskog-Münzing, Chong Shen Ng, Daniel Janes Beutel, Nicholas D. Lane

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant, all-knowing librarian (an AI) who can answer any question you ask. But there's a catch: the books the librarian needs to answer your question are locked in different, private vaults owned by different companies. One vault belongs to a hospital, another to a bank, and another to a university.

The Problem:
Usually, to get the best answer, you'd have to bring all those books to one central room, open the vaults, and let the librarian read them all. But the owners of the vaults say, "No! We can't share our private books because of privacy laws and trade secrets."

The Old Solution (and why it fails):
Some systems try to send the answers from the vaults to the librarian. But often, the person running the librarian's desk (the server) is "honest but curious." They might peek at the books while they are being passed around, or a hacker might steal them.

The New Solution: "Federated RAG" with a Secret Vault
This paper introduces a new, super-secure way to solve this. Think of it as a high-tech, invisible meeting room where the librarian can work without ever seeing the actual books, and the vault owners never have to leave their homes.

Here is how it works, broken down into simple steps:

1. The Local Search (The Private Vaults)

Instead of moving the books, the librarian sends a question to each vault owner.

The Vault Owner: They keep their books safe inside their own building. They search their own shelves, find the top 8 most relevant pages, and write down only the text of those pages.
The Twist: They don't send the pages to the librarian directly. They send them to a Secret, Tamper-Proof Glass Box (called a Trusted Execution Environment or TEE).

2. The Secret Glass Box (Confidential Computing)

This is the magic part. Imagine a glass box that is so secure that even the person who owns the building (the server operator) cannot see inside it while it's working.

Inside this box, the librarian gathers all the pages from the different vaults.
The librarian reads them, combines the information, and writes a final answer.
Crucially: The person running the server never sees the raw data. The data is only visible inside the "glass box" where it is protected by special hardware.

3. The Two Ways to Get the Answer

The paper tests two different ways the librarian can use this setup:

Option A: The "Super-Helper" (Cascading Inference)
The librarian inside the glass box asks a second, very smart (but less secure) AI assistant outside the box for help.
- Analogy: It's like the librarian asking a famous expert for a quick opinion, then combining that opinion with the private notes from the vaults to write the final report.
- Result: This made the answers much better (about 40-50% more accurate) because the librarian got extra "brainpower" without losing privacy.
Option B: The "Mega-Librarian" (Confidential Inference)
Instead of a small librarian, they use a massive, super-powerful AI running inside the secure glass box.
- Analogy: It's like moving the entire library into a high-security bunker and letting a giant robot read everything at lightning speed.
- Result: This was the fastest and most accurate method, proving you can use huge, powerful AI models without ever exposing your private data.

Why This Matters

This system is like a secure, invisible bridge.

For Hospitals and Banks: They can let an AI learn from their private data without ever actually giving the data away.
For the AI: It gets access to a massive amount of knowledge it couldn't reach before.
For You: You get better answers to your questions, knowing your secrets are safe.

In a nutshell: This paper shows how to build a "super-librarian" that can read everyone's private books to give you the best answer, all while keeping those books locked in their original vaults and hidden from prying eyes. It's the best of both worlds: maximum knowledge with maximum privacy.

Based on the paper "Supercharging Federated Intelligence Retrieval," here is a detailed technical summary covering the problem, methodology, contributions, results, and significance.

1. Problem Statement

Traditional Retrieval-Augmented Generation (RAG) systems rely on centralized access to document repositories. This assumption fails in real-world scenarios where sensitive data (e.g., in healthcare or finance) is distributed across private "data silos" due to regulatory, proprietary, or privacy constraints.

While Federated RAG (FedRAG) attempts to solve this by performing retrieval locally within each silo and aggregating results on a server, existing solutions suffer from significant security gaps:

Plaintext Exposure: Retrieved documents are often exchanged in plaintext, exposing sensitive data to the central server.
Server Trust Issues: The server operator is often assumed to be "honest-but-curious" or potentially compromised, meaning they could inspect the aggregated context or the generation process.
Limited Security Scope: Prior work focuses on securing individual stages (like training or encrypted communication) but rarely protects the end-to-end pipeline, specifically the aggregation and generation steps within a trusted environment.

2. Methodology

The authors propose a secure FedRAG architecture built on Flower, a federated learning framework, utilizing Confidential Computing to protect the entire inference pipeline.

Core Architecture

Client-Side (Data Silos): Each client maintains a local private document corpus. Upon receiving a query, the client performs local retrieval using a FAISS index to fetch the top- $k$ relevant documents. These documents and their retrieval scores are sent to the server.
Server-Side (Trusted Execution Environment - TEE):
- The server operates entirely inside a TEE (e.g., via Flower Confidential Remote Compute or standard TEEs).
- Aggregation: The server receives encrypted/secure results from clients, merges them, and re-ranks them using a Reciprocal Rank Fusion (RRF) formula: $Score = 1 / (k_{RRF} + doc_i + 1)$ .
- Generation: The aggregated context is passed to an LLM running within the TEE to generate the final response.
Attestation: Client participation is enforced via remote attestation and authorization to ensure only verified clients interact with the TEE.

Three Inference Modes

The system evaluates three distinct operational modes, all running within a TEE:

Standalone Inference: A server-hosted LLM (SmolLM 1.7B) generates responses using only the retrieved documents as context.
Cascading Inference: The server-hosted LLM generates responses using retrieved documents plus an auxiliary answer from a non-confidential third-party model (AWS Nova Micro). This treats the external model as an additional probabilistic knowledge source.
Confidential Inference: A large-scale LLM (Qwen3 235B) runs inside Flower CRC (Confidential Remote Compute). This ensures that both the prompts and the aggregated context remain encrypted and protected during inference, even from the infrastructure provider.

3. Key Contributions

Secure Federated RAG System: A novel architecture enabling cross-silo retrieval without centralizing raw data, preserving data locality.
Confidential Server-Side Processing: Implementation of server-side aggregation and text generation within a TEE, protecting against honest-but-curious or compromised server operators.
Mixed-Trust Cascading Mechanism: A novel approach that integrates non-confidential third-party models (like AWS Nova) as auxiliary context sources without compromising the end-to-end confidentiality of the primary pipeline.
End-to-End Protected Pipeline: The first system to combine federated retrieval with confidential aggregation and generation using Flower CRC, supporting both CPU-based TEEs and high-capacity GPU-backed confidential inference.

4. Experimental Results

The system was evaluated on the MIRAGE benchmark (PubMedQA, BioASQ, MedQA) using a federation of four clients with diverse medical and general knowledge corpora (PubMed, StatPearls, Textbooks, Wikipedia).

Inference Mode	Model Used	Avg. Time (s)	PubMedQA Acc.	BioASQ Acc.	MedQA Acc.
Standalone	SmolLM 1.7B (CPU)	~43-44	0.32	0.81	0.39
Cascaded	SmolLM + Nova Micro	~45-48	0.45 (+40%)	0.83	0.57 (+46%)
Confidential	Qwen3 235B (GPU/TEE)	~25-26	0.47	0.82	0.78

Key Findings:

Cascading Inference: Even with a lightweight external model (Nova Micro), adding its output as context significantly boosted accuracy (up to 46% improvement on MedQA) compared to standalone inference, demonstrating the value of mixed-trust deployments.
Confidential Inference: Running a massive model (Qwen3 235B) inside Flower CRC on an NVIDIA H100 GPU achieved the highest accuracy across all benchmarks and the lowest latency (approx. 25s), outperforming the CPU-bound SmolLM significantly.
Security vs. Performance: The results prove that high-performance, privacy-preserving RAG is achievable even when the server infrastructure is untrusted, provided TEEs are used.

5. Significance

This work addresses a critical bottleneck in deploying AI for sensitive domains: how to leverage collective knowledge without violating privacy.

Regulatory Compliance: By keeping data local and processing aggregation/generation in TEEs, the system meets strict data sovereignty and privacy regulations (e.g., HIPAA, GDPR).
Trust Model Shift: It shifts the trust model from "trusting the server operator" to "trusting the hardware-enforced TEE," allowing organizations to collaborate on AI without exposing raw data.
Scalability & Flexibility: The architecture supports a flexible mix of inference strategies, from lightweight CPU-based TEEs for cost-effective operations to high-performance GPU-backed confidential inference for complex tasks.
Future Direction: The success of the cascading approach suggests a viable path for integrating powerful, non-confidential commercial APIs (like AWS or Azure) into secure, private federated workflows, expanding the capabilities of local models without sacrificing security.