🤖 AI

FD-RAG: Federated Dual-System Retrieval-Augmented Generation

FD-RAG is a federated dual-system framework that enhances edge-based retrieval-augmented generation by decoupling lightweight memory matching from on-demand LLM reasoning, thereby improving accuracy and reducing latency while preserving data privacy through the aggregation of anonymized semantic memories.

Original authors: Tianhao Gao, Kai Yang, Yiyang Li

Published 2026-05-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Tianhao Gao, Kai Yang, Yiyang Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a complex mystery, but you don't have a single giant library. Instead, you have a team of detectives, each working in their own small, locked office. They can't share their raw notes (because of privacy rules), and they can't afford to call a super-smart, expensive consultant (a Large Language Model) for every single clue they find.

This is the problem FD-RAG solves. It's a new system designed for "edge" devices (like your phone or a local server) that need to answer questions using information spread across many different devices without breaking the bank or violating privacy.

Here is how it works, broken down into simple concepts:

1. The Problem: The "All-or-Nothing" Trap

Most current AI systems work like a student who has to read a whole book every time they get a question. If the book is huge, it takes forever. If the student is in a remote village with no internet, they can't access the book at all.

The Issue: Existing systems assume all knowledge is in one central place and that the AI is powerful enough to reason through everything. In the real world, data is scattered, privacy is strict, and devices are weak.

2. The Solution: The "Dual-System" Detective Team

The authors, inspired by how human brains work (fast intuition vs. slow thinking), built a system with two distinct modes:

System 1: The "Memorizer" (Fast Thinking)
- What it is: A lightweight, super-fast memory bank.
- How it works: Before you even ask a question, the system has already read the local documents and turned them into a set of "Flashcards" (Question-and-Answer pairs).
- The Magic: When you ask a question, the Memorizer checks these flashcards first. If the answer is right there (e.g., "Who is the CEO?"), it snaps it out instantly. No expensive AI reasoning is needed. It's like looking up a word in a dictionary instead of writing an essay about it.
System 2: The "Cognizer" (Slow Thinking)
- What it is: The heavy-duty AI reasoner.
- How it works: If the flashcards don't have the answer (e.g., a complex question requiring connecting three different facts), the system doesn't panic. Instead, it uses the flashcards to find the exact few pages of the original documents that are relevant.
- The Magic: It only calls the expensive AI consultant for those specific pages. This saves massive amounts of time and money compared to reading the whole book again.

3. The Secret Sauce: The "Semantic Hypergraph"

How does the system organize the information so well?

The Analogy: Imagine a standard library catalog is a list of books. A Hypergraph is like a giant, 3D web of sticky notes.
Instead of just linking one sentence to another, this web connects groups of sentences that share a deeper meaning. It captures complex relationships (like "A is related to B, which is related to C") without needing to read every word perfectly.
The system learns this web structure automatically and then "distills" it into those compact Flashcards (QA Memories) mentioned above.

4. The Teamwork: Federated Learning (Without Sharing Secrets)

Since the data is split across different devices (like different hospitals or banks), how do they help each other?

The Problem: Device A has half the puzzle; Device B has the other half. They can't send their raw documents to each other because of privacy laws.
The Solution: Each device creates its own set of "Anonymized Flashcards." Before sending them to the central team, they scramble the sensitive names (like changing "John Smith" to "Patient A" or a similar fake name) so the original identity is hidden.
The Result: The central team combines these scrambled flashcards into a "Global Memory Bank." Now, when a user asks a question, the system can pull clues from all devices without anyone ever seeing the raw, private documents.

5. The Results: Fast, Accurate, and Private

The paper tested this on tough question-answering benchmarks (like multi-step logic puzzles).

Speed: It was 8.4 times faster than other methods because it avoids calling the expensive AI for simple questions.
Accuracy: It was 7.8% more accurate because it uses the "Global Memory" to fill in gaps that a single device might miss.
Privacy: It successfully hid sensitive data while still allowing the system to learn from it.

Summary

FD-RAG is like giving every edge device a personal, ultra-smart assistant that:

Memorizes the most important facts instantly (Fast Path).
Thinks deeply only when absolutely necessary (Slow Path).
Collaborates with neighbors to solve bigger puzzles without ever showing their private notebooks.

It turns a slow, expensive, and privacy-risky process into a fast, efficient, and secure one.

Technical Summary: FD-RAG: Federated Dual-System Retrieval-Augmented Generation

Problem Statement

Retrieval-Augmented Generation (RAG) has become a standard paradigm for grounding Large Language Models (LLMs) in external knowledge. However, existing RAG systems rely on assumptions that fail in edge environments: centralized knowledge access, unrestricted data sharing, and ample computational resources. In real-world domains like healthcare and finance, knowledge is fragmented across devices, data sharing is restricted by privacy regulations, and inference must operate under strict latency and computation constraints.

Current approaches face three primary limitations in these settings:

Fragility with Small Models: Existing methods often rely on LLMs for both knowledge construction and reasoning. When deployed with Small Language Models (SLMs) common in edge devices, knowledge construction degrades, and complex queries become unreliable.
Latency and Cost: Repeated LLM invocations for retrieval, query understanding, and reasoning incur prohibitive latency and computational costs on edge hardware.
Knowledge Fragmentation: Decentralized environments result in incomplete knowledge coverage at individual nodes. Existing frameworks struggle to achieve both efficiency and completeness when evidence is distributed across multiple silos without exposing raw data.

Methodology

The authors propose FD-RAG (Federated Dual-System RAG), a framework that decouples lightweight memory access from on-demand LLM reasoning. Inspired by Dual-Process Theory (System 1: fast, memory-based; System 2: slow, deliberative), FD-RAG separates retrieval and reasoning into distinct computational roles. The framework consists of three core components:

1. Hypergraph-based Memory Construction

To capture higher-order semantic relations without expensive extraction, FD-RAG learns a semantic-aware hypergraph directly from dense representations of the local corpus.

Granularity: The local corpus is segmented into paragraph-level and sentence-level units and encoded using a pre-trained dense encoder (e.g., BGE-M3).
Hypergraph Learning: The system learns a soft incidence matrix to assign nodes to hyperedges, optimizing for intra-hyperedge semantic compactness and inter-hyperedge global discrimination. This process is proven to converge at an $O(1/\epsilon^2)$ rate.
QA Memory Distillation: The learned hypergraph is distilled into a compact set of Question-Answer (QA) memories. Each hyperedge is converted into typed fact sets using a lightweight NLP pipeline (SPACY), which are then used to synthesize QA pairs grounded in the hyperedge structure. This creates a structured access layer rather than a synthetic QA pool.

2. Dual-System Inference

At inference time, the framework routes queries based on their coverage within the QA memory:

The Memorizer (Fast Path): For queries well-covered by the QA memory, the system performs direct matching. It calculates a unified score combining dense semantic similarity and structural overlap (Dice coefficient of entity anchors). If the score exceeds a threshold $\delta$ , the answer is returned directly from memory, bypassing the LLM entirely.
The Cognizer (Slow Path): For queries where memory coverage is insufficient, the system retrieves the top- $K$ relevant memories. It aggregates the supporting hyperedges of these memories to form a structured evidence set. The LLM is then invoked only on this small, hyperedge-grounded evidence set to perform compositional reasoning, rather than searching the full corpus.

3. Federated Knowledge Aggregation

To address knowledge fragmentation across devices without sharing raw data:

Local Export: Each device distills its local hypergraph into QA memories.
Privacy Preservation: Before sharing, sensitive entities in the typed facts are perturbed using a randomized response mechanism to satisfy $\epsilon$ -Local Differential Privacy ( $\epsilon$ -LDP). This replaces entities with semantically similar surrogates from a candidate set.
Global Fusion: Anonymized memories are aggregated into a global memory bank. This allows devices to collaboratively improve coverage and localize evidence for complex queries while keeping raw corpora and full hypergraph structures on-device.

Key Contributions

FD-RAG Framework: A dual-system RAG architecture for edge environments that decouples memory-based retrieval from selective LLM reasoning, enabling collaborative knowledge utilization across distributed devices.
Semantic-Aware Hypergraph Learning: A novel approach for constructing lightweight, expressive knowledge structures that capture higher-order relations, distilled into compact QA memories for efficient inference.
Theoretical Guarantees: A proof establishing an $O(1/\epsilon^2)$ convergence rate for the proposed hypergraph learning procedure, supporting its tractable deployment.
Privacy-Preserving Federation: A mechanism for aggregating anonymized memories across devices to mitigate knowledge fragmentation while adhering to strict privacy constraints.

Experimental Results

The authors evaluated FD-RAG on three multi-hop QA benchmarks: HotPotQA, 2WikiMQA, and MuSiQue, under both local and federated settings.

Performance vs. Baselines: In the local setting, FD-RAG-Local achieved the highest accuracy across all benchmarks, outperforming strong baselines like HippoRAG 2 by 7.8% in accuracy while reducing latency by 8.4×. It also significantly reduced the average number of LLM calls per query (below 0.6).
Federated Performance: In the federated setting, FD-RAG substantially recovered the accuracy gap caused by knowledge fragmentation. It outperformed the federated baseline RAGRoute by 13.6 accuracy points while remaining 2.6× faster.
Dual-System Analysis: Experiments showed that the Memorizer resolved 68.0% of queries with high confidence (77.7% conditional accuracy), effectively bypassing the LLM for the majority of cases. The Cognizer was essential for the remaining complex queries, as a "Memorizer-only" approach failed to handle multi-hop reasoning, while a "Cognizer-only" approach incurred 5× higher latency and lower accuracy.
Privacy-Utility Trade-off: The proposed $\epsilon$ -LDP perturbation mechanism reduced the attacker's ability to restore sensitive entities from 92.4% (no protection) to 8.8%, while retaining 98.3% of the downstream QA utility compared to raw sharing.
Efficiency: FD-RAG demonstrated lower offline construction costs (time, memory, and LLM token usage) compared to other structured RAG methods like GraphRAG and HyperGraphRAG.

Significance and Claims

The paper claims that FD-RAG provides a principled foundation for RAG deployment in real-world edge scenarios where data locality and resource constraints are paramount. By treating retrieval and reasoning as distinct processes, the framework achieves a Pareto-optimal balance between accuracy and latency.

The authors emphasize that the system's effectiveness stems from:

Decoupling: Separating fast memory matching from slow LLM reasoning allows the system to scale efficiently on edge devices.
Structure: Using hypergraphs to capture higher-order relations enables more precise evidence localization than standard vector retrieval.
Collaboration: Federated memory aggregation allows distributed devices to overcome individual knowledge limitations without compromising privacy.

The paper concludes that while the current design relies on offline construction for knowledge base organization, it offers a robust solution for current edge constraints. Future work is suggested to focus on incremental and adaptive mechanisms to reduce the cost of offline reconstruction when adapting to new domains.