Technical Summary: Privacy-Preserving RAG via Sensitive Entity Alias Generator (SEAG)

Problem Statement

Retrieval-Augmented Generation (RAG) enhances Large Language Models (LLMs) by combining internal knowledge with external data sources. While existing privacy research in RAG primarily focuses on preventing unauthorized users from accessing sensitive data, this paper identifies a critical, often overlooked vulnerability: the exposure of sensitive information to external third-party generators.

In many scenarios, an authorized user (e.g., a physician) may query their own organization's data using a powerful external LLM (e.g., GPT-5 or Claude-4). While the user has access rights, the external generator processes the query and retrieved documents, potentially misusing or leaking confidential information such as patient records, financial details, or contract terms. Local deployment of high-performance LLMs (e.g., >100 billion parameters) is often cost-prohibitive due to hardware requirements (e.g., NVIDIA H100 GPUs) and maintenance overhead, forcing organizations to rely on third-party APIs. This paper addresses the need to utilize powerful external generators without disclosing sensitive entities within the query or retrieved documents.

Methodology: The SEAG Framework

The authors propose the Sensitive Entity Alias Generator (SEAG), a privacy-preserving framework that inserts a lightweight, locally deployed model between the RAG retriever and the external generator. The workflow operates as follows:

1. Retrieval: The user submits a query, and the retriever fetches relevant documents from external data sources.
2. Entity Recognition & Alias Generation: The query and retrieved documents are passed to a locally fine-tuned SEAG model (3–4 billion parameters). This model identifies sensitive entities and generates a consistent Entity Replacement Table.
   • The table maps original sensitive entities to semantically consistent aliases (e.g., replacing "Saudi Arabia" with "Qatar" and "Riyadh" with "Doha" to maintain logical relationships).
   • The model ensures consistency across all variations of an entity (e.g., "John," "John Jordan," and "John's" are replaced with corresponding variations of a new name).
3. Anonymization: The Entity Replacement Table is applied to both the user query and the retrieved documents, replacing sensitive entities with their aliases before transmission to the external generator.
4. Generation: The external generator (e.g., GPT-5, Claude-4) processes the anonymized prompt and generates a response based on the aliases.
5. Restoration: The SEAG framework intercepts the generator's output, maps the aliases back to their original entities using the same table, and delivers the final, accurate answer to the user.

Datasets and Experimental Setup

To support this framework, the authors constructed two distinct datasets:

• Fine-tuning Dataset: 640 samples across 8 domains (news, legal, medical, finance, biology, chemistry, history, resume). This dataset was used to fine-tune the SEAG models to generate consistent entity replacement tables.
• Evaluation Dataset: 600 samples across 6 domains (economic, education, cars, legal, finance, business), with 300 documents and 600 questions. This dataset was used to evaluate the entire SEAG pipeline.

Three lightweight models were fine-tuned using QLoRA (Parameter-Efficient Fine-Tuning): Qwen-3 (4B), LLaMA-3.2 (3B), and Phi-4 (3B). These were tested against two external generators: GPT-5 and Claude-4 sonnet.

Evaluation Metrics

The framework was assessed using three metrics:

1. ORG (Original Response Generation): Measures the generator's ability to answer the original question without aliasing, ensuring the baseline questions are solvable.
2. Privacy: Measures the success rate of the SEAG model in hiding sensitive entities from the external generator (i.e., the generator cannot answer based on the original entity).
3. User: Measures the end-to-end success of the framework: the sensitive information is hidden from the generator, yet the correct original answer is restored and presented to the user.

Key Results

• Overall Performance: All SEAG models achieved over 80% accuracy on the User metric, demonstrating the framework's ability to provide correct answers while protecting privacy.
• Best Performing Model: LLaMA-3.2 consistently outperformed the others.
  • Privacy Score: LLaMA-3.2 achieved the highest privacy score of 89.67% (with Claude-4 as the generator) and 88.50% (with GPT-5).
  • User Score: The combination of LLaMA-3.2 and Claude-4 sonnet achieved the highest User score of 83.67%.
• Comprehensive Entity Hiding: When evaluated on the ability to hide all sensitive entities in documents (not just those required for the answer), the models achieved total accuracies of:
  • Qwen-3: 77.83%
  • LLaMA-3.2: 76.73%
  • Phi-4: 74.91%
• Domain Variance: The Economic domain yielded the highest accuracy (86–87%) due to the explicit nature of its entities (countries, organizations, dates). Conversely, Legal and Finance domains showed lower performance due to the complexity and hierarchical nature of entities (e.g., long policy titles, interconnected financial reports).

Limitations

The authors acknowledge specific constraints in their current work:

• Document Length: The framework was tested on documents averaging ~300 words. Performance on substantially longer documents with high entity density remains unverified.
• Entity Density: The current evaluation assumes approximately 15 sensitive entities per document. Scalability to documents with significantly higher entity counts is a subject for future research.

Significance and Contributions

The paper claims to be the first to introduce a framework specifically designed to protect private documents from being shared with external third-party generators within RAG systems. Unlike previous works focusing on user access control or synthetic data generation, SEAG enables organizations to leverage powerful, cloud-based LLMs while retaining control over sensitive data through a lightweight, local intermediary.

The primary contributions are:

1. Framework Innovation: Introducing SEAG to decouple the retrieval of sensitive data from the generation of answers by external models.
2. Dataset Construction: Creating two specialized datasets for fine-tuning entity replacement models and evaluating the full RAG-privacy pipeline.
3. Empirical Validation: Demonstrating that fine-tuned 3–4B parameter models can effectively mask sensitive entities while maintaining high answer fidelity, with LLaMA-3.2 emerging as the most effective candidate for this task.

The authors conclude that while the framework shows promising results, future work must address scalability regarding document length, entity density, and more complex contextual dependencies to ensure robustness in real-world, large-scale deployments.