CTIGuardian: A Few-Shot Framework for Mitigating Privacy Leakage in Fine-Tuned LLMs

1. Problem Statement

Large Language Models (LLMs) are increasingly fine-tuned on proprietary, domain-specific datasets, particularly in Cyber Threat Intelligence (CTI). While this improves task performance (e.g., extracting vulnerability reports), it introduces a critical privacy risk: memorization.

• The Risk: Fine-tuned models can inadvertently memorize sensitive details from training data, such as internal IP addresses, employee emails, unpatched software versions, and specific attack methodologies.
• The Threat: Adversaries with black-box access can exploit data extraction attacks (using crafted prefixes) to trigger the model to "regurgitate" these sensitive tokens, leading to privacy leakage.
• Limitations of Existing Defenses:
  • Pre-training Sanitization (NER): Named Entity Recognition tools struggle with obfuscated CTI data and specific formats, often leaving residual sensitive data.
  • Differential Privacy (DP): Injecting noise degrades the model's utility and disrupts the contextual integrity required for CTI analysis.
  • Machine Unlearning: Requires access to model internals and is computationally expensive, making it impractical for collaborative or shared model settings.

2. Methodology: CTIGuardian

The authors propose CTIGuardian, a privacy-preserving framework that operates at the inference stage without requiring costly retraining. Inspired by "safety alignment," the framework introduces "privacy alignment" using a few-shot supervision approach.

Core Architecture

CTIGuardian functions as a two-stage pipeline powered by a supervisory LLM (e.g., GPT-4o mini or Mistral-7B):

1. Privacy Classifier (Input Filter):
   • Analyzes user prompts to determine if they are "harmful" (attempting to extract sensitive data) or "harmless."
   • It categorizes requests into four types: Direct, Indirect, Disguised, and Harmless.
   • If a prompt is classified as harmful, the system returns a refusal message, preventing the query from reaching the fine-tuned model.
2. Privacy Redactor (Output Filter):
   • Even benign prompts can trigger the model to leak memorized sensitive data (e.g., completing a partial IP address).
   • The redactor post-processes the model's output to detect and remove sensitive entities (IPs, emails, ports, domains, software versions).
   • Key Innovation: Unlike rigid NER rules, the redactor uses few-shot learning to rewrite the text. It removes sensitive data while maintaining semantic coherence and fluency, rather than just replacing entities with generic placeholders (e.g., <IP_Address>).

Experimental Setup

• Datasets:
  • APTQA: A new dataset of 5,093 records created by mapping APT campaign notes to CVEs/CWEs using GPT-4. It retains realistic sensitive entities.
  • CTI-MITRE: A public dataset of 12,945 records mapping attack descriptions to MITRE ATT&CK techniques.
• Base Models: Llama-2-7B was fine-tuned on these datasets.
• Defense Models: GPT-4o mini (closed-source) and Mistral-7B-Instruct-v0.3 (open-source) were used as the classifier and redactor.
• Attack Simulation: A black-box data extraction attack was performed using prefix-based prompting with top-k sampling to extract memorized training data.

3. Key Contributions

1. APTQA Dataset: The creation of a novel, QA-style dataset containing realistic, sensitive CTI information (IPs, emails, ports, etc.) to facilitate ground-truth evaluation of privacy leakage.
2. Privacy Alignment Framework: The introduction of CTIGuardian, which shifts the paradigm from "preventing unsafe responses" (safety) to "preventing sensitive data leakage" (privacy) via few-shot prompting, avoiding retraining costs.
3. Defense-in-Depth Strategy: A combined approach using both input filtering (classifier) and output sanitization (redactor) to handle both explicit extraction attempts and accidental leakage from benign queries.
4. Comprehensive Evaluation: A rigorous benchmark comparing CTIGuardian against the state-of-the-art NER baseline (Microsoft Presidio) across multiple metrics.

4. Results

The evaluation was conducted on GPT-4o mini and Mistral-7B, benchmarked against Microsoft Presidio.

• Privacy Leakage Reduction:

  • Before Defense: Fine-tuned models leaked significant amounts of sensitive data (e.g., ~19% leakage for ports and software versions).
  • After CTIGuardian:
    • GPT-4o mini: Achieved near-total redaction of IPs, emails, and domains. Leakage was reduced to <2% for ports and <7% for software versions.
    • Mistral-7B: Also effective but slightly higher residual leakage (~7-10%) compared to GPT.
  • Comparison with Presidio: CTIGuardian significantly outperformed Presidio. Presidio failed to detect obfuscated entities (e.g., john[at]gmail.com) and left residual leakage. CTIGuardian successfully generalized to obfuscated formats using few-shot examples.

• Utility Preservation:

  • CTIGuardian maintained high semantic similarity with the original outputs.
  • Cosine Similarity: GPT-4o mini achieved 97.6% (APTQA) and 96.07% (CTI-MITRE).
  • BLEU/ROUGE: While slightly lower than Presidio (which preserves exact sentence structure by using placeholders), CTIGuardian's rewriting approach resulted in more natural, readable text that did not reveal the type of data removed (unlike Presidio's <IP_Address> tags).

• Efficiency:

  • GPT-4o mini demonstrated lower latency (~2.28s total) and extremely low operational cost (0.008¢ per inference) compared to Mistral, making it a cost-effective solution.

5. Significance and Conclusion

• Practicality: CTIGuardian offers a practical solution for organizations that cannot afford to retrain models or access model internals. It works as a "wrapper" around any fine-tuned LLM.
• Robustness: The few-shot approach allows the system to adapt to new obfuscation techniques and domain-specific formats without manual rule engineering, addressing a major weakness of traditional NER tools.
• Generalizability: While demonstrated on CTI, the framework is domain-agnostic and applicable to other sensitive fields like healthcare, finance, and law.
• Trade-off: The paper successfully demonstrates that strong privacy protection can be achieved without sacrificing the utility of the model, provided the redactor is powerful enough to rewrite contextually coherent sentences.

In summary, CTIGuardian establishes a new standard for protecting fine-tuned LLMs by moving away from static, rule-based sanitization toward dynamic, AI-driven privacy alignment that balances security, utility, and cost.