Tug-of-War within A Decade: Conflict Resolution in Vulnerability Analysis via Teacher-Guided Retrieval-Augmented Generations

1. Problem Statement

Large Language Models (LLMs) are increasingly used for cybersecurity vulnerability analysis, specifically regarding Common Vulnerabilities and Exposures (CVEs). However, they face a critical "Knowledge Conflict" problem:

• Temporal Lag: LLMs have a fixed "knowledge cutoff date." CVE data evolves rapidly (often updated hourly/daily), meaning an LLM's internal training data becomes outdated quickly.
• Hallucination & Inconsistency: When queried about updated CVEs, LLMs often retrieve outdated internal knowledge, leading to factual errors, hallucinations, or the generation of conflicting information (e.g., citing an old CVE version instead of the latest patch).
• Retrieval Difficulties: Traditional Retrieval-Augmented Generation (RAG) systems struggle with CVE data because adjacent CVE numbers often have high semantic similarity, causing retrieval systems to confuse similar but distinct vulnerabilities.
• Lack of Conflict Datasets: There was a prior lack of curated datasets specifically designed to test and resolve conflicts between old and new vulnerability knowledge.

2. Methodology: CRVA-TGRAG Framework

The authors propose CRVA-TGRAG, a two-stage framework designed to resolve these conflicts by combining optimized retrieval with teacher-guided preference tuning.

Stage I: Enhanced Retrieval and Augmentation

This stage focuses on improving the accuracy of retrieving the correct external knowledge before it reaches the LLM.

• Data Pre-processing:
  • Sources: Official NVD API and CVE GitHub datasets.
  • Format: Converted from JSON to CSV for better metadata handling.
  • Conflict Labeling: Used an LLM to compare "Old" vs. "New" values for modified CVEs, generating a structured dataset highlighting specific changes.
• Semantic Parent Document Segmentation:
  • Instead of fixed-size chunking, the authors use semantic similarity to split documents.
  • Goal: Ensure that all information related to a specific CVE-ID remains within the same text chunk to prevent context fragmentation.
  • Mechanism: Uses cosine similarity on embeddings; splits occur only when the distance between sentences exceeds a high percentile threshold (e.g., 90th percentile).
• Ensemble Retrieval:
  • Combines BM25 (inverted indexing) and Vector Retrieval (semantic similarity).
  • Weighting: Set to [0.25 BM25, 0.75 Vector] to prioritize semantic understanding while maintaining keyword precision for specific CVE IDs.
  • Query Expansion: Performs synonym replacement on user queries to generate joint queries, improving the retrieval of highly similar documents.

Stage II: Teacher-Guided Preference Optimization (Generation)

This stage fine-tunes the LLM to prefer the updated knowledge retrieved in Stage I over its outdated internal parameters.

• Teacher-Guided Approach: Analogous to a teacher guiding a student through textbook updates. The "Teacher" is the curated dataset containing the correct "New Value" vs. "Old Value."
• Direct Preference Optimization (DPO):
  • The system constructs a preference dataset where the Updated CVE content is the "Winning" response ($y_w$) and the Outdated/Incorrect content is the "Losing" response ($y_l$).
  • The LLM is fine-tuned using DPO to maximize the likelihood of generating the updated content while minimizing the outdated content, effectively "unlearning" the conflict.

3. Key Contributions

1. First CVE Conflict Dataset: Created and released a dataset of 1,260 pairwise conflict CVE items (published and modified over the last decade), specifically focusing on high-severity vulnerabilities. This fills a gap in vulnerability analysis research.
2. CRVA-TGRAG Framework: Proposed a novel two-stage architecture that integrates Semantic Parent Document Segmentation, Ensemble Retrieval, and Teacher-Guided DPO.
3. Evaluation Metrics: Established a comprehensive scoring criteria including standard metrics (BLEU, ROUGE) and RAG-specific metrics (Faithfulness, Context Precision, Context Recall) to rigorously measure conflict resolution.
4. Significant Performance Gains: Demonstrated that their method significantly outperforms baselines in resolving knowledge conflicts.

4. Experimental Results

The framework was evaluated against various baselines (No Prompt, Prompt Only, Naive RAG, DPO only) using models like GPT-4o-mini, Llama-3, and Mistral.

• Answer Correctness (AC): Increased from 0.49 (baseline) to 0.76 (+27%).
• Faithfulness (Faith): Increased from 0.36 to 0.88 (+52%).
• Context Precision (CP) & Recall (CR): Showed substantial improvements over Naive RAG, indicating the system retrieves the most relevant updated context rather than just any context.
• Ablation Study: Confirmed that both Stage I (optimized retrieval) and Stage II (DPO fine-tuning) are necessary. Using only DPO without RAG resulted in poor accuracy, while using only RAG without DPO failed to resolve internal knowledge conflicts effectively.

5. Significance and Conclusion

• Practical Impact: The framework provides cybersecurity researchers and engineers with reliable, up-to-date vulnerability analysis, reducing the risk of acting on outdated or hallucinated threat intelligence.
• Generalizability: While focused on CVEs, the "Teacher-Guided" approach to resolving knowledge conflicts via preference tuning offers a blueprint for other rapidly evolving domains (e.g., medical guidelines, legal statutes).
• Future Work: The authors suggest that Agentic RAG (using online search engines like Google/Bing alongside official databases) could further mitigate timeliness issues, though they note the risk of SEO manipulation in open web searches.

In summary, CRVA-TGRAG effectively solves the "Tug-of-War" between an LLM's static internal knowledge and dynamic external reality by ensuring the retrieval system finds the right data and the generation model is explicitly trained to prioritize that new data.