Automatic Simplification of Common Vulnerabilities and Exposures Descriptions

1. Problem Statement

Cybersecurity incident reports, specifically Common Vulnerabilities and Exposures (CVE) descriptions, are critical for organizational security but are often written in highly technical language. This creates a significant barrier for non-expert stakeholders (e.g., strategic decision-makers) who need to understand risks but lack domain expertise.

• The Gap: While Automatic Text Simplification (ATS) is well-studied in medical and scientific domains, it has not been effectively applied to the rapidly evolving and jargon-heavy field of cybersecurity.
• The Challenge: Existing Large Language Models (LLMs) can simplify text but often fail at meaning preservation. In cybersecurity, omitting or altering a technical detail (e.g., a specific version number or exploit condition) can lead to false security assumptions, eroding trust and creating liability gaps.
• The Goal: To investigate the efficacy of LLMs in simplifying CVE descriptions while maintaining factual accuracy and to establish a baseline for future research in this domain.

2. Methodology

Dataset Construction

• Source: 100 CVE descriptions published in 2025 from the cvelistV5 repository.
• Preprocessing: Non-natural language (e.g., log excerpts) was manually removed.
• Split:
  • 40 CVEs: Used for human evaluation (Test Set).
  • 60 CVEs: Used for developing and training local models.
• Human Evaluation: Two rounds of surveys involving cybersecurity experts (12 participants in Round 1, 10 in Round 2) using a 3-point Likert scale to assess "ease of understanding" and "meaning preservation."

Model Approaches

The study compared three distinct approaches:

1. GPT-4o (Commercial LLM):
   • Round 1: Sentence-level simplification via Azure OpenAI API (April 2025).
   • Round 2: Document-level resimplification via ChatGPT GUI (July 2025), incorporating feedback from Round 1 participants.
2. Gemma3:4b (Open-Source Baseline): A 4-billion parameter model tested locally for performance relative to size.
3. GemmaAgent (Custom Agent System): An agent-based pipeline using the Gemma 3 (4B) model, designed to enhance factual accuracy. The pipeline consists of:
   • Term Extraction Agent: Uses the AITSecNER model to identify specific cybersecurity terms (labels: CON, MALWARE, TACTIC, TECHNIQUE, TOOL).
   • Retrieval Augmented Generation (RAG) Agent: Retrieves definitions/explanations for extracted terms from a cybersecurity lexicon database.
   • Simplification Agent: Generates the simplified text using the original CVE, the extracted terms, and their RAG-retrieved explanations as context.

Evaluation Metrics

• Reference-Based:
  • D-SARI: Document-level SARI score (measures addition, deletion, and keeping of n-grams against a reference).
  • BERTScore & Semantic Similarity: Measures meaning preservation using token-level precision/recall and cosine similarity (via MeaningBERT and Sentence-BERT).
• Reference-Free (Simplicity):
  • Flesch-Kincaid Grade Level (FKGL): Estimates readability grade.
  • Structural Metrics: Average sentence length, word count, syllable count, and named entity count.

3. Key Results

Simplicity vs. Complexity

• GPT-4o was the only model that successfully reduced the FKGL score (from 12.45 in original text to 9.49), making the text significantly easier to read. It achieved this by increasing sentence count and reducing syllables per word.
• Gemma3:4b and GemmaAgent actually increased the FKGL score (to ~14.2–14.5), making the text technically "harder" to read by standard metrics, likely due to the inclusion of detailed term explanations.
• GemmaAgent was the only model that increased the average number of named entities (2.85 vs. 2.6 in original), indicating it retained more specific technical identifiers rather than removing them.

Meaning Preservation

• GemmaAgent and Semi-synthetic data performed best on meaning preservation metrics (highest BERTScore and MeaningBERT scores).
• Gemma3:4b (without RAG) had the lowest scores for meaning preservation, suggesting that RAG significantly aids in retaining factual accuracy in technical domains.
• GPT-4o struggled with specific technical details. In one instance (CVE-2025-322027), it incorrectly rounded a version number (4.3000000025 to 4.3), highlighting a risk of hallucination or over-simplification in critical data.

Human Evaluation

• Only 5 out of 40 initial GPT-4o simplifications were accepted without modification.
• The primary source of disagreement among human evaluators was meaning preservation, not just readability.
• GPT-4o occasionally failed to return outputs for prompts involving "exploitation" due to safety guardrails, requiring manual intervention.

4. Key Contributions

1. First ATS Baseline for Cybersecurity: The paper establishes the first dedicated dataset and evaluation framework for simplifying CVE descriptions, a domain previously unexplored in ATS literature.
2. Agent-Based RAG Architecture: Demonstrates that a multi-agent system combining NER and RAG (GemmaAgent) outperforms standard LLM prompting in preserving technical facts, even if it does not reduce readability scores as aggressively as direct prompting.
3. Empirical Evidence of Metric Limitations: Shows that standard simplicity metrics (like FKGL) do not correlate with human perception of "good" simplification in technical fields. A lower FKGL score (GPT-4o) did not guarantee better meaning preservation compared to the RAG-based approach.
4. Dataset Release: Creation of a semi-synthetic, human-evaluated dataset of 40 simplified CVEs for future research.

5. Significance and Conclusion

The study concludes that while out-of-the-box LLMs (like GPT-4o) can make text appear simpler, they pose a significant risk of factual distortion in cybersecurity contexts.

• Implication for Industry: Relying solely on automated simplification for strategic decision-making is risky without human oversight or specialized architectures (like RAG).
• Future Work: The authors emphasize the need for "Human-in-the-Loop" systems where domain experts validate simplifications. They also note that expanding the RAG database could further improve reliability.
• Ethical Note: The paper highlights the "responsibility gap" in ATS; if a simplified report omits a critical vulnerability detail, the liability for the resulting security breach is unclear.

In summary, the paper argues that meaning preservation is the primary bottleneck in cybersecurity ATS, and successful solutions require specialized architectures (RAG) and human validation rather than just generic LLM prompting.