Enabling Transparent Cyber Threat Intelligence Combining Large Language Models and Domain Ontologies

1. Problem Statement

Cyber Threat Intelligence (CTI) relies on extracting actionable insights from cybersecurity system logs. However, current methodologies face significant challenges:

• Unstructured Data: Log entries are often unstructured, syntactically inconsistent, and semantically ambiguous, making automated interpretation difficult.
• Limitations of Traditional Methods: Rule-based or heuristic approaches lack adaptability to evolving threat behaviors and novel attack patterns.
• Limitations of LLMs: While Large Language Models (LLMs) show promise in natural language processing, they often lack precision in domain-specific contexts, suffer from hallucinations, and produce outputs that lack structural coherence or strict adherence to cybersecurity schemas.
• Transparency Gap: Existing AI-driven extraction often operates as a "black box," lacking the explainability required for human analysts to trust and verify extracted threat indicators.

The specific challenge addressed is the need for a method that can accurately, transparently, and semantically structure unstructured log data (particularly from honeypots) into a format suitable for automated reasoning and querying.

2. Methodology: OntoLogX

The authors propose OntoLogX, an AI agent methodology that integrates Domain Ontologies and Large Language Models (LLMs) to generate structured Knowledge Graphs (KGs) from raw logs. The workflow operates as follows:

A. Core Components

1. Custom Log Ontology: A lightweight, flat ontology was designed specifically for parsing free-text logs. Unlike complex standards like STIX or UCO (which assume pre-structured data), this ontology captures core entities (Event, User, Application, File, Network Address, Source, URL) and relationships directly from raw text.
2. SHACL Constraints: The ontology is paired with Shapes Constraint Language (SHACL) specifications to enforce structural and semantic validity (e.g., cardinality, data types, required properties) on the generated graphs.
3. AI Agent Pipeline:
   • Retrieval (Few-Shot Learning): When a new log event arrives, the system queries a vector store to retrieve semantically similar, previously generated KGs using Maximal Marginal Relevance (MMR). This ensures diversity in examples while maintaining relevance.
   • Generation: An LLM is prompted with the raw log, context, retrieved examples, and strict rules derived from the ontology. The LLM is instructed to generate a KG in a specific JSON format.
   • Correction (Iterative Feedback): The system employs a validation loop. If the generated output violates SHACL constraints or the ontology schema, a tailored correction prompt is sent back to the LLM to fix specific errors. This cycle repeats (up to 3 times) until a valid graph is produced or the attempt is abandoned.

B. Technical Implementation

• Framework: Built using Python and LangChain.
• Storage: Neo4j graph database with vector indexing for semantic retrieval.
• Output Control: Uses structured output tools (Pydantic) to force the LLM to adhere to a strict schema, reducing formatting errors.

3. Key Contributions

• Novel Hybrid Architecture: The paper introduces a novel approach combining prompt-driven LLMs with rigid ontology-driven constraints and SHACL validation, creating a "self-correcting" agent for CTI extraction.
• Lightweight Ontology Design: The creation of a specialized, minimal ontology tailored for unstructured log parsing, bridging the gap between raw text and complex CTI standards (like STIX/SEPSES).
• Transparent AI Pipeline: By using natural language prompts to guide the extraction and allowing for iterative correction, the methodology enhances the explainability and auditability of the AI's decision-making process.
• Iterative Correction Mechanism: The inclusion of a feedback loop where the system validates its own output against SHACL constraints and requests specific corrections significantly improves data quality compared to single-pass generation.

4. Experimental Results

The methodology was evaluated using the AIT-LDS dataset (70 diverse log entries) across five different LLMs (including Claude 3.5, Llama 3.3, Mistral Large, and Qwen 2.5 Coder).

• Comparison: OntoLogX was compared against a Baseline (prompt-only approach without retrieval or correction).
• Performance Metrics: Precision, Recall, F1 Score, G-Eval (semantic faithfulness), and SHACL violation rates.
• Key Findings:
  • Superior Accuracy: OntoLogX consistently outperformed the baseline across almost all models and metrics.
  • Recall & Precision: Significant improvements in both Precision (fewer hallucinations) and Recall (more complete extraction) were observed. For example, Claude 3.5 Sonnet achieved the highest F1 score (0.859) and G-Eval score (0.767) under OntoLogX.
  • Robustness: The method significantly reduced SHACL violations, ensuring the output was structurally valid.
  • Model Variance: While most models improved, Mistral Large was an exception, performing slightly better on the baseline for some metrics, suggesting it may be less compatible with the specific structured prompting or tool-assisted workflow used.
  • Efficiency: The approach prioritizes extraction quality over speed, with processing times ranging from 1.4s (Llama 3.3) to 9.8s (Mistral Large) per log entry.

5. Significance and Future Work

• Impact on CTI: The study demonstrates that combining semantic constraints with LLMs creates a robust, transparent, and highly accurate system for converting noisy log data into actionable, queryable intelligence. This is crucial for proactive threat hunting and automated response.
• Scalability: The use of a lightweight ontology allows the system to handle diverse log sources (honeypots, audit logs, etc.) without the overhead of complex pre-processing.
• Future Directions:
  • Human-in-the-Loop: Developing interactive interfaces for analysts to validate and refine the KGs.
  • Adaptive Learning: Implementing mechanisms for the system to learn from analyst feedback to adapt to new adversarial tactics.
  • Standard Alignment: Formalizing mappings between the custom ontology and established standards (SEPSES, UCO, MISP) to ensure interoperability with existing CTI platforms.
  • Specialized Models: Exploring task-specialized LLMs (e.g., code-generation models like Qwen 3) for even better graph generation and correction capabilities.

In conclusion, OntoLogX represents a significant step forward in making AI-driven Cyber Threat Intelligence both accurate and explainable, moving beyond "black box" extraction toward a verifiable, ontology-guided intelligence pipeline.