🤖 AI

GRID: Graph Representation of Intelligence Data for Security Text Knowledge Graph Construction

The paper presents GRID, an end-to-end framework that constructs security knowledge graphs from cyber threat intelligence by leveraging traceable article-graph alignments and a scripted task-bank reward system to train efficient 4B-parameter extractors that achieve superior precision, recall, and cost-effectiveness compared to traditional LLM-based evaluation methods.

Original authors: Liangyi Huang, Zichen Liu, Fei Shao, Shang Ma, Mengshi Zhang, Zihao Chen, Yanfang Ye, Xusheng Xiao

Published 2026-05-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Liangyi Huang, Zichen Liu, Fei Shao, Shang Ma, Mengshi Zhang, Zihao Chen, Yanfang Ye, Xusheng Xiao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a cybersecurity detective trying to solve a massive, complex crime. You have a stack of thousands of messy, long-winded police reports (the "security text"). Your goal is to turn these messy reports into a clean, organized map (a "Knowledge Graph") that shows exactly who did what, to whom, and how.

The problem is that the detectives you usually hire (standard AI models) are great at general conversation but terrible at understanding specific security jargon. They often miss crucial details or make things up. Furthermore, teaching them to do this job is incredibly expensive and difficult because it's hard to grade their work without hiring a human expert for every single attempt.

Enter GRID (Graph Representation of Intelligence Data). Think of GRID not just as a new detective, but as a complete training academy and grading system designed specifically to turn a small, affordable AI into a security expert.

Here is how GRID works, broken down into simple steps:

1. The "Self-Correcting" Textbook (Automatic Annotation)

Usually, to teach a student, you need a teacher to write the perfect answer key. But for security reports, no one has written these answer keys yet.

The GRID Solution: GRID acts like a smart tutor that writes its own textbook. It reads a messy security report, pulls out the facts to make a rough map, and then rewrites the original report.
The Analogy: Imagine a student highlighting a textbook. If the highlighter marks something that isn't actually in the text, the teacher (GRID) erases that highlight and rewrites the sentence to match the facts. This creates a perfect "pair" of a story and its map without needing a human to write it from scratch.

2. The "Multiple Choice" Shortcut (Task-Bank Rewards)

Teaching the AI to draw a whole map from scratch is like asking a student to write a 50-page essay and then grading it. It takes forever and costs a fortune if you use a human grader (or a very expensive AI grader) every time.

The GRID Solution: Instead of grading the whole essay, GRID breaks the lesson down into tiny, easy-to-check quizzes.
The Analogy: Instead of asking the student, "Draw the entire crime scene," GRID asks, "Did the hacker use Tool A or Tool B?" or "Is this specific connection between two people true or false?"
Why it helps: These are like multiple-choice questions with a "checklist" answer. They are cheap to grade instantly. By practicing on thousands of these tiny quizzes, the AI learns the rules of the game much faster and cheaper than if it had to write full essays every time.

3. The "Rulebook" (Ontology-Guided Extraction)

Security terms are tricky. A "virus" might be called a "malware," a "threat," or a "bug" in different reports.

The GRID Solution: GRID gives the AI a strict rulebook (ontology). It tells the AI: "If you see a 'virus' or a 'trojan,' they both belong in the 'Malware' folder."
The Analogy: It's like giving the detective a standardized filing cabinet. Instead of guessing where to put a file, the AI knows exactly which drawer to use based on the specific type of threat, ensuring the final map is organized and consistent.

4. The "Grading" Results

The researchers tested this system on 249 real-world security reports from five different sources. They compared their "trained" AI against other top systems.

The Result: The GRID-trained AI was the best at finding all the missing clues (Recall). It found more of the actual threats than any other system tested.
The Efficiency: While other systems were like expensive, slow-moving trucks, GRID was a nimble, fast car. It achieved nearly the same top scores but used less than half the computing power (tokens) to get there.

Summary

Think of GRID as a system that takes a small, affordable AI and gives it:

Self-made study guides (rewritten articles).
Drill-down practice tests (multiple-choice quizzes) instead of expensive final exams.
A strict filing system (the rulebook) to keep things organized.

The result is a security expert AI that is cheaper to run, faster to train, and better at finding the hidden details in security reports than the previous generation of tools.

Technical Summary: GRID – Graph Representation of Intelligence Data for Security Text Knowledge Graph Construction

1. Problem Statement

The paper addresses the critical need for Cyber Threat Intelligence (CTI) to be structured into computable, traceable knowledge graphs (KGs) to aid security agents and Large Language Models (LLMs) in reasoning over connected evidence. While unstructured CTI articles (blogs, reports) contain rich details on attack behaviors and malware evolution, existing approaches face four primary challenges:

Lack of Domain Integration: General-purpose LLMs often lack specific security domain knowledge, especially at smaller model scales, forcing reliance on expensive commercial APIs.
Data Scarcity: There is a lack of high-quality, publicly available datasets that provide aligned pairs of CTI text and their corresponding ground-truth knowledge graphs for end-to-end training.
Expensive Reward Signals: End-to-end document-to-graph extraction is an open-ended generation task. Designing reinforcement learning (RL) rewards is difficult; using an LLM-as-a-judge to score full graph outputs at every training step incurs prohibitive costs.
Shallow Learning: LLMs tend to rely on superficial lexical overlaps rather than deep semantic understanding of entity aliases, hierarchies, and relation constraints within CTI narratives.

2. Methodology

The authors propose GRID (Graph Representation of Intelligence Data), an end-to-end framework designed to construct security knowledge graphs from long-form text using a 4B-parameter LLM (Qwen3-4B-Instruct-2507). The framework consists of four core components:

A. Automatic Annotation of Article-Graph Alignment

To solve the data scarcity issue without massive manual labeling, GRID introduces a two-stage annotate-and-revise loop:

Traceable Extraction: An LLM extracts a knowledge graph from a raw CTI article under strict "text-provable" constraints, preserving verbatim evidence anchors (source text spans).
KG-Conditioned Revision: The original article is rewritten by an LLM to remove security content not captured by the extracted graph while retaining non-security context and the protected anchors.
This process yields high-quality $(a', G')$ pairs where the revised article $a'$ is explicitly aligned with the graph $G'$ .

B. Task-Bank Reformulation for RL Training

Instead of rewarding full graph generation directly, GRID reformulates the learning objective into a scripted task bank to reduce cost and stabilize rewards:

Multi-Select Questions: Generates four-option questions where the ground truth is a subset of correct options. Distractors are designed as "near-miss" or "hallucination" cases (e.g., correct entities but wrong relations).
Regex Targets: Converts each ground-truth graph edge into a triple-level regex target.
Reward Mechanism: Rewards are calculated based on exact matches for choice tasks and normalized matching ratios for regex tasks. This replaces expensive whole-graph scoring with cheaper, local task-level checks.

C. Article Complexity Modeling

The framework assigns a scalar complexity score to each article based on length, entity/relation counts, and graph connectivity statistics. This enables article-complexity-ordered training, allowing the RL agent to progress from easier to harder examples.

D. Ontology-Guided Inference

GRID employs a CTI-specific ontology (inspired by STIX but refined for practical extraction) that defines entity types (e.g., threat-actor, malware, vulnerability) and relation families. This ontology guides the LLM to:

Infer implicit relations based on typed categories and hierarchy.
Enforce text-provable truth by forbidding external completion or subject elevation.
Normalize aliases and relations to ensure structural consistency.

E. Evaluation Protocol

The system uses an LLM-as-judge evaluator for automatic assessment. The judge scores precision and recall based on "text-provable" rules, requiring the model to cite supporting text evidence for every predicted edge. This evaluator was calibrated against three human reviewers, achieving 86.0% agreement.

3. Key Contributions

Automatic Data Pipeline: A novel algorithm to generate traceable article-graph alignments without large-scale manual annotation, effectively creating a supervised dataset from raw CTI articles.
Low-Cost RL Strategy: A "Task-Bank" approach that reformulates open-ended graph extraction into multi-select questions and regex matching, significantly reducing the computational cost of RL training compared to online LLM-as-judge scoring.
Ontology-Guided Extraction: A framework that integrates a security-specific ontology to guide the LLM, improving semantic understanding and reducing hallucinations without relying on closed-label classification.
Unified Benchmark: The creation of a benchmark comprising 249 CTI articles from five distinct sources (GRID, CASIE, CTINexus, MalKG, SecureNLP) with a trustworthy automatic evaluation protocol.

4. Experimental Results

The authors evaluated GRID on the unified benchmark using a 4B-parameter model.

Performance vs. Baselines: The primary Task-bank Reward model achieved:
- 84.62% source-averaged precision.
- 64.91% source-averaged recall.
- 68.53% Avg F1.
- It achieved the best recall among all baselines and a near-tied top F1 score compared to the strongest competitor (CTINexus), while using roughly 40% of the inference tokens.
Ablation Studies (RQ2 & RQ3):
- The Task-bank Reward approach outperformed an End2End Reward model (trained with online LLM-as-judge scoring) by 11.06 recall points and 10.47 F1 points, while being significantly cheaper ($60 vs. $942 in training costs).
- Removing the regex branch (Choice-only) dropped recall by 31 points, proving the necessity of triple-level matching.
- Article rewriting and complexity-ordered training were both shown to be necessary for optimal performance; removing either resulted in lower training rewards and test scores.

5. Significance and Claims

The paper claims that GRID provides a low-cost, scalable solution for equipping smaller, open-source LLMs with security domain knowledge. Its significance lies in:

Cost Efficiency: By decoupling supervision construction (offline, one-time cost) from training (reusable task bank), GRID avoids the prohibitive costs of online RL with full-graph judging.
Performance: It demonstrates that a 4B model, when properly supervised via this framework, can outperform or match larger or more complex pipelines (like CTINexus) in recall and F1, while being more efficient.
Reliability: The framework emphasizes "text-provable" extraction and evaluation, ensuring that the generated knowledge graphs are grounded in the source text rather than hallucinated from external knowledge.

The authors conclude that their approach successfully bridges the gap between unstructured security text and structured knowledge graphs, offering a practical pipeline for security agents and threat intelligence systems.