OrgForge-IT: A Verifiable Synthetic Benchmark for LLM-Based Insider Threat Detection

1. Problem Statement

Insider threat detection is a complex multi-signal reasoning problem requiring the correlation of heterogeneous data (e.g., Slack messages, IDP logs, host events) across time. Existing benchmarks face critical limitations:

• The CERT Dataset: While canonical, it is static, lacks cross-surface correlation (e.g., identity provider logs), predates the LLM era, and was generated without an external factual constraint, leading to potential cross-artifact contradictions.
• LLM-Generated Corpora: Purely LLM-generated synthetic data suffers from "hallucination at the corpus level," where inconsistencies (e.g., a Slack message at 3 AM vs. a JIRA ticket at 9 AM for the same event) cannot be ruled out without an external ground truth.
• Evaluation Gaps: Current evaluations often rely on single-surface or single-day triage strategies, failing to test the multi-day, multi-signal reasoning required for real-world detection.

2. Methodology: The OrgForge-IT Framework

The paper introduces OrgForge-IT, a verifiable synthetic benchmark built on a novel "physics-cognition boundary" architecture.

A. Architectural Innovation

• Deterministic Ground Truth Engine: A Python-based simulation engine maintains all factual state (system health, social graphs, event timestamps, and ground truth labels).
• Surface Prose Generation: Large Language Models (LLMs) are restricted to generating only the surface prose (e.g., the text of a Slack message or email body) based on validated proposals from the engine.
• Guaranteed Consistency: Because the engine controls the "facts" and LMs only render the "text," cross-artifact contradictions are structurally impossible. The observable telemetry and ground truth labels are derived from the same simulation run.

B. Corpus Properties

• Scale: 51 simulated days, 2,904 observable records.
• Noise Level: 96.4% noise rate (2,798 benign records vs. 106 true positives), simulating a realistic enterprise signal-to-noise ratio of 1:26.4.
• Threat Subjects: Three distinct profiles:
  1. Negligent (Jordan): Single-surface, no concealment (secret in commit).
  2. Disgruntled (Tasha): Multi-day sentiment drift, cross-department snooping, unusual hours access.
  3. Malicious (Jax): Complex, multi-surface attack involving data exfiltration, social engineering, and active concealment (neutral tone).
• Injectable Behaviors: Eight behaviors across five surfaces (PR, Slack, Email, Host, IDP), including "Ghost Logins" (auth without downstream activity) and "Host Data Hoarding" (multi-day staging).

C. Detection Scenarios

Four scenarios are designed to defeat standard single-surface/single-day triage:

1. Ghost Logins: Requires reasoning about the absence of expected downstream activity.
2. Vishing: A phone call followed by an IDP auth on a victim's account. Requires cross-actor temporal joins (detecting the attacker via the victim's anomaly).
3. Host Data Hoarding: A three-phase trail (copy, compress, exfiltrate) spanning days. Single-day windows miss the full sequence.
4. Trust Building: A benign initial contact with a delayed follow-up attack, requiring long-term context retention.

D. Evaluation Pipeline

A three-stage pipeline evaluates 10 models (via AWS Bedrock):

1. Baseline Calibration: Establishes False Positive (FP) rates on clean pre-onset data.
2. Triage: A 7-day sliding window applies a "two-signal escalation threshold" to flag suspects.
3. Verdict: A Tier 2 investigator correlates full telemetry and produces a structured JSON verdict (Innocent, Suspicious, Likely Threat) with evidence citations.

3. Key Contributions

• Verifiable Synthetic Benchmark: The first insider threat corpus where ground truth is an architectural guarantee via a deterministic engine, eliminating cross-artifact contradictions.
• Multi-Surface/Temporal Scenarios: Introduction of scenarios (Ghost Logins, Vishing, Multi-day Hoarding) that specifically target the weaknesses of current single-surface detection strategies.
• SIEM Interoperability: Export formats (JSONL, CEF, ECS, LEEF) allow direct ingestion into major SIEMs (Splunk, Elastic, QRadar, Sentinel).
• Open-Source Leaderboard: A public leaderboard and evaluation harness (eval insider threat.py) enabling reproducible testing of any AWS Bedrock model.

4. Results & Findings

A leaderboard of 10 models (including Devstral 2, Claude Opus/Sonnet/Haiku, Llama 3.3/4, Mistral, DeepSeek, Qwen3-Coder) revealed four critical findings:

A. Triage/Verdict Dissociation

• Finding: Eight models achieved identical Triage F1 = 0.80, yet their Verdict F1 split into two tiers: Tier A (1.0) and Tier B (0.80).
• Implication: High triage performance (flagging suspects) does not guarantee high verdict accuracy (correctly identifying the threat vs. the victim).

B. Baseline False-Positive Rate (FP) is Critical

• Finding: Models with identical Verdict F1 (0.80) differed by two orders of magnitude in baseline FP rates.
  • Tier B (Operational): DeepSeek, Mistral Large, etc., had FP rates ≤ 0.023.
  • Operational Failure: Llama 3.3 70B had an FP rate of 0.813 (escalating 39 innocent employees), rendering it operationally useless despite matching the 0.80 Verdict F1.
• Implication: Verdict F1 alone is insufficient; baseline FP rate must be a primary metric.

C. Victim Attribution as the Tier A/B Boundary

• Finding: The specific capability separating Tier A from Tier B is victim attribution in the Vishing scenario.
  • Tier A (Devstral 2, Opus): Correctly identified that the anomalous session on "Chris's" account was inconsistent with his normal device profile (iOS vs. macOS) and concurrent normal activity, exonerating him as the victim.
  • Tier B: Detected the vishing link but misclassified the victim (Chris) as the threat, producing the single false positive that capped their Verdict F1 at 0.80.
• Implication: Detecting an anomaly is not enough; the model must reason about behavioral baselines to distinguish attacker access from victim compromise.

D. Prompt Sensitivity & Vocabulary Hallucination

• Finding: Unstructured prompts induced "vocabulary hallucination" across all tiers. Models reasoned correctly but used non-canonical labels (e.g., "ghost logins" instead of "unusual hours access"), resulting in zero credit under exact-match scoring.
• Implication: A two-track scoring framework is proposed: one for exact-match (leaderboard) and one for semantic similarity (capability auditing).

5. Significance

• Operational Reality Check: The paper demonstrates that high accuracy metrics (F1) can mask operational failures (high false positives) and reasoning gaps (victim misattribution).
• Architectural Necessity: It proves that SOC thresholds must be threat-class-specific; a "two-signal" threshold designed for APTs will miss negligent insiders (single-signal events).
• Future of Evaluation: OrgForge-IT establishes a new standard for synthetic data generation where factual consistency is enforced by a state machine, not just hoped for in LLM generation. It provides a rigorous, reproducible testbed for advancing LLMs from "pattern matchers" to true "SOC investigators" capable of multi-hop reasoning and victim attribution.

Availability: The code, simulation framework, and dataset are open-source under the MIT license at github.com/aeriesec/orgforge and huggingface.co/datasets/aeriesec/orgforge-insider-threat.