Technical Summary: Agentic and Generative AI for Open-Source Intelligence and Cyber Investigations

1. Problem Statement

The rapid proliferation of publicly available digital information has rendered traditional manual Open-Source Intelligence (OSINT) analysis insufficient for contemporary cybersecurity and cyber investigation requirements. While Large Language Models (LLMs) and Agentic AI systems offer the potential to automate multi-step reasoning, tool selection, and intelligence synthesis, a critical structural imbalance exists in the literature: capability demonstrations have substantially outpaced evaluation infrastructure.

The field suffers from three primary deficits:

1. Evaluation Gap: Systems are proposed and described, but lack standardized benchmarks, reproducible datasets, and adversarial test conditions.
2. Hallucination-Validation Gap: While hallucination is widely acknowledged as a primary reliability concern in over twenty studies, it is empirically measured in only one OSINT-specific system (a 4% rate under favorable conditions).
3. Workflow Imbalance: Research heavily favors collection and analysis stages, while verification, reporting, dissemination, and decision support remain systematically underexplored.

Furthermore, there is a tension between cloud-based deployment (which offers scalability but introduces operational security and GDPR risks) and on-premise deployment (which ensures data sovereignty but may limit model scale). Similarly, benchmark performance (e.g., multiple-choice knowledge recall) often fails to predict operational readiness in complex, open-ended threat intelligence workflows.

2. Methodology

This systematic literature review follows PRISMA-style principles to construct and analyze a corpus of 74 unique studies published between 2014 and 2026.

• Search Strategy: The corpus was assembled via expert curation across ACM Digital Library, IEEE Xplore, arXiv, Google Scholar, and institutional repositories, guided by a conceptual search string covering OSINT, LLMs, agentic AI, RAG, and related domains.
• Inclusion Criteria: Studies were included if they applied LLMs, generative AI, or agentic systems to OSINT, Cyber Threat Intelligence (CTI), or Cyber Investigation, or made direct methodological contributions (e.g., prompt engineering, evaluation frameworks) transferable to these domains.
• Analytical Tiers: The 74 studies were classified into four tiers:
  • Primary/Core (37 studies): Direct applications of AI to OSINT/CTI tasks.
  • Methodological Support (11 studies): Contributions to prompting, fine-tuning, or benchmarking.
  • Background/Context (22 studies): Foundational LLM papers (e.g., scaling laws, chain-of-thought).
  • Excluded/Peripheral (4 studies): Non-OSINT or non-AI relevant works.
• Data Extraction: A structured 21-field evidence matrix was applied to the 48 primary and methodological papers. Key fields included hallucination measurement status, evaluation metrics, workflow stages covered, and adversarial robustness.
• Synthesis: The review employs thematic and comparative synthesis rather than quantitative meta-analysis, organizing findings into an 11-category taxonomy and mapping them against the OSINT workflow lifecycle.

3. Key Contributions

The paper makes four primary contributions to the literature:

1. 11-Category Taxonomy: It organizes the literature into a distinct taxonomy separating Agentic AI (autonomous tool selection, multi-step reasoning) from simple LLM prompting. The categories span:
   • Foundations (LLMs, Transformers).
   • OSINT Workflows (Collection, Extraction).
   • Agentic Architectures (ReAct, Tool Orchestration).
   • Knowledge Infrastructure (RAG, Knowledge Graphs).
   • Domain Applications (CTI, Dark Web).
   • Risks and Ethics (Hallucination, Adversarial Attacks).
2. Identification of the Hallucination-Validation Gap: The review establishes as a corpus-level finding that while hallucination is cited as a major risk in >20 studies, only one study [6] provides an empirical, end-to-end hallucination rate (4%) for an OSINT-specific system. All other error measurements are derived from general-domain QA or reasoning chains, not OSINT-specific pipelines.
3. Workflow Coverage Analysis: The review maps the corpus to the OSINT lifecycle, revealing a structural imbalance. Collection and Analysis are well-served, whereas Verification, Reporting, Dissemination, and Decision Support are significantly underexplored, creating a gap between technical capability and operational intelligence reliability.
4. Adversarial Risk Synthesis: It synthesizes evidence showing that every major agentic capability has a corresponding adversarial dual-use risk (e.g., Knowledge Graph poisoning, adversarial CTI generation) that is documented in the risk literature but never addressed in the evaluation design of the system papers themselves.

4. Key Results and Findings

A. Technical Capabilities vs. Operational Reality

• Collection & Extraction: LLMs demonstrate high performance in binary classification (F1 ~0.94 for GPT-4 on CTI data) and basic entity extraction. However, Named Entity Recognition (NER) for cybersecurity-specific entities (IPs, CVEs, malware families) falls approximately 25% below specialized pre-trained models in zero-shot/few-shot settings.
• Agentic Systems: Systems utilizing the ReAct (Reason + Act) paradigm can autonomously orchestrate tools (Shodan, Maltego, VirusTotal) and integrate Knowledge Graphs (Neo4j) and Dark Web (Tor) sources. However, these systems are evaluated exclusively under benign conditions, assuming input data is genuine.
• Hallucination: The only empirical measurement of hallucination in an OSINT system is 4% (in a RAG-augmented architecture using a private, curated dataset). This figure represents an empirical floor under favorable conditions, not a transferable benchmark.
• Evaluation Discrepancy: A stark contrast exists between benchmark performance and operational performance. CyberMetric [17] shows GPT-4o achieving 91.25% accuracy on multiple-choice cybersecurity questions (surpassing human experts), whereas CyberThreat-Eval [3], using a realistic three-stage analyst workflow (triage, deep search, report drafting), finds no LLM achieves analyst-acceptable performance on complex threat attribution or report quality.

B. Risks and Vulnerabilities

• Adversarial CTI: Fine-tuning a modest GPT-2 model on OSINT-sourced data can generate fake threat intelligence that professional threat hunters judge as genuine at rates comparable to authentic CTI.
• Knowledge Graph Poisoning: Adversarial content ingested via standard NER pipelines can corrupt Knowledge Graphs, turning grounding mechanisms (RAG) into amplification mechanisms for error.
• Privacy & Legal: Cloud-based API deployment poses GDPR risks for personal data. Automated Dark Web collection lacks legal frameworks for chain-of-custody and admissibility.
• Human Oversight: Analysts with high CTI experience often have low LLM familiarity (median 2/5), and current human oversight mechanisms are not empirically validated to detect adversarial content or hallucinations effectively.

C. Deployment Architecture

The review finds no standardized, open benchmark for cross-study comparison. Consequently, the most defensible near-term deployment architecture is a structured human-AI co-pilot model. In this model:

• LLMs support collection, triage, and preliminary analysis.
• Human analysts retain responsibility for verification, reporting, and final decision support.
• This architecture is necessary given the lack of validated reliability, provenance tracking, and adversarial robustness in current autonomous systems.

5. Significance and Claims

The paper claims its significance lies in shifting the field's focus from capability demonstration to validated reliability. It argues that the current literature is characterized by a "structural imbalance" where architectural ambition outpaces the methodological infrastructure required for responsible operational deployment.

The review does not argue against the use of LLMs in OSINT but asserts that capability without validated reliability, provenance tracking, ethical safeguards, and structured human oversight is insufficient. It identifies the hallucination-validation gap as the central failure of the field's evaluation culture.

Based on the evidence, the authors propose a ten-point research agenda to address these gaps, including:

1. Standardized hallucination measurement protocols.
2. Open, community-adopted OSINT-AI benchmark suites.
3. Evaluation of API reliability and orchestration security under operational conditions.
4. Development of multilingual, multi-platform datasets.
5. Legally admissible Dark Web collection methodologies.
6. Responsible disclosure norms for offensive/defensive AI research.
7. Structured LLM literacy training for practitioners.
8. Empirical evaluation of human oversight effectiveness.
9. Mandatory open datasets and code repositories.
10. Comprehensive legal and governance frameworks.

The paper concludes that Agentic and Generative AI for OSINT is a present capability waiting to be responsibly validated, and the path forward requires a shift from technical demonstration to operational trustworthiness through rigorous, adversarial, and reproducible evaluation.