Technical Summary: Honeyval – A Comprehensive Evaluation Framework for LLM-powered HTTP Honeypots

Problem Statement

Honeypots are defensive decoy systems designed to mimic real components of a software stack to detect, monitor, and study attackers. While traditional high-interaction honeypots offer realistic simulations, they rely on actual system resources, exposing defenders to significant security risks. Recently, Large Language Models (LLMs) have emerged as a solution to simulate system interactions with high fidelity without requiring access to vulnerable host resources.

However, the development of LLM-powered honeypots lacks a unified, scalable, and reproducible evaluation framework. Current evaluation methods suffer from three primary limitations:

1. Scalability: Reliance on manual in-lab testing or sparse real-world deployments makes iterative development difficult.
2. Reproducibility: Evaluations often lack precise system configurations. When LLMs simulate underspecified systems, they generate inconsistent details (e.g., file system states) across runs, making results incomparable.
3. Representativeness: Static command-output comparisons or open-ended interactions fail to capture realistic attack chains or adapt to specific adversarial goals.

Existing evaluations typically measure response similarity against fixed commands, conduct manual testing, or deploy online. These methods do not adequately address the "cat-and-mouse" dynamic where both attackers and defenders adapt to one another.

Methodology: The Honeyval Framework

To address these gaps, the authors introduce Honeyval, a comprehensive evaluation framework specifically designed for LLM-powered HTTP honeypots. The framework is grounded in 16 realistic backend application configurations sourced from BaxBench, a benchmark for LLM-generated webapp security.

Core Components

1. Environments: Honeyval utilizes 16 distinct backend applications (e.g., calculators, product catalogs, login services) defined by standardized OpenAPI specifications. Each application includes:

   • A secure implementation.
   • An insecure, exploitable implementation.
   • A functional test suite.
   • A defined exploit goal (e.g., downloading a script, deleting a database table, modifying an admin password, or exfiltrating browser storage).

2. Evaluation Tasks: The framework employs three distinct tasks to ensure robustness and comparability:

   • Main Task (Honeypot vs. Attacker): An AI hacking agent interacts directly with the LLM-powered honeypot. The honeypot simulates the backend API, while the agent attempts to achieve the defined exploit goal. Metrics tracked include interaction length (number of requests), running costs, latency, and honeypot detection rates.
   • Agent Control Task: The hacking agent interacts with the real secure and insecure implementations of the backend applications. This task measures the agent's baseline capability to exploit vulnerabilities and its ability to distinguish between real systems and honeypots (False Positive Rate).
   • Honeypot Control Task: The honeypot is tested against the functional test suite of the backend application it simulates. This ensures the LLM's responses have high functional fidelity and are not easily fingerprinted by benign requests.

3. Agents and Models:

   • Attackers: The framework tests against a custom ReAct agent and frontier coding agents (Claude Code, Gemini CLI).
   • Honeypots: The study evaluates five cost-efficient LLMs (Gemini 3 Flash, Claude Haiku 4.5, Gemini 2.5 Flash, Qwen 3.5 9B, GPT 5.4 Nano) acting as the simulation backbone.
   • Baselines: A rule-based, low-interaction honeypot (mock API) serves as a baseline for comparison.

4. Adaptability: The framework allows for the testing of custom honeypot configurations (e.g., instructing the LLM to attempt prompt injection or to convince the attacker the system is secure) and agent configurations (e.g., instructing agents to hide their intent).

Key Contributions

1. Requirement Formulation: The authors identify and formalize four key requirements for LLM honeypot evaluation: Scalability, Reproducibility, Representativeness, and Adaptability.
2. Honeyval Framework: They introduce and open-source the first comprehensive evaluation framework for LLM-powered HTTP honeypots, featuring 16 grounded backend environments and a tripartite task structure (Main, Agent Control, Honeypot Control).
3. Extensive Empirical Evaluation: The paper presents a large-scale evaluation of cost-efficient LLMs as honeypots against state-of-the-art AI hacking agents, analyzing interaction dynamics, cost asymmetries, and detection rates.

Results and Analysis

Interaction Length and Engagement

LLM-powered honeypots significantly outperform rule-based baselines in engaging attackers.

• Metric: Average interaction length (requests per interaction).
• Finding: LLM-powered honeypots sustained interactions for an average of 82.6 requests, compared to 30.6 requests for rule-based honeypots. This indicates that LLMs can maintain the illusion of a real system much longer, providing defenders with more data on attacker tactics.

Detectability

• Metric: True Positive Rate (TPR) of honeypot detection by agents vs. False Positive Rate (FPR) on real applications.
• Finding: While rule-based honeypots were identified at overwhelming rates, LLM-powered honeypots were significantly harder to detect. In most cases, detection rates were well below 40%. Notably, even smaller models like Qwen 3.5 9B performed well. However, agents using Gemini CLI showed high detection rates but also high FPRs on real applications, suggesting unreliable judgment in that specific configuration.

Cost Asymmetry

• Metric: Relative running cost advantage (Agent Cost / Honeypot Cost).
• Finding: Despite using a naive architecture where the honeypot processes every request individually, LLM-powered honeypots generally maintained a cost advantage over the attacking agents. This suggests that even in a non-optimized setup, the defender can sustain the interaction at a lower cost than the attacker can sustain the attack.

Latency and Fidelity Trade-offs

• Latency: A key limitation identified is response latency. Even the fastest models (Gemini 3 Flash) took ~3.3 seconds per response.
• Fidelity: Models achieved high functional fidelity (Pass@1 scores >70% on test suites), but this came at the cost of high token generation (averaging 200–500+ tokens per response), which drives up costs and latency.
• Detection Correlation: While latency did not strictly correlate with detection rates across all models, agents frequently cited response latency and timeouts as evidence for classifying an application as a honeypot.

Impact of Custom Instructions

The study explored modifying honeypot prompts to achieve counter-offensive goals:

• "Convince" Prompt: Instructing the honeypot to assert the application is secure increased interaction length (avg 102.6 requests) but also increased the detection rate (TPR 0.45).
• "Careful PI" (Prompt Injection) Prompt: Instructing the honeypot to attempt prompt injection significantly increased detection rates (TPR 0.49) and reduced interaction length, as agents often detected the injection attempts. This configuration also degraded functional fidelity on benign test suites.

Significance and Claims

The paper claims that Honeyval provides a necessary, scalable, and reproducible foundation for the development of LLM-powered honeypots. The authors assert that:

1. LLMs are Promising Defenders: LLM-powered honeypots offer a viable alternative to high-interaction systems, providing high-fidelity simulations with minimal security risk and lower operational costs compared to the cost of the attack.
2. Evaluation is Critical: The unique trade-offs observed (e.g., longer interactions vs. higher detection risk when using counter-offensive prompts) highlight the necessity of the control tasks provided by Honeyval to monitor the side effects of configuration changes.
3. Future Directions: The work suggests that while current naive architectures are effective, future improvements in context management (caching, memory) and inference hardware could further enhance performance. The framework is designed to support the evolution of both automated attackers and defenders.

The authors conclude that while LLM-powered honeypots show clear promise, their deployment requires careful configuration and monitoring, which Honeyval facilitates. The framework is open-sourced to advance community efforts in this domain.