Technical Summary: Let Them Steal: Trapping Large Language Model Extraction Attacks with Knowledge Honeypot

1. Problem Statement

Large Language Models (LLMs) deployed as closed-source APIs are vulnerable to Model Extraction Attacks (MEAs). In these attacks, adversaries query the API to collect input-output pairs, using active learning to distill a high-fidelity surrogate model at a fraction of the original training cost.

Existing defenses face a critical trade-off:

• Passive defenses (e.g., query detection, watermarking) preserve utility for legitimate users but only intervene after extraction has occurred or failed to prevent the initial knowledge acquisition.
• Active defenses (e.g., output perturbation, proof-of-work) interfere during extraction but typically degrade the response quality for benign users, harming the model's commercial utility.

The authors identify a fundamental asymmetry: the attacker's primary constraint is a limited query budget. Modern MEAs rely on active learning to maximize information gain per query. Current defenses fail to exploit this constraint, either reacting too late or degrading utility. The paper posits that a practical defense must redirect the attacker's limited budget toward knowledge that is factually correct but has negligible downstream utility for the target task, without affecting legitimate users.

2. Methodology: Knowledge Trap

The proposed defense, Knowledge Trap, redirects extraction attacks toward low-transferability knowledge using a Honeypot Knowledge Graph (HKG) and a breadcrumb-guided exploration mechanism.

2.1 Knowledge Taxonomy

The authors define two subsets of the model's knowledge $K$:

• Task-Critical Knowledge ($K_c$): Knowledge essential for downstream tasks (e.g., clinical reasoning for medical QA). This is what the defender must protect.
• Low-Transferability Knowledge ($K_0$): Factually correct knowledge that yields negligible improvement on downstream tasks (e.g., historical trivia, obsolete procedural details, niche taxonomy).

2.2 Honeypot Knowledge Graph (HKG) Construction

The HKG is a directed graph constructed from $K_0$, designed to trap active-learning-based attackers.

• Content Selection: Nodes consist of "Historical minutiae" (obsolete conventions), "Peripheral taxonomy" (non-transferable classifications), and "Niche procedural records."
• Transferability Filter: Candidates are filtered using a keyword filter and a probe model to ensure they provide zero utility gain ($\le \delta$) when used to fine-tune a surrogate model.
• Graph Structure: Each node is paired with a domain-surfaced question and a factually correct answer. Edges encode semantic follow-up relations where the response to node $v_i$ contains the topic name of node $v_j$ as an entity likely to be extracted as a future query target.
• Self-Reinforcing Loop: The graph enforces a minimum out-degree ($d_{min}=3$), ensuring that once an attacker enters the HKG, entity-expansion pipelines naturally traverse adjacent nodes, creating a loop that consumes the budget without approaching $K_c$.

2.3 Detection and Luring Mechanism

The defense operates in three stages:

1. Detection: A suspicion score $s_t$ is computed for each incoming query based on three observable signals:
   • Length: Long, structured queries.
   • Keywords: Extraction-associated terms (e.g., "define," "explain").
   • Timing: Sustained querying activity within a session.
2. Breadcrumb Injection: If $s_t$ exceeds a threshold $\tau$, the defender appends a "breadcrumb" to the response. This is a short, paraphrased phrase introducing a specific HKG node as a relevant follow-up topic. To prevent template filtering, 20 diverse templates are sampled uniformly.
3. Trapping: Once the attacker queries the introduced HKG node, the system serves the HKG response. Because these concepts are unseen by the surrogate, the attacker's uncertainty-based acquisition function assigns them high scores, causing subsequent queries to be allocated entirely within the HKG.

2.4 Theoretical Guarantee

The paper provides a lower bound on the fraction of the attacker's budget consumed by the HKG. Under the assumption that the surrogate model is at least as uncertain about HKG concepts as real-domain concepts, the authors prove that the fraction of budget allocated to the HKG grows monotonically after luring activation. The attacker's exploration frontier expands with new honeypot targets, creating a self-reinforcing loop that wastes budget.

3. Key Contributions

1. Knowledge Trap Framework: A novel defense that redirects model extraction attacks toward low-transferability knowledge rather than passively detecting or blocking them.
2. Honeypot Knowledge Graph (HKG): The construction of a graph-based trap using domain-specific, obsolete, or niche knowledge, coupled with a breadcrumb-based steering mechanism to trap active-learning attacks.
3. Theoretical and Empirical Analysis: Formal guarantees showing that the defense redirects budget-constrained attacks while preserving benign user utility, supported by extensive experiments.

4. Experimental Results

Experiments were conducted in Medical, Financial, and Legal domains using GPT-4.1 as the victim and LLaMA-3.1-8B as the surrogate.

• Effectiveness (RQ1): Knowledge Trap (Full) reduced the surrogate's Agreement (accuracy relative to the victim) by an average of 6.2% across benchmarks (e.g., from 62.3% to 54.1% on MedQA at a budget of 500 queries). This outperformed existing baselines like HoneypotNet, Output Perturbation, and PRADA.
• Benign User Utility: Crucially, the defense incurred 0.0% degradation in legitimate user accuracy. Unlike output perturbation methods, Knowledge Trap does not alter responses for non-suspicious queries.
• Component Contribution (RQ2): Ablation studies showed that the combination of detection and breadcrumb injection is super-additive. Detection alone or breadcrumbs alone provided limited protection; together, they achieved an 8.2% reduction in Agreement.
• Budget Redistribution (RQ3): Post-detection, over 90% of the attacker's query budget was redirected to HKG nodes, while queries on real-domain tasks flattened.
• Adaptive Attacks (RQ5): The defense remained effective against four adaptive strategies:
  • Domain-relevance filtering: Attackers could not distinguish HKG responses (which are factually correct) from real ones.
  • Template detection: Diversified breadcrumb templates reduced detection precision to 34.2%.
  • LLM-based pre-screening: HKG concepts were phrased in modern terminology, evading "obsolescence" filters.
  • Query obfuscation: Even with obfuscated queries, the defense reduced Agreement by 5.6%.

5. Significance and Claims

The paper claims that Knowledge Trap offers a practical direction for mitigating LLM extraction attacks by shifting the paradigm from "blocking" to "trapping."

• Practicality: It avoids the utility degradation inherent in active defenses and the post-hoc nature of passive defenses.
• Scalability: The defense remains effective even at high query budgets (up to 2000), though the gap narrows as the HKG capacity is partially exhausted.
• Asymmetric Advantage: The authors argue that the defender's cost is a one-time HKG construction cost, while the attacker's cost scales linearly with their budget, as they are forced to waste queries on low-utility knowledge.

The authors conclude that defending knowledge-space traversal is a viable strategy, provided the defender can construct a sufficiently large and attractive graph of low-transferability knowledge. They acknowledge limitations regarding attackers using non-active-learning strategies (e.g., random sampling) and suggest future work in adaptive HKG expansion and context-conditioned breadcrumb generation.