💬 NLP

Black-Box Forensics for Conversational LLM Agents

This paper presents a black-box forensic framework for conversational LLM agents that achieves high-accuracy attribution of base models and robust fingerprinting of unseen system prompts using only a few turns of non-adversarial conversation, thereby enabling the tracing of AI-enabled scams to their providers and the linking of criminal networks.

Original authors: Isadora White, Yasaman Jafari, Taylor Berg-Kirkpatrick

Published 2026-06-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Isadora White, Yasaman Jafari, Taylor Berg-Kirkpatrick

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the internet is a giant, bustling marketplace. In this market, there are thousands of "chatbots" (conversational AI agents) that look and sound exactly the same to a human user. They might be helpful customer service bots, or they might be scammers trying to trick you. The problem is that these bots are black boxes: you can talk to them, but you can't see inside their brains. You don't know which computer model they are running, and you don't know the secret instructions (the "system prompt") the owner gave them.

This paper introduces a new set of "detective tools" that can figure out who these bots really are just by having a normal, polite conversation with them. No hacking, no special codes, just talking.

Here is how the paper's two main tools work, explained with simple analogies:

1. Attribution: Identifying the "Brand" of the Bot

The Analogy: Imagine you are tasting a soup. You can't see the chef, and you don't know the recipe. But, you know that Chef A always uses a lot of salt, Chef B always makes the broth very thick, and Chef C always adds a hint of lemon. Even if you don't know the exact dish, you can guess which chef made it based on the flavor.

What the paper does:
The researchers built a system that acts like a "flavor taster" for text. They trained it to listen to a few turns of conversation and guess which base model (the underlying AI engine, like GPT-4 or Llama) is powering the bot.

The Result: They can identify the specific "brand" of the AI with 98% accuracy just from a few sentences of normal chat. They can even tell the difference between two very similar models from the same family (like a small version vs. a large version).

2. Fingerprinting: Catching the "Secret Recipe"

The Analogy: Now, imagine two different restaurants both claim to serve "Grandma's Secret Chili." You can't ask for the recipe (it's a trade secret), but you want to know: Are these two restaurants actually using the exact same secret recipe, or are they just pretending?

What the paper does:
This is the paper's big breakthrough. Usually, to know if two things are the same, you need to see them both before. But here, the researchers built a tool that can look at two completely new chatbots (that it has never seen before) and say, "Yes, these two are running the exact same secret instructions," or "No, they are different."

How it works: They use a "Cross-Encoder," which is like a super-observer that reads two conversations side-by-side and looks for subtle, invisible patterns in how the bots speak, pause, and structure their answers.
The Result: Even if the secret instructions have never been seen by the detective before, the tool can match them with about 77% accuracy using just a few turns of chat. If the detective collects 50 conversations from the same bot, the accuracy jumps to 94%.

Why is this important?

The paper suggests three main uses for these tools, all focused on catching bad actors or ensuring safety:

Cracking Scam Networks: If a scammer uses a bot today, and then uses a different bot next week, this tool can tell investigators, "Hey, these two bots are using the same secret instructions." This links individual scams together into a single criminal network.
Spotting Silent Switches: Companies sometimes quietly swap the AI model they use (e.g., switching from a smart, expensive model to a cheaper, dumber one) without telling their customers. This tool can detect that "silent drift" just by listening to the bot's behavior change over time.
Better Security Testing: If you are trying to test a bot for weaknesses (like "jailbreaks"), you need to know exactly which model you are fighting. This tool tells defenders, "You aren't fighting a generic bot; you are fighting a specific model with a specific personality," so they can tailor their defense.

The "Detective" Method

A key part of this paper is how they talk to the bots. Traditional hacking tries to trick the bot with nonsense or confusing questions. This paper says, "No, let's be polite."

They use a "Detective Agent" that just has a normal, friendly conversation (like asking about customer support or negotiating a price).
Because the conversation is normal, the bot doesn't get suspicious and try to hide. This allows the detective to hear the bot's "natural voice" and find the hidden fingerprints.

The Catch (Limitations)

The paper is very honest about its limits:

It's a Simulation: They didn't test this on real, live scammers on the internet (because that's illegal and dangerous). They created a massive library of 240,000 fake conversations using different models and made-up secret instructions to train their tools.
It Needs a Little Data: While it works with just a few sentences, it gets much better if you have more conversations (up to 50) to analyze.
It's Not Magic: If someone tries to disguise the bot by rewriting its answers using another AI, the tool gets a bit less accurate, but it's still better than many other methods.

In short: This paper gives investigators a way to identify the "make and model" of a chatbot and link different bots together by their "secret recipe," all by just having a polite chat with them. It turns the invisible world of AI backdoors into something that can be tracked and held accountable.

Technical Summary: Black-Box Forensics for Conversational LLM Agents

Problem Statement

As Large Language Model (LLM) adoption expands, so does the proliferation of AI-assisted fraud, social engineering, and misinformation campaigns. These malicious agents often operate behind anonymous endpoints where investigators lack access to model weights, system prompts, or internal configurations. The paper addresses the challenge of black-box forensics: recovering identifying information about an agent's base model and system prompt solely through conversational interaction, without adversarial probing or access to internal states.

The authors define two complementary capabilities required for this task:

Attribution: Identifying which base model or system prompt powers a specific endpoint from a fixed, closed set of candidates.
Fingerprinting: Determining whether two distinct conversation transcripts originate from the same hidden configuration (base model + system prompt), even if that specific configuration was never seen during training (zero-shot).

This capability is critical for linking individual scams to criminal networks, detecting silent API changes (e.g., providers swapping backend models without disclosure), and tailoring red-teaming efforts to specific model vulnerabilities.

Methodology

The paper proposes a framework that relies on active elicitation rather than passive observation. A "detective" agent (powered by QWEN-4B-INSTRUCT) initiates and steers ordinary, non-adversarial conversations with a "target" agent. This approach avoids the detectability of traditional adversarial strings or prompt injections, allowing the forensic tool to remain covert.

Data Generation

Due to the infeasibility of obtaining labeled real-world data (which would require extracting proprietary prompts or relying on unverified scraped data), the authors constructed a synthetic corpus of 240,000 labeled transcripts.

Base Models: Six models were used: GPT-OSS-20B, GPT-OSS-120B, LLAMA-3.1-8B-INSTRUCT, QWEN-4B-INSTRUCT, GPT-4O-MINI, and GPT-4.1-NANO.
System Prompts: 40 distinct system prompts were curated, covering customer support and negotiation scenarios.
Variations: To test sensitivity, the authors created controlled variants of prompts, including severe truncation, semantic paraphrasing, and high-density summaries.
Control: All interactions used standardized generation settings (temperature 0.7, max 512 tokens) to isolate stylistic and structural fingerprints from semantic noise.

Technical Approaches

1. Attribution

The authors adapted authorship attribution techniques to conversational data:

Sparse Baseline: Utilized unigram/character-level TF-IDF and stylometric features (punctuation frequency, utterance length) with a multinomial logistic regression classifier.
Dense Baseline: Fine-tuned QWEN-4B-INSTRUCT using LoRA adapters to act as a classifier.
Task: The system attempts to map a conversation to a specific base model or system prompt from a known candidate set.

2. Fingerprinting (Zero-Shot)

The core innovation is a cross-encoder approach designed to match two conversations to the same hidden configuration without prior training on that specific configuration.

Architecture: The authors leveraged ELECTRA-large and BERT-base as cross-encoders. These models encode pairs of conversations and output log probabilities for a "same vs. different" classification.
Training Objective: The model is trained to distinguish between pairs of conversations from the same agent configuration versus pairs from different configurations (different prompts on the same base model).
Input Handling: Due to context window limits (512 tokens), conversations were truncated to the first three turns.
Scaling: The method evaluates performance by aggregating multiple conversation pairs ( $k$ ) from the same target to boost confidence.

Key Results

Attribution Performance

Base Model Attribution: The system achieves 98% accuracy in identifying the base model from a few turns of non-adversarial conversation. It successfully distinguishes between models of the same family but different sizes (e.g., GPT-OSS-20B vs. GPT-OSS-120B) with 95% accuracy.
System Prompt Attribution: While possible, accuracy is highly dependent on the base model's responsiveness to instructions and the semantic similarity of the prompts.
- On models like QWEN-4B-INSTRUCT and GPT-4.1-NANO, pairwise prompt attribution reaches >90% accuracy.
- On models like GPT-OSS-20B and GPT-OSS-120B, attribution accuracy drops significantly, and the correlation between prompt semantic similarity and detectability is negligible ( $r < 0.2$ ).
- Multi-way classification of prompts proved significantly harder than binary classification, with fine-tuned classifiers struggling to exceed 63% accuracy.

Fingerprinting Performance

Zero-Shot Capability: The cross-encoder method achieves an AUC of 0.768 and an F1 score of 0.703 on entirely unseen system prompts (zero-shot setting).
Scaling with Evidence: Performance improves substantially with more data. Aggregating 50 conversation pairs per target agent boosts the AUC to 0.943 and F1 to 0.77. Even with only 5 pairs, the AUC reaches 0.83.
Baseline Comparison: The cross-encoder significantly outperformed baselines, including N-gram overlap (AUC 0.59), simple logistic regression (AUC 0.667), and bi-encoders (AUC ~0.75).

Robustness

Topic and Agent Shifts: The method is robust to changes in conversation topics and the identity of the detective agent (e.g., using GPT-OSS-20B as the detective instead of QWEN-4B-INSTRUCT), with AUC drops of less than 0.03.
Sampling Parameters: Variations in temperature and token limits caused minimal performance degradation (<0.02 AUC drop).
Adversarial Perturbations:
- Paraphrase Attacks: Using a proxy LLM to rephrase utterances reduced performance by 0.10 AUC, but the method remained effective (AUC 0.94 with $k=10$ ).
- Context Length: Performance degrades significantly with shorter contexts (1-turn conversation drops AUC by 0.20) and in later stages of long interactions, though majority voting can recover some performance.

Significance and Claims

The paper claims to be the first to study system prompt attribution and fingerprinting in a conversational black-box setting without access to model weights, system prompts, or ground-truth training conversations.

Key contributions include:

Covert Forensics: Demonstrating that high-fidelity attribution and fingerprinting can be achieved through ordinary, non-adversarial dialogue, avoiding the detection risks associated with prompt injection or adversarial probing.
Scalability to the Open World: Unlike attribution methods that require a fixed candidate set, the fingerprinting approach scales to the "open world," allowing investigators to cluster distinct scam campaigns or detect silent API drifts without retraining on the new configuration.
Practical Utility: The results suggest that aggregating a small number of interactions (e.g., 50 turns) is sufficient to link disparate scams to criminal networks or identify silent model swaps with high confidence.

The authors position this work as a necessary tool for platform governance and security teams, enabling them to trace AI-enabled scams back to providers and detect unannounced backend changes that could materially shift downstream behavior.

Limitations and Ethical Considerations

The authors acknowledge that their evaluation relies on a synthetic corpus, which may serve as a lower bound for real-world performance given the diversity of wild system prompts. They emphasize that the methodology must be strictly restricted to auditing public-facing commercial APIs and investigating coordinated, mass-scale malicious operations. To mitigate surveillance risks, they propose operational protocols including pre-flight target validation, data minimization, redaction of PII, and mandatory auditability.