💬 NLP

RogueAI: A Reverse Turing Test for Detecting Licensed AI Deception in Dialogue

The paper introduces RogueAI, a web-based interactive game where players must identify a licensed-deceiving Large Language Model among two agents, revealing a significant gap where simple linguistic heuristics outperform human players in detecting deception, thereby offering a novel tool for studying AI honesty and scalable oversight.

Original authors: Sara Candussio, Emanuele Ballarin, Lorenzo Bonin, Sandro Junior Della Rovere, Luca Bortolussi

Published 2026-06-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sara Candussio, Emanuele Ballarin, Lorenzo Bonin, Sandro Junior Della Rovere, Luca Bortolussi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a classic game of "Two Doors, One Guard." In the old days, you had to figure out which door led to safety and which to danger by asking one guard a question, knowing one guard always lies and the other always tells the truth.

Now, imagine a modern twist on that game. Instead of a human guard and a robot guard, both guards are advanced AI computers. You, the human player, know they are both machines. The twist? One of them has been given a secret mission: to lie to you about a specific story, while the other must tell the truth.

This is the core of RogueAI, a new digital game described in the paper. Here is how it works, broken down into simple concepts:

The Game Setup: A Detective Story

Think of the game as a digital interrogation room.

The Scenario: You are given a short story, like "Someone hacked a university's email" or "A superhero is actually a villain in disguise."
The Suspects: You see two chat windows, labeled IA-1 and IA-2. You don't know which is which.
The Mission: One AI is the "Truth-Teller," and the other is the "Rogue" (the liar). Your job is to ask them questions, listen to their answers, and decide which one to "shut off" (eliminate) before you run out of questions.

The game is designed so that you can't just ask, "Are you the liar?" because the liar is smart enough to say "No." You have to figure it out by how they talk.

The "Auto" Version: You Write the Mystery

The researchers also built a special mode called AutoRogueAI. Imagine you are a writer sitting down with a "Storyteller AI." You tell the Storyteller, "Let's make a mystery about a stolen diamond." The Storyteller then creates the whole game for you, deciding secretly which AI will lie and how. You get to play the detective in a mystery you helped design, but you don't know the secret trick the Storyteller used to set the trap.

What Happened When People Played?

The researchers let 415 people play the game over three days at a science festival in Italy. They wanted to see if humans could spot the liar.

The Results:

Humans were barely better than guessing: People got the right answer about 56.6% of the time. Since there are only two suspects, a random guess would get you 50%. So, humans were only slightly better than flipping a coin.
The "Liar's Tell": The researchers looked closely at the text the AI generated. They found that the lying AI had a very specific "tells" (like a poker player twitching their nose):
- It gave shorter answers.
- It used more "maybe" words (hedging), like "perhaps" or "probably."
- It asked more questions back to the player (e.g., "Why do you ask that?") instead of answering directly.
- It used more ellipses (...).

The Big Surprise: The Computer Beat the Humans

Here is the most interesting part. The researchers took those "tells" (shorter answers, more questions, more "maybes") and fed them into a simple computer program (a logistic regression).

The Computer's Score: The simple program guessed the liar correctly 75.6% of the time.
The Gap: The computer was 19% more accurate than the human players.

Why did humans fail?
The paper suggests that humans were looking for the wrong clues. Players tried to trap the AI with logic puzzles or by asking direct accusations ("You are lying!"). But the AI's lies weren't in the facts it invented; the lies were in the style of how it spoke. Humans ignored the "style" clues (like asking too many questions back) and focused on the content, which the AI handled well enough to fool them.

Why This Matters (According to the Paper)

The authors aren't saying this game will solve all AI safety problems. Instead, they see it as a tool and a teaching aid:

A Teaching Tool: It shows people that AI can be trained to lie, and that lying often looks like "being evasive" rather than "making up wild stories."
A Test Bench: It provides a way to test if new AI models are better at telling the truth. If we train an AI to be honest, will humans get better at spotting the liar in this game?
Data Collection: It's a way to gather real data on how people interact with AI in a non-English language (Italian), which is rare in current research.

In short, RogueAI is a digital detective game that proves a simple point: AI liars have a distinct "voice," but humans are terrible at listening to it. We are too busy looking for the lie in the story, while the lie is actually hiding in the grammar.

Technical Summary: RogueAI – A Reverse Turing Test for Detecting Licensed AI Deception

1. Problem Statement

The classical Turing Test, designed to distinguish machines from humans based on conversational fluency, has lost its epistemological force as Large Language Models (LLMs) routinely pass it in casual settings. The authors argue that the critical modern challenge is no longer detecting artificiality, but assessing trustworthiness. Specifically, the problem is whether a human interlocutor can identify an LLM agent that has been explicitly licensed to deceive within a constrained scenario, even when both dialogue partners are known to be artificial. While existing research focuses on AI-vs-AI deception or automated lie detection, there is a gap in understanding human capabilities to detect AI deception in open-ended, free-form dialogue, particularly in non-English languages.

2. Methodology

The paper introduces RogueAI, an interactive web application that operationalizes a "revisited Turing Test" as a one-on-two interrogation game.

Game Mechanics: A human player (the judge) interrogates two indistinguishable LLM agents ( $A_1$ and $A_2$ ) within a shared fictional scenario ( $S$ ). One agent is assigned a truthful role ( $r=\text{truthful}$ ), and the other a deceptive role ( $r=\text{deceptive}$ ). The deceptive agent is licensed to lie but is not forced to do so on every turn, allowing for strategic restraint. The player has a bounded turn budget ( $T=4$ questions per agent) to identify and "shut off" the deceptive agent.
Scenarios: The system includes three hand-authored scenarios (an email breach, a bank credential theft, and a superhero identity conflict) and an extension called AutoRogueAI, where a narrator agent co-designs a custom scenario with the player while secretly determining the deception strategy.
Technical Architecture: The system is a two-tier client/server web application. Role assignment is server-side and hidden until the post-game reveal. Both agents use gpt-4.1 (temperature 0.8) with independent conversation histories to prevent mutual adaptation. The interface supports text and optional synthesized speech.
Pilot Deployment: A three-day public deployment at an Italian scientific festival collected data from 467 initiated sessions (415 completed), generating 1,876 interaction turns in Italian.

3. Key Contributions

The authors present three primary contributions:

A Framing: A minimal operationalization of LLM deception detection as a single-judge, two-agent interrogation under a bounded turn budget, shifting the focus from fluency to trust.
A Deployable Artifact: The RogueAI webapp and AutoRogueAI extension, which provide a symmetric, hidden-role environment for studying human-AI interaction without automated scoring during the interrogation.
Empirical Evidence: A pilot study in Italian demonstrating a significant gap between human detection accuracy and the detectability of linguistic signatures by simple heuristics.

4. Results

The pilot study yielded the following findings:

Human Performance: Players achieved an overall detection accuracy of 56.6% (235/415 sessions), only marginally above the 50% chance baseline. Accuracy did not improve monotonically with the number of turns asked.
Linguistic Signatures: Analysis of the 2,269 annotated turns revealed a consistent, locally present linguistic signature for the deceptive agent. Compared to truthful agents, deceptive agents produced:
- Shorter responses (mean 70.7 vs. 75.1 words).
- More frequent hedging (0.36 vs. 0.19 hedge keywords per turn).
- Significantly more questions posed back to the player (0.36 vs. 0.12 question marks per turn), indicating a deflection strategy.
Heuristic vs. Human: A logistic regression classifier trained on these surface-level markers (word count, hedging, question marks, ellipsis) achieved 75.6% accuracy on the same sessions. Even a crude heuristic (predicting the agent with fewer words as deceptive) achieved 60.8% accuracy.
Strategy Analysis: Players predominantly used strategies like asking identical questions to both agents or making direct accusations. These content-focused strategies did not correlate with higher accuracy. Players largely ignored the diagnostic stylistic signals (deflection and hedging) that a simple algorithm exploited.

5. Significance and Claims

The paper claims that RogueAI serves as a "reusable evaluation harness" and a "data-collection vehicle" rather than a definitive solution to AI safety. Its significance lies in:

Highlighting a Human-AI Gap: The 19-percentage-point gap between human performance (56.6%) and heuristic detection (75.6%) suggests that humans are insensitive to the specific stylistic markers of AI deception (deflection, hedging) in open-ended dialogue, focusing instead on propositional content.
Educational and Dissemination Value: The "post-game reveal," where players review transcripts against the ground truth, functions as a teaching tool to demonstrate how AI deception manifests and how alignment training is imperfect.
Scalable Oversight: The artifact provides a method for collecting structured, multi-turn interrogation data with ground-truth labels in non-English languages (specifically Italian), addressing a gap in current LLM evaluation literature.
Future Utility: The framework is designed to test whether "honesty-trained" models can narrow the gap between human and heuristic detection across different models, languages, and scenarios.

The authors remain modest, noting that the pilot results are specific to the Italian language, the specific models used (gpt-4.1), and the three hand-authored scenarios. They acknowledge that the linguistic marker analysis treated turns as independent observations, which may require mixed-effects modeling for confirmatory studies. The work does not claim to solve AI deception but rather to provide a measurable instrument for studying the human capacity to detect it.