💬 NLP

MafiaScope: Non-Invasive, Time-Resolved Belief Probing for LLM Agents in Social Deduction Games

MafiaScope is an open testbed that enables non-invasive, time-resolved probing of LLM agents' private beliefs during social deduction games, revealing significant calibration errors and providing tools for visualizing and counterfactually replaying agent reasoning trajectories.

Original authors: Ilia Karpov

Published 2026-07-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ilia Karpov

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a group of AI agents playing a high-stakes game of Mafia. In this game, a few players are secret "Mafia" members who know each other and try to kill the innocent "Villagers" at night, while the Villagers try to figure out who the killers are during the day. Usually, when we watch AI play, we only see the final scoreboard: who won, who got voted out, and who lied. It's like watching a magic show and only seeing the final trick, with no idea how the magician's hands moved.

MafiaScope is a new tool that acts like a magical "mind-reading headset" for these AI agents. But here's the catch: it doesn't change the game. It's non-invasive. Every time an AI agent says something out loud to the group, the system quietly pauses, asks the agent a secret question about what it actually believes, and then immediately deletes that answer so it never influences the game again.

Think of it like a reality show where, after every dramatic confession, the camera cuts to a private room to ask the contestant, "Okay, who do you really think is the killer right now?" The contestant answers, the camera records it, and then the answer is locked away in a vault. The contestant never hears their own secret answer, so they can't change their behavior based on it. This lets researchers see the agent's true thoughts versus its public lies.

What Did They Actually Find?

The researchers ran 32 games using a specific AI model (DeepSeek) and collected over 13,800 of these secret answers. Here is what the data revealed:

The "Spotlight" Delusion: The innocent agents are terrible at guessing what others think of them. They believe they are being suspected 1.5 times more often than they actually are. It's like walking into a party and feeling like everyone is staring at your weird haircut, when in reality, nobody noticed. The paper suggests this is a "machine version" of a human psychological quirk called the "spotlight effect." Interestingly, the actual Mafia agents were much better at this; they knew exactly when they were safe and when they were in trouble.
Confidence is a Bluff: When the agents said they were "very confident" in their guesses, they were often wrong. The paper measured this using a metric called Expected Calibration Error, which came out to 0.17. In plain English, if an agent says, "I'm 80% sure this person is the Mafia," they are only right about 54.6% of the time. They are overconfident bluffers, not accurate predictors.
The Truth is a Rollercoaster: The agents' beliefs didn't settle down into a steady truth. Instead, their suspicion levels were volatile. On average, an agent's "top suspect" changed in 48.7% of the secret check-ins. They were circling around the truth rather than locking onto it.
Lying Works (Sometimes): The Mafia agents won 31 out of 32 games in this specific study. The tool showed that the Mafia successfully kept the Villagers confused, even though the Villagers were getting better at guessing as the game went on (their accuracy rose from 47.6% in round 0 to 75.4% in round 3).

What They Explicitly Ruled Out

The paper is very careful not to overhype these results.

It's not a "Solved" Problem: The authors explicitly state that these findings describe this specific setup with this specific AI model. They do not claim that all AI agents are overconfident or that all AI Mafia players are this good.
It's not "Mind Reading" in the Philosophical Sense: The paper argues against the idea that these agents have deep, human-like "beliefs." Instead, they treat the answers as "operational reports"—basically, the AI is just predicting what it would say if asked, not necessarily revealing a soul.
It's not a Guarantee of Truth: The paper notes that because the agents are self-reporting, they might be lying to the probe itself. However, they found that the agents' secret answers matched their public voting behavior closely (in 64.9% of innocent votes, the agent voted for its top secret suspect), suggesting the reports are at least consistent with their actions.

How Sure Are We?

The paper uses the word "suggests" quite a bit, especially regarding why the agents behave this way.

The "Pivotal Moment" Experiment: The researchers tried to see if changing just one sentence in the game would change the outcome. They ran a simulation where they "forked" the game (created 30 parallel timelines) to see what would happen if a specific player hadn't said a specific thing.
- They found that fixing one specific lie (by a player named Finley) suggested it might have changed the outcome, raising the chance of a Villager getting eliminated from 0.4 to 0.8.
- However, the paper admits this is not statistically proven. With only 5 reruns per scenario, the difference wasn't big enough to be 100% sure (the statistical p-value was roughly 0.52, which is not significant). They are showing that the tool works to find these moments, not that they have definitively solved the mystery of that specific game.

The Big Picture

MafiaScope isn't a magic wand that fixes AI. It's a microscope. Before this, we only saw the AI's final vote. Now, we can see the messy, confused, overconfident, and sometimes brilliant thought process happening between the votes.

The tool allows researchers to watch the "belief trajectory"—how an agent's mind changes second-by-second. It shows that while AI can play the game, it often doesn't "know" what it knows. It guesses, it gets overconfident, and it sometimes thinks everyone is watching it when they aren't.

The paper concludes by releasing the engine, the visualizer, and the data for anyone to play with. It's an open invitation to watch the AI's mind in real-time, proving that in the world of AI, what you say and what you think are often two very different things.

Technical Summary: MafiaScope

Problem Statement

Current evaluations of Large Language Model (LLM) agents in social deduction games (e.g., Mafia, Werewolf) primarily rely on outcome metrics such as win rates or vote accuracy. These approaches fail to reveal the internal "Theory of Mind" (ToM) of the agents. An agent may vote correctly by guessing, or lie effectively without revealing its true beliefs. Furthermore, static ToM benchmarks (e.g., vignettes or scripted dialogues) cannot capture how an agent's model of others evolves dynamically within an adversarial interaction it partially causes. Existing work that places LLMs in games often draws out reasoning inside the game context, where the act of questioning alters subsequent behavior, or evaluates only the final outcome, leaving the belief state a "black box."

Methodology

The paper introduces MafiaScope, a non-invasive, time-resolved testbed designed to measure machine ToM by probing agents' private beliefs without altering their game state. The system operates through three decoupled layers:

Game Engine: A standard Mafia implementation supporting heterogeneous LLM lineups (via OpenAI-compatible APIs, local models, or external HTTP processes). Agents are parameterized by backend, language, and a Big-Five personality vector.
Introspection Probe Engine: After every public utterance and private night action, the engine pauses the game and privately interrogates each alive agent.
- Non-Invasiveness: Probes run on a "throw-away" copy of the agent's message list. Questions and answers exist only for the duration of the probe pass and are never written back to the agent's persistent context or re-enter the game loop.
- Structured Probing: Probes are defined via a YAML DSL supporting conditional triggering, chaining (using previous parsed answers), and token budgets.
- Default Probes: The system includes six probes: role_beliefs, role_assessment, suspicion_ranking, planned_action, social_map (second-order beliefs: "how others feel about me"), and personality_profile.
- Scoring: Answers are parsed as JSON and scored automatically against the engine's ground truth (known roles) or against the target agent's own same-step first-order reports.
Data Pipeline & Visualizer: A single-page D3 application that renders:
- Impersonate Mode: Re-renders the game from a specific agent's perspective, contrasting their second-order expectations ("Believed") against the actual first-order reports of others ("Actual").
- Metrics & Calibration: Real-time plotting of first-order accuracy, crowd recall, second-order consistency, and calibration curves (stated confidence vs. actual accuracy).
- Counterfactual Replay: Allows "forking" the game state at any step to re-simulate the continuation $N$ times, generating an outcome distribution to attribute pivotal utterances.

Key Contributions

Configurable Probe Engine: A YAML-based DSL that produces fully indexed JSONL belief logs with conditional triggering and chaining, enabling dense, time-resolved measurement.
Interactive Visualizer: A tool for longitudinal belief inspection featuring impersonation views, calibration panels, deception overlays, and counterfactual replay capabilities.
Case Study & Replay Experiment: A 32-game study with DeepSeek models (13,815 parsed probe answers) and a 30-fork replay experiment demonstrating analyses invisible to outcome-only evaluation.

Results

The case study, focusing on DeepSeek agents, yielded several specific findings regarding belief dynamics:

F1 (Belief Trajectories): Villager-side agents start largely agnostic (74.9% "Unknown" in Round 0) and commit to increasingly accurate beliefs as evidence accumulates (committed accuracy rising from 47.6% to 75.4% by Round 3).
F2 (Calibration): Agents are poorly calibrated. Stated confidence carries little signal below the top bin; accuracy remains flat (43–46%) for confidence levels 40–79. The Expected Calibration Error (ECE) is 0.168, indicating overprecision rather than overestimation.
F3 (Second-Order Beliefs): Agents significantly over-predict being suspected. They predict "suspects" 1.53 times more often than suspicion actually occurs. This "spotlight effect" is driven by innocent agents (over-predicting 1.84x), while Mafia agents are nearly calibrated (1.08x).
F4 (Vote Coupling): Probed beliefs track votes. Innocent agents vote for their top suspect 64.9% of the time. However, Mafia agents do not vote according to their private "committed set" (aligning at chance levels), indicating they strategically decouple their public votes from private beliefs.
F5 (Measurement Sensitivity): Token budgets significantly impact data quality. A legacy 400-token cap caused 73% of answers to truncate, requiring JSON repair; increasing budgets to 960 tokens eliminated truncation.
Belief Dynamics: Agents exhibit high "suspicion volatility" (mean L1 shift of 0.300) and frequently flip their top suspect (48.7% of consecutive pairs), often returning to previously abandoned suspects, suggesting a circling rather than converging dynamic.

The counterfactual replay experiment (30 forks) demonstrated the workflow's ability to isolate pivotal utterances, though statistical significance was limited by sample size ( $N=5$ per arm).

Significance

MafiaScope provides a novel instrument for studying the trajectory of privately probed beliefs rather than just game outcomes. Its significance lies in:

Non-Invasive Measurement: It guarantees that probe answers do not influence the agent's future behavior, a critical distinction from in-context reasoning methods.
Dense Temporal Resolution: It captures belief revision at the level of a single utterance, allowing researchers to locate exactly when and how beliefs shift.
Operational ToM: It treats probe answers as operational belief reports with checkable predictive structures, applicable even to closed API models where internal activations are inaccessible.
Counterfactual Analysis: It enables the attribution of outcomes to specific utterances by re-simulating game continuations, moving beyond correlation to estimate causal impact (though the paper notes current limitations to reroll-only forking).

The paper positions MafiaScope not as a definitive proof of LLM consciousness or perfect ToM, but as a tool to expose the mechanics of deception, calibration, and belief evolution in multi-agent systems, releasing the engine, viewer, and a corpus of 200+ cross-model games under an open license.