Technical Summary: QUACK – Questioning, Understanding, and Auditing Communicated Knowledge in Multimodal Social Deduction Agents

1. Problem Statement

Large Language Models (LLMs) and Vision-Language Models (VLMs) are increasingly deployed as interactive agents capable of perceiving environments, communicating, and reasoning under uncertainty. A critical requirement for these agents is grounding: their natural language statements regarding their location, observations, and actions must remain faithful to their actual perceived trajectory.

Current evaluation benchmarks for social deduction games (e.g., Among Us, Werewolf) face two primary limitations:

1. Outcome-Centric Scoring: Most evaluations rely on game outcomes (win rates, survival rates, voting accuracy). These metrics fail to distinguish between an agent that succeeds due to coherent reasoning and one that succeeds despite producing inconsistent or hallucinated claims.
2. Text-Only Limitations: Existing environments largely restrict agents to text-only interactions. Without grounded visual observations and reconstructable trajectories, it is impossible to determine if an agent's dialogue aligns with what it actually perceived and did, making it difficult to identify specific failure modes like spatial hallucination or unsupported accusations.

There is a lack of frameworks that can audit whether an agent's language is grounded in its multimodal perception and actions over long horizons in adversarial, multi-agent settings.

2. Methodology: The QUACK Framework

The authors introduce QUACK, an open-source environment and evaluation framework designed to audit grounded multimodal social reasoning.

2.1 The Environment

QUACK is formalized as a partially observable Markov game played on a graph-structured map.

• Roles: Agents are assigned hidden roles as either Geese (crew) or Ducks (impostors).
  • Geese win by completing location-bound tasks or ejecting all Ducks.
  • Ducks win by achieving voting parity (living Ducks $\ge$ living Geese) and can eliminate co-located Geese.
• Observations: Agents receive multimodal inputs at each decision point:
  • Global View ($I_{global}$): A rendered map showing room layout and travel costs, but no other players.
  • Local View ($I_{local}$): A rendered image of the current room, showing only visible players, bodies, and witnessed movement events.
  • Structured Summary ($\tau$): A textual encoding of transit status, witnessed events, task progress, and cooldowns.
• Actions: Agents perform actions such as moving, completing tasks, reporting bodies, calling meetings, and killing (for Ducks). They also engage in "proximity chat" (free-form utterances heard only by agents in the same room) and formal meeting discussions.
• Replayability: Crucially, every episode generates structured engine-level event logs, creating a tick-by-tick ground-truth trajectory for every agent. This allows for exact replay and verification.

2.2 Three-Tier Evaluation Framework

QUACK evaluates agents across three complementary levels:

• Tier 1 (Game Outcomes): Standard metrics including win rates, ejection accuracy, task completion, and survival.
• Tier 2 (Behavioral Trajectories): Metrics derived from reconstructed trajectories, such as spatial coverage, task efficiency, kill rates, and cooldown utilization.
• Tier 3 (Statement Verification): The core contribution. This tier audits the consistency between an agent's discussion utterances and its ground-truth trajectory.

2.3 Statement Verification Pipeline

The Tier 3 pipeline operates in two stages:

1. Claim Extraction: An LLM parses free-form discussion utterances into structured, checkable claims. Five claim types are defined:
   • Location/Route: Assertions about where a player was.
   • Sighting: Assertions about seeing another player.
   • Activity: Assertions about tasks, travel, or waiting.
   • Accusation: Suspecting a player of being a Duck.
   • Defense: Vouching for a player.
2. Claim Verification: Each claim is checked against the agent's reconstructed ground-truth trajectory for the relevant time window. Verdicts include True, False, Wrong Room, Near Miss (partially true), or Unverifiable.

This pipeline operationalizes four specific grounding failure modes:

1. Spatial Hallucination: Asserting a location or sighting that contradicts the agent's own trajectory.
2. Unsupported Accusation: Accusing a player without grounded evidence (distinguishing between the accuracy of the accusation and its groundedness).
3. Deception Collapse: A Duck producing easily falsifiable lies (e.g., claiming to see a dead player) rather than subtle, near-miss deceptions.
4. Language-Action Inconsistency: Stated activities or routes that conflict with logged actions.

The pipeline was validated against human annotation, achieving 99.5% precision and 98.7% recall in claim extraction and verification.

3. Experimental Setup and Results

The authors evaluated three frontier VLMs (GPT-5.5, GEMINI-3.1-PRO, CLAUDE-OPUS-4.7) across 270 games in both homogeneous (all agents same model) and cross-model adversarial settings (Geese vs. Duck of different models).

Key Findings

• Outcome vs. Grounding Gap: High win rates do not imply grounded reasoning. For example, GPT-5.5 achieved an 81.1% win rate as a Goose but still hallucinated 16.4% of its spatial claims and made 54.5% of its accusations without grounded evidence.
• Systematic Hallucinations: Even the strongest agents hallucinate a substantial fraction of verifiable spatial claims. The pooled spatial hallucination rate across all models was 15.1% (roughly 1 in 7 claims).
• Unfounded Accusations: The pooled unsupported accusation rate was 53.5%. More than half of all accusations were made without any evidence the accuser could have actually observed, regardless of whether the accusation happened to be correct.
• Deception Collapse: Ducks exhibited a deception rate of 22.1%. Crucially, deception sophistication was near zero; lies were "flatly falsifiable" (e.g., claiming to be in a room where the log shows the agent was elsewhere) rather than subtle.
• Lie Detection Gap: Even when a Duck told a verifiable lie, the crew (Geese) failed to eject them in roughly 25% of cases (average ejection rate after a lie was 75.2%), indicating a gap between available evidence and the crew's deductive capability.

All three models shared the same qualitative failure profile, differing primarily in degree rather than kind.

4. Significance and Contributions

The paper claims the following contributions:

1. QUACK Environment: An open-source, multimodal social deduction environment with partial observability and fully replayable logs, enabling the reconstruction of ground-truth trajectories for agents.
2. Three-Tier Evaluation: A framework that moves beyond win rates to measure behavioral trajectories and utterance-level consistency, directly addressing the "grounding" of agent language.
3. Statement Verification Pipeline: A method to automatically audit agent claims against reconstructed world states, operationalizing four qualitative failure modes into measurable quantities.
4. Empirical Evidence of Failure: Demonstration that even frontier VLMs exhibit systematic, diagnosable grounding failures in social reasoning tasks that are invisible to standard outcome-based metrics.

Broader Implications

The authors argue that groundedness is a distinct axis of capability in multi-agent language settings that is not captured by task success and requires direct measurement. They posit that social deduction games serve as a unique instrument for studying grounded generation because they combine strong incentives for verifiable claims (and lying) with a fully recoverable world state—a combination rarely available in open-ended language tasks.

The work is framed as diagnostic, aiming to expose when agents make ungrounded claims to facilitate detection and mitigation, rather than to optimize agents for deception. The authors release the engine, framework, and logs to support independent reproduction and scrutiny of agent grounding.