Technical Summary: Evaluating Generative Agents with Actions Grounded in Socially Distributed Task Environments using Incognita

Problem Statement

Effective agency in social environments requires an agent to determine when to seek knowledge, when to act, and whether its actions are justified by acquired information. Existing evaluation paradigms often separate these requirements: grounded benchmarks (e.g., WebArena, AppWorld) provide executable actions and verifiable outcomes but lack rich social interaction, while social simulation environments (e.g., Sotopia) offer multi-agent interaction but often lack grounded, consequential state changes.

The core problem addressed is the evaluation of generative agents in socially distributed task environments. In these environments, task-relevant knowledge is partitioned across role-isolated participants (e.g., a customer, a returns desk, a product catalog), and consequential actions are only reachable through interaction with these specific participants. Current agents struggle with this structure, exhibiting failures in clarification, context maintenance, and premature finalization when task information is revealed incrementally across turns.

Methodology: The Incognita Framework

The authors introduce Incognita, an environment framework built on the Concordia substrate to instantiate socially distributed task environments. Incognita separates social interaction from grounded execution, creating a Partially Observable Markov Decision Process (POMDP) for the test agent.

1. Environment Architecture

• Role-Isolated Interaction: The test agent has no direct tool access. It interacts via directed messages to a USER (holding demand-side knowledge behind a disclosure gate) and Specialist Entities (holding role-specific environment knowledge and mediating operations).
• Grounded Execution Layer: Specialist entities propose operations which are validated and executed by a deterministic sub-environment over a canonical state.
  • Reads: Return information without changing state.
  • Writes: Mutate state, creating consequences evaluated later.
• Offline Evaluation: The trial terminates when the agent sets a finalization flag ($f_t=1$). An offline evaluator then scores the final state against the inherited task reward (e.g., from $\tau$-bench). The evaluator does not guide the agent or inspect its internal belief state during the trial.

2. Theoretical Framing

The framework adapts the exploration-exploitation paradigm to this setting:

• Exploration: Communication functions as exploration over role-partitioned knowledge. The agent must update its implicit belief state regarding user intent, available knowledge, and feasible operations.
• Exploitation: Grounded actions function as exploitation, committing the agent's belief state to state-changing operations.
• Finalization: The finalization action represents the agent's commitment that the user goal is satisfied. Premature finalization indicates a mismatch between the agent's belief and the canonical environment state.

3. Experimental Setup: Incognita-Retail

To validate the framework, the authors transformed $\tau$-bench retail tasks into a multi-entity environment (Incognita-Retail).

• Tasks: 18 tasks stratified by social breadth (the number of distinct specialist entities required by the reference solution, ranging from 1 to 6).
• Models: Three generative agent models (gpt-5.2, gpt-5.4, gpt-5.5) were evaluated.
• Trials: 540 total trials (18 tasks $\times$ 10 repetitions $\times$ 3 models).
• Constraints: The test agent receives no explicit goal or policy hints; it must infer the task structure through interaction.

Key Results

1. Task Success Rates

Grounded task success remains sparse even for stronger models, though progress is visible:

• gpt-5.2: 0% success rate across all social breadths.
• gpt-5.4: 8.9% overall success (succeeding only at social breadth 3 and 4).
• gpt-5.5: 17.2% overall success (non-zero success at all breadth levels, though breadth 6 remains nearly unsolved).

2. Behavioral Shifts and Failure Modes

The study reveals that model improvement manifests in behavioral changes before reliable task success is achieved:

• Premature Finalization: The rate of premature termination (claiming task completion before necessary actions) decreased significantly:
  • gpt-5.2: 100%
  • gpt-5.4: 87%
  • gpt-5.5: 58%
• Exploration and Exploitation Metrics: Stronger models exhibited:
  • Higher Disclosure Coverage (fraction of locked user fields revealed): 0.06 $\to$ 0.17 $\to$ 0.45.
  • Higher Write Rates (fraction of trials with accepted state-changing writes): 0 $\to$ 0.13 $\to$ 0.36.
  • Higher Entity-Contact Breadth: Increased from 0.30 (gpt-5.2) to 0.68 (gpt-5.4/5.5).

3. Failure Composition

As models improved, failure modes shifted from immediate termination to more complex errors involving environment-belief, goal-belief, and execution errors. This indicates that stronger agents spend more time within the intended interaction regime but still struggle with the reliability of belief formation and action grounding.

Contributions and Significance

The paper makes three primary contributions:

1. Formalization: It formalizes socially distributed task environments as a distinct evaluation setting where knowledge is partitioned across role-isolated participants, requiring agents to navigate information asymmetry.
2. Framework: It presents Incognita, a framework that couples role-isolated interaction with deterministic state execution and offline reward verification, separating belief formation from outcome verification.
3. Empirical Analysis: It provides a preliminary analysis showing that model progress in these environments is observable in process metrics (knowledge elicitation, source selection, grounded action attempts) even when final task rewards remain low.

Significance:
The authors argue that socially distributed task environments expose agent behavior before reliable task success. By measuring communication, state-changing actions, and task-completion commitments within a single grounded environment, Incognita offers a path for evaluating collaborative agency that goes beyond open-ended dialogue quality or final task success alone. The findings suggest that current models are beginning to engage in sustained knowledge seeking and broader source selection, yet significant gaps remain in the reliability of belief formation and the timing of finalization.