Technical Summary: CollabSim

Problem Statement

While Multi-Agent Systems (MAS) built on Large Language Models (LLMs) show promise in complex domains like software engineering and scientific research, their effectiveness often hinges on the agents' ability to coordinate rather than their individual task-solving capabilities. Recent studies indicate that MAS failures frequently stem from a lack of collaborative competence—the capacity to establish common ground, maintain shared task understanding, balance individual and collective incentives, and repair misalignment during interaction.

Current evaluation methodologies are insufficient for diagnosing these process-level failures. Existing benchmarks primarily focus on outcome-centric task performance or single-agent proficiency in reasoning and tool use. They fail to isolate the coordination breakdowns that arise from agent interactions. Furthermore, decades of research in Computer-Supported Cooperative Work (CSCW) have established that human collaborative competence is highly sensitive to interaction conditions (e.g., communication bandwidth, information visibility, and team structure), yet these variables are rarely manipulated in LLM agent evaluations.

Methodology: CollabSim Framework

To address these gaps, the authors introduce CollabSim, a configurable simulation framework grounded in CSCW theory for systematically investigating LLM agents' collaborative competence. The framework is designed to isolate specific dimensions of collaboration through controlled manipulation of interaction conditions and action-level probing of agents' internal states.

System Architecture

CollabSim operates on two layers:

1. Interaction Layer: Governs how agents perceive the environment and act. It utilizes an Agent Context Protocol that constructs observations by combining public updates (filtered by information-visibility rules) and private states (agent-specific facts).
2. Control Layer: An Experiment Controller orchestrates the simulation via a turn-based loop. It manages state transitions, validates actions, and triggers a Probing Module after every agent action.

Configurable Interaction Conditions

Drawing on CSCW literature, CollabSim manipulates three key experimental variables:

• Communication Bandwidth: Controls message frequency and length (paralleling media richness manipulations).
• Information Visibility: Controls access to shared state information (e.g., awareness dashboards or canvas visibility).
• Group Size: Varies the number of agents to examine scaling effects on coordination.

Probing Module

To move beyond observable behavior, the framework includes a probing module that queries agents after every action regarding:

1. Perceived task state.
2. Perceived teammate intentions.
3. Self-reasoning/plans.
   Quantitative measures derived from these probes include self-reported confidence scores and pairwise response similarity (via SBERT) to track the convergence of shared mental models.

Experimental Tasks

CollabSim implements four tasks adapted from established CSCW and social science paradigms, each targeting a distinct collaborative mechanism:

1. Shape Factory: Resource coordination under interdependence and cost asymmetry.
2. DayTrader: Incentive management in a social dilemma (balancing individual vs. collective returns).
3. Hidden Profile: Information pooling under asymmetric knowledge (where optimal solutions require integrating distributed private facts).
4. Map Task: Referential grounding under spatial asymmetry (text-based navigation instruction).

Key Contributions

1. Theory-Grounded Framework: CollabSim is the first framework to systematically apply CSCW experimental paradigms to LLM agent evaluation, moving beyond outcome metrics to process-level analysis.
2. Granular Probing: It introduces an action-level probing mechanism that captures agents' internal mental models (task state, partner intent, and reasoning), enabling the diagnosis of alignment failures that are invisible in final task outcomes.
3. Systematic Variable Manipulation: The framework allows for the controlled variation of communication bandwidth, information visibility, and group size to isolate their specific effects on collaborative behavior.
4. Comprehensive Benchmarking: The authors validate the framework across four distinct LLM backbones (Qwen3.6, Llama-4, GPT-5.5, Claude 4.6) and two agent designs (Persona-based vs. Collaboration-Theory-Informed).

Experimental Results

The authors conducted large-scale experiments across the four tasks, manipulating conditions and comparing agent designs.

Effects of Interaction Conditions

• Communication Bandwidth: Reducing bandwidth consistently lowered raw message volume and generally degraded cooperation metrics (e.g., trade acceptance, cooperation rates), suggesting agents struggle to prioritize grounding exchanges when channels are constrained.
• Information Visibility: Enabling visibility (e.g., dashboards, canvas views) increased agent engagement (e.g., higher trade acceptance, more drawing revisions) but did not uniformly improve task outcomes. In the Map Task, visibility significantly improved route accuracy, whereas in Shape Factory, it did not increase accumulated wealth.
• Group Size: Larger groups created a tension between opportunity and coordination strain. In Shape Factory, larger groups increased potential wealth but decreased order fulfillment rates due to coordination overhead. In DayTrader, larger groups sometimes led to a divergence between high self-reported confidence and low actual cooperation.

Model Backbone Differences

No single model dominated across all tasks.

• GPT-5.5 was the most consistently strong performer but showed the lowest average wealth in DayTrader.
• Claude 4.6 excelled in sustained cooperative investment (DayTrader) but failed completely in structured deliberation tasks (Hidden Profile).
• Qwen3.6 showed high sensitivity to condition changes, improving under specific constraints but failing in others (e.g., 0% order fulfillment in Shape Factory).
• Llama-4 generally underperformed compared to proprietary models.

Agent Design Impact

Comparing Persona-Based agents against Collaboration-Theory-Informed agents (explicitly prompted with shared mental model and common ground theories) revealed mixed results:

• Theory-informed agents often improved specific metrics (e.g., higher trade acceptance in Shape Factory, higher cooperation in DayTrader).
• However, the benefits were task-dependent. In some conditions (e.g., Hidden Profile with limited bandwidth), theory-informed agents actually reduced task accuracy, suggesting that explicit theoretical guidance can sometimes lead to over-cautiousness or misapplied strategies.

Significance and Claims

The paper claims that CollabSim provides stable and interpretable measurements of agent collaborativeness. Its primary significance lies in demonstrating that collaborative competence cannot be reduced to stronger individual task-solving ability.

By connecting task outcomes, interaction traces, and agents' self-reported internal states, CollabSim offers a process-level evaluation framework. This allows researchers to diagnose when and why LLM agents fail to collaborate effectively under realistic constraints. The authors assert that their findings highlight the necessity of evaluating agents not just on what they achieve, but on how they coordinate, maintain shared understanding, and repair misalignments during the interaction process. The framework serves as a tool to separate model performance patterns and reveal task-dependent effects of agent design, providing a more nuanced view of multi-agent system capabilities than outcome-only benchmarks.