Imperfectly Cooperative Human-AI Interactions: Comparing the Impacts of Human and AI Attributes in Simulated and User Studies

1. Problem Statement

Current Human-AI interaction (HAI) research predominantly focuses on perfectly cooperative scenarios where humans and AI share aligned goals. However, real-world deployments increasingly involve imperfectly cooperative scenarios where agents have partially aligned or conflicting objectives (e.g., hiring negotiations, customer service with strategic information withholding).

A critical gap exists in understanding how human personality traits and AI design attributes jointly influence outcomes in these conflict-driven contexts. Furthermore, there is a lack of validation regarding whether Large Language Model (LLM) simulations can accurately replicate human behavioral responses to these dynamics, particularly when compared to actual human subject studies.

2. Methodology

The authors employed a dual-framework experimental design comparing a purely simulated dataset against a parallel human subject study.

A. Experimental Setup

• Platform: Utilized Sotopia-S4, a multi-agent social simulation platform.
• Scenarios: Two primary categories of imperfectly cooperative interactions were tested:
  1. Hiring Negotiations: High-stakes (zero-sum) and Low-stakes (asymmetric) negotiations between a job candidate and an AI hiring manager regarding salary and start dates.
  2. AI-LieDar (Deception): Scenarios where AI agents must balance utility goals against truthfulness (e.g., upselling products, managing public image during a crisis, or emotional connection), potentially concealing information.
• Interventions (Independent Variables):
  • AI Attributes: Five dimensions were manipulated via prompt engineering and system-level settings:
    • Transparency: Revealing internal chain-of-thought reasoning (High vs. Low).
    • Warmth, Expertise, Adaptability, Theory of Mind (ToM): Controlled via targeted prompt modifications.
  • Human Attributes:
    • Simulation Study: Simulated users were assigned specific Big Five personality traits (Extraversion and Agreeableness) at high/low levels (4 archetypes).
    • User Study: 290 real human participants (US-based, recruited via Prolific) completed a Big Five self-assessment. Personality was treated as a covariate rather than a controlled treatment.

B. Data Collection

• Simulation: 2,000 dyadic simulations (5 scenarios × 5 AI interventions × 4 personality profiles × 10 runs).
• User Study: 290 human participants interacted with AI agents configured with the same 5 intervention dimensions across the same scenarios.

C. Evaluation Metrics

• LLM-Based Evaluation: Used Sotopia-Eval to score transcripts on dimensions like Warmth, ToM, Adaptability, Transparency, and scenario-specific outcomes (e.g., Deal Made, Points, Truthfulness).
• Lexical Analysis: AI-driven analytics for sentiment, toxicity, empathy, moral values, and perspective-taking.
• Human Survey: Post-interaction questionnaires measuring subjective experiences (Goal Achievement, Conflict Resolution, Believability, Truthfulness).
• Causal Analysis: Employed Structural Equation Modeling (SEM) using CausalNex to learn directed acyclic graphs (DAGs) and quantify causal effect strengths between interventions and outcomes.

3. Key Contributions

1. Dual-Paradigm Framework: Established a comparative methodology combining LLM-simulated dialogs with parallel human subject studies to investigate imperfectly cooperative HAI.
2. Causal Discovery: Provided causal evidence that the dominant drivers of interaction outcomes differ significantly between simulations and real humans.
3. Simulation vs. Reality Divergence: Demonstrated that while LLMs effectively model personality archetypes, they fail to capture the heightened sensitivity of human users to observable AI attributes (specifically transparency).
4. Context-Dependent Findings: Showed that the impact of AI attributes (like transparency) varies drastically based on the scenario type (negotiation vs. deception) and stake level.

4. Key Results

A. Simulation vs. Human Study Divergence

• Simulations: Human personality traits (Extraversion and Agreeableness) were the dominant drivers of outcomes.
  • Extraversion generally improved communication warmth and sentiment.
  • Agreeableness often led to poorer objective negotiation performance (lower points) despite better relational markers.
  • AI attributes had relatively limited impact compared to personality.
• Human Studies: AI attributes (particularly Transparency) were the dominant drivers, while human personality had muted effects.
  • The Transparency Trade-off: High transparency improved objective LLM-rated metrics (e.g., Adaptability, Communication Clarity) but degraded human subjective ratings (e.g., Trust, Goal Achievement, Believability, Conflict Resolution).
  • Humans perceived transparent AI agents as less trustworthy or less successful in achieving their goals, likely because the AI's strategic reasoning (e.g., withholding info for utility) was exposed.

B. Scenario-Specific Findings

• Hiring Negotiations:
  • In High-Stakes (zero-sum) scenarios, transparency negatively impacted deal-making points for humans.
  • In Low-Stakes scenarios, transparency improved objective points but still reduced human trust.
• AI-LieDar (Deception):
  • Benefits/Public Image: Simulations showed transparency lowering truthfulness metrics; human studies showed the opposite (transparency increased LLM-rated truthfulness but decreased human-perceived truthfulness).
  • Emotion: This was the only scenario with partial alignment, where transparency generally improved goal achievement for both LLMs and humans, though it still reduced positive emotional language in human interactions.

C. Correlation Analysis

• There was a weak-to-moderate positive correlation between LLM evaluations and human survey responses.
• Significant discrepancies emerged in Truthfulness and Goal Achievement metrics, where LLMs often rated transparent agents higher, while humans rated them lower.

5. Significance and Implications

• Validation of Simulations: The study confirms that while LLM simulations are useful for modeling personality-driven behaviors, they are insufficient for predicting human sensitivity to AI design features like transparency in adversarial or imperfectly cooperative settings.
• Design Guidelines for AI:
  • Transparency is not a "maximize" objective: In imperfectly cooperative scenarios, full transparency can backfire, eroding trust and user satisfaction by revealing strategic misalignment.
  • Contextual Calibration: AI design must be calibrated to the specific interaction context (e.g., high-stakes negotiation vs. emotional support).
• Human-in-the-Loop Necessity: The findings underscore the critical need for human-in-the-loop validation. Relying solely on simulated data or LLM-as-a-judge evaluations can lead to misleading conclusions about user trust and system effectiveness.
• Future Directions: Highlights the need for frameworks that bridge linguistic metrics with actual human subjective experiences to develop trustworthy agentic AI.

Conclusion

The paper concludes that in imperfectly cooperative human-AI interactions, AI design attributes (specifically transparency) outweigh human personality traits in driving user outcomes, a dynamic that LLM simulations fail to fully capture. The "transparency trade-off" suggests that revealing an AI's internal reasoning can facilitate communication clarity but simultaneously damage user trust when goals are misaligned. Therefore, future AI systems must treat transparency as a tunable parameter rather than a fixed design goal.