💻 computer science

Measuring Successful Cooperation in Human-AI Teamwork: Development and Validation of the Perceived Cooperativity and Teaming Perception Scales

This paper introduces and validates two new scales, the Perceived Cooperativity Scale (PCS) and the Teaming Perception Scale (TPS), which effectively measure the subjective quality of human-AI teamwork across diverse contexts through three empirical studies.

Original authors: Christiane Attig, Christiane Wiebel-Herboth, Patricia Wollstadt, Tim Schrills, Mourad Zoubir, Thomas Franke

Published 2026-04-28

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Christiane Attig, Christiane Wiebel-Herboth, Patricia Wollstadt, Tim Schrills, Mourad Zoubir, Thomas Franke

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to bake a cake with a new, high-tech robot assistant. Sometimes, the robot is amazing: it knows exactly when to mix, it tells you if the oven is too hot, and you feel like you're a perfect baking team. Other times, the robot is confusing: it burns the batter, ignores your instructions, or acts like it has no idea what it's doing.

This paper is about creating two special "report cards" (called scales) to measure exactly how good that baking partnership feels, from your perspective. The researchers, working with Honda and the University of Lübeck, wanted to move beyond just asking, "Do you like this robot?" to asking, "How well did we actually work together?"

Here is a breakdown of their work using simple analogies:

The Two Report Cards

The researchers realized that working with AI happens in two different ways, so they built two different measuring tools:

1. The "Single Moment" Report Card (Perceived Cooperativity Scale - PCS)

What it measures: How well the AI acted during one specific task.
The Analogy: Think of this like judging a single dance move. Did your partner lead clearly? Did they follow your lead? Did they stay in rhythm just for this one song?
The Theory: It's based on "Joint Activity Theory." It checks if the AI was transparent (you knew what it was thinking), reliable (it did what it said), and responsive (it listened to you).
The Result: They tested this on 409 people in three different scenarios: playing a cooperative card game, chatting with a text-bot, and using a car's trip planner. The card worked great! It could clearly tell the difference between a human partner (who got high scores), a smart-but-predictable rule-based robot (medium scores), and a robot that learned by trial and error and got confused (low scores).

2. The "Long-Term Team" Report Card (Teaming Perception Scale - TPS)

What it measures: The feeling that you and the AI have become a genuine team over time.
The Analogy: This isn't just about one dance move; it's about whether you feel like you are part of a dance troupe. Do you feel like you are both investing effort? Do you feel like you share a common goal? Do you feel like the robot is a "teammate" rather than just a tool?
The Theory: It's based on "Evolutionary Cooperation Theory." It looks at whether both sides are paying a "cost" (effort) to help the other succeed.
The Result: This scale has three parts:
- The Team: "We are a unit."
- The Partner: "You are helping me."
- The Self: "I am helping you."
- The Twist: The "Self" part was tricky. People tended to give themselves high scores no matter how bad the robot was. It's like a student who thinks, "I studied hard!" even if they didn't actually study. The researchers found this part of the scale is sensitive to the situation; it works better when people have more freedom to choose how hard they work.

How They Tested It (The Three Studies)

To make sure their report cards were accurate, they used three different "training grounds":

The Card Game (Hanabi): Imagine a game where you can't see your own cards and must rely on hints from your partner.
- The Test: People played with a human, a robot that followed strict rules, and a robot that learned on its own.
- The Finding: The report cards successfully ranked them: Human > Rule-Based Robot > Learning Robot. The learning robot was often confusing, so people didn't feel like they were cooperating well.
The Chatbot (LLM): People used a text-bot to fact-check articles.
- The Test: They used bots that were helpful, bots that were helpful but added unwanted changes, and bots that failed.
- The Finding: The scales picked up on the differences in how "cooperative" the bots felt.
The Car Trip Planner: People used a Tesla trip planner to find charging stations.
- The Test: This was a "boundary case." The car isn't really a "teammate"; it's just a tool.
- The Finding: The "Single Moment" card (PCS) still worked fine here. However, the "Long-Term Team" card (TPS) wasn't used because a car trip planner doesn't really have a "mind" or "goals" to team up with. You can't really feel like a team with a GPS.

What They Learned (The Takeaways)

The "Single Moment" card is rock solid. It is a very reliable way to measure how well an AI acts during a specific task. It works for everything from chatbots to game bots.
The "Team" card is powerful but needs care. It measures the deep feeling of partnership. However, the part where people rate their own contribution is a bit biased. People tend to think they are great teammates even when the AI is terrible. This suggests that in rigid situations (like a card game where you must cooperate), people's self-ratings don't change much.
Context matters. If you are just using a tool (like a calculator or a GPS), the "Team" feeling doesn't really apply. But if you are working on a complex project where you and the AI need to rely on each other, these scales tell you if that partnership is working.

Why This Matters

Before this, we mostly had tools to ask, "Do you trust AI?" or "Is this AI smart?" This paper gives us a way to ask, "How well are we working together right now?" and "Do we feel like a team?"

This helps designers build better AI. If a robot gets low scores on "predictability," the designers know to make the robot's actions clearer. If the "Team" score is low, they know the robot needs to show more support and shared goals.

In short: The researchers built two rulers. One measures how well a robot dances in a single song. The other measures if you feel like you and the robot are in a dance troupe together. They tested these rulers on 409 people and found they work well, though the "dance troupe" ruler needs to be used carefully depending on how much freedom the human has to move.

1. Problem Statement

As Artificial Intelligence (AI) becomes increasingly integrated into daily life and hybrid work environments, there is a critical need for reliable instruments to assess the subjective quality of human-AI cooperation. While existing scales measure general attitudes toward AI or AI literacy, they fail to capture the specific, process-oriented dynamics of cooperative interactions.

The authors identify a gap in measuring two distinct dimensions of human-AI integration:

Synchronic Integration: The degree of interdependence within a single task sequence (e.g., a specific interaction).
Diachronic Integration: The continuity and stability of cooperation across repeated interactions over time (e.g., long-term partnerships).

Current tools do not adequately differentiate between the perception of an agent's cooperative capability in a single moment versus the emergent sense of "teaming" that develops over time. Furthermore, there is a lack of validated instruments that allow for direct comparison between human-human and human-AI cooperation.

2. Methodology

Theoretical Framework

The study introduces two theoretically grounded scales:

Perceived Cooperativity Scale (PCS): Based on Joint Activity Theory. It measures the perceived capability and practice of an agent within a single interaction sequence. It operationalizes eight characteristics of effective agents: observability, monitorability, communicativeness, self-assessment, reliability, directability, selectivity, and coordination.
Teaming Perception Scale (TPS): Based on Evolutionary Cooperation Theory and shared action frameworks. It measures the emergent sense of teaming arising from mutual contribution and support. It is conceptualized as having three facets:
- Team: The emergent sense of the group as a whole.
- Self: The human's own cooperative investment and support.
- Partner: The human's perception of the AI's support and contributions.

Scale Construction

Item Generation: Initial item pools were derived from theoretical constructs. Items were phrased in accessible language (e.g., "I had the feeling that...") to ensure intuitive response.
Adaptation: Both scales were developed with parallel versions for AI partners (PCS-A, TPS-A) and human partners (PCS-H, TPS-H) to enable cross-agent comparisons.
Translation: Rigorous translation and back-translation procedures were used to ensure linguistic equivalence between German and English versions.

Study Design and Samples

The scales were validated across three independent studies involving a total of N = 409 participants with diverse demographics and contexts:

Study 1 (Hanabi Card Game): A cooperative, high-interdependence game. Participants played with a human partner, a rule-based AI agent (RB), and a reinforcement learning AI agent (RL). This tested the scales' ability to differentiate between partners with varying levels of cooperative transparency and predictability.
Study 2 (LLM Interaction): Participants fact-checked articles using four versions of an LLM chatbot with varying degrees of autonomy and goal alignment (Focused, Augmented, Conflicted, Failed). This tested sensitivity to different cooperative qualities in a lower-interdependence context.
Study 3 (Tesla Trip Planner): A real-world interaction with a decision-support system (low agency). This served as a boundary case to test the applicability of the PCS in contexts where the "partner" is a tool rather than an agent.

Psychometric Analysis

The authors employed standard scale evaluation procedures:

Dimensionality: Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis (CFA) to determine factor structure.
Reliability: Cronbach's $\alpha$ and McDonald's $\omega$ .
Validity: Discriminant validity (correlations with SIPA, trust, warmth, competence, acceptance) and criterion validity (ability to differentiate between partner types).

3. Key Contributions

Development of Two Novel Scales: The creation of the PCS (14 items after refinement) and TPS (9 items, 3 subscales) specifically designed for human-AI teamwork.
Theoretical Grounding: The scales are explicitly linked to Joint Activity Theory (synchronic) and Evolutionary Cooperation Theory (diachronic), providing a robust theoretical basis for future research.
Cross-Agent Comparability: The inclusion of human-partner versions allows researchers to benchmark AI performance directly against human performance using identical metrics.
Empirical Validation: Comprehensive validation across three distinct contexts (gaming, LLM chat, real-world navigation) and diverse populations (students, Prolific workers, EV drivers).

4. Results

Dimensionality and Item Analysis

PCS: CFA and EFA supported a unidimensional structure. Item PCS14 ("I had the feeling that the person/agent was able to realize that we had different approaches...") showed poor discrimination and low confidence ratings across contexts and was excluded, resulting in a final 14-item scale.
TPS: While EFA results were mixed, the authors retained a theoretical three-factor structure (Team, Self, Partner).
- The Team and Partner subscales showed strong internal consistency and good item discrimination.
- The Self subscale showed poor to questionable reliability (especially in the game context) and low item discrimination. Analysis suggests this subscale may capture a stable, positively biased self-evaluation rather than interaction-specific dynamics, particularly when task structures limit behavioral variance.

Reliability

PCS: Demonstrated excellent internal consistency ( $\alpha \approx .93$ ) across all samples and contexts.
TPS: The composite scale and Team/Partner subscales showed good to excellent reliability ( $\alpha \approx .88 - .96$ ). The Self subscale reliability improved in non-game contexts (LLM study) where participants had more agency.

Validity

Construct Validity: Both scales showed expected positive correlations with perceived competence, system trustworthiness, and subjective information processing awareness (SIPA).
Differentiation (Criterion Validity): Both scales successfully differentiated between cooperation partners of varying quality:
- Ordering: Human Partner > Rule-Based AI > Reinforcement Learning AI (Study 1); Aligned LLM > Misaligned/Failed LLM (Study 2).
- Effect Sizes: Large effect sizes were found for the differences between partner types, confirming the scales' sensitivity to cooperative quality.
Boundary Case: In Study 3 (Tesla Trip Planner), the PCS remained meaningful even with a low-agency system, though the TPS was not administered as the conditions for "teaming" were not met.

5. Significance and Implications

Diagnostic Tool for AI Design: The PCS items map directly to specific cooperative capabilities (e.g., observability, reliability). Low scores on specific items can provide actionable feedback to designers on which agent behaviors need improvement.
Differentiating "Tool" vs. "Teammate": The scales help distinguish between interactions where the AI is merely a tool (measured well by PCS) versus interactions where a genuine sense of teaming emerges (measured by TPS).
Guidance for Researchers: The paper provides specific recommendations for scale usage, including:
- Assessing the degree of synchronic/diachronic integration before choosing a scale.
- Administering scales immediately after interaction episodes rather than retrospectively.
- Interpreting the TPS "Self" subscale with caution due to potential positivity bias.
Foundation for Future Research: The scales enable rigorous empirical investigation into hybrid team dynamics, trust calibration, and the long-term evolution of human-AI partnerships.

Conclusion: The study successfully validates the Perceived Cooperativity Scale and Teaming Perception Scale as robust, theoretically grounded instruments. They fill a critical gap in human-AI interaction research by providing reliable metrics to evaluate the subjective quality of cooperation, applicable across diverse contexts and capable of distinguishing between varying levels of AI cooperative performance.