💬 NLP

Measuring Inclusion in Interaction: Inclusion Analytics for Human-AI Collaborative Learning

This paper introduces "inclusion analytics," a discourse-based framework designed to measure inclusion in human-AI collaborative learning as a dynamic, moment-to-moment process across participation, affect, and epistemic dimensions.

Original authors: Jaeyoon Choi, Nia Nixon

Published 2026-02-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jaeyoon Choi, Nia Nixon

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The "Dinner Party" Problem: Making Sure Everyone Has a Seat at the Table

Imagine you are hosting a dinner party. You want it to be inclusive, so you check your guest list: "Great, I have a diverse group of people here!" You feel good. But then, the party actually starts.

As the night goes on, you notice something:

The Loud Talker: One guest is doing 90% of the talking, while another guest is sitting quietly, barely getting a word in.
The Cold Shoulder: When one person makes a joke, everyone laughs, but when another person shares a serious thought, the room goes silent.
The Idea Thief: Someone suggests a great idea, but the group ignores it—only to have a different person suggest the exact same thing five minutes later, and that person gets all the credit.

Even though your guest list was "diverse," the experience of the party was not inclusive.

What is this paper about?

In the world of Artificial Intelligence (AI), we are increasingly asking humans and AI to work together to solve problems (like a student and an AI tutor working on a science project).

Currently, researchers usually measure "inclusion" by looking at the guest list (demographics) or asking people afterward, "Did you feel included?" (self-reports). But the authors of this paper, Jaeyoon Choi and Nia Nixon, argue that this is too late. Inclusion isn't just a checkbox; it’s a living process that happens moment-by-moment during a conversation.

They propose a new toolkit called "Inclusion Analytics." Instead of asking how people felt, they use math and language technology to watch how people actually interact in real-time.

The Three "Lenses" of Inclusion

The researchers look at collaboration through three specific lenses:

1. Participation Equity (The "Microphone" Test)

The Analogy: Imagine a group of people sitting around a single microphone.
The Goal: Is the microphone being passed around fairly? Or is one person hogging it? The researchers use math to track how many turns each person takes and how many words they say. If one person is "hogging the mic," the system flags an imbalance.

2. Affective Climate (The "Vibe" Check)

The Analogy: Think of the "social temperature" of a room. Is it warm and welcoming, or cold and prickly?
The Goal: They look at politeness. But they don't just look for "please" and "thank you." They look for "politeness uptake." If I am respectful to you, do you reflect that respect back to me? If I am polite and you respond with coldness, the "social temperature" is dropping, and the environment might be becoming unsafe for people to take risks or ask questions.

3. Epistemic Equity (The "Echo" Test)

The Analogy: Imagine you throw a ball into a group, and everyone ignores it. Then, someone else throws the same ball, and everyone catches it.
The Goal: This is about ideas. Does the group actually "pick up" what you are saying? The researchers use AI to see if the meaning of your words is echoed or supported in the next few sentences. If your ideas are constantly ignored or "dropped," you are being excluded from the group's brainpower, even if you are allowed to speak.

What did they find?

The researchers tested this using both computer-simulated conversations and real experiments where humans worked with AI.

The AI is a "Chatty Cathy": They found that when an AI joins a human team, the "Microphone Test" often fails. The AI tends to be much more talkative (using way more words) than the humans, which can throw off the balance.
The AI is a "Polite Ghost": While the AI is very polite, humans don't always "echo" that politeness back. Humans often treat the AI like a tool (giving direct commands) rather than a teammate.
The "Idea Gap": They noticed that while humans tend to build on each other's ideas, the AI's ideas are often accepted but not really "discussed." It’s like the AI says something, everyone says "okay," and moves on—there’s no real collaborative "dance."

Why does this matter?

If we want AI to be a true partner in education and work, we can't just hope it's inclusive. We need to be able to measure the dance.

By using these "Inclusion Analytics," we can build systems that alert us when a student is being silenced, when the "vibe" is turning toxic, or when an important idea is being ignored. It moves us from simply hoping for equality to actually seeing it in action.

Technical Summary: Measuring Inclusion in Interaction

Title: Measuring Inclusion in Interaction: Inclusion Analytics for Human-AI Collaborative Learning
Authors: Jaeyoon Choi & Nia Nixon (University of California, Irvine)

1. Problem Statement

In Artificial Intelligence in Education (AIED), inclusion, equity, and access are core values. However, current assessment methods are often coarse and retrospective, relying on demographic descriptors (representation counts) or post-hoc self-reports (surveys on belonging).

The authors argue that these methods fail to capture how inclusion is actually enacted and experienced moment-by-moment during Collaborative Problem Solving (CPS). This is particularly critical in human-AI teaming, where AI agents act as teammates rather than just tools. AI can redistribute authority and attention in ways that might reinforce existing social inequities (e.g., marginalizing certain human voices), but current tools cannot "see" these dynamic shifts as they happen.

2. Methodology: The Inclusion Analytics Framework

The paper proposes Inclusion Analytics, a discourse-based framework that operationalizes inclusion through three observable, interactional dimensions:

Participation Equity: Measures whether all members have comparable access to the "conversational floor."
- Metric: Inequality of Participation (IP), a group-level index based on turn counts and word counts (where 0 is perfect equality and 1 is maximal imbalance).
Affective Climate: Measures whether the interactional environment supports social safety and mutual respect.
- Metric: Politeness Uptake. Rather than just measuring if a speaker is polite, this measures the reciprocity of politeness. It uses a bag-of-words approach (11 dimensions like gratitude, apology, and hedging) and calculates the cosine similarity between a speaker's politeness vector and the politeness vectors of subsequent teammates.
Epistemic Equity: Measures whether learners' contributions are treated as legitimate knowledge resources.
- Metric: Idea Uptake. This is split into two components:
  1. Adjusted Semantic-based Uptake: Uses sentence-BERT embeddings to measure semantic similarity in subsequent turns, adjusted by a Monte Carlo null-score baseline to filter out "noise" caused by the shared task topic.
  2. Endorsement-based Uptake: Uses regular expressions to detect explicit agreement (e.g., "yeah," "great idea") and applies an exponential decay weight to prioritize immediate feedback.

3. Key Contributions

Conceptual Shift: Moves the definition of inclusion from a static "internal state" (feeling included) to a dynamic "interactional process" (being heard and respected).
Multi-Dimensional Framework: Provides a theoretically grounded taxonomy (Participation, Affect, Epistemic) for analyzing collaborative discourse.
Scalable Metrics: Introduces computational, non-annotated, and temporally sensitive measures that can be applied to large-scale interaction logs without manual coding.

4. Results

The authors validated the framework using two datasets: Simulated Conversations (generated by GPT-5.2 to test controlled imbalances) and Empirical Human-AI Teaming Data.

Simulated Data (Calibration):
- Participation Equity: Successfully distinguished between balanced and imbalanced groups (significant $p < .005$ ).
- Affective Climate: Successfully showed that an impolite speaker reduces the politeness uptake of others.
- Epistemic Equity: Showed limitations; semantic uptake was sometimes "inflated" because even when an idea was rejected (e.g., "Idea A is wrong"), the semantic similarity remained high.
Human-AI Teaming Data (Empirical Validation):
- Participation Equity: Found that teams with an AI teammate had significantly higher word-count imbalance ( $p < .005$ ), confirming that AI teammates tend to be much more verbose than humans.
- Affective Climate: No significant difference in politeness uptake between humans and AI, likely due to the informal, low-politeness nature of anonymous chat.
- Epistemic Equity: Humans showed significantly higher adjusted semantic uptake and endorsement uptake than AI ( $p < .005$ ). This suggests that while AI provides many ideas, human contributions are more likely to trigger deep discussion or explicit validation.

5. Significance

This work serves as a proof-of-concept for a new methodological direction in AIED. It demonstrates that while measuring social dynamics is technically challenging (especially distinguishing between "topic similarity" and "idea uptake"), it is possible to move beyond "checkbox" evaluations.

The significance lies in the ability to design proactive AI systems: if an AI teammate can detect low participation equity or low epistemic uptake in real-time, it could theoretically adjust its behavior to facilitate a more inclusive and equitable learning environment.