🤖 AI

Surfacing Isolated Learners with Outcome-Independent Mediation of Feedback between Teachers and Students Using AI

This paper proposes an interpretable, AI-driven decision layer that identifies isolated learners and prioritizes course topics by integrating student self-reports, observed difficulties, and teacher concerns without relying on graded outcomes, demonstrating its effectiveness in aligning with instructor insights and surfacing at-risk students in a graduate CS course.

Original authors: Junsoo Park, Youssef Medhat, Htet Phyo Wai, Ploy Thajchayapong, Ashok K. Goel

Published 2026-05-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Junsoo Park, Youssef Medhat, Htet Phyo Wai, Ploy Thajchayapong, Ashok K. Goel

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a classroom as a busy kitchen where a chef (the teacher) is trying to cook a complex meal for a large group of diners (the students). Usually, the chef only knows if the meal is good after the diners have eaten it and sent back their plates with comments. But by then, it's too late to fix the soup that was too salty or the steak that was undercooked.

This paper proposes a new way to cook: a "Smart Sous-Chef" (an AI system) that helps the main chef spot problems while the food is still being prepared, without waiting for the final taste test (grades).

Here is how the system works, broken down into simple parts:

1. The Problem: Cooking Blindfolded

In a normal class, teachers often don't know which topics are confusing students until they see the test scores weeks later. By then, the students have already moved on. The paper asks: How can we fix the "recipe" while we are still cooking, using only the signals we have right now?

2. The Solution: The "Three-Legged Stool"

The researchers built a transparent "decision layer" (a smart dashboard) that acts like a three-legged stool. It balances three different sources of information to decide which topics need attention right now:

Leg 1: The "Crowd's Struggle" (Gap Prevalence): The AI looks at how many students are asking questions or getting stuck on a specific topic. If 50% of the class is asking about "planning algorithms," that's a red flag.
Leg 2: The "Confusion Gap" (Survey Disagreement): This is the most interesting part. Sometimes students think they understand a topic, but their actions (like asking for help) show they don't. Or, they say a topic is hard, but they aren't asking for help. The system spots this mismatch. It's like a student saying, "I'm fine!" while clearly dropping their fork.
Leg 3: The "Chef's Gut Feeling" (Teacher Friction): The system listens to the teacher's worries. If the teacher says, "I'm worried about how the students are handling 'analogical reasoning'," the system takes that concern seriously.

3. How It Decides What to Fix

The system doesn't just guess. It combines these three signals into a ranked list of topics.

The Output: It gives the teacher a list like: "1. Analogical Reasoning (High Priority), 2. Planning (Medium Priority)..."
The "Receipt" (Decision Record): Crucially, for every item on the list, the system provides a "receipt" explaining why. It says, "We ranked 'Analogical Reasoning' #1 because 40% of students struggled with it, the teacher mentioned it in an interview, and students said it was easy but acted like it was hard." This makes the AI's thinking transparent, not a "black box."

4. Finding the "Invisible" Students

The paper highlights a special group called "Isolated Learners."
Imagine a student who is quiet, doesn't ask for help, and says they are doing fine. A single signal (like "did they ask for help?") would miss them completely.

The Magic: When the system combines all three signals, it found students who were slipping through the cracks. For example, a student might have a high "self-confidence" score but a hidden pattern of "unresolved help requests." Alone, these signals look normal. Together, they reveal a student who is silently struggling.
The Result: The system found these "invisible" students much better than looking at just one signal alone.

5. Did It Work?

The researchers tested this in one graduate computer science class.

Teacher Match: The topics the AI flagged as "most urgent" matched what the teacher was actually worried about.
Student Match: The topics the AI flagged also matched what students said were difficult in their surveys.
Stability: Even if the researchers tweaked how much weight they gave to each signal (e.g., caring more about the teacher vs. the students), the top priorities on the list stayed mostly the same.

The Bottom Line

This paper doesn't claim the AI replaces the teacher. Instead, it acts as a transparent coordinator. It takes the teacher's worries, the students' actions, and the students' self-reports, and weaves them into a clear, ranked list of "what to fix next."

It's like having a dashboard in a car that doesn't just tell you the engine broke after you crashed (the grade), but warns you, "Hey, the tire pressure is low, the oil light is flickering, and the driver looks worried," so you can pull over and fix it before the trip is ruined.

Technical Summary: Surfacing Isolated Learners with Outcome-Independent Mediation of Feedback

Problem Statement

In AI-augmented classrooms, teachers and students generate rich feedback signals (e.g., interaction logs, surveys, interviews) prior to the availability of graded outcomes. However, these signals are often difficult to synthesize into timely instructional decisions. Traditional approaches relying on post-hoc analytics or graded outcomes arrive too late to support proactive learning interventions. The core problem addressed is how to coordinate teacher and learner feedback to identify instructional priorities and support needs when outcome data (grades) are unavailable.

Methodology

The authors propose an interpretable decision layer (or mediation layer) that acts as a transparent mechanism to rank course topics requiring attention. This system does not use grades or post-hoc outcome labels. Instead, it integrates three distinct signal sources into a priority score ( $P_t$ ) for each topic $t$ :

Gap Prevalence ( $R_t$ ): The fraction of students identified as having difficulty with a specific topic, derived from AI teaching assistant (Jill Watson) interaction traces.
Survey Disagreement ( $D_t$ ): The absolute difference between observed learning difficulties ( $R_t$ ) and student self-reported difficulty ( $s_t$ ) from mid-course surveys ( $D_t = |R_t - s_t|$ ).
Teacher Friction ( $F$ ): A summary measure of unresolved instructor concerns derived from coded themes in semi-structured interviews (specifically usability friction and trust uncertainty).

The Mediation Process:
The system computes a topic priority score using a weighted linear combination:
$P_t = w_r R_t + w_d D_t + w_f F$
Where weights ( $w_r, w_d, w_f$ ) are researcher-defined based on co-design priorities (default: 0.70, 0.20, 0.10). The output is a ranked list of topics accompanied by a decision record (or decision trace) that explains the inputs and logic behind each ranking.

Study Context:
The approach was evaluated in a graduate-level AI course at Georgia Institute of Technology ( $N=279$ students). Data sources included:

Teacher: 5 semi-structured interviews following the deployment of an analytics dashboard.
Student: Mid-course survey responses and Jill Watson interaction logs.
Validation: Reflective thinking (RTQ), help-seeking behavior, and self-efficacy measures were used as complementary evidence for learning-related constructs but were not inputs to the priority score.

Key Contributions

Outcome-Independent Coordination: The paper introduces a mechanism to prioritize instructional support without relying on grades, addressing the latency problem in educational feedback loops.
Interpretable Decision Layer: Unlike "black box" analytics, the system provides per-topic decision records, making the ranking logic transparent and inspectable by teachers, students, and researchers.
Surfacing "Isolated Learners": The framework defines and identifies "isolated learners"—students who do not trigger alerts in any single signal source (e.g., they report high understanding and have no help-seeking logs) but are surfaced when multiple signals are synthesized.
Human-AI Co-Agency: The work organizes teacher, student, and researcher roles around a shared decision process, moving beyond simple dashboard reflection to active coordination of perspectives.

Results

The study provided preliminary evidence supporting three hypotheses:

Stability of Priorities (H1): The ranked topic priorities remained stable across reasonable variations in weighting schemes. In the specific course offering, the top-5 prioritized topics were analogical reasoning, frames, planning, case-based reasoning, and knowledge-based AI.
Alignment with Stakeholders:
- Instructor Agreement: The mediated priorities showed high rank agreement with instructor concerns (Spearman $\rho = 0.80$ , $p < .001$ ). Three of the top-5 mediated topics overlapped with instructor-identified concerns.
- Student Alignment: The priority scores were moderately associated with student self-reported difficulty ( $\rho = 0.46$ , $p = .048$ ), outperforming gap prevalence alone ( $\rho = 0.29$ ).
Multi-Signal Integration (H3): Combining signals revealed learners missed by single-source approaches.
- For "isolated learners" (those not flagged by individual signals), exposure to high-priority topics during weeks they reported difficulty resulted in a significantly higher Area Under the Curve (AUC = 0.96) compared to using gap prevalence alone (AUC = 0.91).
- In the cohort, 3 students (2.2% of baseline-unidentified students) were surfaced only through the synthesis of multiple signals.
Behavioral Constructs: Student behavioral signals (self-efficacy, reflective thinking, help-seeking) aligned with topic understanding, validating that these signals capture meaningful learning behaviors.

Significance and Claims

The paper positions this work as a preliminary step toward Human-AI co-agency in educational settings. The authors claim that:

Transparent coordination mechanisms can support decision-making when feedback is incomplete or outcome data is absent.
The mediation layer does not replace instructor judgment but serves as a mechanism to organize and make visible the convergence of teacher, student, and researcher perspectives.
By surfacing "isolated learners" and providing decision records, the system enables proactive intervention before graded outcomes are available.

The authors explicitly note limitations, including the study's scope (one course offering), the small number of instructor interviews, and the cross-sectional nature of the survey data. Consequently, the findings are presented as descriptive and exploratory evidence rather than population-level estimates. The authors provide a lightweight reference implementation to facilitate replication and future exploration of richer classroom signals and human oversight workflows.