💻 computer science

Human–AI collaboration in deductive coding of classroom dialogue: prompt engineering and collaboration patterns

This study employs design-based research to demonstrate that a customised GPT coding assistant, when integrated with structured prompts and a collaborative human-first workflow, effectively supports deductive coding of classroom dialogue while highlighting that successful human–AI collaboration depends on the dynamic interplay between prompt architecture, workflow sequencing, and the researcher's stance toward AI.

Original authors: Luwei Bai, Dongkeun Han, Sara Hennessy

Published 2026-07-13

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Luwei Bai, Dongkeun Han, Sara Hennessy

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to decode a secret language spoken in a classroom. Teachers and students are having a deep, complex conversation, and your job is to sort every single sentence into specific categories like "Reasoning," "Invitation to think," or "Agreement." Doing this by hand is like trying to drink from a firehose: it's exhausting, takes forever, and your brain gets tired.

Enter the Customised GPT Coding Assistant (CGCA). Think of this AI not as a magic robot that knows everything, but as a very smart, very fast intern who has been given a specific rulebook (called the Cambridge Dialogue Analysis Scheme) and a few sample sentences to study. The researchers from the University of Cambridge wanted to see: Can we train this intern to do the heavy lifting? And how do we work together without the intern taking over or the human getting annoyed?

Here is what they found, served up with a few metaphors to keep things clear.

1. The "Intern" Needs the Right Instructions (Prompt Engineering)

At first, the researchers just handed the AI the entire rulebook and said, "Go!" The result? The AI got confused. It was like giving a new employee a 500-page manual and expecting them to know exactly what to do on day one. The AI's accuracy was low, sometimes as low as 6.7% for certain categories.

The researchers realized the AI has a "brain" that works differently than a human's. It gets overwhelmed if you give it too much text at once. So, they started prompt engineering. This is like rewriting the instructions for the intern. Instead of a giant wall of text, they:

Broke the rules down into a "decision tree" (If X happens, then do Y).
Gave the AI fewer, but better, examples (about 5–15 carefully chosen ones instead of hundreds).
Made the instructions step-by-step.

The Result: After these tweaks, the AI's accuracy jumped to 64.9%. That's not perfect (humans usually agree with each other about 70–75% of the time), but it's good enough to be a helpful sidekick. The paper suggests that simply pasting a human's training manual into a computer prompt doesn't work; you have to restructure the instructions to fit how the AI thinks.

2. Who Goes First? The "Human-First" vs. "AI-First" Dance

The researchers tested two ways to work with this AI intern. They had six educational researchers (some new to coding, some experts) try both methods on 55 turns of classroom dialogue.

Condition A (Human-First): The human codes the sentence first, then asks the AI, "What do you think?"
Condition B (AI-First): The human asks the AI, "What do you think?" first, then the human decides if they agree.

The Findings:
When the human went first, the agreement between the human and the "gold standard" (the original expert coder) was around 0.40 (a measure of how well they matched).
When the AI went first, that number bumped up slightly to 0.45.
Note: The paper says this difference is small and not statistically "proven" to be a huge win because the group was so small, but the trend suggests starting with the AI might help people stick to the rules better.

Also, when the AI went first, the humans finished the job faster. However, the researchers noticed something tricky: in the "AI-First" group, some people felt like they were just checking the AI's work, while in the "Human-First" group, they felt like they were being checked by the AI.

3. The Secret Ingredient: How You Treat the AI

This is the most interesting part. The researchers found that how the human felt about the AI mattered more than the order of operations. They spotted two distinct "personalities" in how the coders worked:

The "Dialogic" Partner (The Collaborators)
About half the team (Participants C, D, and E) treated the AI like a critical friend or a debate partner.

The Metaphor: Imagine playing tennis with a partner who hits the ball back to you so you can hit it harder. They didn't blindly trust the AI, but they used its suggestions to test their own thinking. "Hmm, the AI says this is 'Reasoning.' Why? Oh, I see, because of the word 'because.' Okay, I agree."
The Outcome: Even though some of these people were beginners at coding, they actually improved the most. They treated the AI as a tool to help them learn and refine their decisions.

The "Supervisory" Watchdog (The Skeptics)
The other half (Participants A, B, and F) treated the AI like a suspect or a machine that needed constant checking.

The Metaphor: Imagine a security guard who thinks the camera is broken and is constantly trying to prove it's wrong. They said things like, "I'm checking ChatGPT" or "I don't think it's a colleague." They kept their distance, refusing to have a "conversation" with the tool.
The Outcome: These people didn't improve as much. Their coding accuracy stayed lower, and they spent less time engaging with the AI's reasoning. They were so focused on verifying the AI that they missed the chance to collaborate with it.

4. What the Paper Rules Out

It is important to know what this study says does not work:

It is NOT a magic fix: You cannot just dump a huge pile of data and a standard rulebook into an AI and expect it to work perfectly. The paper explicitly argues that "directly transferring a human training codebook into a prompt is insufficient."
It is NOT a replacement for skill: The study does not claim that the AI makes humans instantly experts. In fact, one participant (Participant E) started by deferring to the AI because they lacked confidence, and only later learned to judge the AI's work critically. The paper suggests this might be a temporary boost in confidence, not necessarily a permanent leap in independent skill.
It is NOT a "one-size-fits-all" solution: The paper argues that there is no single perfect way to use the AI. What works for a "dialogic" partner might not work for a "supervisory" watchdog.

5. How Sure Are We?

The researchers are careful with their words. They say their findings suggest that a "dialogic" approach (treating the AI as a partner) leads to better results. They suggest that starting with the AI (AI-first) might help consistency.

The Confidence Level: The study is a "small-scale" exploration with only six people. The authors admit that because the group is so small, they cannot say for sure that the "AI-first" method is definitely better; they can only say the numbers trended that way.
The Accuracy: The AI reached 64.9% precision. The paper states this is "approaching" the human agreement level of 70–75%, but it is not there yet. It is a "structured decision-support tool," not a final answer key.

The Big Picture

This study paints a picture of human-AI collaboration not as a simple "human vs. machine" or "machine doing the work" scenario. Instead, it's a dynamic dance.

The success of the dance depends on three things moving together:

The Steps (The AI Design): The instructions must be simple, step-by-step, and tailored to the AI's brain (like a decision tree).
The Music (The Context): The order of operations (who goes first) changes the rhythm.
The Dancer's Mindset (The Human): If you treat the AI as a partner to learn from, you dance better. If you treat it as an enemy to be checked, you stumble.

The paper concludes that AI in research is a socio-technical process. It's not just about the code; it's about how the human feels, thinks, and interacts with the tool. For a curious teenager, the takeaway is this: The tool is only as good as the relationship you build with it. If you treat it like a smart teammate, it helps you grow. If you treat it like a suspicious stranger, it just sits there waiting to be checked.

Technical Summary: Human–AI Collaboration in Deductive Coding of Classroom Dialogue

Problem Statement
Classroom dialogue is a critical mechanism for knowledge co-construction and cognitive development, yet its systematic analysis via deductive coding remains a bottleneck. Manual coding using established frameworks like the Cambridge Dialogue Analysis Scheme (CDAS) is cognitively demanding, time-intensive, and difficult to scale. While Large Language Models (LLMs) offer potential for automating qualitative analysis, conventional supervised approaches require large annotated datasets often unavailable in specialized educational research. Furthermore, there is a lack of empirical understanding regarding how to practically implement lightweight, configured conversational systems for deductive coding and how the human–AI collaboration process unfolds in this specific context.

Methodology
The study employed a Design-Based Research (DBR) approach combined with a within-subject quasi-experimental design to address three research questions: (1) how to optimize a Customised GPT Coding Assistant (CGCA) for deductive coding with limited data; (2) how workflow sequencing (Human-first vs. AI-first) affects collaboration; and (3) what user-level factors influence collaboration patterns.

System Development (RQ1): The researchers iteratively developed a CGCA grounded in the CDAS framework. Initial attempts using a direct transfer of the human codebook failed to achieve reliable performance. Subsequent iterations focused on prompt engineering strategies, including:
- Instruction Architecture: Moving from simple rule lists to decision-tree logic, if–else structures, and modular segmentation to reduce cognitive load and align with LLM processing constraints.
- Exemplar Selection: Curating a small set of high-quality, diverse examples (5–15 per category, including core, ambiguous, and edge cases) rather than relying on large volumes of undifferentiated data.
- Feedback Loops: Incorporating interactive verification steps to encourage rule-based classification over surface-level pattern matching.
- Evaluation: The final CGCA was tested against 1,386 authentic, human-coded dialogue turns from primary school lessons, evaluated using precision, recall, accuracy, and F1 scores.
Collaboration Study (RQ2 & RQ3): Six educational researchers (varying in coding proficiency and AI experience) participated in two workflow conditions:
- Condition A (Human-first): Coders assigned labels independently before consulting the CGCA.
- Condition B (AI-first): Coders reviewed CGCA outputs before finalizing their own labels.
- Data Collection: Data included coding records, full GPT chat logs, think-aloud protocols, and post-task interviews (including stimulated recall).
- Analysis: Quantitative analysis measured inter-rater reliability (Cohen's $\kappa$ ) against a reference coding. Qualitative analysis used an inductive thematic approach to identify factors shaping collaboration, characterized on an ordinal scale (weak–moderate–strong) based on discourse evidence.

Key Results

System Performance (RQ1): The initial baseline CGCA showed low precision (6.7%–38.2%). Through iterative refinement of prompt structure and exemplar curation, the final system achieved 64.9% turn-level precision, approaching the 70–75% inter-coder agreement typical of human experts. The study found that structural clarity (decision trees, stepwise reasoning) and high-quality, context-rich examples were more critical than the sheer volume of examples.
Workflow Impact (RQ2): The AI-first condition (Condition B) yielded slightly higher mean Cohen's $\kappa$ values (M ≈ 0.45) compared to the Human-first condition (M ≈ 0.40), though the difference was not statistically significant due to the small sample size. Condition B was associated with reduced coding time per turn. Qualitative analysis revealed that Condition B often triggered anchoring effects, where initial AI outputs influenced subsequent human judgment, whereas Condition A involved more extensive self-reasoning and rule-based negotiation.
User Factors and Collaboration Patterns (RQ3): The study identified two distinct orientations in how researchers engaged with the CGCA:
1. Dialogic Orientation: Participants treated the AI as a "critical friend" or collaborative interlocutor. They used AI outputs to externalize reasoning and recalibrate their decisions. This group, which included novice coders, demonstrated the strongest coding improvements and learning gains.
2. Supervisory Orientation: Participants viewed the AI as an object of scrutiny, maintaining epistemic distance and focusing on verifying the model's reliability rather than engaging in dialogue. This group showed less iterative reasoning and smaller performance gains.
- Dynamic Shift: Positionality was not static; for example, a novice participant shifted from a deferential stance to a more critical, negotiated stance as their confidence grew through interaction.

Key Contributions

Prompt Engineering for Deductive Coding: The study demonstrates that effective AI-assisted deductive coding requires adapting human coding procedures to fit LLM constraints. Specifically, decomposing complex judgments into decision-tree logic and using curated, diverse exemplars yields better results than simply inputting large codebooks or extensive examples.
Socio-Technical Framework: The paper proposes a preliminary framework conceptualizing human–AI collaboration in qualitative coding as a dynamic, reciprocal process. It posits that outcomes are jointly determined by the CGCA system design (prompt architecture), contextual conditions (workflow sequencing), and human coder characteristics (positionality, trust, prior knowledge).
Role of Positionality: The findings challenge the notion that human authority must be strictly supervisory. Instead, a "dialogic" stance—where the human retains final evaluative control but treats the AI as a reasoning partner—appears to yield better coding accuracy and learning outcomes, particularly for less experienced coders.

Significance and Claims
The authors claim that AI-supported dialogue coding should be understood not merely as a technical automation task but as a dynamic, socio-technical process. The study suggests that:

Design Implications: Building effective AI tools requires more than translating human rules; it demands prompt architectures that align with cognitive load theory and LLM processing limits.
Collaborative Dynamics: The effectiveness of human–AI collaboration is heavily influenced by the coder's positionality (how they view their relationship to the AI). Treating the AI as a collaborative partner rather than a subordinate tool can lead to stronger analytic outcomes and learning gains.
Equity in Coding: The CGCA may function as a "levelling mechanism," reducing the performance gap between novice and experienced coders when the novice adopts a dialogic orientation.

The authors remain modest in their claims, acknowledging the study's small scale (n=6), the specific context of UK primary classrooms, and the lack of long-term evidence regarding independent skill development. They emphasize that uniform implementation strategies are unlikely to be effective and that future research is needed to validate these findings across larger samples and different coding frameworks.