Automated Coding of Communications in Collaborative Problem-solving Tasks Using ChatGPT

1. Problem Statement

Collaborative Problem Solving (CPS) is a critical 21st-century skill, yet assessing it at scale is hindered by the bottleneck of coding communication data.

• The Bottleneck: Traditionally, extracting evidence from text-based chat logs requires human raters to meticulously categorize conversational turns based on complex coding frameworks. This process is labor-intensive, expensive, and limits the scalability of research and large-scale assessments (e.g., PISA).
• The Gap: While traditional automated coding (using supervised machine learning) exists, it requires large, human-coded training datasets, which perpetuates the labor burden. Large Language Models (LLMs) like ChatGPT offer the potential for zero-shot or few-shot coding, bypassing the need for extensive training data. However, it remains an empirical question whether LLMs can accurately code complex, domain-specific CPS communication data compared to human experts, and how factors like task type, coding framework design, and model architecture influence performance.

2. Methodology

The study employed a systematic empirical investigation using five distinct CPS datasets and two coding frameworks to evaluate four leading OpenAI models: GPT-4, GPT-4o, GPT-o1-mini, and GPT-o3-mini.

Datasets

• Science Tasks (2): Condensation and Volcano tasks. Pairs of participants solved scientific problems via text chat.
• General Cognitive Tasks (3): Negotiation, Decision-Making, and Letter-to-Number. Teams of four participants collaborated on reasoning and negotiation tasks.
• Data Volume: Approximately 1,500 chat turns per task (randomly sampled from crowdsourced sessions on MTurk and Prolific).

Coding Frameworks

• Framework 1 (Theory-driven): Developed from CSCL research and PISA 2015. Used for Science tasks. Categories: Sharing ideas, Negotiating ideas, Regulating problem solving, Maintaining communication.
• Framework 2 (Data-driven): Developed combining theory and empirical negotiation data. Used for General Cognitive tasks. Categories: Maintaining communication, Staying on task, Eliciting information, Sharing information, Acknowledging.

Experimental Design

• Prompt Engineering: Prompts included task goals, framework definitions, ~10 expert examples per category, and input/output specifications.
• Evaluation Metric: Cohen's Kappa was used to measure agreement between LLMs and human expert raters (used as ground truth).
• Iterative Refinement: The study tested if incorporating feedback from miscoded cases (error analysis) into the prompt could improve accuracy.
• Control Variables: Temperature set to 0; fixed random seeds; batching of ~70 turns per API call to manage context windows.

3. Key Contributions

1. Feasibility of Zero-Shot Coding: Demonstrated that ChatGPT can effectively code CPS communication data without prior supervised training, significantly reducing the need for human-coded training sets.
2. Model Selection Insights: Found that newer "reasoning" models (GPT-o1-mini, GPT-o3-mini) did not outperform GPT-4o on this specific task, suggesting that advanced reasoning capabilities do not necessarily translate to better classification accuracy in this domain.
3. Framework Sensitivity: Identified that the design of the coding framework is a critical determinant of success. Frameworks grounded in empirical data (Framework 2) yielded higher LLM accuracy than purely theoretical frameworks (Framework 1).
4. Prompt Refinement Strategy: Provided empirical evidence on the efficacy of feedback-driven prompt engineering, showing it yields mixed results depending on the task and initial performance levels.

4. Key Results

Performance by Model (RQ1)

• GPT-4o achieved the highest overall performance.
• Reasoning Models (o1/o3): Did not outperform GPT-4o. In some cases, they performed slightly worse, indicating that the "chain-of-thought" internal processing did not benefit this specific classification task and may have introduced noise or cost inefficiencies.

Performance by Task Type (RQ2)

• General Cognitive Tasks: LLMs achieved agreement levels comparable to, or even exceeding, human-to-human agreement (e.g., Letter-to-Number Kappa ~0.73).
• Science Tasks: LLMs consistently underperformed compared to human raters (Kappa ~0.57–0.60 vs. Human-Human ~0.68–0.78).
• Scientific Terminology: While turns containing scientific terms showed lower agreement, the performance gap between humans and LLMs persisted even in turns without scientific terms. Thus, terminology alone did not explain the discrepancy; the nature of the task and framework was the primary driver.

Impact of Coding Framework (RQ3)

• Framework 2 (Data-driven) resulted in significantly higher accuracy for LLMs across all three general tasks.
• Framework 1 (Theory-driven) proved difficult for LLMs to interpret, leading to lower accuracy. This suggests LLMs struggle with abstract, theoretically dense definitions compared to concrete, empirically derived categories.

Prompt Refinement (RQ4)

• Condensation Task: Adding feedback on miscoded cases to the prompt did not improve overall Kappa (remained ~0.64). Errors shifted to other categories, suggesting the original prompt was near-optimal.
• Volcano Task: Incorporating feedback improved Kappa from 0.604 to 0.637 (on the full dataset). This indicates that prompt refinement is task-dependent and effective only when the initial errors are systematic and correctable via examples.

5. Significance and Implications

• Scalability: This research offers a viable pathway to scale CPS assessments by reducing the reliance on expensive human coding. LLMs can process large volumes of chat data efficiently.
• Cost-Efficiency: Researchers should prioritize GPT-4o over newer, more expensive reasoning models for coding tasks, optimizing the balance between cost and performance.
• Framework Design: The study highlights that for automated coding to succeed, coding frameworks must be designed with the "machine reader" in mind—favoring clear, empirically grounded definitions over abstract theoretical constructs.
• Practical Application: While LLMs are not yet a full replacement for human raters (especially in complex, theory-heavy domains), they serve as a powerful complement. They can handle the bulk of coding, with human raters focusing on edge cases or validation, thereby accelerating the assessment of 21st-century skills.

Limitations Noted: The study acknowledges that results are specific to the snapshot versions of models used; fairness/bias across demographic groups requires further investigation; and LLMs currently do not fully meet the validity standards of human raters for high-stakes assessment without human oversight.