💬 NLP

Using LLMs for Knowledge Component-level Correctness Labeling in Open-ended Coding Problems

This paper proposes an automated framework leveraging large language models to generate fine-grained knowledge component-level correctness labels from open-ended student code, demonstrating that this approach yields more cognitively consistent learning curves and improved predictive performance compared to traditional problem-level labeling methods.

Original authors: Zhangqi Duan, Arnav Kankaria, Dhruv Kartik, Andrew Lan

Published 2026-03-31

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zhangqi Duan, Arnav Kankaria, Dhruv Kartik, Andrew Lan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a teacher grading a student's essay. If the essay has a few typos and a wrong conclusion, you might give it a "B" or a "C." That's a problem-level grade. It tells you the final result, but it doesn't tell you what the student actually learned. Did they understand the grammar? Did they grasp the main argument? Or did they just get lucky with the conclusion?

In computer science education, this is a huge problem. When a student writes code to solve a problem, the computer usually just says "Pass" or "Fail." If the code fails, we assume the student failed at everything. But in reality, a student might have perfectly understood how to use a "loop" (a repeating action) but messed up the "string manipulation" (handling text).

The Problem: The "All-or-Nothing" Mistake
Traditional systems treat coding like a light switch: it's either ON (correct) or OFF (incorrect). This is like grading a math test by only looking at the final answer. If a student gets the answer wrong, the system assumes they don't know any of the math steps leading up to it. This confuses the computer models that try to predict how students learn, making their "learning curves" look jagged and unrealistic.

The Solution: The "AI Teaching Assistant"
The authors of this paper built a smart system using Large Language Models (LLMs)—the same type of AI that powers chatbots—to act as a super-observant teaching assistant. Instead of just looking at the final "Pass/Fail" result, this AI reads the student's code line-by-line to check specific skills, which they call Knowledge Components (KCs).

Think of a coding problem as a recipe for a cake.

KC 1: Do you know how to preheat the oven?
KC 2: Do you know how to mix the batter?
KC 3: Do you know how to frost the cake?

If the cake comes out burnt (the code fails), a traditional system says, "You failed at baking."
The new AI system says, "Wait! The student preheated the oven perfectly and mixed the batter correctly. They just forgot to frost the cake. Let's give them credit for the first two skills and only mark the frosting as incorrect."

How It Works (The Magic Trick)

The Detective Work: The AI reads the student's code and the list of skills required for the problem. It acts like a detective, asking: "Did the student use a loop? Yes. Did they use it correctly? Yes. Did they use string manipulation? Yes, but they made a mistake here."
The Context Clue: Sometimes, a student's first attempt is a messy draft, and their last attempt is a polished solution. The AI looks at the student's final attempt to figure out what strategy they were trying to use, then goes back to grade their first attempt based on that strategy. It's like looking at a student's final essay to understand what they were trying to say, then grading their first rough draft with that context in mind.
The Chain of Thought: The AI doesn't just guess. It is prompted to "think out loud" (Chain of Thought), explaining its reasoning before giving a grade. This makes the AI much more accurate, similar to how a human teacher explains why an answer is wrong.

The Results: Smoother Learning Curves
When the researchers tested this, the results were like watching a blurry photo suddenly come into focus.

Old Way: The learning curve looked like a messy scribble. It didn't make sense because the system thought students were failing skills they actually knew.
New Way: The learning curve became a smooth, downward slope. This matches how humans actually learn: as you practice a specific skill (like loops), you make fewer mistakes over time.

Why This Matters
This isn't just about better grades. It's about personalized learning.

If a system knows a student is great at loops but bad at strings, it can give them practice problems that focus only on strings.
If the system thinks the student failed everything, it might force them to redo the whole lesson, wasting their time.

In a Nutshell
This paper introduces a way to use AI to grade the ingredients of a student's code, not just the final dish. By giving credit for the parts a student got right, even if the final code failed, the system creates a much clearer, more accurate picture of how students learn. It turns a blunt "Fail" into a helpful roadmap for improvement.

1. Problem Statement

In educational data mining and student modeling, Knowledge Components (KCs) represent fine-grained skills or concepts. Accurate labeling of student mastery at the KC level is crucial for:

Generating interpretable learning curves (e.g., adhering to the Power Law of Practice, where error rates decrease with practice).
Providing personalized feedback and curriculum sequencing.
Training predictive models like the Additive Factors Model (AFM).

The Core Challenge:

Data Scarcity: Real-world datasets for open-ended programming tasks typically only provide problem-level correctness (binary: correct/incorrect for the whole submission), not KC-level labels.
The "Coarse-Grained" Flaw: Simply propagating problem-level correctness to all associated KCs is flawed. A student may master some KCs (e.g., loop logic) while failing others (e.g., string manipulation) within a single incorrect submission. Treating all KCs as "incorrect" obscures partial mastery, leading to poorly fitted learning curves and inaccurate predictive models.
Manual Labeling Limitations: Manually annotating KCs for open-ended code is labor-intensive, prone to bias, and does not scale.
Existing Automated Limitations: Previous automated methods rely on Abstract Syntax Trees (AST) to match syntactic nodes against reference code. These methods capture surface-level syntax but often miss higher-level algorithmic concepts and lack interpretability.

2. Methodology

The authors propose an automated framework leveraging Large Language Models (LLMs) to generate fine-grained KC-level correctness labels directly from student code.

A. Prompting Strategy

Model: The framework utilizes GPT-4o and Qwen3 Coder 30B.
Input: For each problem-submission pair, the LLM receives:
1. The problem statement.
2. The student's code submission.
3. The set of KCs mapped to the problem.
4. In-context examples: Few-shot demonstrations manually selected to guide the output format.
Chain-of-Thought (CoT): The model is instructed to reason step-by-step:
1. Determine if a specific KC is present in the code.
2. Evaluate if the KC is applied correctly.
3. Output a binary label (Correct/Incorrect) for each KC.
Handling Missing KCs: If a required KC is not used in the code, it is labeled as incorrect.

B. Automated KC Generation

To address the variability in student solution strategies (which human-defined KC sets often miss), the framework supports LLM-generated KCs:

Clustering: Correct solutions are clustered using CodeBERT embeddings.
Generation: GPT-4o is prompted with these diverse exemplars to generate candidate KCs.
Refinement: Generated KCs are clustered and summarized to create a refined set at a controlled abstraction level.

C. Temporal Context-Aware Code-KC Mapping

A critical innovation is aligning KCs specifically to the student's intended solution strategy rather than just the problem definition.

Problem: A student's first attempt (used for learning curve analysis) is often incomplete, while their last attempt is near-correct.
Mechanism:
1. Identify the "closest" correct reference code to the student's last attempt using cosine distance on CodeBERT embeddings.
2. Determine which subset of KCs that specific reference code uses.
3. Apply these specific KCs to label the student's first attempt.
Rationale: The last attempt reveals the student's full solution strategy, ensuring the KCs being evaluated are relevant to what the student was actually trying to build, even if their first attempt failed.

3. Key Contributions

First LLM-based KC Labeling: This is the first study to systematically apply LLMs for KC-level correctness labeling in open-ended programming.
Context-Aware Mapping: Introduction of a mechanism that aligns KCs to individual student solution strategies using temporal context (comparing first and last attempts).
Chain-of-Thought Integration: Demonstrating that CoT prompting significantly improves label accuracy compared to direct binary classification.
Framework Flexibility: The method works with both human-defined and automatically generated KC sets.

4. Experimental Results

Dataset: CodeWorkout (10,834 submissions from 246 students on 50 Java problems).

Evaluation Metrics:

Learning Curve Fit: Root Mean Square Error (RMSE) and $R^2$ against the Power Law of Practice.
Predictive Performance: Area Under Curve (AUC) using the Additive Factors Model (AFM).

Key Findings:

Superior Fit: LLM-based labeling (both GPT-4o and Qwen3) significantly outperformed the baseline (problem-level correctness) in fitting the Power Law of Practice.
- Example: Using GPT-4o with "Selected" KCs, $R^2$ improved from 0.253 (Baseline) to 0.383.
Predictive Power: AFM models trained on LLM-generated labels showed higher AUC (up to 0.676 with Qwen3) compared to the baseline (0.529).
Ablation Study: Removing Chain-of-Thought (CoT) prompting resulted in a significant drop in performance, confirming that intermediate reasoning is essential for accurate KC diagnosis.
Human Evaluation:
- Inter-annotator agreement (Human vs. Human): Cohen's Kappa = 0.86 (Almost perfect).
- LLM vs. Human agreement: Cohen's Kappa = 0.74 (Substantial agreement).

Visual Evidence:
Learning curves generated with LLM labels showed a consistent downward trend in error rates (adhering to cognitive theory). In contrast, curves based on problem-level correctness showed erratic behavior, failing to decrease over time.

5. Significance and Implications

Scalability: Provides a scalable alternative to manual expert annotation for open-ended coding tasks.
Cognitive Grounding: The resulting data aligns better with cognitive theories of learning (Power Law of Practice), validating the quality of the labels.
Instructional Impact: Accurate KC-level labels enable adaptive learning systems to:
- Detect partial mastery (e.g., a student understands loops but not string manipulation).
- Avoid misdiagnosis that leads to skipping necessary practice or providing redundant support.
- Deliver targeted, fine-grained feedback rather than generic "incorrect" messages.
Generalizability: The framework is robust across different KC definitions (human vs. generated) and different LLM architectures (proprietary vs. open-weight).

Conclusion

The paper demonstrates that LLMs, when guided by Chain-of-Thought prompting and temporal context-aware mapping, can effectively replace coarse problem-level labels with fine-grained KC-level correctness labels. This shift significantly improves the fidelity of student learning curves and the predictive power of educational models, offering a scalable path toward more intelligent and personalized coding education.