💻 computer science

Identifying Quality Indicators in Student Self-Reflections in Software Engineering

This study proposes an eight-indicator framework for assessing student self-reflections in software engineering and validates a fine-tuned RoBERTa model that achieves human-level agreement in automatically classifying these reflections to provide scalable, timely, and structured feedback.

Original authors: Matthew Minish, Matthias Galster, Fabian Gilson

Published 2026-06-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Matthew Minish, Matthias Galster, Fabian Gilson

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a teacher grading a stack of diaries. But these aren't just any diaries; they are written by software engineering students who are building complex apps in teams. The goal of these diaries isn't just to say "I did this," but to show how the student thinks, feels, and plans to improve. This is called reflection.

The problem? Grading these diaries is exhausting. If a teacher reads 50 of them, they might just write a generic note like, "Good job, try to think deeper next time." That's not very helpful. The student doesn't know what to think deeper about. Did they forget to consider their teammate's feelings? Did they fail to explain why a bug happened?

This paper is about building a smart robot assistant that can read these diaries instantly and tell the teacher (and the student) exactly which parts of the thinking process are missing.

Here is how they did it, broken down into simple steps:

1. The "Checklist" (The Framework)

First, the researchers couldn't just ask the robot to "grade the diary." They needed a specific checklist. They created 8 specific ingredients that make a good reflection, kind of like a recipe for a perfect cake:

Description: Just stating the facts (e.g., "We fixed the bug").
Understanding: Showing you get the big picture (e.g., "Teamwork was great").
Feelings: Sharing emotions (e.g., "I felt proud" or "I was frustrated").
Reasoning: Explaining the cause and effect (e.g., "The bug happened because we didn't talk about the files").
Perspective: Thinking about what others think (e.g., "My teammate was confused by my code").
New Learning: Admitting what you learned (e.g., "I learned how to use this new tool").
Hindsight: Looking back and saying, "I should have done it differently" (e.g., "I should have spoken up sooner").
Future Intention: Planning ahead (e.g., "Next week, I will check the code more").

They spent a lot of time teaching three human experts how to use this checklist so they all agreed on what counted. It was like training three judges to score a gymnastics routine so they all gave the same score.

2. The "Race" (Testing the Robots)

Next, they built two different types of "robots" (computer programs) to read the diaries and check for these 8 ingredients.

The Specialized Worker (Encoder-Only Models): Think of this like a specialized librarian. It's been trained specifically on this task. It's fast, efficient, and knows exactly what to look for. They used a model called RoBERTa.
The General Genius (Decoder-Only Models): Think of this like a super-smart but slow oracle. It's a massive Large Language Model (like the ones powering chatbots) that hasn't been specifically trained on this task but is told, "Here are the rules, please grade this." They tested models like GPT and Qwen.

3. The Results: Speed vs. Brains

The race results were clear:

The Specialized Worker (RoBERTa) won. It was incredibly accurate, matching the human experts' grades almost perfectly. Even better, it was lightning fast. It could grade a diary in less than a tenth of a second. It's like a calculator that solves a math problem instantly.
The General Genius (LLMs) were slow and less accurate. While they were smart, they took 20 to 45 seconds to grade just one diary. That's like waiting for a slow elevator every time you want to check a student's work. Also, they missed more details than the specialized worker.

4. Why This Matters

The paper concludes that if you want to give students instant, specific feedback, you need the "Specialized Worker."

Instead of a teacher saying, "Your reflection is vague," the robot can instantly say:

"You mentioned what you did (Description), but you didn't explain why it happened (Reasoning)."
"You talked about your own feelings, but you didn't mention how your teammate felt (Perspective)."

This turns a vague "try harder" into a specific "try thinking about your teammate's perspective."

The Catch (Limitations)

The authors are honest about the limits:

The "Tricky" Ingredients: Two of the 8 ingredients (Reasoning and Perspective) were hard even for the human experts to agree on. The robot is good, but it's not perfect on these specific tricky parts.
One School Only: They tested this on students from just one university in New Zealand. We don't know if it works exactly the same way for students in Japan or for students writing free-form essays instead of answering specific questions.
Not for Grades: The authors say this tool is for helping students learn (formative), not for giving them a final grade (summative). It's a coach, not a referee.

In a nutshell: The researchers built a fast, specialized robot that can read student diaries and spot exactly what kind of thinking is missing. It's much faster and more accurate than using a giant, slow AI chatbot, and it helps teachers give students the specific advice they need to get better at thinking.

Technical Summary: Identifying Quality Indicators in Student Self-Reflections in Software Engineering

Problem Statement
Reflection is a critical skill in software engineering education and professional practice, yet assessing written reflections to provide actionable feedback is resource-intensive for instructors. Manual assessment often yields broad, non-specific feedback (e.g., "your reflection could be deeper") rather than identifying specific dimensions of reflection that a student has addressed or neglected. Furthermore, in project-based software engineering courses where reflection occurs iteratively throughout a project lifecycle, delayed feedback is often ineffective. There is a lack of automated approaches specifically designed to classify reflective writing within the collaborative, iterative context of software engineering project courses.

Methodology
The study employs a multi-stage approach to develop a reflective writing framework and an automated classifier:

Framework Development: The authors adapted existing reflective writing frameworks (specifically those by Ullmann et al. and Alrashidi et al.) to the software engineering context. Through iterative refinement, they established an eight-indicator scheme:
- Description: Factual recounting of events.
- Understanding: Subjective interpretation of goals or experiences.
- Feelings: Identification of emotions (e.g., pride, frustration).
- Reasoning: In-depth analysis of concrete causes and effects.
- Perspective: Consideration of alternative viewpoints (peers, team members).
- New Learning: Acquisition of skills or insights.
- Hindsight: Retrospective analysis of how a situation could have been approached differently.
- Future Intention: Prospective planning for future actions.
- Note: The framework splits the traditional "Future Action" indicator into "Hindsight" (retrospective) and "Future Intention" (prospective) to align with the specific three-prompt structure used in the study.
Data Annotation: A dataset of 1,518 student responses (6,704 sentences) was collected from a third-year software engineering project course at the University of Canterbury. Students responded weekly to three prompts regarding what went well, what did not, and what they would do differently. Three annotators labeled the data, achieving moderate to reliable inter-rater agreement (Krippendorff's $\alpha$ ) after iterative guideline refinement.
Model Training and Evaluation:
- Encoder-only Models: Five transformer models (BERT, DistilBERT, ALBERT, DeBERTa, RoBERTa) were fine-tuned on the annotated dataset using a multi-label classification approach with Binary Cross Entropy loss and inverse frequency weighting to handle class imbalance. Hyperparameter tuning was performed using Optuna.
- Decoder-only Models: Two large language models (GPT-OSS:20b and Qwen3:14b) were evaluated using zero-shot prompting with extended reasoning capabilities. These were run locally to ensure privacy.
- Comparison: Performance was measured using Hamming loss, exact match accuracy, and per-indicator F1-scores. Computational efficiency was also compared.

Key Contributions

Adapted Framework: An eight-indicator reflective writing framework tailored for software engineering project courses, validated by human instructors with moderate to strong inter-rater agreement.
Automated Classifier: A fine-tuned RoBERTa model capable of identifying reflection indicators with human-level agreement on most dimensions.
Architectural Comparison: A comprehensive comparison demonstrating that fine-tuned encoder-only models substantially outperform zero-shot decoder-only LLMs in both accuracy and computational speed for this specific task.
Deployment Guidelines: Practical guidance for educators, including two model variants (F1-optimised for accuracy and Recall-optimised for safety) to suit different pedagogical priorities.

Results

Model Performance: The fine-tuned RoBERTa model achieved the strongest performance, with a Macro F1-score of 0.809 and an exact match accuracy of 0.380. It significantly outperformed the decoder-only models, which achieved exact match accuracies of 0.229 (Qwen3:14b) and 0.189 (GPT-OSS:20b).
Indicator Difficulty: Classification performance closely mirrored human inter-rater agreement. Indicators with high human agreement (e.g., Future Intention, Hindsight) were classified well by the models. Indicators with lower human agreement (Reasoning and Perspective) remained challenging for both humans and machines, though the RoBERTa model still achieved the best results among all tested approaches.
Efficiency: The encoder-only RoBERTa model classified a single reflection in under 0.1 seconds on consumer-grade hardware. In contrast, the decoder-only models required 20–45 seconds per classification, making them impractical for real-time feedback.
Human Agreement: When treated as a fourth annotator, the F1-optimised RoBERTa model maintained or exceeded human-to-human agreement levels on six of the eight indicators.

Significance and Claims
The paper claims that its framework and automated classifier offer a viable means to provide timely, structured feedback on student reflections in software engineering education. By automating the identification of specific reflection dimensions, the approach addresses the limitation of broad, non-specific feedback, allowing educators to direct student attention to overlooked aspects of their reflection (e.g., prompting for reasoning or perspective).

The authors emphasize that the fine-tuned encoder-only models enable efficient, scalable assessment that reduces instructor workload while supporting formative assessment. They explicitly state that the system is designed for formative feedback (guiding learning) rather than summative assessment (grading), citing the inherent subjectivity of reflection and the risk of students "gaming" the system if grades were directly tied to automated classifications. The study concludes that while the framework is robust for high-agreement indicators, the moderate agreement on Reasoning and Perspective suggests these should be used with caution, potentially as prompts for further reflection rather than definitive assessments.

1. The "Checklist" (The Framework)

2. The "Race" (Testing the Robots)

3. The Results: Speed vs. Brains

4. Why This Matters

The Catch (Limitations)

More like this