Fine-Tuning Large Language Models for Codebook-Guided Coding of Students' Mathematics Metaphor Responses

Imagine you are a detective trying to solve a mystery, but instead of fingerprints, you are looking for clues in the way people talk about their feelings. In the world of education, researchers often ask students to write down their thoughts in their own words—like, "If math were a food, what would it be?" These open-ended answers are gold mines for understanding how kids feel about school, but they are a nightmare to read through one by one. It takes a long time for human experts to read thousands of these stories and sort them into categories, like "happy," "frustrated," or "confused." This is where Artificial Intelligence (AI) steps in. Specifically, Large Language Models (LLMs) are like super-smart robots that can read and understand human language. The big question researchers have been asking is: Can we teach these robots to act like expert human detectives? And if we do, can we make them work on our own computers to keep student secrets safe, rather than sending private data to big tech companies?

This study dives into that exact question. The researchers took a collection of 2,265 real student responses about math metaphors and taught two smaller, open-source AI models to code them just like a human expert would. They used a special training technique called "fine-tuning," which is like giving the robot a crash course using the human experts' answer keys. The results were surprising: after this training, the smaller, cheaper AI models became so good at the job that they often beat the massive, expensive, "black-box" AI models that companies usually sell. The study suggests that with the right training, we can have powerful, private, and accurate AI tools right in our own classrooms to help understand how students feel about math, without needing to send their data anywhere else.

The Detective's Dilemma: Reading Between the Lines

Think of a student's metaphor as a secret code. When a student says, "Math is like a dog because it's a loyal friend," they aren't just talking about pets; they are telling us they feel safe and happy with math. But if they say, "Math is like a mosquito because it won't leave me alone," they are saying math is annoying and inescapable. These are complex feelings wrapped in simple words.

For decades, researchers have relied on human experts to decode these messages. It's a bit like having a team of translators trying to translate a library of books by hand. It's slow, expensive, and hard to scale up. If you have 100 students, a human can do it. If you have 100,000, the humans get tired, and the process grinds to a halt.

Enter the Large Language Model (LLM). You can think of an LLM as a robot that has read almost everything on the internet. It's incredibly good at guessing what comes next in a sentence and understanding context. But, just like a smart student who hasn't taken the specific class you are teaching, a generic robot might not know exactly how you want it to sort these metaphors. It might guess "happy" when you wanted "neutral," or it might miss the subtle difference between "challenging" and "threatening."

Furthermore, there's a privacy problem. Most of the super-powerful robots (like the ones from big tech companies) live on their servers. To use them, you have to send your students' private writing to them. For schools, this is like handing a diary full of secrets to a stranger. The researchers wanted to know if they could build a robot that lives on their own computer, is trained specifically on the job, and keeps the secrets safe.

The Experiment: Teaching the Robot the Rules

The researchers set up a race. On one side, they had the "Big Tech" champions: two powerful, proprietary models (GPT-4o mini and GPT-5 mini) that were used "as is." They were given a set of rules (a codebook) and asked to sort the student metaphors. They didn't get any special training; they just had to figure it out from the instructions.

On the other side, they had two "Open-Weight" models (DeepSeek-R1 1.5B and Mistral 7B). These are smaller, less expensive models that anyone can download and run on their own servers. The researchers took these models and gave them a "crash course" using a technique called LoRA-based supervised fine-tuning.

Imagine you are teaching a dog to fetch. The "prompt-only" method is like just telling the dog, "Go get the ball!" over and over. The "fine-tuning" method is like taking the dog through a training camp where you show it exactly how to grab the ball, bring it back, and drop it in your hand, using a thousand examples of a human doing it perfectly. The researchers fed the open-weight models 2,265 examples of student metaphors that had already been correctly sorted by human experts. The models learned to mimic the human experts' decisions.

The task had two parts:

1. Valence-Intensity Coding: How strong is the feeling? Is it very negative (1), neutral (3), or very positive (5)?
2. Thematic Coding: What is the story? Is math a "tool," a "threat," a "journey," or a "boring chore"?

The Results: The Underdogs Win

The results of the race were clear and exciting. Before the training, the smaller open-weight models were terrible at the job. They were guessing wildly. But after the "crash course" (fine-tuning), they transformed.

The Performance Leap:
The fine-tuned models didn't just get a little better; they got much better.

• For the "food" metaphors, the DeepSeek model's accuracy jumped from 0.369 to 0.787.
• The Mistral model's accuracy on food metaphors went from 0.207 to 0.778.
• On the "animal" metaphors, the Mistral model improved from 0.267 to 0.766.

In the world of AI, these numbers are huge. It means the trained robots were now agreeing with the human experts almost as often as two human experts agree with each other.

Beating the Giants:
Here is the twist: The trained, smaller models actually beat the big, expensive, "prompt-only" models in many categories.

• The fine-tuned Mistral 7B model outperformed both GPT-4o mini and GPT-5 mini on almost every metric for both food and animal metaphors.
• The only time the big models won was on very rare, specific themes where the smaller models still struggled a bit, but even then, the gap was small.

The "Stability" Test:
One of the biggest problems with AI is that if you ask it the same question twice, it might give you two different answers. This is bad for science. The researchers tested this by running the models three times.

• The untrained DeepSeek model was all over the place, with a stability score of only 0.592.
• But after training, its stability skyrocketed to 0.992.
• The fine-tuned Mistral model was perfect, scoring 1.000.
• Even the big commercial models weren't as stable as the trained open models (GPT-5 mini scored 0.644 on thematic stability).

This suggests that training the model on specific examples makes it not only smarter but also more consistent and reliable.

What This Means for the Future

The study suggests that we don't need to rely on massive, expensive, and privacy-risky AI systems to analyze student feelings. By taking smaller, open models and teaching them with human examples, we can create tools that are:

1. Accurate: They match human experts.
2. Private: They can run on a school's own computer, keeping student data safe.
3. Scalable: They can handle thousands of responses in seconds.

However, the researchers are careful to note that this isn't a magic wand for everything. The models still struggled a bit with very rare themes (like specific societal views on math), suggesting that if a topic is very uncommon, the robot might need even more examples to learn it. Also, the study only looked at 6th to 8th graders talking about food and animals, so we don't know yet if it works for high schoolers talking about algebra or college students talking about calculus.

But the main takeaway is a hopeful one: We can build our own "detective robots" that are smart, safe, and ready to help teachers understand the hidden feelings in their students' words. The future of educational measurement might not be in the cloud, but right there on the school server, trained by the very experts who know the students best.