So Many Opinions, So Many LLMs: Comparing Large Language Models to Traditional Machine Learning for Open- Ended Survey Analysis
This study compares large language models (LLMs) to traditional machine learning for analyzing open-ended survey responses, finding that while LLMs achieve superior classification accuracy in understanding complex themes and sentiments, they present significant trade-offs regarding consistency and explainability in their reasoning.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you have a giant box of thousands of letters from students, all written in their own messy, emotional handwriting. Some are happy, some are angry, some are sarcastic, and some are just plain confused. Your job? To read every single one and sort them into six emotion bins: Joy, Surprise, Love, Anger, Sadness, and Fear.
For a long time, researchers tried to do this with "Old School" computer programs. Think of these programs like a very strict librarian who only knows how to count specific words. If the librarian sees the word "great," they slap a "Joy" sticker on the letter. If they see "bad," they slap a "Sad" sticker. It's fast, and you can see exactly why the librarian made that choice. But here's the problem: these librarians are easily fooled. They miss the sarcasm, the exhaustion hidden behind a thank-you, or the subtle hints that a student is actually furious even though they used polite words.
In this study, the researchers decided to test a new team of helpers: Large Language Models (LLMs). You can think of these as super-smart, well-read robots that have read almost everything on the internet. Unlike the strict librarians, these robots try to understand the vibe and the context of the sentence, not just the individual words.
The Big Showdown
The researchers pitted these two teams against each other using real student survey data from 2019 and 2021 (293 letters from 2021 and 142 from 2019). They didn't teach the robots anything new; they just asked them to sort the letters using a simple instruction, a method called "zero-shot" prompting.
What the Old Librarians Did
The traditional computer models (like Support Vector Classifiers and Random Forests) were okay at sorting the letters when they were trained on a specific set of examples. They got about 90% accuracy on their practice tests. But when they faced the real, messy student letters, they started acting a bit crazy.
- In the 2019 batch, one model decided that 99.3% of the letters were "Joy." It basically ignored all other emotions!
- In the 2021 batch, the models disagreed wildly with each other. One thought a letter was "Love," while another thought it was "Sadness."
- The researchers found that these old models were too sensitive to specific words (like "great" or "good") and couldn't handle the tricky, human side of language. They were brittle, meaning if the data changed even a little, their sorting fell apart.
What the New Robots Did
The team of LLMs (including GPT-4-Turbo, Claude 3.5, Meta's LLaMA 3, and Perplexity Sonar Pro) did something different. They didn't just count words; they tried to understand the feeling.
- They agreed with each other: When the four different robots looked at the same pile of letters, they mostly sorted them the same way. For example, in the 2021 data, all four robots agreed exactly on how many letters were "Anger" (46 letters).
- They handled the nuance: They were much better at spotting complex emotions, like when a student is sarcastic or when their "thank you" is actually laced with exhaustion.
- The Numbers: While the paper doesn't give a single "accuracy score" for the robots (because there's no human answer key to compare them to), their consistency suggests they are much more reliable for this kind of messy, real-world data than the old models.
The Catch
The robots aren't perfect magicians. The researchers point out a few things to watch out for:
- The "Love" Glitch: All four robots seemed to sort more letters as "Love" than "Joy." The researchers suspect this is because the robots learned from social media, where people say "I love this class!" to mean "I really like it," not that they have deep romantic feelings.
- The Black Box: Unlike the old librarians, you can't always see why the robot made a choice. It's harder to audit their work.
- Cost and Updates: These robots are expensive to run, and the companies that make them update them often. If you run the same prompt next year, the robot might give a slightly different answer because it's a "new version."
The Verdict
The study suggests that if you want to analyze thousands of open-ended student responses, the new AI robots are the better tool. They are much better at understanding the complex, human side of language and are more consistent across different groups of students.
However, the researchers don't say this is a "solved" problem. They suggest that for high-stakes decisions (like changing school policies), the best approach is a team effort: let the robots do the heavy lifting to sort the letters, but have humans check the work to make sure the robots aren't getting tricked by sarcasm or bias. It's a powerful new way to listen to students, but it still needs a human hand on the wheel.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.