🤖 AI

LLM-assisted sentiment analysis for integrated computational and qualitative mixed methods education research: A case study of students' written reflection assignments

This paper presents a mixed-methods case study demonstrating how LLM-assisted sentiment analysis of student reflections can efficiently identify and explain the impact of multiple demographic variables on study abroad experiences, specifically revealing that prior living abroad experience uniquely influences students' sentiments regarding language and communication.

Original authors: Xiomara Gonzalez, Gabriella Coloyan Fleming, Andrew Katz, Maya Denton, Jessica Deters

Published 2026-05-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xiomara Gonzalez, Gabriella Coloyan Fleming, Andrew Katz, Maya Denton, Jessica Deters

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a teacher trying to understand how 51 students felt about their month-long trip to Japan. You have 151 handwritten journals filled with their thoughts, worries, and "aha!" moments. Reading every single word to find patterns is like trying to drink from a firehose: it's overwhelming, time-consuming, and you might miss the subtle currents in the water.

This paper is about building a smart, digital assistant to help researchers drink from that firehose without drowning.

The Problem: The "Needle in a Haystack"

The researchers wanted to know: Do students feel differently about communicating in a foreign language based on who they are? (For example, does having lived abroad before change how you feel about the language barrier?)

Traditionally, a human researcher would have to read every journal, highlight the relevant parts, guess the "mood" (positive, negative, or neutral), and then try to count the results. Doing this for multiple groups of students is like trying to sort a massive pile of mixed-up Lego bricks by color and size entirely by hand. It takes forever, and you might get tired and make mistakes.

The Solution: The "Sentiment Detective" Robot

The researchers introduced a Large Language Model (LLM)—a type of advanced AI—as a co-investigator. Think of the AI not as a replacement for the human, but as a super-fast, tireless intern who can read thousands of pages in seconds.

Here is how they used this "digital intern":

The Human Sets the Rules: The human researcher first read the journals and pulled out the specific sentences about talking and communicating.
The AI Does the Heavy Lifting: They fed these sentences to the AI (specifically an open-source model called Llama3) and asked it to label each sentence as Happy (Positive), Sad/Anxious (Negative), or Just Factual (Neutral).
The Human Double-Checks: The AI isn't perfect. Sometimes it gets confused, like a student who thinks a neutral statement is actually an opinion. The human researcher reviewed the AI's work, fixing the mistakes. This is like a teacher grading the intern's homework to make sure the logic holds up.

The Results: What the "Firehose" Revealed

Once the AI helped sort the data, the researchers could finally see patterns that were too small to spot by eye. They looked at seven different student identities (like gender, income, or first-generation status) to see if any of them changed how students felt about communication.

The Big Discovery:
Out of all the different backgrounds they checked, only one thing mattered: Had the student lived abroad before?

The "Veterans" (Students who had lived abroad before):
- At the start: They were actually more nervous than the others. They knew the language barrier was real and were worried about making mistakes or offending people.
- At the end: They realized their fears were valid but manageable. They saw communication aids (like translation apps) as a "last resort" when they got stuck.
The "Newbies" (Students who had never lived abroad):
- At the start: They were optimistic but had unrealistic expectations. They thought it would be easy or that everyone spoke English.
- In the middle: They hit a wall. They were frustrated because their preparation didn't work, and they felt unprepared.
- At the end: They had a major turnaround. They went from frustrated to confident. They realized that struggling was part of the learning process and started seeing translation apps as helpful tools rather than crutches.

The Takeaway: A New Way to Cook

The paper argues that this "Human + AI" team is a powerful new way to do research.

The AI is the blender: It processes the raw data quickly and consistently.
The Human is the chef: They decide what ingredients to use, taste the dish, and fix the seasoning when the AI adds too much salt.

The researchers found that while the AI was fast and mostly accurate (agreeing with humans about 60-67% of the time, which is considered "substantial" in this field), it still needed a human to clean up its mess. The AI sometimes got chatty or labeled things incorrectly, so the human had to do the final "post-processing."

In short: This study shows that by letting an AI do the boring, repetitive counting of emotions, human researchers can spend their time understanding the story behind the numbers. In this specific case, the story was that previous travel experience changes how you handle the shock of a new language, turning a scary obstacle into a learning opportunity.

Technical Summary: LLM-assisted Sentiment Analysis for Integrated Computational and Qualitative Mixed Methods Education Research

Problem Statement

Written reflection assignments are a vital pedagogical tool in engineering education, offering students opportunities for critical self-assessment and providing rich qualitative data for researchers. However, analyzing these qualitative datasets is traditionally time-intensive, particularly when attempting to compare findings across multiple demographic variables (e.g., gender, income, prior travel experience). Conventional qualitative analysis often limits comparisons to a single variable due to the labor involved. While Large Language Models (LLMs) have emerged as potential research assistants for sentiment analysis and coding, their integration into rigorous, longitudinal mixed-methods workflows—specifically for sensitive student data—remains under-explored. There is a need to determine how LLMs can effectively augment human researchers in identifying sentiment trends across multiple identity variables without compromising data privacy or analytical rigor.

Methodology

The study employs a "quasi-convergent parallel" mixed-methods design, applying both quantitative and qualitative analyses to the same qualitative dataset (student reflections). The research utilizes a longitudinal case study of 51 undergraduate engineering students participating in a one-month study abroad program in Japan.

Data Collection:

Dataset: 151 written reflections collected at three time points (beginning, middle, and end of the program).
Focus: Responses specifically addressing verbal language and communication behaviors were extracted.
Variables: Seven student identity/lived experience variables were analyzed: prior experience living abroad, foreign language knowledge, foreign-born parents, low-income status, first-generation college attendance, first-generation American status, and Asian/Asian American identity.

Analysis Pipeline (Six Stages):

Data Processing & Familiarization: Human researchers read all reflections and wrote analytic memos.
Manual Quote Extraction: Researchers manually isolated quotes related to verbal language and communication.
Hybrid Sentiment Labeling:
- Human: A researcher assigned initial sentiment labels (Positive, Negative, Neutral).
- LLM: An open-source LLM (MetaAI's Llama3, 8 billion parameters) was prompted to assign sentiment labels to the same quotes. The model was run locally via Ollama to ensure data privacy and ethical compliance, as participants did not consent to third-party data sharing.
- Discrepancy Resolution: The human researcher reviewed LLM outputs. Discrepancies were discussed, and human labels were revised if the initial interpretation was deemed inaccurate. The LLM was not re-prompted.
- Reliability: Inter-rater reliability was measured using Cohen's kappa ( $\kappa$ ).
Qualitative Analysis: Inductive thematic coding was performed on the quotes to identify major themes (e.g., prior experiences, preparation, language dynamics, perceived outcomes).
Quantitative Analysis: Statistical tests (Student's t-test or Wilcoxon rank-sum test, depending on normality) were conducted to compare sentiment counts across the seven identity variables. Effect sizes (Cohen's $d$ or Glass rank biserial correlation) were calculated.
Interpretation: Qualitative findings were interpreted in the context of statistically significant quantitative results.

Key Results

1. LLM Performance and Workflow (RQ1):

Efficiency: The LLM processed quotes significantly faster than the human researcher (approx. 2–3 quotes per human labeling time).
Reliability: Inter-rater reliability between the human coder and Llama3 was "substantial" across all time points ( $\kappa = 0.52$ for Reflection 1; $\kappa = 0.64$ for Reflection 3; $\kappa = 0.67$ for Reflection 4).
Collaborative Value: Discrepancies between human and LLM labels were not random. The LLM often assigned affective labels (positive/negative) to statements the human coder viewed as neutral/descriptive. These divergences prompted deeper reflexive discussion and refinement of coding criteria, acting as a "second coder" that deepened interpretative engagement.
Limitations: The workflow required significant upfront technical setup (Python scripting) and iterative prompt engineering. The LLM occasionally generated extraneous text or mislabeled prompt instructions, necessitating human post-processing.

2. Case Study Findings (RQ2):

Significant Variable: Among the seven identity variables tested, prior experience living abroad was the only factor showing statistically significant differences in sentiment expression across all time points.
Longitudinal Trends:
- Students with Prior Living Abroad Experience (LA): Expressed more negative sentiments regarding communication in Reflection 1 (beginning) compared to those without. They struggled to leverage prior language skills in the Japanese context and felt apprehensive about potential misunderstandings. By Reflection 4, their sentiments remained relatively stable but shifted toward viewing communication aids as a "last resort."
- Students without Prior Living Abroad Experience (Not-LA): Expressed more negative sentiments in the middle of the program (Reflection 3) compared to LA students, driven by misaligned expectations about the ease of navigating language barriers and frustration with preparation efforts. However, by Reflection 4, this group showed a significant shift toward positive sentiments, expressing increased confidence and reframing communication challenges as valuable learning opportunities.

Significance and Claims

The paper claims that this workflow offers a feasible method for qualitative researchers to probe multiple demographic variables simultaneously, a task often limited by the time constraints of traditional analysis.

Methodological Contribution: The study demonstrates that LLMs can serve as effective collaborative partners in qualitative research, specifically for sentiment labeling, provided they are used in a "human-in-the-loop" capacity. The use of open-source, locally run models addresses critical ethical concerns regarding data privacy and participant consent.
Educational Insight: The findings highlight that prior international experience significantly shapes how students perceive and navigate communication barriers during study abroad. Students without such experience undergo a distinct emotional trajectory, moving from frustration to growth, whereas experienced students face different challenges related to the applicability of their existing skills.
Practical Implication: The study suggests that educators and administrators can use these insights to tailor pre-departure support: encouraging students with prior experience to reflect on cross-cultural transferability of skills, while helping students without such experience manage expectations and focus on practical communication strategies rather than just vocabulary.

The authors maintain a modest stance, acknowledging that the effect sizes observed were small to medium, limiting generalizability to other contexts (e.g., longer programs or language-immersion specific programs). They emphasize that LLMs do not replace qualitative analysis but can accelerate the identification of trends to guide deeper human interpretation.