💻 computer science

Artificial Intelligence in Surgical Qualitative Research: A Comparison of Human and AI-Assisted Thematic Analysis

This study compares human and AI-assisted thematic analysis in surgical qualitative research, finding that while both methods identify similar overarching themes, human-generated codebooks yield higher inter-coder reliability due to clearer thematic boundaries, suggesting a hybrid approach combining AI efficiency with human refinement may be optimal.

Original authors: Bonnie E Laingen, Wesley H Iobst, Jarrett Dobbins, Jacob W Johnson, Gabriel E Cambronero, Lucas P Neff, Maggie E Bosley

Published 2026-06-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Bonnie E Laingen, Wesley H Iobst, Jarrett Dobbins, Jacob W Johnson, Gabriel E Cambronero, Lucas P Neff, Maggie E Bosley

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a giant box of 200 handwritten notes from surgeons. Each note answers one question: "What is the hardest thing about doing a specific surgery (LCBDE) at your hospital?"

Your goal is to read all these notes, sort them into piles based on the problems mentioned, and give each pile a label. This process is called Thematic Analysis. Usually, humans do this by reading, arguing, and refining the labels until they agree on what each pile means.

This paper asks a simple question: Can a robot (Artificial Intelligence) do this sorting job just as well as a human?

Here is the story of how they tested it, using some simple analogies.

The Two Teams

The researchers set up a race between two teams to sort the same 200 notes:

The Human Team: Two doctors read the notes. They started with no labels. As they read, they created their own list of categories (like "Not enough money," "Not enough time," "Bad equipment"). They kept refining these labels until they were very clear and distinct. This is their Human Codebook.
The AI Team: They fed all 200 notes into a powerful AI (ChatGPT) in one big batch. They told the AI, "Look at these notes and make your own list of categories." The AI did this instantly without any human help or examples. This is the AI Codebook.

The Results: Who Sorted Better?

Both teams managed to find the same big picture problems. Whether it was a human or a robot reading the notes, they both agreed that the main issues were:

Equipment problems
Money/Cost issues
Time constraints
Lack of support from bosses
Not doing the surgery often enough (Low Volume)

However, there was a difference in how they sorted the notes.

The "Fuzzy Pile" Problem

Think of the Human Team's labels like clear, distinct boxes. If a note said, "We don't have the right tools," it went into the "Equipment" box. If it said, "We don't have time," it went into the "Time" box. The boxes didn't overlap much.

The AI Team's labels were a bit more like fuzzy, overlapping circles. The AI created some very broad categories. For example, it might have a label called "Workflow and Culture." A single note about "The hospital prefers a different surgery" could fit into three different AI categories at the same time.

Because the AI's categories were so broad and overlapping, the two human coders who tried to use the AI's list to sort the notes got confused more often. They didn't always agree on which "fuzzy" pile a note belonged to.

The Scorecard

The researchers measured how often the two coders agreed with each other (Inter-Coder Reliability).

Human Team: They agreed 95.1% of the time. Their "agreement score" (Kappa) was 0.75.
AI Team: They agreed 92.3% of the time. Their "agreement score" was 0.64.

While the AI was still pretty good, the humans were slightly more consistent. The paper notes that the difference was statistically significant, meaning it wasn't just a fluke; the human labels were just a bit clearer.

The "Hybrid" Solution

The paper concludes that the AI is like a fast, helpful assistant who can quickly look at a messy room and say, "Okay, here are the main types of clutter: clothes, books, and dishes."

But the AI might not be perfect at defining exactly where a "sock" goes if it's mixed with a "towel." The humans are needed to come in afterward and say, "Actually, let's make a specific rule for socks so everyone knows exactly where they go."

The Takeaway:
You can use AI to do the heavy lifting of finding the main themes quickly. But to make sure the rules are clear and everyone sorts the notes the same way, a human needs to step in and refine the definitions. The best approach is a hybrid: let the AI do the first pass, then let humans polish the labels.

What the Paper Doesn't Say

It is important to note what this study did not claim:

It did not say AI is ready to replace human researchers entirely.
It did not test the AI on long, complex interviews (only short survey answers).
It did not test different versions of AI or different ways of asking the AI questions (it used a "zero-shot" method, meaning it asked the AI to do it with no prior training or examples).
It did not claim the AI was "wrong" about the themes; the AI found the right big ideas, it just had trouble drawing the lines between them as clearly as humans did.

Technical Summary: Artificial Intelligence in Surgical Qualitative Research

Problem Statement
While Artificial Intelligence (AI) and Large Language Models (LLMs) have established utility in quantitative research, their integration into qualitative data analysis remains limited. Qualitative research, particularly thematic analysis, is historically labor-intensive and relies on human interpretation to distinguish subtle conceptual differences and interpret context-dependent meaning. Although there is growing interest in using AI to accelerate qualitative workflows, concerns persist regarding AI's ability to generate nuanced, well-differentiated thematic structures without compromising interpretive depth. Specifically, there is a lack of empirical comparison between human-derived and AI-derived codebooks within clinical and surgical contexts to determine if AI can produce reliable, defensible qualitative analyses.

Methodology
This study conducted a secondary methodological analysis of 200 de-identified, open-ended survey responses regarding barriers to performing laparoscopic common bile duct exploration (LCBDE). The research compared the inter-coder reliability (ICR) of two distinct codebooks applied to the same dataset:

Human-Derived Codebook (HC): Two authors performed inductive thematic analysis. Through an iterative process of open coding, constant comparison, and consensus discussion, they developed a codebook with explicit definitions to minimize overlap.
AI-Derived Codebook (AIC): The same dataset was processed using ChatGPT (GPT-5.2) via a "zero-shot" prompting strategy. The model was asked to generate a thematic codebook without examples or iterative refinement, simulating a scenario where labeled training data is unavailable.

Both codebooks were subsequently applied deductively to the full dataset by the same two coders, who were blinded to each other's assignments during the application phase. Statistical analysis included percent agreement, Cohen's kappa coefficient (with bootstrapped 95% confidence intervals), and Gwet's AC1 to evaluate ICR. A qualitative comparison of thematic separation, conceptual overlap, and coding ambiguity was also performed.

Key Results

Thematic Convergence: Both the HC and AIC independently identified 13 codes. Five codes were thematically identical across both approaches: Equipment, Global Support, Cost, Time/OR Constraints, and Low Volume.
Inter-Coder Reliability (ICR): The HC demonstrated higher overall ICR than the AIC.
- Percent Agreement: HC was 95.1%; AIC was 92.3%.
- Cohen's Kappa: HC was 0.754 [95% CI: 0.711–0.792]; AIC was 0.638 [95% CI: 0.591–0.684].
- Difference: The difference in kappa ( $\Delta\kappa$ ) was -0.116 [95% CI: -0.168 to -0.059], indicating a statistically significant advantage for the human-derived codebook.
Thematic Structure: Qualitative analysis revealed that while both codebooks identified similar overarching themes, the AIC tended to produce more overlapping, overarching themes, leading to coding ambiguity. In contrast, the HC exhibited clearer thematic boundaries. The AI-generated codebook often grouped multiple related constructs into single definitions (e.g., combining workflow, culture, and personal preference), whereas the human codebook separated these more distinctly.
Statistical Nuance: While some individual codes showed high percent agreement in both groups, the "kappa paradox" was observed in codes with low prevalence or high overlap. Gwet's AC1 values remained high across both codebooks, suggesting that while statistical agreement was strong, the structural ambiguity in the AIC contributed to lower kappa scores.

Significance and Claims
The paper claims that modern LLMs, specifically when used in a zero-shot capacity, are capable of identifying major patterns and generating initial thematic structures that align broadly with human findings in surgical qualitative data. However, the study concludes that AI-generated codebooks currently lack the thematic granularity and boundary clarity achieved through human iterative refinement.

The authors assert that the findings support a hybrid approach to surgical qualitative research. In this model, AI serves as an efficient tool for initial clustering and the identification of overarching concepts, while human researchers remain essential for refining codebook definitions, resolving thematic overlaps, and ensuring interpretive accuracy. The study does not claim that AI can currently replace human analysts in qualitative research but rather positions AI as a viable assistive tool to optimize efficiency in the early stages of thematic analysis, provided human oversight is maintained to ensure analytic validity.