💬 NLP

Multi-LLM Thematic Analysis with Dual Reliability Metrics: Combining Cohen's Kappa and Semantic Similarity for Qualitative Research Validation

This paper introduces a multi-LLM thematic analysis framework that validates qualitative research reliability by combining Cohen's Kappa and semantic similarity metrics, demonstrating through a psychedelic art therapy case study that an ensemble approach using Gemini, GPT-4o, and Claude achieves high inter-rater agreement and robust consensus theme extraction.

Original authors: Nilesh Jain, Hyungil Suh, Seyi Adeyinka, Leor Roseman, Aza Allsop

Published 2026-02-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Nilesh Jain, Hyungil Suh, Seyi Adeyinka, Leor Roseman, Aza Allsop

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to understand a very complex, emotional story told by a friend. In the past, if you wanted to be sure you understood the story correctly, you would ask three or four other friends to read it, write down their own summaries, and then sit down together to argue until they all agreed on the main points. This is how researchers traditionally do "Qualitative Research."

The Problem: This process is slow, expensive, and often, even the human friends can't agree perfectly. They might argue over whether a specific detail was about "sadness" or "disappointment."

The New Idea: What if we could use Artificial Intelligence (AI) to do this? But here's the catch: AI is a bit like a dreamer. If you ask it the same question twice, it might give you two slightly different answers because it's probabilistic (it guesses the next word based on patterns). If you only ask it once, you don't know if the answer is a solid truth or just a random guess.

The Solution: This paper introduces a new "Team of AI Dreamers" approach. Instead of asking one AI to read the story once, they ask six different versions of the same AI (using different "random seeds" like rolling dice) to read the story six times. Then, they compare all six answers to see which ones the AI agrees on.

Here is the breakdown using simple analogies:

1. The "Six Friends" Analogy (Ensemble Validation)

Imagine you hire six different detectives to solve the same mystery.

Old Way: You ask one detective for a report. You have no idea if they got lucky or if they missed something.
New Way: You ask six detectives. If five of them say, "The butler did it," you can be very confident that's the answer. If one says "The butler" and five say "The gardener," you know the "butler" theory is shaky.
In the Paper: The researchers ran the AI six times. They looked for the themes that appeared in at least 3 out of 6 (or 5 out of 6) reports. These are the "Consensus Themes"—the parts the AI is sure about.

2. The "Two Rulers" Analogy (Dual Reliability Metrics)

How do you know the detectives are actually agreeing? The paper uses two different "rulers" to measure agreement:

Ruler #1: The "Exact Match" Ruler (Cohen's Kappa)
- This checks if the detectives used the exact same words. If Detective A says "Sadness" and Detective B says "Sadness," they get a point. If A says "Sadness" and B says "Grief," this ruler might say, "No match!"
- Why it matters: It's the standard, strict way researchers have used for decades to prove reliability.
Ruler #2: The "Meaning Match" Ruler (Cosine Similarity)
- This is the clever part. It understands that "Sadness" and "Grief" mean the same thing, even if the words are different. It looks at the vibe and meaning of the sentences.
- Why it matters: AI often rephrases things. This ruler ensures that if the AI says "The client felt heavy" in one run and "The client felt burdened" in another, it counts as an agreement.

3. The "Magic Recipe" (Configurable Parameters)

The researchers built a tool that lets you tweak the "personality" of the AI:

Temperature: Think of this as the AI's "creativity dial."
- Low Temperature (0.0): The AI is a robot. It gives the exact same answer every time. Good for facts.
- High Temperature (2.0): The AI is an artist. It gets creative and wild. Good for exploring new ideas.
- The researchers set it to a "just right" level to get consistent but thoughtful answers.
Seeds: Think of these as the "starting point" for the AI's brain. By changing the seed, you get a slightly different perspective, just like asking six different people to read the same book.

4. The Results: Who Was the Best Detective?

They tested three famous AI models (Gemini, GPT-4o, and Claude) on a real interview about art therapy and ketamine treatment.

The Winner: Gemini 2.5 Pro was the most consistent detective. It agreed with itself 90.7% of the time (on the strict ruler) and 95.3% of the time (on the meaning ruler).
The Runners Up: GPT-4o and Claude were also very good, scoring above 84% agreement.
The Takeaway: All three models were "Almost Perfect" in their agreement. This proves that if you run an AI multiple times and look for the common threads, you can trust the results almost as much as if you had hired a team of human experts.

Why This Matters to You

This paper is a game-changer for researchers.

Before: To get reliable results, you needed to pay humans hundreds of dollars and spend weeks arguing over notes.
Now: You can use this free, open-source tool to get "almost perfect" reliability in minutes for a few dollars.

It doesn't replace human researchers; instead, it gives them a super-powered assistant. The AI does the heavy lifting of finding patterns and checking its own work, and the human researcher steps in to make the final judgment call on the most important themes.

In short: They taught AI to "double-check its own homework" by asking itself the same question six times, and they proved that when the AI agrees with itself, you can trust the answer.

1. Problem Statement

Qualitative research relies heavily on inter-rater reliability to ensure the trustworthiness of thematic analysis. Traditional methods require multiple human coders to independently analyze data, a process that is:

Time-intensive and expensive: Often requiring weeks of labor.
Inconsistent: Human coders frequently achieve only moderate agreement ( $\kappa = 0.40–0.60$ ).
Limited by subjectivity: Discrepancies require extensive discussion to resolve.

While Large Language Models (LLMs) offer a potential solution for automating thematic analysis, current approaches suffer from a critical gap: lack of systematic reliability quantification. Most existing frameworks present a single-run output as a definitive conclusion without assessing the stochastic stability of the model. A single generation may be an artifact of a specific random seed or interaction path rather than a robust finding. There is no standard metric to distinguish between stable thematic signals and "hallucinatory noise" in AI-assisted qualitative research.

2. Methodology

The authors propose a Multi-Perspective Validation Framework that treats LLM thematic analysis as a statistical ensemble problem rather than a single-generation task.

A. Ensemble Validation Approach

Instead of a single run, the framework executes multiple independent analytical runs (default: 6) using fixed random seeds (e.g., 42, 123, 456, 789, 1011, 1213) and a configurable temperature ( $T=0.7$ ).

Statistical Rationale: Six runs generate 15 pairwise comparisons, providing sufficient data points to estimate true score variance versus error variance, reducing standard error by ~41% compared to three runs.
Consensus Mechanism: An adaptive algorithm extracts themes from each run, groups semantically similar themes (using a cosine similarity threshold $>0.70$ ), and retains only those appearing in $\geq 50\%$ of runs as "consensus themes."

B. Dual Reliability Metrics

The framework introduces two complementary metrics to validate the output:

Cohen's Kappa ( $\kappa$ ): Measures statistical inter-rater agreement for theme presence/absence across runs, accounting for chance. It follows Landis-Koch criteria (where $\kappa > 0.80$ is "almost perfect").
Cosine Similarity: Measures semantic consistency beyond exact lexical matches. Using all-MiniLM-L6-v2 embeddings (384-dimensional), it calculates the semantic overlap between theme descriptions across runs. This captures equivalence even when phrasing differs (e.g., "perfectionist barriers" vs. "creative blocks").

C. Technical Architecture & Flexibility

Structure-Agnostic Parsing: The system uses regex and JSON parsing to handle arbitrary output formats, stripping markdown and validating structure without requiring predefined schemas.
Configurable Parameters: Researchers can adjust seeds (1–6), temperature (0.0–2.0), and use custom prompts with variable substitution (e.g., {seed}, {text chunk}).
Client-Side Processing: The tool (built with Next.js/React) processes data locally in the browser using Transformers.js for embeddings, ensuring raw data privacy. It supports nine LLM providers (Gemini, Claude, GPT-4o, etc.) via unified APIs.

3. Key Contributions

Dual Reliability Framework: The first integration of Cohen's Kappa (statistical agreement) and Cosine Similarity (semantic consistency) specifically for validating LLM-based thematic analysis.
Ensemble Validation Protocol: A methodological shift from single-run generation to multi-run consensus, providing quantifiable stability metrics.
Structure-Agnostic Consensus Extraction: A flexible algorithm that extracts consensus themes from any JSON structure, allowing researchers to use custom analytical frameworks without code modification.
Open-Source Implementation: A fully functional, privacy-preserving tool available on GitHub that supports cross-model validation (comparing outputs across different LLM architectures).

4. Experimental Results

The framework was evaluated on a 28,377-character interview transcript regarding ketamine-assisted art therapy, using three leading LLMs: Gemini 2.5 Pro, GPT-4o, and Claude 3.5 Sonnet. Each model underwent six independent runs.

Model	Cohen's Kappa ( $\kappa$ )	Cosine Similarity	Consensus Themes
Gemini 2.5 Pro	0.907 (Range: 0.745–0.977)	95.3%	6 themes (50–83% consistency)
GPT-4o	0.853 (Range: 0.672–0.988)	92.6%	5 themes
Claude 3.5 Sonnet	0.842 (Range: 0.604–1.000)	92.1%	4 themes

Key Findings:

High Reliability: All models achieved "almost perfect" agreement ( $\kappa > 0.80$ ), validating the ensemble approach.
Semantic Correlation: Cosine similarity strongly correlated with Kappa ( $r=0.97$ ), confirming that semantic embeddings are effective reliability measures.
Consensus Extraction: The framework successfully filtered spurious themes while preserving valid variations. For example, "Integration of Internal Family Systems (IFS)" was identified as a robust theme across all models (appearing in 50–83% of runs per model).
Cost Efficiency: The AI approach cost approximately $0.15–$ 0.20 per transcript, compared to $20–$ 40 for traditional human coding.

5. Significance and Implications

Methodological Rigor: This work establishes a new standard for AI-assisted qualitative research by moving beyond "black box" generation to quantifiable, reproducible reliability. It allows researchers to claim "almost perfect agreement" in their AI-assisted analyses.
Scalability: The framework drastically reduces the time and cost barriers of qualitative research, making rigorous thematic analysis accessible to smaller research teams.
Human-in-the-Loop Validation: The system does not replace human researchers but provides validated starting points. It distinguishes between high-confidence themes (requiring minimal review) and moderate-confidence themes (requiring researcher judgment), effectively balancing automation with human reflexivity.
Future Directions: The authors call for large-scale validation across diverse domains, languages, and cultural contexts to establish boundary conditions and normative benchmarks for AI reliability in social science research.

In conclusion, this paper provides a robust, open-source framework that bridges the gap between the efficiency of LLMs and the rigorous validation standards required in qualitative research, fundamentally changing how reliability is assessed in AI-driven thematic analysis.