Large Language Models for Large-Scale, Rigorous Qualitative Analysis in Applied Health Services Research

Technical Summary: Large Language Models for Large-Scale, Rigorous Qualitative Analysis in Applied Health Services Research

Problem Statement
Qualitative approaches are essential for health services research to understand the contextual and interpersonal factors shaping care delivery. However, large-scale, applied studies often face constraints of time and resources, necessitating timely findings to remain relevant to practitioners. While Large Language Models (LLMs) offer potential for enhancing analytic efficiency, their integration into rigorous qualitative workflows remains under-guided. Existing literature highlights risks such as superficial, context-blind classifications, compromised methodological transparency, and a lack of evidence regarding how LLMs influence real-world research outcomes. Furthermore, current tools often fail to support the specific, context-dependent needs of applied health studies, and commercial platforms frequently lack the security or transparency required for high-risk medical data.

Methodology
The authors propose a model- and task-agnostic framework for designing human-LLM qualitative analysis methods, applied within a multi-site study of diabetes care at Federally Qualified Health Centers (FQHCs). The study involved 12 FQHCs, 167 interviews, and four research teams. The framework consists of four iterative steps:

1. Define Task: Manual completion of the task on a small sample to clarify goals, finalize output formats, and identify components requiring human researcher familiarity.
2. Design Human-LLM Method: Decomposing the task to distinguish between actions requiring human tacit knowledge (e.g., context-dependent judgment) and those suitable for automation.
3. Evaluate at Small-Scale: Comparing human-only and human-LLM outputs on a small dataset against task-specific goals and qualitative rigor criteria (e.g., grounding in data, alignment with research questions) rather than quantitative metrics like inter-rater reliability.
4. Apply and Evaluate: Implementing the validated method on the full dataset and assessing its impact on efficiency and research goals.

The study utilized two distinct tasks to test this framework:

• Task 1 (Qualitative Synthesis): Generating comparative summary reports for FQHCs based on researcher-generated site summaries (~31,200 words). The team used OpenAI's ChatGPT-4o for thematic organization of site summaries and the o1 model for generating cross-site syntheses.
• Task 2 (Deductive Coding): Refining a practice-transformation intervention by coding 167 interview transcripts (~1.3 million words) against 19 pre-defined codes. Due to context window limitations and the need for semantic retrieval, the team implemented Retrieval-Augmented Generation (RAG) using embedding-based retrieval (OpenAI's text-embedding-3-large) and a Qdrant vector database. To mitigate "example bias" and "positivity bias," the team employed a strategy of asking sub-questions with and without examples, and paired questions with counter-perspectives on barriers.

Security and infrastructure were managed via Stanford SecureGPT to access commercial LLMs (ChatGPT-4o, o1, Gemini, Claude, DeepSeek) via API, ensuring compliance with high-risk medical data standards.

Key Results

• Task 1 Outcomes: LLMs successfully organized site-level data into themes, aligning closely with human grouping. However, LLMs lacked the nuance to determine the novelty or actionability of insights, often including irrelevant data or failing to filter out misaligned inputs. The final human-LLM workflow involved LLMs organizing data, which researchers then refined into final syntheses. This approach reduced analysis time by 30–55% while maintaining analytic quality comparable to two human researchers with different perspectives.
• Task 2 Outcomes: The RAG-based human-LLM method successfully identified relevant excerpts across a large dataset. However, LLMs frequently overgeneralized findings, lacked perspective, and missed context due to fragmented, decontextualized excerpts. Human analysis provided more detailed explanations and identified interpersonal dynamics, while LLMs focused on structures and processes. The team found that LLM outputs required significant researcher validation and contextualization.
• Efficiency Gains: Without LLMs, coding the full dataset (12 sites, 19 codes) was estimated to require nearly two years of full-time work. The human-LLM approach made the incorporation of all 167 transcripts into the intervention refinement feasible within a nine-month deadline.

Significance and Claims
The paper claims to demonstrate how LLMs can be integrated into applied health-services research to enhance efficiency while preserving methodological rigor. The authors assert that their framework provides necessary guidance for leveraging the generality of LLMs into formalized, task-specific methods. Key contributions include:

• Methodological Guidance: A structured approach for determining where LLMs add value (organizational, high-level analytic steps) versus where human interpretive control is essential (theory development, final insight generation).
• Interdisciplinary Collaboration: The study highlights the necessity of collaboration between qualitative and computer science researchers to develop custom, secure, and transparent tools that address specific analytic bottlenecks, as off-the-shelf solutions were insufficient.
• Real-World Application: The work offers evidence of LLM utility in analyzing large, heterogeneous datasets typical of applied health research, resulting in timely feedback to practitioners and the refinement of a diabetes care intervention for future implementation.

The authors remain modest regarding the limitations, noting that current LLMs lack the capacity for long-context reasoning across entire datasets and that human researchers must retain interpretive control to ensure transparency and credibility. They suggest that future advances in long-context reasoning and domain knowledge transfer could further improve LLM capabilities in this domain.