Can LLMs Understand the Impact of Trauma? Costs and Benefits of LLMs Coding the Interviews of Firearm Violence Survivors

1. Problem Statement

Firearm violence is a critical public health issue disproportionately affecting Black men in the US. Understanding the lived experiences of survivors through qualitative research (e.g., in-depth interviews) is essential for designing interventions. However, manual thematic analysis and inductive coding are labor-intensive, time-consuming, and underfunded due to historical restrictions on firearm violence research funding.

While Large Language Models (LLMs) offer the potential to automate qualitative coding, there is a significant gap in research regarding their ability to:

• Accurately code long-form, unstructured interviews.
• Handle the specific linguistic nuances of African American English (AAE).
• Ethically process sensitive narratives involving trauma, violence, and marginalized identities without bias or "narrative erasure" (refusal to process content).

2. Methodology

The authors developed a machine coding (MC) pipeline to compare open-source LLMs against human qualitative coders (HC) using a dataset of 21 semi-structured interviews with Black male firearm violence survivors.

Data

• Source: 21 interviews conducted between 2013–2015 with survivors admitted to a Level 2 trauma center in the DC metro area.
• Format: Long-form transcripts (approx. 11,500 words per interview) containing AAE and discussions of trauma, violence, and sexual activity.
• Preprocessing: Transcripts were cleaned, de-identified, and chunked into three strategies:
  1. Full Text: Entire interview passed at once (only for 1B parameter models).
  2. Paired Chunks: Interviewer question paired with subject response.
  3. Question Chunks: Questions extracted and matched to responses via semantic similarity.

Human Coding (Ground Truth)

• Conducted by three Black/African-American women (one undergraduate, two staff) using grounded theory and inductive thematic analysis.
• Resulted in 41 initial codes and 11 formal codes.
• Total time: ~12 hours per coder.

Machine Coding Pipeline

• Models: Open-source instruction-tuned models: Llama-3.2-1B and Llama-3.1-8B.
• Prompting: Zero-shot prompts where the system was assigned an identity (e.g., "African American Studies anthropologist") and context.
• Code Generation:
  1. Initial Codes: LLMs generated themes from interview excerpts.
  2. Clustering: Initial codes were aggregated and clustered using BERTopic (with all-mpnet-base-v2 embeddings) to create "Formal MC codes."
• Evaluation Metrics:
  • Percent Captured: Percentage of formal Human Codes (HC) matched by Machine Codes (MC) (Recall).
  • Percent Relevant: Percentage of MC codes that match any HC code (Precision).
  • Similarity Threshold: Cosine similarity > 0.6 was required for a match.

3. Key Contributions

• Empirical Assessment of LLMs in Trauma Research: One of the first studies to evaluate open-source LLMs on long-form, sensitive qualitative data involving firearm violence survivors.
• Analysis of Narrative Erasure: Quantified the extent to which LLM "guardrails" refuse to process content related to violence, race, and sexuality, leading to the loss of critical data.
• Resource Efficiency vs. Performance: Demonstrated that smaller, low-resource models (1B parameters) can approach the performance of larger models (8B) but with distinct failure modes regarding AAE and colloquialisms.
• Pipeline Architecture: Proposed a two-step pipeline (Generation + Clustering) to manage the volume of LLM outputs, though noting that clustering can degrade semantic alignment.

4. Key Results

Performance Metrics

• Capture Rate: The best models captured 100% of the initial human codes but only 36% of the formal human codes after clustering. The average capture rate across all experiments was 71% (initial) dropping significantly after clustering.
• Relevance: Overall relevance remained low, averaging 10% for initial codes and 28% for formal codes.
• Model Comparison:
  • Llama 8B showed slightly higher relevance (13.8% initial) than Llama 1B (9.0%).
  • Llama 1B actually had a slightly higher (though not statistically significant) capture rate than Llama 8B in initial generation.
  • Clustering Impact: While clustering reduced the code count from thousands to manageable numbers (<100), it decreased the alignment with human codes. The semantic similarity between the final cluster names and the human codes was often weak.

Data Processing Impact

• Chunking Strategy: "Paired chunks" (Q&A pairs) yielded the highest Percent Captured (outperforming full text by 32%).
• Full Text: While it had higher relevance, it failed to capture as many human codes, likely due to context window limitations or loss of detail.

LLM Refusals (Narrative Erasure)

• Refusal Rate: On average, 44% of prompt requests were refused by the LLMs.
• Causes: Refusals were triggered by:
  • Graphic descriptions of violence/injury.
  • Discussions of sexual activity.
  • Use of African American English (AAE) and specific slang (e.g., the n-word), even when non-derogatory.
  • Topics of race and the criminal justice system.
• Consequence: Critical research themes regarding trauma, high-risk sexual behavior, and systemic racism were systematically ignored or erased by the models.

Hallucinations and Bias

• Colloquialism Fixation: Llama 1B often fixated on linguistic characteristics of AAE (e.g., usage of "get" or "like") rather than thematic content.
• Hallucinated Codes: Some generated codes (e.g., "Gang affiliation codes") had no basis in the source text, while others (e.g., "Racial profiling") were valid but not explicitly supported by the specific text excerpts provided to the model.

5. Significance and Conclusion

• Limitations of Automation: The study concludes that current LLMs cannot reliably replace human coders for sensitive, long-form qualitative research involving marginalized communities. The time saved in generation is negated by the time required to validate, correct, and filter hallucinated or biased outputs.
• Ethical Risks: The "guardrails" designed to protect users often result in narrative erasure, silencing the very voices (Black men, trauma survivors) that the research aims to amplify. This perpetuates systemic bias by rendering their experiences invisible in automated analysis.
• Resource Implications: While smaller models (1B) are computationally feasible for underfunded communities, they currently lack the nuance to handle AAE and trauma without significant human oversight.
• Future Directions: The authors argue for the development of low-resourced, community-aligned AI tools that are specifically fine-tuned to understand AAE and trauma narratives, rather than relying on general-purpose, safety-guarded models that prioritize "safety" over accuracy in marginalized contexts.

Final Verdict: While LLMs show potential for extracting some codes, they currently introduce unacceptable risks of bias, data loss, and misrepresentation in firearm violence research. Human expertise remains essential for ethical and accurate qualitative analysis.