Extracting Social Determinants of Health from Electronic Health Records: Development and Comparison of Rule-Based and Large Language Model Methods

This study demonstrates that large language models, particularly advanced "mini" variants, outperform rule-based systems in extracting social determinants of health from unstructured clinical notes, with the highest accuracy achieved by a late-fusion ensemble combining both approaches.

The Gist

Imagine you are trying to understand a patient's life story to help them stay healthy. You have their medical chart, but it's like a library with two very different sections:

1. The Structured Section: This is the neat, organized filing cabinet. It has checkboxes for "High Blood Pressure" or "Diabetes." It's easy to read, but it's missing a huge chunk of the story.
2. The Unstructured Section: This is the messy, handwritten diary. Doctors write paragraphs here about a patient's job, whether they can afford rent, if they have food in the fridge, or if they feel lonely. This is where the real "Social Determinants of Health" (SDoH) live, but because it's written in plain English, computers can't easily read it.

The Problem:
If we want to help patients, we need to read that messy diary. But reading millions of pages of doctor's notes by hand is impossible. We need a robot to do it.

The Experiment:
The researchers in this paper built two different kinds of robots to read these notes and find the social issues:

Robot A: The "Rule-Bot" (Rule-Based System)

Think of this robot as a strict librarian with a giant, pre-written list of keywords.

• How it works: If the doctor writes the word "unemployed," the librarian checks their list, sees "unemployed" is on it, and flags it.
• The Flaw: This librarian is very literal. If the doctor writes, "The patient is struggling to pay rent," the librarian might miss it because the word "unemployed" isn't there. It's great at finding exact matches but terrible at understanding context or nuance. It's like a metal detector that only beeps for gold coins, missing gold bars or jewelry.

Robot B: The "Brain-Bot" (Large Language Models)

Think of this robot as a super-smart graduate student who has read the entire internet.

• How it works: You give it a paragraph and say, "Find any social problems here." Because it understands human language, it knows that "struggling to pay rent" means the same thing as "housing insecurity." It gets the meaning, not just the words.
• The Catch: These robots are expensive to run and can sometimes get confused if you don't give them clear instructions.

The Showdown

The researchers tested these robots against a "Gold Standard" (a group of human experts who manually read the notes and marked the answers).

1. The Rule-Bot was very careful. When it said it found something, it was usually right (High Precision). But it missed a lot of things because it was too rigid (Low Recall).
2. The Brain-Bot was much better at finding the hidden social issues. It caught almost everything the humans found. The newest, most advanced "mini" versions of these Brain-Bots (like GPT-5-mini and o4-mini) were the champions. They were fast, cheap, and incredibly accurate without needing to be "taught" specific rules first.

The Secret Weapon: The "Team-Up" (Ensemble)

The researchers realized that the best team isn't just one robot; it's a hybrid squad.

• They let the Rule-Bot scan the notes first to catch the obvious, easy hits.
• Then, they let the Brain-Bot take a second look to catch the tricky, nuanced stuff.
• Finally, they combined the results. If either robot found something, they counted it.

The Result: This team-up was the most successful method. It caught almost everything (92% accuracy) and balanced the strengths of both robots.

Why This Matters

In the past, doctors might have missed that a patient is homeless or hungry because that info was buried in a paragraph of text. With this new system:

• Hospitals can automatically flag patients who need extra help (like food banks or housing assistance).
• Researchers can study how social issues affect health on a massive scale.
• Cost: It's much cheaper than hiring humans to read every note, and it's faster than waiting for a super-computer to process everything.

The Bottom Line:
We finally have a way to turn the messy, handwritten stories in medical charts into useful data. By using a smart "Brain-Bot" (specifically the new, efficient mini-models) and teaming it up with a simple "Rule-Bot," we can uncover the hidden social struggles of patients, helping doctors treat the whole person, not just their symptoms.

Technical Summary

1. Problem Statement

Social Determinants of Health (SDoH)—factors like financial stability, housing, and food security—are critical drivers of health outcomes but are frequently under-documented in structured Electronic Health Record (EHR) data. Instead, this information resides in unstructured clinical notes.

• The Challenge: Extracting SDoH from narrative text is difficult. Traditional Rule-Based Systems (RBS) suffer from low recall due to rigid keyword matching and inability to handle semantic nuance. Supervised Machine Learning approaches require large, costly, and labor-intensive annotated datasets for fine-tuning.
• The Gap: While Large Language Models (LLMs) offer a potential solution, there is a lack of systematic evaluation comparing state-of-the-art reasoning models (specifically newer OpenAI GPT models) against optimized RBS, particularly regarding cost-efficiency, prompting strategies, and ensemble methods. Furthermore, many studies focus only on common SDoH domains (e.g., smoking), neglecting others like social resources and health insurance.

2. Methodology

Data Source and Annotation

• Dataset: Clinical notes (discharge summaries and progress notes) from 171 patients in the Mass General Brigham Research Patient Data Registry (RPDR).
• Gold Standard: A manually annotated corpus of 226 text segments (with 150 characters of context on either side) covering 7 SDoH domains and 23 subcategories.
  • Domains: General financial status, Employment status, Housing status, Health insurance status, Food security status, Social resources, and Physical activity.
  • Contextual Attributes: Annotations included temporality, experiencer, hypothetical status, and uncertainty to filter out non-relevant mentions (e.g., family history or hypothetical scenarios).
• Inter-Annotator Agreement: High reliability achieved (Krippendorff's $\alpha$ = 0.95 for domains; 0.78–0.84 for subcategories).

Approaches Evaluated

1. Rule-Based System (RBS):
   • Built using medspaCy.
   • Utilized a lexicon created via seed terms, UMLS expansion, and semantic similarity search (using MPNet embeddings).
   • Included iterative manual refinement and context disambiguation rules.
2. Large Language Model (LLM) Approaches:
   • Models: Seven OpenAI GPT models were tested: GPT-4o, 4.1, 4.1-mini, o4-mini, GPT-5, GPT-5-mini, and o3.
   • Settings: Evaluated in Zero-shot and Few-shot (5 examples) configurations.
   • Prompting Strategies:
     • Styles: Strict, Balanced, and Liberal (varying evidence thresholds).
     • Example Types: Easy (high agreement), Hard (requiring adjudication), and Synthetic examples, with and without reasoning explanations.
3. Ensemble (Late-Fusion) Strategy:
   • Combined RBS and GPT outputs to leverage complementary strengths.
   • Domain Level: Aggregated RBS output with a "GPT-committee" (multiple prompting styles) using Union, Intersection, or Majority Voting.
   • Subcategory Level: Aggregated GPT-committee outputs only (as RBS performed poorly at this granular level).

3. Key Contributions

• Comprehensive Taxonomy: Developed a fine-grained classification schema covering 7 domains and 23 subcategories, including under-explored areas like social resources and health insurance.
• Systematic LLM Benchmarking: First extensive comparison of the latest "reasoning" GPT models (including "mini" variants) for SDoH extraction without task-specific fine-tuning.
• Cost-Efficient Framework: Demonstrated that a hybrid approach (RBS pre-screening + LLM classification) is highly cost-effective compared to processing entire notes with LLMs.
• Ensemble Validation: Validated that late-fusion ensembles significantly improve domain-level performance by balancing the high precision of RBS with the high recall of LLMs.

4. Key Results

Performance Metrics

• Rule-Based System (RBS):
  • High Precision (0.96 for domains) but low Recall (0.68).
  • Failed to capture implicit meanings (e.g., "owns multiple homes" = secure housing) and struggled with negation and hypotheticals.
• LLM Performance:
  • Consistently outperformed RBS in recall and F1 scores.
  • Best Domain Performance: GPT-5 and GPT-5-mini (5-shot) achieved an F1 of 0.89.
  • Best Subcategory Performance: o4-mini (5-shot) achieved the highest F1 of 0.88 (Precision: 0.88, Recall: 0.88).
• Ensemble Results:
  • Domain Level: The late-fusion ensemble (Majority Voting GPT-committee + Union with RBS) achieved the highest F1 of 0.92 (Precision: 0.93, Recall: 0.90).
  • Subcategory Level: The ensemble did not improve performance over the best single GPT model (o4-mini), suggesting the GPT models were already optimal for granular extraction.

Efficiency and Cost

• Latency: "Mini" models (o4-mini, GPT-5-mini) were significantly faster (7–9 seconds/segment) compared to flagship GPT-5 (33–34 seconds/segment).
• Cost: Mini models offered 72–82% cost reductions compared to GPT-5 while maintaining competitive performance.
• ICD Code Comparison: Both RBS and LLMs identified significantly more SDoH mentions than structured ICD (V/Z) codes, highlighting the value of unstructured text extraction.

Error Analysis

• RBS Errors: Primarily false negatives due to vocabulary gaps and false positives due to failure to detect negation or context (e.g., misclassifying medical eating issues as food insecurity).
• LLM Errors: Included missing implicit evidence, over-interpretation of weak evidence, and occasional failure to adhere to strict prompt instructions regarding temporality.

5. Significance and Conclusion

• Clinical Impact: The study provides a validated, cost-efficient framework to unlock SDoH data from clinical notes, enabling better risk stratification, population health research, and personalized care.
• Methodological Shift: It demonstrates that state-of-the-art reasoning LLMs (specifically "mini" variants) can outperform traditional rule-based systems without the need for expensive fine-tuning or massive annotated datasets.
• Practical Application: The proposed hybrid pipeline (RBS for pre-screening + LLM for classification + Late Fusion) offers a scalable solution for healthcare systems to integrate SDoH into clinical decision-making and algorithmic auditing.
• Future Directions: The authors suggest future work should focus on generalizing these models to diverse healthcare systems, incorporating richer contextual event extraction, and validating the downstream impact on clinical outcomes (e.g., suicide prediction).

Conclusion: Recent GPT models with advanced reasoning capabilities represent a superior, cost-effective alternative to rule-based systems for SDoH extraction, particularly when combined in an ensemble framework to maximize both precision and recall.