💬 NLP

Benchmarking LLMs for Pairwise Causal Discovery in Biomedical and Multi-Domain Contexts

This paper introduces a comprehensive benchmark evaluating 13 open-source large language models on pairwise causal discovery across 12 diverse datasets, revealing significant performance gaps in both causal detection and extraction—particularly for complex, implicit, and multi-sentence relationships—while providing a unified framework and open-source resources to advance research in this critical area.

Original authors: Sydney Anuyah, Sneha Shajee-Mohan, Ankit-Singh Chauhan, Sunandan Chakraborty

Published 2026-03-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sydney Anuyah, Sneha Shajee-Mohan, Ankit-Singh Chauhan, Sunandan Chakraborty

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a team of very smart, well-read interns (the Large Language Models or LLMs) to help you organize a massive library of medical records, news articles, and financial reports. Your goal isn't just to find keywords; you want them to understand the story of cause and effect.

You want them to answer questions like: "Did the medication cause the side effect?" or "Did the storm cause the power outage?"

This paper is essentially a report card given to 13 different AI models to see how good they are at this specific job. The researchers from Indiana University put these models through a rigorous "driving test" to see if they can safely navigate the complex roads of biomedical and general text.

Here is the breakdown of their findings, using some everyday analogies:

1. The Two Main Skills Tested

The researchers tested the models on two distinct skills, like testing a driver on two different things:

Skill A: The "Spotter" (Causal Detection)
- The Task: Look at a sentence and simply say, "Yes, this is a cause-and-effect story," or "No, this is just a coincidence."
- The Analogy: It's like a security guard at a door. They just need to raise a hand and say "Go" or "Stop."
- The Result: The best model (DeepSeek-R1-Distill-Llama-70B) got it right about 50% of the time. That's basically a coin flip. It's not great, but it's better than some others.
Skill B: The "Archivist" (Causal Extraction)
- The Task: If the story is about cause and effect, the model must pull out the exact Cause and the exact Effect.
- The Analogy: Now the guard has to not only stop the car but also read the driver's license, write down their name, and write down their destination perfectly.
- The Result: The best model here (Qwen2.5-Coder-32B) got it right about 47% of the time. Again, barely better than guessing.

2. The "Easy Mode" vs. "Hard Mode"

The researchers realized that these AI models are like students who are great at memorizing flashcards but terrible at critical thinking.

Easy Mode (Explicit Markers):
- The Scenario: The text says, "The patient took aspirin, which caused the rash."
- The Result: The models did okay here. They saw the word "caused" and knew what to do. It's like a student recognizing a math problem because it has a plus sign.
Hard Mode (Implicit & Complex):
- The Scenario: The text says, "The patient took aspirin. An hour later, a rash appeared." (No word "caused" is used).
- The Result: The models crashed and burned. They couldn't connect the dots.
- The Analogy: It's like asking a student to solve a riddle instead of a math problem. If the answer isn't written in big bold letters, the AI gets confused.
- Even Harder: When the cause is in one sentence and the effect is three sentences later, or when one cause leads to three different effects, the models often gave up or hallucinated (made things up).

3. The "Magic Prompt" Didn't Work

The researchers tried to help the models by giving them different "cheat sheets" (Prompting Strategies):

Zero-Shot: Just asking the question.
Few-Shot: Giving them a few examples first.
Chain-of-Thought: Asking them to "think step-by-step" before answering.

The Surprise: None of these tricks made a huge difference. Even when the AI was told to "think hard" or given examples, it still struggled with the complex, implicit relationships found in real-world medical notes. It's like giving a calculator a harder math problem; if the calculator doesn't know the formula, giving it a bigger screen doesn't help.

4. Why This Matters for Medicine

Imagine a doctor using an AI to scan a patient's history to find out why they got sick.

If the AI is too confident but wrong, it might tell the doctor, "This drug caused the heart attack," when it actually didn't. This could lead to dangerous medical decisions.
If the AI is too cautious, it might miss a real link, like "Smoking led to the lung damage," because the text didn't use the word "led to."

The paper concludes that while these AI models are amazing at writing poems or summarizing news, they are not yet safe to trust with high-stakes medical reasoning. They are currently "pattern matchers" (looking for the word "because") rather than true "reasoners" (understanding the logic of the situation).

The Bottom Line

The researchers built a giant, high-quality test (a "benchmark") to prove that current AI is not ready to be the primary doctor's assistant for figuring out cause and effect in complex texts.

The Takeaway: We need to train these models specifically on medical logic and teach them to look beyond simple words like "causes" or "results." Until then, we must keep a human expert in the loop to double-check the AI's work, because right now, the AI is guessing about half the time.

1. Problem Statement

The paper addresses the critical challenge of deploying Large Language Models (LLMs) in high-stakes fields like biomedicine, where the ability to distinguish causation from correlation is essential for safety. While LLMs excel at pattern matching, their capacity for genuine causal reasoning remains underexplored.

The specific task investigated is Pairwise Causal Discovery (PCD): determining if a causal link exists between two entities in text and extracting the specific cause and effect spans. This is a foundational step for building causal graphs used in clinical decision support, drug discovery, and adverse event detection. The authors highlight that current methods struggle with:

Implicit causality: Relationships without explicit markers (e.g., "because," "causes").
Inter-sentential scope: Causal links spanning multiple sentences.
Complexity: Texts containing multiple causal pairs or nested relations.
Domain specificity: The nuance of biomedical language where "causality" must be factual, not merely reported or hypothetical.

2. Methodology

A. Dataset and Annotation Protocol

The study utilizes 12 diverse datasets spanning biomedical, financial, news, and general domains (e.g., MedCaus, FinCausal, SemEval2010, PubMed).

Annotation Rigor: Eight human annotators were recruited and screened (score $\ge$ 83%). They worked in two groups to create a "golden dataset" with high inter-annotator agreement ( $\kappa \ge 0.758$ ).
Strict Criteria:
- Causality must be stated as a fact (not a claim, hypothesis, or negation).
- Cause and effect spans must be unambiguous and contiguous.
- Causal cue words (e.g., "because") are excluded from the extracted spans.
- Ambiguity defaults to "non-causal."

B. Experimental Design

The authors designed 12 experimental tasks categorized by three axes of difficulty:

Marker Presence ( $M$ ): Explicit (contains cues) vs. Implicit (no cues).
Textual Scope ( $B$ ): Intra-sentential (within one sentence) vs. Inter-sentential (across sentences).
Cardinality ( $r$ ): Single pair vs. Multiple pairs.

The tasks are split into two phases:

Causal Detection (Tasks 1a–1d): Binary classification (Causal vs. Non-Causal).
Causal Extraction (Tasks 2a–2h): Extracting exact text spans for cause and effect pairs.

C. Models and Prompting Strategies

Models: 13 open-source LLMs were tested, ranging from 3B to 70B parameters, including various architectures (Dense, Mixture-of-Experts) and families (Llama 3.x, Qwen 2.5, Mistral, DeepSeek-R1).
Prompting Strategies:
- Zero-shot: Direct instructions.
- Few-shot In-Context Learning (FICL): Providing canonical examples.
- Chain-of-Thought (CoT): Asking the model to reason step-by-step.
- Hybrid CoT+FICL: Combining examples with reasoning rationales.
- Advanced Extraction: Least-to-Most (LtM) and ReAct frameworks for complex extraction.

D. Evaluation Metrics

Detection Competence ( $D_{m,b}$ ): Probability of correct causal detection.
Extraction Competence ( $X_{m,b,r}$ ): Probability of extracting all gold pairs given successful detection.
Scoring: Used Hungarian matching for multi-pair alignment and a calibrated SentenceLM-based similarity scheme (probabilistic score map) to handle approximate string matching for extracted spans, rather than strict exact-match.

3. Key Contributions

Unified Benchmark: A comprehensive evaluation framework for PCD across 12 datasets and 12 distinct difficulty configurations, filling a gap in prior work that often aggregated results or focused on single domains.
Rigorous Annotation: A high-quality, human-validated dataset with strict definitions of factual causality, distinguishing it from correlation or reported claims.
Systematic Analysis: A granular breakdown of model performance based on linguistic complexity (implicit vs. explicit, intra vs. inter-sentential) and model architecture.
Open Resources: Public release of all data, code, prompts, and annotation guidelines to facilitate reproducibility and further research.

4. Key Results

A. Overall Performance

Current open-source LLMs show major deficiencies in pairwise causal discovery:

Best Detection: DeepSeek-R1-Distill-Llama-70B achieved a mean score of 49.57% ( $C_{detect}$ ).
Best Extraction: Qwen2.5-Coder-32B-Instruct achieved a mean score of 47.12% ( $C_{extract}$ ).
Human Baseline: Human annotators achieved ~95% accuracy, highlighting a significant gap between current LLMs and human reasoning.

B. Performance Trends

Explicit vs. Implicit: Models perform significantly better on explicit, single-sentence relations (Task 1a). Performance drops precipitously for implicit relationships (Task 1b) and inter-sentential tasks.
Domain Sensitivity: Biomedical datasets often contain implicit, multi-sentence causal links. Models struggle most here, indicating they rely heavily on surface-level lexical cues rather than deep discourse understanding.
Instruction Tuning: Instruction-tuned models (e.g., Llama-3.2-3B-Instruct) vastly outperform their base counterparts, proving the importance of alignment for causal reasoning.
Architecture Impact:
- DeepSeek Models: Excelled at detection (binary decisions) but failed at extraction, often due to token constraints causing them to stop generating before providing the answer.
- MoE Models (Mixtral): Showed advantages in handling complex extractions due to dynamic routing to specialized "experts."
- Qwen Models: Demonstrated the best balance, particularly in extraction tasks.

C. Error Analysis

Missing Relations: The most common error (35.7%) was failing to detect existing causal links (False Negatives).
Hallucinations: ~4.5% of extractions were hallucinated data.
Swapped Pairs: ~4% of errors involved swapping cause and effect.
False Positives: Rare, indicating models are conservative but lack recall.

5. Significance and Conclusion

The paper concludes that while LLMs show promise for simple, explicit causal extraction, they are not yet reliable for the complex, implicit, and multi-sentence reasoning required in clinical settings (e.g., Electronic Health Records).

Implications:

Safety: Deploying current LLMs for automated clinical decision support without human-in-the-loop verification is risky due to the high rate of missed causal links and hallucinations.
Future Directions: To bridge the gap, future work must prioritize:
1. Domain-specific fine-tuning on biomedical corpora (EHRs, abstracts).
2. Hybrid workflows combining LLM speed with human precision.
3. Architectural improvements to handle long-context reasoning and reduce hallucination in multi-pair extraction.

This benchmark serves as a critical reality check for the field, establishing that "causal reasoning" in LLMs is currently a fragile capability that requires significant advancement before safe deployment in high-stakes biomedical applications.