💬 NLP

Can Large Language Models Infer Causal Relationships from Real-World Text?

This paper introduces ReCITE, the first benchmark derived from real-world academic literature to evaluate large language models' causal reasoning capabilities, revealing that current models struggle significantly with inferring causal relationships in complex, natural texts compared to their performance on synthetic datasets.

Original authors: Ryan Saklad, Aman Chadha, Oleg Pavlov, Raha Moraffah

Published 2026-04-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ryan Saklad, Aman Chadha, Oleg Pavlov, Raha Moraffah

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

🕵️‍♂️ The Big Question: Are AI Detectives or Just Parrots?

Imagine you hire a super-smart detective (a Large Language Model, or LLM) to solve a mystery. You give them a thick, messy police report (a real-world academic paper) and ask: "Who caused what?"

The Parrot: The detective just repeats exactly what the report says. If the report says "A caused B," the detective says "A caused B."
The Detective: The detective reads between the lines. The report might say, "It rained, and the grass got wet," without explicitly saying "Rain caused the grass to get wet." A real detective infers the cause.

The Problem: Most previous tests for AI were like giving the detective a cheat sheet where the answers were written in big, bold letters. This paper asks: Can these AI detectives solve the mystery when the clues are hidden, scattered, and written in complex, real-world language?

🏗️ The New Test: "ReCITE" (The Real-World Maze)

The researchers built a new test called ReCITE. Instead of using simple, made-up sentences, they gathered 292 real academic papers from fields like economics, engineering, and environmental science.

Think of these papers as dense, tangled forests.

The Trees: These are the "events" (e.g., "rain," "crop growth," "income").
The Roots: These are the "causes."
The Goal: The AI must draw a map (a causal graph) showing how the trees are connected by their roots.

Why is this hard?
In a real forest, you don't see a signpost saying "Rain causes Growth." You have to look at the wet soil, the green leaves, and the weather report to figure it out. Sometimes the clues are missing entirely, or the text is 100 pages long.

📉 The Results: The AI Got Lost

The researchers tested the world's smartest AIs (like Claude, GPT-4, Gemini, etc.) on this forest maze.

The Scorecard:

The Best AI: Got a score of 0.535 (on a scale where 1.0 is perfect).
The Average AI: Did even worse.

What does this mean?
It's like giving a GPS a map of a city it has never seen, but the streets are named in a code it doesn't fully understand. The AI can guess some streets, but it misses the main highways and often connects the wrong neighborhoods.

Key Findings:

The "Explicit" Trap: When the text clearly says "A causes B," the AI does okay. But when the text implies it (e.g., "A happened, then B happened"), the AI's performance drops by half. It struggles to read between the lines.
It's Not a Vocabulary Problem: The researchers gave the AI the list of all the "trees" (nodes) it needed to find. Even with the list, the AI still couldn't figure out how they were connected. This proves the problem isn't that the AI doesn't know the words; it's that it doesn't understand the logic of cause and effect.
Size Doesn't Save You: Bigger, more powerful models did slightly better, but they still failed at the core task. Being "smarter" didn't make them better detectives.

🧩 The Analogy: The "Recipe" vs. The "Cooking Class"

Previous Tests (Synthetic Data): Imagine teaching a student to cook by giving them a recipe that says: "Step 1: Add salt. Step 2: Salt makes food salty." The student just memorizes the steps.
This Paper (ReCITE): Imagine giving the student a 50-page memoir of a grandmother describing her life in a village. The student has to figure out that "drought" caused "poor harvest," which caused "low income," which caused "migration." The text never explicitly states "Drought caused migration." The student has to infer the whole chain.

The Result: The AI students are great at memorizing the recipe, but they fail miserably at the cooking class. They can't connect the dots in the real world.

🔍 Why Does This Matter?

If we want AI to be truly helpful in the future (like diagnosing diseases, fixing climate change, or managing economies), it needs to understand cause and effect, not just word patterns.

Current AI: "I see the word 'fire' and the word 'smoke,' so I'll say fire causes smoke." (It's just matching patterns).
Future AI (What we need): "I understand that heat ignites fuel, which creates smoke. If I stop the heat, the smoke stops." (It understands the mechanism).

🏁 The Conclusion

This paper is a reality check. It shows that while Large Language Models are amazing at writing, summarizing, and chatting, they are currently bad at deep reasoning when faced with messy, real-world information.

They are like brilliant librarians who can find any book instantly, but they aren't yet philosophers who can understand the deep, hidden connections between the ideas in those books. To build "Artificial General Intelligence" (AI that thinks like a human), we need to teach them how to be better detectives, not just better parrots.

1. Problem Statement

Current research evaluating Large Language Models' (LLMs) causal reasoning capabilities relies heavily on synthetic datasets, short narratives, or texts with explicitly stated causal links. These benchmarks fail to capture the complexity of real-world reasoning, where:

Causal relationships are often implicit rather than explicit.
Texts are long-form and complex, requiring the integration of information scattered across documents.
Causal structures involve large graphs with numerous nodes and edges, rather than simple pairwise relationships.

The core question addressed is: Can state-of-the-art LLMs infer accurate causal graphs from lengthy, real-world academic texts where causal links are not explicitly stated?

2. Methodology

A. The ReCITE Benchmark

The authors introduce ReCITE (Real-world CausalIty from Textual Evidence), the first benchmark designed to evaluate causal reasoning on real-world data.

Source Data: 292 open-access academic papers from MDPI and PLOS that contain "causal loop diagrams" (CLDs). These papers provide both the source text and a human-authored ground-truth causal graph.
Dataset Characteristics:
- Length: Texts range from 684 to 171,141 characters (mean ~40k).
- Complexity: Graphs contain 5–140 nodes and 6–205 edges.
- Explicitness: Only ~87.7% of causal events are explicitly mentioned; many require inference.
- Diversity: Spans 17 distinct fields (e.g., Environmental Science, Engineering, Business), with a strong representation of STEM over traditional Economics.
Construction Pipeline:
1. Collection: Automated search for "causal loop diagram" followed by manual filtering to ensure single, legible graphs.
2. Annotation: Human annotators converted image-based graphs into text-based edge lists (source -> sink).
3. Post-Processing: Automated correction of formatting errors and a second annotator verified ~13% of the data, achieving high inter-annotator agreement (Edge F1 = 0.984).
4. Text Normalization: PDFs were converted to Markdown, and explicit references to the graph (e.g., "see Figure 1") were removed to prevent the task from becoming trivial.

B. Evaluation Framework: LLM-as-a-Judge

Traditional metrics (Precision/Recall against a fixed graph) are insufficient because real-world texts use synonyms and varying levels of abstraction. The authors developed an LLM-as-a-Judge framework:

Dual Comparison: The judge compares generated graphs against both the Ground-Truth Graph and the Source Text.
Semantic Scoring: It evaluates presence, semantic similarity, and abstraction levels (broader/narrower) to award partial credit for valid but differently phrased concepts.
Metrics:
- Node/Edge Precision & Recall: Weighted by the importance of the element (Core vs. Peripheral).
- Structural Hamming Distance (SHD): Measures structural errors (false positives, false negatives, direction reversals).
Reliability: Validated through cross-judge agreement (correlation $r > 0.78$ ) and checks for model family bias.

C. Experimental Setup

Models: Evaluated 10 State-of-the-Art (SOTA) LLMs, including reasoning-tuned models (Claude Opus 4.5, GPT-5.2, DeepSeek R1, Kimi K2) and instruction-tuned models (Llama 3.1, Qwen 2.5).
Task: Zero-shot generation of a causal graph (JSON format) from the text, with the expected number of nodes provided to control for granularity.

3. Key Results

A. Overall Performance

Significant Struggle: All models performed poorly. The best-performing model, Claude Opus 4.5, achieved an average F1 score of only 0.535.
Recall vs. Precision: Models exhibited high precision (0.85–0.97) but very low recall (0.11–0.57). This indicates models can generate plausible causal links supported by the text but fail to recover the specific relationships defined in the ground truth.
Reasoning Models: Reasoning-tuned models generally outperformed instruction-tuned models, and larger parameter counts correlated with better performance.

B. Ablation Study: Node Identification vs. Reasoning

To determine if failure was due to entity recognition or causal logic, the authors provided models with the ground-truth node names (Name-Assisted setting).

Result: Performance did not significantly improve (Edge F1 increased by only ~0.04 for top models).
Conclusion: The primary bottleneck is causal reasoning, not entity recognition. Even when the "what" is known, the "how" (the causal link) remains difficult to infer.

C. Impact of Text Characteristics

Explicitness: This is the strongest predictor of difficulty. As explicitness decreases (more implicit reasoning required), F1 scores drop by approximately 50%.
Text Length & Complexity: Counter-intuitively, there was a weak positive correlation between text/graph size and performance. This is likely because larger samples tended to have higher explicitness, which aided the models. Graph topology (cycles, density) had minimal impact on difficulty.
Domain: Performance was consistent across domains (Economics, Engineering, Health), suggesting the challenge is generalizable and not domain-specific.

D. Error Analysis

Direction Reversals: Rare (<1.1% of matched edges). When models identify a relationship, they rarely invert the causality.
Grounding Gap: ~85–90% of generated edges had some textual support, but only 17–33% matched the ground-truth edges. Models generate "plausible" but incorrect causal structures.

4. Key Contributions

ReCITE Benchmark: The first dataset for evaluating LLM causal reasoning on real-world, long-form academic texts with implicit relationships and complex graph structures.
LLM-as-a-Judge Framework: A novel evaluation method that handles semantic variation and abstraction, allowing for automated, scalable assessment of causal graphs.
Empirical Evidence of Limitations: Demonstrated that current SOTA LLMs fail to perform robust causal reasoning on realistic tasks, with performance heavily dependent on the explicitness of the text.
Diagnostic Insights: Isolated causal reasoning as the primary failure mode (distinct from entity recognition) and identified explicitness as the dominant difficulty factor.

5. Significance and Future Directions

AGI Roadmap: The inability to infer implicit causal structures from complex text is a major barrier to Artificial General Intelligence (AGI), as real-world decision-making relies on this skill.
Research Focus: The findings suggest that future improvements in causal reasoning will not come solely from scaling parameters but require architectural changes or training strategies specifically targeting implicit inference and long-context integration.
Cost-Effectiveness: The benchmark is designed to be computationally efficient (total experiment cost < $1,000), making it accessible for further research.

In conclusion, the paper establishes that while LLMs are excellent at retrieving explicit information, they currently lack the robust causal reasoning capabilities required to reconstruct complex, implicit causal graphs from real-world documents.