💬 NLP

Causal Claims in Economics

This paper introduces evidence-annotated claim graphs to map causal and non-causal relationships across 44,852 economics papers from 1980 to 2023, revealing a significant rise in causal claims over time and demonstrating that papers with stronger causal narrative structures and novelty are more likely to be published in top journals and receive long-term citations.

Original authors: Prashant Garg, Thiemo Fetzer

Published 2026-02-25

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Prashant Garg, Thiemo Fetzer

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the world of economics as a massive, chaotic library containing nearly 45,000 books (research papers) written over the last 40 years. For a long time, if you wanted to know what these books were actually saying, you had to read every single one of them. But as the library grew, reading them all became impossible.

This paper introduces a new way to organize that library. The authors, Prashant Garg and Thiemo Fetzer, built a "Claim Graph" system. Think of it as turning every dense, complicated economics paper into a simple, visual flowchart or a family tree of ideas.

Here is how it works, broken down into simple concepts:

1. The "Claim Graph" (The Family Tree of Ideas)

Instead of reading paragraphs of text, the authors used Artificial Intelligence (AI) to break every paper down into its core ingredients:

The Nodes (The People): These are the economic concepts, like "Inflation," "Education," or "Minimum Wage."
The Edges (The Relationships): These are the arrows connecting the concepts. They show what the author claims causes what. For example, an arrow pointing from "Education" to "Higher Wages."
The Color Code (The Proof): This is the most important part. The arrows are colored based on how strong the proof is.
- 🔵 Blue Arrows: These represent claims backed by "Causal Inference." This is the "gold standard" of proof. It means the author used a clever experiment or a natural disaster (like a sudden policy change) to prove that A actually caused B, not just that they happened at the same time.
- 🟠 Orange Arrows: These are claims backed by weaker evidence, like simple observation, theory, or just saying "they seem related."

2. The Big Discovery: The "Credibility Revolution"

The authors looked at the library from 1980 to 2023 and found a massive shift in how economists write.

In 1990: Only about 8% of the arrows in these flowcharts were Blue (strong proof). Most were Orange (weak proof).
In 2020: That number jumped to 32%.

The Analogy: Imagine a courtroom. In the 1990s, most lawyers were just saying, "My client was at the scene, so they must be guilty!" (Orange). Today, more lawyers are bringing in security camera footage and DNA tests to prove it (Blue). The entire field of economics has become much more rigorous about proving cause-and-effect.

3. Who Gets Rewarded? (The "Popularity Contest")

The paper asks: Does being "rigorous" (having more Blue arrows) actually help a paper get published in top journals or get cited by others?

The Good News: Yes! Papers with more Blue Arrows (strong causal proof) and New Blue Arrows (discovering new cause-and-effect relationships) are much more likely to get published in the "Top 5" elite journals and get cited by other researchers.
The Twist: Simply making a paper longer or more complex with Orange Arrows (weak proof) doesn't help. In fact, it might even hurt your chances.
The "Mainstream" Effect: There is one exception. If a paper talks about very popular, central topics (like "Inflation" or "Growth"), it gets cited a lot, even if the proof is weak. It's like a celebrity: everyone talks about them regardless of the quality of their latest work. But to get into the "Hall of Fame" (top journals), you need the Blue Arrows.

4. How They Did It (The AI Chef)

You might wonder, "How did they read 45,000 papers?"
They didn't read them all manually. They used a "Smart Chef" (AI) in a three-step kitchen:

The Taster: The AI reads the first 30 pages of a paper and summarizes the main ingredients.
The Chef: The AI draws the flowchart, connecting the ingredients and coloring the arrows based on the proof used.
The Taste Test: To make sure the AI didn't make mistakes, they asked it to cook the same dish three times. If the flowchart looked the same in all three tries, they kept it. If it looked different, they threw it out. This ensured the data was reliable.

Why Does This Matter?

This paper is like building a GPS for economic research.

Before, if you wanted to know "Does education cause higher wages?", you had to read hundreds of papers and guess which ones were trustworthy.
Now, with Claim Graphs, we can instantly see: "Here are 50 papers that prove it with strong evidence (Blue), and here are 20 that just guess (Orange)."

It helps us understand that the field of economics is maturing. It is moving away from "storytelling" and toward "hard evidence." And the best part? The system rewards the scientists who do the hard work of finding that hard evidence.

1. Problem Statement

Economic research is scaling rapidly in volume, driven by expanding working paper repositories and AI-assisted drafting. However, a critical bottleneck exists in representing and aggregating scholarly claims. Currently, economic arguments are embedded in heterogeneous narrative prose, making systematic comparison, synthesis, and measurement of "what papers claim" and "how they support it" difficult.

The authors aim to solve this representation problem by moving beyond simple paper-level classification (e.g., topic or method tags) to a claim-level representation. This allows researchers to:

Distinguish between causal and non-causal relationships within a single paper.
Map how specific claims chain together into mechanisms.
Quantify novelty and "gap filling" conditional on evidentiary support.
Analyze how incentives (publication, citations) shape the production of causal vs. non-causal evidence.

2. Methodology: Evidence-Annotated Claim Graphs

The core innovation is the construction of Claim Graphs, a structured, machine-readable representation of economic papers.

A. Data Corpus

Source: 44,852 working papers from the National Bureau of Economic Research (NBER) and the Centre for Economic Policy Research (CEPR).
Timeframe: 1980–2023.
Preprocessing: PDFs converted to text; analysis limited to the first 30 pages (where core claims and identification strategies are typically stated) to ensure consistency and manage compute costs.

B. Extraction Workflow (LLM-Based)

The authors employ a structured multi-stage retrieval workflow using GPT-4o-mini, designed to minimize hallucinations by acting as a constrained information-retrieval engine rather than an open-ended generator.

Stage 1 (Qualitative Summary): Extracts a structured summary of research questions, identification strategies, and metadata from the first 30 pages. This is run 3 times per paper to assess stability.
Stage 2 (Edge Extraction): Uses the Stage 1 summaries to extract candidate "edges" (relationships between concepts). Each edge is labeled with its evidentiary basis (causal vs. non-causal) and method (e.g., DiD, IV, RCT, OLS). This is run 3 times for each Stage 1 output, resulting in 9 independent extraction passes per paper.
Stage 3 (Standardization): Maps free-text source and sink variables to standardized JEL (Journal of Economic Literature) codes using semantic embeddings (cosine similarity).

C. Aggregation and Validation

Edge-Overlap (EO) Criterion: To convert the 9 candidate edge lists into a stable graph, the authors use an overlap threshold. An edge is retained only if it appears in at least 4 out of 9 runs (EO ≥ 4). This balances precision (filtering out noise) and recall (capturing valid claims).
Validation: The pipeline is validated against:
- Brodeur et al. (2024): For method and field classification.
- Plausibly Exogenous Galore Dataset: For exogenous variation sources.
- Snippet-based self-consistency checks: Comparing extraction from summaries vs. verbatim snippets.
- Stability diagnostics: Ensuring trends hold across different EO thresholds (EO ≥ 1 to EO ≥ 9).

D. Graph Definition

Nodes: Standardized economic concepts (JEL codes).
Edges: Directed relationships (Source $\to$ Sink) stated by authors.
Annotation: Edges are labeled as Causal (supported by canonical designs: DiD, IV, RCT, RDD, Synthetic Control) or Non-Causal (theoretical, descriptive, correlational).

3. Key Contributions

Methodological Infrastructure: Creation of a scalable, evidence-annotated claim graph dataset for 44,000+ papers, enabling claim-level meta-research.
New Metrics: Introduction of paper-level graph measures:
- Narrative Complexity: Breadth (edge count), depth (longest path), and interconnection (unique paths).
- Novelty: New edges, new paths, and "gap filling" (bridging historically rare concept pairs).
- Conceptual Positioning: Centrality (importance in prior literature) and diversity (mix of central/peripheral concepts).
- Architecture: Source-sink ratio (fan-in vs. fan-out structures).
Empirical Evidence: The first large-scale quantification of the "Credibility Revolution" at the level of specific claims rather than just paper-level method tags.

4. Key Results

A. The Rise of Causal Claims

Trend: The share of causal edges per paper rose sharply from 7.7% in 1990 to 31.7% in 2020.
Heterogeneity: The shift is most pronounced in fields like Health (9.7% $\to$ 41.9%), Urban (9.0% $\to$ 37.4%), and Development. Fields like Macroeconomics and Finance show smaller increases, though the gap is narrowing.
Method Adoption: DiD, IV, RCTs, and RDDs have proliferated, while theoretical/non-empirical work has declined from 51% (1980) to 28% (2023).

B. Rewards for Causal Structure (Publication & Citations)

The authors regress publication outcomes (Top 5 journals, Top 6–20, Top 21–100) and long-run citations on claim-graph measures.

Causal vs. Non-Causal:
- Causal Edges: Strongly positive association with Top 5 publication and citations. Doubling causal edge volume increases Top 5 probability by ~1.36 percentage points (+12% relative) and citations by ~11.2%.
- Non-Causal Edges: Weakly related or negatively associated with Top 5 publication and citations.
Novelty:
- Causal Novelty: Introducing new causal links or paths is strongly rewarded (Top 5 probability +1.71 pp; Citations +10.7%).
- Non-Causal Novelty: Exhibits weak or negative effects on outcomes.
Conceptual Positioning:
- Citations: Strongly linked to non-causal centrality (anchoring in mainstream topics).
- Editorial Selection: More closely aligned with causal depth and novelty.
- Interpretation: There is a "wedge" between what gets published in top journals (credibility/novelty) and what gets widely cited (conceptual familiarity).

C. Narrative Architecture

Papers using causal designs to support multiple drivers for a focal outcome ("fan-in" structure) are more likely to be published in top outlets and cited.
Non-causal fan-in expansion is not similarly rewarded.

5. Significance and Implications

For the "Credibility Revolution": The paper provides quantitative evidence that the shift toward causal inference is not just a change in methodology tags but a fundamental restructuring of economic arguments, with the profession increasingly rewarding causal depth.
For Research Incentives: The findings suggest that journals prioritize causal novelty and identified mechanisms, while the broader diffusion of ideas (citations) still relies heavily on conceptual centrality. This highlights a potential tension between rigorous identification and broad accessibility.
For Scientific Infrastructure: The "Claim Graph" framework offers a reusable, portable infrastructure for cumulative science. It allows for:
- Automated evidence synthesis across literatures.
- Tracing how claims are reframed in policy and media.
- Detecting gaps in the literature ("gap filling") to guide future research.
Limitations & Future Work: The approach extracts stated claims, not truth; it relies on the first 30 pages (potentially missing robustness checks); and JEL mapping compresses nuance. Future work could extend this to other domains, incorporate data provenance, and analyze external validity statements.

In summary, Garg and Fetzer demonstrate that by converting economic narratives into standardized, evidence-annotated graphs, researchers can rigorously measure the evolution of the discipline, revealing that causal depth and causal novelty are the primary drivers of high-impact publication, while conceptual centrality drives broad citation impact.