Technical Summary: EconCausal

Problem Statement

Socio-economic causal effects are inherently context-dependent; the same intervention can yield divergent or even opposite outcomes depending on regulatory regimes, market conditions, time periods, or population characteristics. While Large Language Models (LLMs) have demonstrated promise in general reasoning, their application to economic decision-support is hindered by a critical gap: the inability to reliably infer causal directions under specific, high-dimensional contexts and to revise those judgments when the context shifts. Existing benchmarks fail to address this because they typically focus on generic, context-light relations, omit the institutional and empirical context necessary for interpretation, or rarely test robustness under context shifts. Consequently, current models risk producing superficial or misleading causal judgments in socio-economic settings.

Methodology

Dataset Construction: EconCausal

The authors introduce EconCausal, a large-scale benchmark comprising 10,490 context-annotated causal triplets derived from 2,595 high-quality empirical studies published in top-tier economics and finance journals (e.g., American Economic Review, Journal of Finance). The dataset construction follows a rigorous four-stage pipeline:

1. Triplet Extraction with Multi-run Consensus: Using GPT-5 mini, the system extracts treatment–outcome pairs from papers three times independently. Pairs appearing in at least two runs are retained. Minor wording variations are merged via cosine similarity (threshold $\ge$ 0.8) on text embeddings. Signs ($+, -, \text{None}, \text{Mixed}$) are assigned by majority vote, with supporting evidence paragraphs stored for traceability.
2. Metadata and Global Context: Paper-level metadata (empirical vs. theoretical), global context summaries (time, location, unit of analysis), and identification methods (e.g., DiD, IV, RCT) are extracted to serve as defaults.
3. Triplet-Specific Refinement: For empirical papers, the system verifies if the global context applies to specific triplets. If a triplet requires a more specific context or identification method, a minimal edit is applied; otherwise, global defaults are retained.
4. Multi-Critic Evaluation and Filtering: Three independent LLM critics (Gemini, Grok, Qwen) score each triplet on six dimensions (variable extraction, direction, sign, causality, main claim, context appropriateness) using a 0–3 rubric. Triplets failing conservative thresholds (average score $\ge$ 2.0 per criterion and total sum $\ge$ 15) are filtered out, removing 27.3% of candidates.

The dataset was validated by three economics professors, who approved 86.6% of the retained triplets, confirming the filtering rule's conservatism.

Benchmark Tasks

EconCausal evaluates LLMs through three progressively complex tasks:

• Task 1 (Causal Sign Identification): Given a specific context $c$ and a treatment–outcome pair $(T, O)$, predict the causal sign. This tests if models have internalized established economic causalities.
• Task 2 (Context-Dependent Sign Prediction): Given a known sign for $(T, O)$ under context $c_1$, predict the sign under a different context $c_2$. This tests the ability to recognize that causal signs shift with context.
• Task 3 (Misinformation-Robust Sign Prediction): Similar to Task 2, but the provided example for $c_1$ contains a deliberately incorrect sign. This tests robustness against misleading evidence.

Key Results

Performance Overview

Experiments across diverse proprietary (Gemini, GPT, Grok) and open-source (Llama, Qwen) models reveal a significant performance gap between fixed-context reasoning and context-dependent transfer:

• Fixed Context (Task 1): Top models achieve high accuracy ($\sim$88% for Gemini 3 Flash), indicating strong internalization of causal facts in explicit settings.
• Context Shift (Task 2): Accuracy drops significantly when models must revise judgments across contexts. For closed-source models, overall accuracy falls from 73.9% to 41.3% on "sign-mismatch" cases (where the source and target contexts imply opposite signs).
• Misinformation Robustness (Task 3): Performance degrades further when misleading evidence is introduced, with closed-source and open-source averages dropping to 49.3% and 44.3%, respectively.

Systematic Failures

• Over-Transfer of Source Signs: In Task 2, models exhibit a strong bias toward "over-transferring" the sign from the source example, even when it contradicts the target context. Adding a signed source example reduced correct predictions by 15.08 percentage points while increasing predictions matching the source sign by 21.95 percentage points.
• Bias Toward Directional Signs: Models systematically over-commit to binary signs ($+$ or $-$), struggling to identify null ($\text{None}$) or heterogeneous ($\text{Mixed}$) effects. Accuracy on $\text{None}$ cases is only 13.8%, and on $\text{Mixed}$ cases is 22.8%, compared to 74.8% and 61.1% for positive and negative signs, respectively.
• Poor Calibration: Models display "over-commitment," assigning high confidence to incorrect predictions, particularly for $\text{None}$ and $\text{Mixed}$ categories. Expected Calibration Error (ECE) is severe for these ambiguous categories (e.g., 0.839 for $\text{None}$ in Finance), and models rarely select "Unknown" even when contextual information is entirely removed.

Significance and Claims

The paper claims that EconCausal provides a rigorous foundation for evaluating how LLMs integrate institutional and environmental constraints into economic judgments. The authors argue that current systems are not yet robust enough for high-stakes economic or policy-sensitive analyses because they conflate "plausibility" with "portability." While models can generate plausible causal signs, they fail to determine whether those signs apply to a specific target context, often anchoring on empirically valid signs from source contexts where they no longer hold.

The benchmark highlights that socio-economic reasoning requires more than pattern recognition; it demands the ability to condition judgments on high-dimensional, partially observed, and potentially endogenous contexts. The authors position EconCausal as a necessary step toward developing systems that can judge not only whether a causal sign is plausible, but whether it transfers to the context at hand.

Limitations

The authors acknowledge several limitations:

• Extraction Dependence: The pipeline relies on LLMs for extraction and filtering, which may introduce systematic biases or omit subtle identification details, despite multi-run consensus and expert validation.
• Corpus Bias: The dataset is drawn from top-tier, English-language journals, potentially overrepresenting influential research while underrepresenting other socio-economic claims.
• Label Granularity: The directional label space ($+, -, \text{None}, \text{Mixed}$) compresses complex empirical findings, omitting effect sizes, magnitudes, and confidence intervals.
• Context Scope: Contexts are tied to specific empirical settings, limiting the direct evaluation of external validity across unannotated features like market structure or enforcement environments.
• Behavioral Scope: The evaluation is behavioral and does not probe internal model representations or attention mechanisms.