NoisyCausal: A Benchmark for Evaluating Causal Reasoning Under Structured Noise

Technical Summary: NoisyCausal

Problem Statement
Causal reasoning in natural language requires identifying variables, understanding their interactions, and reasoning about effects and interventions, often under conditions of noise or ambiguity. While Large Language Models (LLMs) demonstrate strong general reasoning capabilities, they frequently struggle to disentangle correlation from causation, particularly when observations are partially incorrect or contain irrelevant information. Existing benchmarks (e.g., COPA, CausalQA, Cladder) primarily rely on clean, idealized scenarios with simple or implicit causal structures. These datasets often fail to reflect real-world conditions where data is noisy, incomplete, or confounded by spurious variables. Consequently, current models often succeed via pattern matching or shallow heuristics rather than genuine causal understanding, limiting their robustness and interpretability.

Methodology
The paper introduces NoisyCausal, a synthetic benchmark designed to evaluate causal reasoning under structured noise, alongside a modular graph-guided reasoning framework.

1. NoisyCausal Benchmark Construction:

   • Pipeline: The dataset is generated through a five-stage pipeline: (1) sampling a Directed Acyclic Graph (DAG) with diverse topological motifs (chains, forks, colliders); (2) semantic grounding by assigning real-world labels (e.g., medicine, infection) and metadata; (3) sampling variable values from a Structural Causal Model (SCM) to ensure causal consistency; (4) injecting structured noise; and (5) assembling natural language questions.
   • Noise Types: The benchmark introduces six controllable noise categories to simulate real-world imperfections:
     • Value Perturbation (VP): Flipping variable values.
     • Irrelevant Variable Injection (IV): Adding non-causal, spurious variables.
     • Partial Masking (PM): Hiding observed variable values.
     • Causal Swap (CS): Swapping values of linked nodes to break conditional logic.
     • Latent Confounders (CI): Simulating hidden variables influencing multiple nodes.
     • Question Perturbation (QP): Modifying questions to contradict the underlying SCM.
   • Scale: The dataset contains 10,617 question-answer pairs covering various noise combinations and difficulty levels.

2. Graph-Guided Reasoning Framework:

   • Instead of end-to-end black-box inference, the proposed method decomposes reasoning into three steps:
     1. Variable Extraction: The LLM identifies key entities and latent factors from the context.
     2. Causal Graph Construction: The LLM constructs a task-specific DAG, predicting causal directionality between variable pairs.
     3. Structured Inference: The background context, observed values, and the constructed graph are formatted into a structured prompt. The LLM then performs inference grounded in this explicit symbolic structure rather than relying solely on statistical patterns.

Key Contributions

• NoisyCausal Benchmark: A new evaluation suite featuring explicit causal graphs, natural language scenarios, and controllable structured noise, designed to test robustness against distractors, confounders, and partial observability.
• Modular Framework: A reasoning architecture that integrates LLMs with explicit causal abstractions. It uses the causal graph as an interpretable intermediate to guide the LLM, separating structure extraction from inference.
• Empirical Validation: Demonstration that combining causal structure with LLM reasoning significantly improves performance and robustness compared to standard prompting and Chain-of-Thought (CoT) baselines.

Experimental Results

• Performance on NoisyCausal: The graph-guided method achieved 80.7% accuracy on clean data, outperforming GPT-4 (62.8%) and Causal CoT (73.4%). Under structured noise, the proposed method maintained high stability (e.g., 77.3% on Value Perturbation, 73.5% on Latent Confounders), whereas standard LLMs and CoT baselines showed significant degradation (e.g., GPT-3.5 dropped to 43.8% on Question Perturbation).
• Generalization: The model generalized to the external Cladder benchmark without task-specific tuning, achieving 82.3% accuracy, surpassing all baselines.
• Ablation Studies:
  • Removing the causal graph or using a randomly rewired graph caused significant performance drops (approx. 15–20%), confirming the necessity of correct structural guidance.
  • Performance declined gracefully as noise types were combined, dropping from 73.47% (1 noise type) to 57.98% (all 6 noise types).
  • The framework was found to be sensitive to incorrect causal directionality (Direction Reversal) more than edge deletion.
• Structure Discovery: GPT-4 achieved high F1 scores (>85%) in extracting causal graphs from clean text, though performance degraded with noise. Structured prompts yielded better extraction quality than natural language instructions.
• Belief-Inconsistent Perturbation (BIP): When exposed to prompts containing common misconceptions contradicting the causal graph, standard models suffered accuracy drops (up to 6%), while the graph-guided model remained robust (dropping only 1.2%).
• Transferability: Fine-tuning LLaMA-7B on NoisyCausal improved its performance on the human-authored CausalQA benchmark by 5.1 percentage points, suggesting the learned skills transfer to real-world tasks.

Significance and Claims
The paper argues that current LLMs often fail at causal reasoning because they rely on statistical associations from pretraining corpora rather than mechanistic dependencies. The authors claim that NoisyCausal serves as a critical diagnostic tool to isolate specific failure modes (e.g., distractor sensitivity, confounding) that are difficult to disentangle in natural corpora.

The primary significance lies in demonstrating that combining causal abstractions with language-driven reasoning yields more faithful and robust inference. By guiding LLMs with explicit symbolic structures, the proposed framework mitigates the tendency to confuse correlation with causation, even under noisy conditions. The authors position this approach as a step toward "causally aware" LLMs, emphasizing that structural grounding is essential for achieving robust AI in uncertain environments.

Limitations
The authors acknowledge that the framework relies on LLM-based variable extraction and graph construction, which can introduce cascading errors. The current design lacks end-to-end optimization, and while the benchmark is diverse, real-world generalization to multimodal or open-domain discourse remains untested. Future work is suggested to include confidence estimation, joint training, and evaluation on real-world multimodal data.