Using large language models for sensitivity analysis in causal inference: cases studies on Cornfield inequality and E-value

1. Problem Statement

In observational studies, establishing causal effects is hindered by unmeasured confounding, which can lead to biased effect estimates and misleading conclusions. To address this, researchers use sensitivity analysis methods like the Cornfield inequality and the E-value.

• The Challenge: While these methods are robust, their calculation and interpretation are often mathematically complex and difficult for clinicians and interdisciplinary researchers to apply correctly.
• The Gap: Although Large Language Models (LLMs) are increasingly used for causal discovery and reasoning, their reliability and utility in performing sensitivity analysis (specifically calculating E-values and interpreting robustness against unmeasured confounding) have not been systematically evaluated.

2. Methodology

The authors conducted a case study evaluation involving four widely used LLMs: ChatGPT (GPT-5 mini), Claude 4.5 Opus, DeepSeek V3, and Gemini-2.5-pro.

Data Sources

The study utilized four published observational studies across different domains (Smoking/Pulmonary Fibrosis, Back Pain/BMI, Alzheimer's Treatment, and Environmental Health/PCBs). From these studies, the authors extracted:

• Exposure and Outcome variables.
• Measured confounders.
• Estimated effect sizes (Risk Ratios, Hazard Ratios, or Odds Ratios).
• Ground Truth: The original E-values and qualitative conclusions reported in the papers.

Experimental Design

The researchers developed a structured prompting strategy to guide the LLMs through three specific tasks without providing the explicit E-value formula (to test independent reasoning):

1. Quantitative Calculation: Compute the E-value based on the provided effect size.
2. Qualitative Interpretation: Assess the robustness of the association to unmeasured confounding using two perspectives:
   • Cornfield Inequality: Evaluate if a single unmeasured factor could plausibly explain the association.
   • E-value: Evaluate the magnitude of the calculated E-value.
   • Synthesis: Provide a final conclusion ("unlikely," "possibly," or "highly likely" that confounding explains the result).
3. Exploratory Suggestion: Identify three potential unmeasured confounders relevant to the specific exposure-outcome relationship that were not measured in the original study.

Implementation Details:

• Inference was conducted via APIs.
• Temperature was set to 0 to minimize randomness and hallucinations.
• Maximum token limit was set to 2000 to ensure complete reasoning steps.

3. Key Contributions

• First Evaluation of LLMs in Sensitivity Analysis: This is the first study to investigate the capability of LLMs to perform sensitivity analyses for unmeasured confounding.
• Structured Prompting Framework: The authors developed a reproducible prompting strategy that guides LLMs to act as epidemiological experts, separating calculation, qualitative reasoning, and confounder identification.
• Comprehensive Benchmarking: The study provides a comparative analysis of four major LLMs across quantitative accuracy, qualitative nuance, and domain-specific knowledge generation.

4. Key Results

A. Quantitative Accuracy (E-value Calculation)

• ChatGPT, Claude, and Gemini: Demonstrated zero bias. They accurately reproduced the ground-truth E-values for all 11 test cases (covering 11 exposure-outcome pairs) without being given the formula.
• DeepSeek: Showed small but consistent biases ranging from -0.01 to 0.23 compared to the true values.

B. Qualitative Interpretation (Robustness Assessment)

• Differentiation: ChatGPT, Claude, and Gemini successfully differentiated their conclusions based on the magnitude of the effect size and E-value.
  • High E-values (e.g., 4.25): All three correctly concluded the association was "unlikely" to be explained by confounding.
  • Low E-values (e.g., <1.4): Nuanced differences emerged. Claude tended to be more conservative, frequently concluding "highly likely" that confounding could explain the result for low E-values. ChatGPT and Gemini were more moderate, often concluding "possibly."
• DeepSeek: Failed to differentiate; it consistently concluded "possibly" regardless of the E-value magnitude or effect size, indicating a lack of sensitivity to the quantitative inputs.

C. Suggestion of Unmeasured Confounders

• All four models generated biologically and epidemiologically plausible unmeasured confounders (e.g., genetic predisposition, occupational exposures, sleep quality).
• Specificity: Claude, DeepSeek, and Gemini provided more specific suggestions (e.g., "fish consumption" vs. general "dietary patterns"). ChatGPT tended to provide broader, less detailed categories.

5. Significance and Implications

• Accessibility: The study demonstrates that LLMs, when guided by structured prompts, can serve as accessible tools for clinicians and researchers to perform complex sensitivity analyses, lowering the barrier to entry for robust causal inference.
• Study Design Support: By suggesting plausible unmeasured confounders, LLMs can assist researchers in identifying critical variables to measure during the design phase of future observational studies.
• Nuanced Interpretation: Unlike original papers that often provide a single qualitative conclusion for multiple analyses, LLMs can generate distinct interpretations for different exposure-outcome pairs within the same study, aligning more closely with the mathematical reality of the E-values.
• Future Directions: The authors suggest that with larger datasets and refined prompting, LLMs could become standard assistants in clinical and public health research for assessing the credibility of observational findings.

Conclusion: The paper concludes that ChatGPT, Claude, and Gemini are effective tools for assisting in sensitivity analysis, capable of accurate calculation and nuanced interpretation, provided they are used with structured prompting strategies.