🤖 machine learning

Causal methods for LLM development and evaluation

This paper argues that causal inference methods are currently underutilized yet essential for addressing confounding, bias, and non-stationarity in large language model development and evaluation, offering a principled framework to guide interventions across the entire LLM pipeline from pretraining to deployment.

Original authors: Dennis Frauen, Marie Brockschmidt, Konstantin Hess, Haorui Ma, Yuchen Ma, Abdurahman Maarouf, Maresa Schröder, Jonas Schweisthal, Yuxin Wang, Athiya Deviyani, Sonali Parbhoo, Rahul G. Krishnan, Stefan

Published 2026-05-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Dennis Frauen, Marie Brockschmidt, Konstantin Hess, Haorui Ma, Yuchen Ma, Abdurahman Maarouf, Maresa Schröder, Jonas Schweisthal, Yuxin Wang, Athiya Deviyani, Sonali Parbhoo, Rahul G. Krishnan, Stefan Feuerriegel

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef trying to create the perfect new recipe for a massive restaurant chain. You have thousands of ingredients (data), different cooking methods (models), and a team of critics (evaluators). Right now, most chefs just guess: "Let's add more salt," "Let's try a different oven," or "Let's ask the critics what they liked." They taste the dish, make a note, and try again. This is how Large Language Models (LLMs) are currently built—through trial and error.

This paper argues that this "guess and check" approach is risky. Instead, the authors suggest we should use Causal Methods, which are like a scientific detective kit. Instead of just seeing what happened, these tools help us answer the question: "What caused this result, and what would have happened if we did something different?"

Here is the paper's vision, broken down into simple analogies:

1. The Core Problem: The "Fake" Connection

Imagine you notice that customers who order the most expensive wine also tip the most.

The Naïve View: "If I give everyone free expensive wine, they will tip more!"
The Causal View: "Wait. Maybe the rich people who can afford the wine are the ones tipping well, not the wine itself. If I give cheap wine to a rich person, they might still tip well. If I give expensive wine to a broke person, they might not tip at all."

In LLMs, this happens all the time.

The Scenario: A system sends difficult questions to a "smart" (large) model and easy questions to a "fast" (small) model.
The Mistake: The large model gets lower scores because the questions were harder, not because the model is worse.
The Causal Fix: We need to separate the difficulty of the question from the quality of the model to see who is actually doing the best job.

2. The Three Main Tools (The "How-To")

The paper explains that to solve these puzzles, we need three specific skills:

Identifiability (The "Can We Know?" Check): Before we start, we ask: "Do we actually have enough information to solve this?" If our data is biased (like only asking happy customers for reviews), we might never know the true answer, no matter how much data we collect. Causal methods tell us when we are stuck and what assumptions we are making.
Estimation (The "How to Calculate" Check): Even if we can know the answer, the data is messy and huge. We need special math (called "Double Machine Learning") to clean out the noise and get a precise answer, similar to how a noise-canceling headphone filters out background chatter to hear the music clearly.
Counterfactuals (The "What If?" Machine): This is the superpower. It lets us simulate a different reality. "What if we had routed this question to the small model instead?" We can answer this without actually breaking the system to test it.

3. Where This Helps (The Kitchen Tour)

The authors map out where these detective tools fit into building an AI:

Pre-training (Mixing the Ingredients):
- The Problem: We mix books, code, and forums to train the AI. But if we filter out "toxic" text, we might accidentally remove important dialects or facts.
- The Fix: Use causal tools to predict how changing the "recipe" (the mix of data) will change the final taste (model performance) without having to cook the whole meal again.
Alignment (Teaching the AI to be Polite):
- The Problem: We ask humans to pick the better answer between two options. But humans have biases (e.g., they prefer longer answers).
- The Fix: Use causal tools to strip away the human bias so we can see which answer was actually better, not just which one looked nicer.
Routing (The Waiter's Decision):
- The Problem: A waiter (router) decides which chef (model) gets which order. If they send hard orders to the head chef, the head chef looks slow.
- The Fix: Causal methods help the waiter learn the true speed and quality of each chef, regardless of the order difficulty, so they can assign tasks efficiently.
Agents (The AI Team):
- The Problem: Modern AI doesn't just answer; it takes steps (searches the web, uses tools, calls other AIs). If the final result is bad, was it the search tool? The tool user? Or the first step?
- The Fix: Causal methods act like a "black box" recorder that traces exactly which step caused the success or failure, helping us fix the specific broken link in the chain.
Evaluation (The Critics):
- The Problem: Sometimes we use one AI to judge another AI. But the judge AI might be biased (e.g., it likes its own style of writing).
- The Fix: Use a mix of human and AI judges with causal math to cancel out the bias and get a fair score.

4. The Safety Check

Finally, the paper talks about safety. If an AI says something harmful, we need to know why. Was it the training data? The reward system?

The Analogy: If a car crashes, we don't just say "it crashed." We look at the black box to see if it was the brakes, the driver, or the road.
The Fix: Causal methods help us trace harmful behavior back to its source in the training pipeline, rather than just guessing.

The Bottom Line

The authors aren't saying "Causal methods will make AI smarter." They are saying: "Causal methods will make the process of building and testing AI more reliable."

Right now, we are building these complex systems in the dark, guessing what works. This paper argues we should turn on the lights, use scientific tools to understand cause and effect, and build AI systems that are safer, cheaper, and more trustworthy because we actually know why they work.

Technical Summary: Causal Methods for LLM Development and Evaluation

Problem Statement

The development and evaluation of Large Language Models (LLMs) currently rely heavily on large-scale empirical iteration over data mixtures, reward models, routing strategies, and evaluation pipelines. However, this process faces a fundamental challenge: many central decisions are made based on logged observational data rather than randomized experiments. This data is subject to selection bias, distribution shifts, and noisy or biased labels (e.g., from human raters or learned judges).

Consequently, purely predictive approaches are often fragile. Observed performance differences may reflect confounding factors (e.g., a larger model appearing worse because it is routed harder queries) rather than the true causal effect of a design decision. The paper argues that the LLM development pipeline is inherently a causal problem, yet causal methods remain underutilized. The core problem is how to reliably infer the counterfactual effects of interventions (such as changing data mixtures, routing policies, or prompt templates) when only historical, non-randomized logs are available.

Methodology

The paper proposes a unifying framework that applies causal inference principles—specifically identifiability and estimation—to the entire LLM lifecycle.

1. Formalization as Causal Interventions

The authors formalize LLM development decisions using the potential outcomes framework:

Context ( $X$ ): Observed variables (e.g., user prompts, pre-training corpus, retrieved documents).
Treatment ( $A$ ): The system choice or intervention (e.g., data mixture weights, model routing assignment, prompt template).
Outcome ( $Y$ ): The metric of interest (e.g., task success, preference score, latency, safety).
Goal: Estimate causal quantities such as $E[Y(a)]$ (the expected outcome if treatment $A$ were set to $a$ ) or $E[Y(a) - Y(a')]$ (the effect of switching treatments).

2. Core Principles

Identifiability: Determining whether the causal effect can be uniquely recovered from observed data under explicit structural assumptions. The paper highlights that in LLMs, data is rarely randomized; it is policy-induced. Challenges include selective observation (only some prompts are labeled) and biased judges. The framework clarifies when assumptions (e.g., no unobserved confounding) are plausible and when they are not, suggesting the use of sensitivity analysis and partial identification when assumptions fail.
Estimation: Even when identifiable, effects must be estimated from high-dimensional, complex data. The paper advocates for Debiased/Double Machine Learning (DML). DML uses flexible machine learning models to predict nuisance components (e.g., the historical policy or reward model) and estimates the causal effect on the residuals. This approach removes bias from confounding and provides valid statistical inference (e.g., confidence intervals) even when nuisance models are misspecified.

3. Application Across the Pipeline

The methodology is mapped to specific stages of LLM development:

Pre-training: Treating domain mixture proportions as continuous treatments to predict downstream performance without full retraining.
Alignment & Preference Learning: Addressing the limitations of the Bradley-Terry (BT) model, which assumes a single latent score. The paper suggests using DML and semiparametric extensions to handle cyclic preferences and unobserved confounding in annotation data.
Deployment (Routing & Agents): Applying off-policy evaluation to routing systems where queries are non-randomly assigned to models. For agentic workflows (sequential decisions), the framework addresses dynamic confounding and credit assignment across intermediate steps (e.g., retrieval, tool use).
Evaluation: Using causal methods to correct for biases in "LLM-as-a-judge" systems and to design optimal data collection strategies (active learning) that minimize variance in annotation costs.
Safety Audits: Framing the discovery of latent reward functions as a partial identification problem, acknowledging that multiple reward functions can produce identical behaviors, and seeking bounds on plausible objectives rather than point estimates.

Key Contributions

The paper makes three primary contributions:

Conceptual Framework: It explains why and when causal methods are necessary for LLM development, identifying common engineering decisions as causal interventions and highlighting the fragility of non-causal approaches due to confounding and distribution shifts.
Pipeline Mapping: It provides a comprehensive map of opportunities for causal methods across the entire LLM lifecycle, including pre-training, alignment, routing, agentic workflows, and evaluation. It identifies specific research opportunities (ROs) in each domain, such as using DML for preference learning and off-policy evaluation for routing.
Research Agenda: It outlines a "Blue Sky" vision for future research, particularly focusing on agentic systems and world models where actions unfold through sequential decisions. It emphasizes the need for methods that handle structured mediators, dynamic confounding, and the identification of latent reward functions.

Results and Claims

As a position and vision paper, it does not present new experimental results or a specific algorithmic benchmark. Instead, its "results" are the identification of gaps and the proposal of a methodological shift:

Current State: LLM development relies on heuristics and observational data, leading to potential spurious correlations and biased evaluations (e.g., larger models appearing worse due to routing bias).
Proposed State: By adopting causal inference, developers can move from correlation-based heuristics to principled identification and estimation.
Specific Claims:
- Causal methods can ensure reliable and scientifically grounded design by isolating the effect of decisions from artifacts of data collection.
- Techniques like DML can improve data efficiency, robustness to misspecification, and the validity of statistical inferences (e.g., confidence intervals) in high-dimensional settings.
- Simply adding "causal methods" to training objectives does not automatically grant LLMs causal reasoning abilities; rather, causal methods should be used as inputs to improve the development and evaluation process itself.

Significance

The paper argues that causal methods are potentially underutilized in the LLM development pipeline. Its significance lies in:

Reliability: Providing a framework to distinguish true causal effects from confounding, thereby reducing the risk of deploying models based on spurious correlations.
Scientific Rigor: Introducing formal assumptions (identifiability) and statistical guarantees (valid confidence intervals) to a field often driven by empirical iteration.
Future-Proofing: Offering tools to handle the non-stationary nature of LLM environments where models, users, and tasks co-adapt over time.
Safety: Enabling more robust safety audits by characterizing the set of plausible reward functions rather than assuming a single identified objective.

The authors conclude that bridging causal inference and LLM development is a promising foundation for building reliable, data-efficient, and deployment-aware AI systems.