Technical Summary: Validity Threats for Foundation Model Research

Problem Statement

The advent of foundation models has dramatically increased the computational cost of training state-of-the-art machine learning systems, often requiring tens of thousands of GPUs. Consequently, the ideal controlled experiment—training multiple models with varying recipes to isolate causal effects—is computationally prohibitive. To circumvent these costs, the research community has adopted approximation strategies, including proxy experiments (using smaller models or cheaper treatments), observational studies (analyzing public meta-data), and single-run designs (leveraging internal variation within a single training run).

The central problem addressed in this work is that these cost-saving strategies introduce validity threats: hidden and sometimes untestable assumptions. When these assumptions are violated, research claims regarding model behavior, optimal hyperparameters, or data composition can become fundamentally incorrect or inapplicable. The authors argue that there is "no free lunch" in approximating large-scale experiments; savings in compute inevitably come at the cost of validity.

Methodology

The paper proposes an evaluation framework that formalizes foundation model research as a causal inference problem. The goal is to approximate the "ideal experiment" that identifies the recipe-specific treatment effect, defined as:
$$\tau(a, y; \psi) = E[y(a) | \psi] - E[y(\emptyset) | \psi]$$
where $\psi$ is the training recipe, $a$ is a treatment (change to the recipe), and $y$ is the outcome.

The framework decomposes the analysis into two layers:

1. Research Strategies: The authors categorize current practices into three primary strategies:

   • The Proxy Approach: Conducting experiments on cheaper proxies (smaller models, proxy treatments like fine-tuning instead of pre-training, or proxy outcomes like validation loss) and extrapolating to the target scale.
   • The Observational Approach: Analyzing meta-data from existing public models to infer relationships between training choices and outcomes without new training runs.
   • The Single-Run Approach: Emulating multiple experiments within a single training run by treating data points or subwords as independent units or using theoretical controls (e.g., influence functions, canary data).

2. Validity Types: Adapting a taxonomy from empirical social sciences, the framework evaluates strategies through four lenses:

   • Statistical Validity: Reliability of estimates given finite, noisy samples.
   • Internal Validity: Whether an estimate reflects a genuine causal effect rather than mere association.
   • External Validity: Whether findings generalize from the proxy or sample to the target training recipe (e.g., across scales or model families).
   • Construct Validity: Whether the operationalization of treatment and outcome accurately reflects the intended research question.

Key Contributions and Results

1. Characterization of Validity Profiles

The paper synthesizes the validity threats inherent to each strategy into a "validity profile," classifying threats as "rather not affected," "requiring careful consideration," or "key threat."

• Proxy Models:
  • Strengths: Generally preserve statistical and internal validity (if controlled experiments are used) and construct validity (if the original treatment/outcome is used).
  • Key Threat: External Validity. The assumption that treatment effects scale predictably (e.g., via scaling laws) is often unverified. Predictability breaks down across different training recipes or when emergent behaviors occur.
• Proxy Treatments and Outcomes:
  • Strengths: Do not inherently threaten statistical, internal, or external validity regarding the target model.
  • Key Threat: Construct Validity. Substituting the treatment (e.g., fine-tuning for pre-training) or outcome (e.g., benchmark subsets) often fundamentally alters the causal estimand. For instance, fine-tuning proxies may overestimate privacy leakage due to different memorization dynamics compared to single-epoch pre-training.
• Observational Approach:
  • Strengths: Retains evidence at the target scale and can offer high statistical validity by pooling data across families.
  • Key Threat: Internal Validity. The approach relies on the untestable assumption of exchangeability (no confounding). Confounders like "calendar time" (where newer models benefit from both architectural advances and curated data mixes) can induce spurious associations. Additionally, External Validity is threatened by effect heterogeneity across model families; pooling data averages out these differences, potentially obscuring recipe-specific effects.
• Single-Run Approach:
  • Strengths: Can provide fine-grained insights and maintain statistical validity with large within-run populations.
  • Key Threat: Internal Validity due to interference. The assumption that treating one unit (e.g., a data point) does not affect others is strained because all units share the same model parameters; training on point $i$ updates parameters used for point $j$. Additionally, External Validity across units is threatened if effects are heterogeneous.

2. Identification of Under-Addressed Threats

The analysis reveals specific validity threats that have received insufficient attention in the literature:

• The violation of exchangeability in observational studies due to confounding by calendar time and data composition.
• The interference between units in single-run designs, which violates the Stable Unit Treatment Value Assumption (SUTVA).
• The fundamental shift in construct validity when proxy treatments (like multi-epoch fine-tuning) are used to approximate pre-training dynamics.

3. Practical Toolkit

The paper provides a matrix (Table 1) serving as a practical toolkit for researchers and reviewers. It allows them to:

• Explicitly identify which validity types are at risk for a given design.
• Make hidden assumptions explicit.
• Understand the trade-offs required when combining strategies (e.g., an observational study using a proxy treatment accumulates validity threats).

Significance and Claims

The authors claim that their framework provides a common language for discussing validity in foundation model research, bridging the gap between causal inference theory and empirical machine learning practice.

• For Authors: The framework offers a vocabulary to defend necessary trade-offs and make hidden assumptions explicit, ensuring that claims are aligned with the active validity threats of the chosen design.
• For Reviewers: It provides a structured basis to assess whether a paper's conclusions are supported by its underlying methodology, particularly when multiple approximation strategies are combined.
• For the Field: The paper argues that while the ideal experiment is often infeasible, researchers must be aware that "savings in compute come at the cost of validity threats." The framework does not propose a single solution to eliminate these threats but rather a systematic method to scrutinize them, suggesting that robust conclusions may require pursuing multiple strategies in parallel or relying on accumulated scientific knowledge about the reliability of specific threats.

The work emphasizes that validity is not a binary property but a spectrum of risks that must be managed through careful design and explicit acknowledgment of assumptions.