Spend Less, Fit Better: Budget-Efficient Scaling Law Fitting via Active Experiment Selection

Technical Summary: Spend Less, Fit Better: Budget-Efficient Scaling Law Fitting via Active Experiment Selection

1. Problem Statement

Scaling laws are essential for predicting the performance of large-scale models (e.g., LLMs) to guide multi-million-dollar training decisions. However, the process of "fitting" these laws—running pilot experiments to collect data points—is itself becoming prohibitively expensive.

The authors identify a critical gap in current workflows: most researchers use manual, non-optimized experimental designs. In modern, complex scaling scenarios (such as optimizing data mixing, sparsity, or inference-time compute), the cost of pilot runs can reach millions of dollars. The core challenge is formulated as a budget-aware sequential experimental design problem: Given a finite pool of candidate experiments with heterogeneous costs, how can we select a subset of runs to maximize the predictive accuracy of the fitted law in a high-cost, large-scale target region?

2. Methodology

The paper proposes a novel sequential, uncertainty-aware design method. Unlike classical Optimal Experimental Design (OED), which focuses on parameter estimation, this method focuses on target-region extrapolation accuracy.

Key components of the methodology include:

• Multimodal Posterior Approximation: Recognizing that scaling-law fitting is highly non-linear and prone to local optima, the authors model the parameter uncertainty using a Mixture of Gaussians. Each component (or "basin") represents a plausible local fit of the scaling law.
• Target-Aware Uncertainty Objective: The objective is to minimize the Mean Squared Prediction Error (MSPE) in the target region ($X_{tar}$). The authors decompose this uncertainty into two distinct terms:
  1. Intra-basin Uncertainty ($V_{intra}$): The local predictive variance within each plausible fit.
  2. Inter-basin Uncertainty ($V_{inter}$): The disagreement between different plausible fits (basins) regarding their predictions in the target region.
• Acquisition Function: At each step, the algorithm selects the next experiment $x$ that maximizes a cost-aware score:
  $$S(x) = \frac{\Delta V_{intra}(x) + \Delta V_{inter}(x)}{c(x)^\alpha}$$
  where $\Delta V$ represents the expected reduction in uncertainty and $c(x)^\alpha$ penalizes the experiment's cost.
• Sequential Execution: The process is iterative: (1) Estimate current basins $\rightarrow$ (2) Score all candidates $\rightarrow$ (3) Run the highest-scoring experiment $\rightarrow$ (4) Update the dataset and repeat until the budget is exhausted.

3. Key Contributions

• Formalization: The first systematic study of scaling-law fitting treated as a budget-constrained, sequential decision-making problem.
• New Acquisition Framework: A mathematical framework that explicitly accounts for both local parameter refinement (intra-basin) and global model disambiguation (inter-basin) while considering heterogeneous costs.
• Comprehensive Benchmark: The creation of a diverse benchmark spanning 8 tasks (e.g., vocabulary design, MoE, data concentration, and hyperparameter tuning) and 65 scaling-law instances to rigorously test the method.

4. Results

The method was evaluated against several baselines, including Random, Cheapest, Cost-weighted Random, and classical design criteria (D-optimality and V-optimality).

• Superior Efficiency: The proposed method consistently outperforms all baselines. Notably, it can approach the performance of fitting on the full experimental set while using only approximately 10% of the total training budget.
• Robustness in Low-Budget Regimes: The advantage is most significant when the budget is extremely tight (e.g., 1% or 5%), where the method's ability to intelligently pick "informative" points prevents the wasted expenditure seen in random or heuristic approaches.
• Handling Non-linearity: As shown in the t-SNE visualizations (Figure 3), the method successfully navigates complex, multi-modal landscapes where traditional methods (like V-opt) fail because they only consider a single local linearization.
• Ablation Insights: Ablation studies revealed that while both terms are useful, reducing intra-basin uncertainty is the dominant driver of performance, though inter-basin disambiguation is crucial for resolving high-level model ambiguity early in the process.

5. Significance

This work shifts the paradigm of scaling-law analysis from a "modeling problem" to an "experimental design and budget allocation problem." For industrial AI labs, this methodology provides a mathematically grounded way to significantly reduce the R&D costs associated with planning large-scale training runs. It demonstrates that smart selection is more important than massive data collection when the goal is to predict behavior at the frontier of scale.