Benchmarking the Benchmarks: Testing the Predictive Validity of Commonsense Benchmarks

Technical Summary: Benchmarking the Benchmarks

Problem Statement
Standardized evaluation setups, particularly static multiple-choice commonsense benchmarks, are widely used to infer Large Language Model (LLM) capabilities. However, the extent to which performance on these benchmarks predicts downstream behavior in real-world tasks remains underspecified. While the flaws of existing benchmarks (e.g., artifacts, contamination, unstable annotations) are well-documented, the critical question has shifted: do benchmark scores serve as valid indicators of downstream model competence? Specifically, it is unclear whether "reworked" benchmarks—designed to mitigate known flaws—actually improve criterion validity (predicting external outcomes) or merely preserve the relative rankings of models without enhancing their utility for capability inference.

Methodology
The study evaluates 23 open-weight LLMs from six families (Llama 3, Gemma 2, Pythia, OPT, Qwen 2.5, Mistral) ranging from 1B to 72B parameters. The experimental design involves a comprehensive comparison across four dimensions:

1. Benchmark Pairs: The authors evaluate models on four established commonsense benchmarks and their reworked variants:
   • WinoGrande vs. WinoWhat (paraphrased)
   • HellaSwag vs. GoldenSwag (filtered)
   • Social IQA vs. Social IQA filtered
   • Physical IQA vs. Physical IQA-RUP (paraphrased)
2. Downstream Tasks: Models are tested on eight downstream tasks requiring implicit reasoning, categorized into three conceptual axes:
   • Social/Emotional: CEI, SARC7, False Beliefs.
   • Pragmatic: Implicature, Presupposition, Indirect Requests.
   • Event/Physical: TimeDial, TRIP.
3. Control Benchmarks: To distinguish commonsense-specific ability from general model capability, the study includes non-commonsense controls: BLiMP, GPQA-Diamond, and MMLU-Redux.
4. Evaluation Protocol: All tasks utilize a log-likelihood-based multiple-choice setup to ensure uniformity. The analysis employs:
   • Pairwise Spearman correlations to assess model rankings.
   • Partial Spearman correlations controlling for model size, family, and control benchmark performance.
   • Leave-one-family-out cross-validation (LOFOCV) using Ridge regression to test predictive validity for unseen model families.
   • Fisher-z transformations and nonparametric bootstrapping to estimate confidence intervals.

Key Contributions and Results

• Preservation of Rankings vs. Validity: The study finds that reworked benchmarks largely preserve the model rankings of their original counterparts. The exception is WinoWhat, which correlates less strongly with WinoGrande and other benchmarks, suggesting it captures different model behaviors. However, these revisions do not systematically improve downstream predictive validity. In most cases, the difference in correlation between original and reworked benchmarks regarding downstream tasks is negligible. Notably, the original WinoGrande is substantially more predictive of the False Beliefs task than its reworked variant WinoWhat.
• Narrow Transfer to Downstream Tasks: Commonsense benchmarks show consistent cross-family predictive validity only for a narrow subset of downstream tasks.
  • Strongest Predictors: False Beliefs and TRIP show the strongest and most consistent associations with commonsense benchmark scores.
  • Metric-Specific Gains: Implicature, Indirect Requests, and TimeDial show gains only for specific metrics (e.g., macro-F1 or accuracy), not both.
  • Poor Predictors: CEI (Emotional Inference) and SARC7 (Sarcasm) remain poorly predicted by commonsense benchmarks.
• Lack of Domain Specificity: The hypothesis that benchmarks transfer best to downstream tasks within the same conceptual axis (e.g., Social IQA predicting social tasks) is not supported. While Event/Physical benchmarks show some alignment with physical plausibility tasks, the pragmatic axis shows the opposite pattern. Globally, matched pairs are often weaker than mismatched pairs, indicating that benchmark labels do not reliably map to fine-grained downstream capabilities.
• General Capability vs. Specific Skill: When controlling for model size, family, and performance on non-commonsense controls, the statistical significance of most correlations diminishes. This suggests that observed transfers often reflect broad model capability rather than distinct commonsense-specific reasoning. However, LOFOCV results indicate that commonsense scores do retain some incremental, cross-family information for False Beliefs and TRIP.

Significance and Claims
The paper argues that standardized commonsense benchmarks provide task-dependent rather than broad evidence of downstream commonsense competence. The authors claim that:

1. Revisions do not guarantee validity: Cleaning datasets to remove artifacts or ambiguity improves internal quality but does not automatically translate to better criterion validity or more useful indicators for model selection.
2. Limited Generalizability: Performance on benchmarks like HellaSwag, Social IQA, or Physical IQA should not be interpreted as broad evidence of pragmatic, social, or contextual reasoning abilities. The relationship between benchmark scores and downstream tasks is uneven and rarely aligns with the subdomains implied by benchmark metadata.
3. Shift in Evaluation Priorities: The study calls for a shift in the benchmarking paradigm. Rather than repeatedly refining isolated static datasets, the community should prioritize building evaluation suites where scores are demonstrably informative about specific behaviors and applications. Claims about model capabilities require evidence of criterion validity: scores must generalize to external tasks that instantiate the intended capability in different contexts and formats, contributing information beyond general model ability.

The authors conclude that while commonsense benchmarks are informative for specific tasks like False Beliefs and TRIP, they do not reliably support fine-grained claims about distinct commonsense subskills or broad pragmatic competence.