BenchBrowser -- Collecting Evidence for Evaluating Benchmark Validity

1. Problem Statement

Language Model (LLM) practitioners often rely on public benchmarks as proxies for their specific use-cases. However, this reliance is fraught with validity risks due to two primary issues:

• Content Validity Failure: Benchmarks often lack comprehensive coverage of a capability's facets. For example, a "coding" benchmark might overwhelmingly test Python syntax while ignoring Go concurrency, leading to a narrow and misleading assessment of a model's general coding ability.
• Convergent Validity Failure: Different benchmarks claiming to measure the same capability (e.g., "reasoning") often produce conflicting model rankings. A model might excel at scientific reasoning but fail at narrative reasoning, yet standard benchmarks may not distinguish these operationalizations, leaving practitioners unsure which signal to trust.

Current methods to diagnose these issues (manual auditing, keyword searches) are too slow and expensive to scale to modern benchmark suites containing tens of thousands of items. There is a lack of tools that allow practitioners to retrieve evidence matching their specific, natural-language intent without relying on rigid, pre-defined schemas.

2. Methodology: BenchBrowser

The authors introduce BenchBrowser, a retrieval-based tool designed to surface evaluation items relevant to a practitioner's natural-language use-case across 20+ benchmark suites (containing ~70k examples). The system operates in three stages:

Stage 1: Anchor Generation

To bridge the gap between abstract user queries and specific benchmark examples, the system rewrites the user's use-case into "retrieval anchors." The paper evaluates four strategies, ultimately selecting Example Synthesis as the most effective:

• Original: Uses the raw user query.
• Rephrasing: Decomposes the query into sub-topics or skills.
• Example Synthesis (Selected): Uses an LLM (GPT-5-mini) to generate $n=3$ synthetic test cases in diverse formats (e.g., MCQ, long-form QA, fill-in-the-blank) that mimic the distribution of the benchmark database. This maximizes format alignment.
• Shorthand: Converts queries and database items into a structured ⟨skill⟩&⟨key1⟩&⟨key2⟩ format using a fine-tuned Llama-3-8B model to enforce consistent skill verbalization.

Stage 2: Semantic Retrieval

• Embedding: The generated anchors are embedded using BAAI/bge-en-icl, an in-context learning embedding model chosen for its superior performance in handling diverse query intents compared to standard bi-encoders or rerankers.
• Search: The system uses Faiss to perform efficient similarity search, retrieving the top-$k$ nearest neighbors for each anchor.

Stage 3: Filtering

• An LLM judge (GPT-5-mini) filters the retrieved examples to remove obviously off-topic items.
• The judge assesses relevance based on three dimensions: Topical Overlap, Skill Overlap, and Practical Utility (whether success on the example implies competence for the user's specific goal).

3. Key Contributions

1. A Novel Retrieval Tool: BenchBrowser is the first tool to enable interactive, evidence-based auditing of benchmark validity by retrieving specific items relevant to free-form user intents.
2. Diagnostic Framework for Validity: The paper operationalizes the diagnosis of validity failures:
   • Content Validity: Measured by retrieving examples for different "facets" of a capability (e.g., "coding in Python" vs. "coding in Go") and observing disparities in coverage.
   • Convergent Validity: Measured by comparing model rankings on retrieved subsets against rankings on established "gold" test sets to detect rank inversions.
3. Human-in-the-Loop Validation: The system was validated via a user study with 10 NLP practitioners, confirming high retrieval precision and the tool's utility in diagnosing validity gaps.

4. Results and Findings

Retrieval Performance

• Precision: The Example Synthesis anchor method achieved the highest system precision (Prec_sys@20 ≈ 0.89 on Topics/Skills sets), significantly outperforming non-neural baselines (BM25, Random) and the raw "Original" query baseline.
• Stability: Precision remained stable as the number of retrieved examples ($k$) increased for broad topics and skills, though it dropped for highly contextual "Applications" queries.
• Human Alignment: Automatic relevance judgments by the LLM judge correlated strongly with human annotators (Spearman's $\rho \approx 0.50$ for Applications, higher for others), validating the system's ability to surface relevant evidence.

Content Validity Findings

• Facet Disparities: The tool revealed stark coverage gaps. For instance, while "Write a Python function" yielded a high precision of relevant examples (~0.87), "Write a function in Go" dropped precipitously to 0.10. Similarly, "Fantasy stories" were well-represented, while "Historical fiction" was significantly underrepresented.
• Implication: Aggregate benchmark scores often reflect the performance on dominant facets (e.g., Python) rather than the full capability, distorting the perceived competence of models on under-represented facets.

Convergent Validity Findings

• Rank Divergence: When evaluating models on retrieved examples versus gold test sets, significant rank divergences were observed.
• Root Causes: The paper identified three drivers for ranking inversions:
  1. Rubric Shifts: Differences in format (e.g., multiple-choice vs. open-ended) or scoring metrics (e.g., win-rate vs. accuracy).
  2. Divergent Operationalization: Different benchmarks measuring the same nominal skill (e.g., "puzzle solving") actually test distinct sub-skills.
  3. Context Shifts: Moving from definitional recall to applied reasoning.
• Example: Models ranked highly on BIG-bench's "Hindu Knowledge" (multiple-choice) dropped to the bottom when evaluated on HellaSwag examples (prompt completion) regarding the same topic.

5. Significance

• Shift from "Score-Centric" to "Evidence-Centric": BenchBrowser moves the field away from treating benchmark scores as self-explanatory truths. Instead, it forces practitioners to inspect the underlying items, rubrics, and framing that generate those scores.
• Actionable Diagnostics: It provides a practical workflow for practitioners to identify why a benchmark might be failing their specific use-case (e.g., "This benchmark only tests Python, not Go") before deploying a model.
• Foundation for Future Benchmarks: By highlighting under-explored facets and conflicting operationalizations, the tool guides the construction of more robust, representative, and valid future benchmarks.
• Open Source: The tool and code are publicly available, promoting transparency in LLM evaluation.

In summary, BenchBrowser addresses the "black box" nature of benchmark validity by providing a scalable, retrieval-based interface that empowers practitioners to audit the evidence behind model rankings, thereby reducing the risk of deploying models that are competent only on narrow, unrepresentative test sets.