🤖 AI

MINCE: Shrinking LLM Evaluation Datasets via Few-Model Monte Carlo Calibration

The paper introduces MINCE, a Monte Carlo-based method that efficiently shrinks LLM evaluation datasets by determining minimum subset sizes from a small calibration pool to bound accuracy drift, achieving significant speedups and lower drift compared to existing techniques without requiring learned prediction layers.

Original authors: Devleena Das, Rajeev Patwari, Vikram Kumar Bukka, Nithin Kumar Guggilla, Elliott Delaye, Ashish Sirasao

Published 2026-06-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Devleena Das, Rajeev Patwari, Vikram Kumar Bukka, Nithin Kumar Guggilla, Elliott Delaye, Ashish Sirasao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef who has developed a new recipe. Before you serve it to the public, you need to taste it to make sure it's good. Now, imagine you have to test hundreds of different versions of this recipe (some with less salt, some with different spices, some cooked on a tiny stove instead of a giant industrial oven).

If you taste every single dish from every single version, it would take you weeks. You'd be so tired you couldn't cook anything new.

This is the problem computer scientists face with Large Language Models (LLMs). They create thousands of slightly different versions of AI models. To check if they work well, they run them through massive "tests" (benchmarks) that contain thousands of questions. Running these tests on small, energy-efficient chips (like those in phones or laptops) can take tens of hours per model.

The paper introduces a solution called MINCE. Here is how it works, explained simply:

The Core Idea: "How Many, Not Which"

Most previous methods tried to be smart detectives. They asked: "Which specific 100 questions are the most important? Let's pick those and ignore the rest." To do this, they needed a massive library of past test results (a "calibration pool") and complex math to figure out the perfect mix.

MINCE changes the question. It asks: "How many questions do we actually need to get a reliable score?"

It assumes that if you have enough questions, it doesn't really matter which specific ones you pick. A random handful of questions will give you almost the same score as the whole test, as long as the handful is big enough.

The Recipe: How MINCE Works

The authors used a method called Monte Carlo Simulation. Think of this like a "virtual taste test."

The Small Group: They took results from just 7 different AI models (the "calibration models") that had already taken the full, long tests.
The Simulation: They ran a computer simulation that randomly picked different numbers of questions (e.g., 100 questions, then 200, then 300...) and checked how much the score changed compared to the full test.
Finding the Sweet Spot: They looked for the point where adding more questions stopped making a big difference.
- Analogy: Imagine filling a bucket with water. The first cup fills it up a lot. The second cup adds a bit. By the tenth cup, the water level barely rises. MINCE finds the exact moment where the "extra water" isn't worth the effort.
The Result: They found a "magic number" (a subset size). For example, instead of testing 14,000 questions (MMLU), they only needed 1,500. Instead of 1,300 questions (GSM8K), they only needed 400.

The Payoff: Speed and Accuracy

Once they found this "magic number," they just picked that many questions at random. They didn't need a super-computer to figure out which ones were "special."

Here is what they achieved:

Huge Time Savings: They reduced the test size by 54% to 89%.
Speed: On standard computer chips (GPUs), the tests ran 2.7 to 8 times faster. On small, edge chips (NPUs), they ran 1.7 to 2 times faster.
Accuracy: The scores didn't change much. The "drift" (the difference between the short test and the long test) was tiny—less than 2.6 percentage points.

Why It's Better Than Old Methods

The paper compares MINCE to a popular method called tinyBenchmarks.

tinyBenchmarks is like a master chef who needs to taste 395 different dishes to figure out which 100 questions to keep. It's very precise but requires a huge amount of data to start.
MINCE is like a smart estimator who only needs to taste 7 dishes to know exactly how many questions to ask. It works even if you don't have a massive history of past tests.

The Bottom Line

MINCE is a practical tool for engineers. It says, "You don't need to run the whole marathon to know if you're in good shape; you just need to run a specific distance that we calculated using a few practice runs."

It allows companies to test their AI models much faster and cheaper, especially on the small chips found in everyday devices, without needing complex AI systems to help them pick the questions. It works reliably even if you only have a small group of models to test against.

Technical Summary: MINCE – Shrinking LLM Evaluation Datasets via Few-Model Monte Carlo Calibration

Problem Statement

Rigorous evaluation of Large Language Models (LLMs) across numerous variants (e.g., quantized, fine-tuned, or architecture-specific) necessitates running large-scale benchmarks repeatedly. This process creates a significant bottleneck, particularly on edge hardware like NPUs where inference throughput is orders of magnitude lower than on datacenter GPUs. Full evaluations can take tens of hours per model, hindering rapid iteration.

Existing methods for reducing evaluation costs through benchmark subsetting—such as those based on Item Response Theory (IRT), genetic algorithms, sparse optimization, or surrogate models—often rely on two impractical prerequisites:

Large Calibration Pools: Methods like tinyBenchmarks require hundreds or thousands of pre-evaluated models to fit statistical models or select items.
Learned Components: Many approaches require learned prediction layers or specialized item selectors that must be trained on specific data distributions.

For new benchmarks or scenarios where large calibration pools do not exist, assembling such data is impractical. The paper poses the question: Can benchmark subsets be determined reliably from a small number of calibration models without learned item selection or prediction layers?

Methodology: MINCE

The authors propose MINCE (Monte Carlo Informed N-sizing for Compact Evaluation), a method that reframes benchmark subsetting from "which items to select" to "how many items to select." The core hypothesis is that if a subset contains enough items to keep scores close to the full-benchmark scores, the specific identity of those items matters less.

The MINCE pipeline operates in two stages:

1. Monte Carlo Sizing (Determining $n^*$ )

Given per-item evaluation logs from a small set of calibration models ( $K$ models), MINCE uses Monte Carlo simulation to determine the minimum subset size ( $n^*$ ) required to bound accuracy drift.

Drift Definition: Accuracy drift $\delta$ is the absolute difference between the subset accuracy and the full-benchmark accuracy for a given model.
Simulation Process: The algorithm sweeps candidate subset sizes ( $n_t$ ). For each size, it draws $R=10,000$ random subsets without replacement.
Worst-Case Estimation: For each subset, it calculates the drift for every (model, metric) pair and retains the maximum drift. It then computes the 95th percentile ( $P_{95}$ ) of these worst-case drifts across all random draws.
Stopping Criterion: The algorithm identifies the point of diminishing returns. $n^*$ is defined as the first size where the marginal gain in drift reduction ( $\Delta(n_t) = P_{95}(n_{t-1}) - P_{95}(n_t)$ ) drops below a threshold $\tau$ . In the main experiments, $\tau$ is set to 1 percentage point (pp).

2. Subset Construction

Once $n^*$ is determined, a subset of that size is constructed.

Sampling Strategy: The paper demonstrates that at the calculated $n^*$ , uniform random sampling performs comparably to more complex strategies like stratified sampling or k-means clustering on sentence embeddings.
Output: A "frozen" subset file is generated, which is then reused for all future evaluations of that benchmark, requiring no further learning or optimization.

Key Contributions

Efficient Calibration with Minimal Data: MINCE requires only 7 calibration models (spanning 3B–35B parameters and 4 architecture families) to determine optimal subset sizes, drastically reducing the data requirement compared to prior work.
Significant Benchmark Reduction: The method reduces benchmark sizes by 54% (IFEVAL), 89% (MMLU), and 70% (GSM8K).
Controlled Accuracy Drift: The reduced subsets bound accuracy drift to a maximum of 2.62 pp on BF16 models and maintain a mean drift of 0.77–3.59 pp on held-out quantized NPU models.
Ranking Preservation: The method preserves model rankings with high fidelity (Spearman $\rho \geq 0.96$ ).
Robustness to Calibration Pool Size: The selected subset size ( $n^*$ ) remains stable even when the calibration pool is reduced from 7 to 3 models, offering practitioners flexibility.
No Learned Layers: Unlike competing methods, MINCE does not require a learned item selector, prediction layer, or large pre-evaluated pools.

Experimental Results

Performance on Calibration Models (BF16)

Evaluated on 7 BF16 models across IFEVAL, MMLU, and GSM8K:

Drift: The mean absolute drift was 0.44 pp (IFEVAL), 0.44 pp (MMLU), and 1.40 pp (GSM8K). The maximum drift observed was 2.62 pp.
Speedups: On GPU hardware, MINCE delivered median evaluation speedups of 2.7× (IFEVAL) to 8.1× (MMLU).

Generalization to Edge Hardware (INT4 NPU)

The method was validated on 3 held-out INT4 NPU models (Llama-3.1-8B-Inst, Phi-4-mini, Qwen3-8B) not seen during calibration:

Drift: Mean drift was 3.59 pp (IFEVAL), 0.77 pp (MMLU), and 0.77 pp (GSM8K). While IFEVAL showed higher drift for specific models (e.g., Phi-4-mini at 8.17 pp), MMLU and GSM8K remained highly stable.
Speedups: Median speedups on NPU hardware were 1.7× (IFEVAL) and 2.0× (MMLU), reaching 3.4× for specific models.

Comparison with State-of-the-Art

Compared to tinyBenchmarks (which uses 395 models for calibration):

Drift Reduction: MINCE achieved 12× lower mean drift on MMLU and 3.3× lower on GSM8K.
Calibration Efficiency: MINCE used 57× fewer calibration models (7 vs. 395).
Trade-off: While tinyBenchmarks minimizes subset size (100 items) at the cost of a large calibration pool and a prediction layer, MINCE accepts a slightly larger subset size in exchange for extreme calibration efficiency and applicability to any benchmark without pre-existing large-scale data.

Ablation Studies

Sampling Strategy: At the determined $n^*$ , uniform random, stratified, and k-means sampling yielded comparable drift, confirming that the sizing step is the critical factor, not the sampling complexity.
Calibration Pool Size: Reducing the calibration pool from 7 to 3 models resulted in stable $n^*$ values for MMLU and GSM8K, with only minor shifts for IFEVAL.

Significance and Claims

The paper positions MINCE as a scalable foundation for compact evaluation in an era of increasing model variants and diverse hardware targets. Its primary significance lies in:

Democratizing Benchmark Subsetting: By removing the need for large calibration pools or learned models, MINCE makes rigorous subset selection accessible for new benchmarks and specific deployment scenarios.
Practical Efficiency: It offers a direct, non-learned mechanism for practitioners to control calibration effort, balancing evaluation speed against accuracy drift.
Robustness: The method's stability across different sampling strategies and calibration pool compositions suggests it is a reliable heuristic for real-world deployment.

The authors acknowledge limitations, noting that the current calibration pool does not include frontier-scale (>100B) or non-transformer architectures, and that the 1 pp threshold is a heuristic. They also note that each new benchmark requires an initial calibration run with per-item logs, which remains a cost for benchmarks requiring human annotation. However, the paper concludes that MINCE effectively addresses the compounding time costs of full benchmark evaluation, particularly on edge hardware.