🤖 AI

Industrializing Prediction-Powered Inference: The GLIDE Library for Reliable GenAI and Agentic Systems Evaluation

The paper introduces GLIDE, an open-source Python library that unifies various prediction-powered inference methods and samplers under a standardized API to enable reliable, unbiased evaluation of agentic systems with valid uncertainty estimates while significantly reducing annotation costs.

Original authors: Grégoire Martinon, Ibrahim Merad, Mohammed Raki

Published 2026-06-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Grégoire Martinon, Ibrahim Merad, Mohammed Raki

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the manager of a massive factory that produces thousands of unique, complex robots every day. Your job is to figure out how many of these robots are "safe" and how many are "dangerous."

The Problem: The Costly Inspector vs. The Fast Robot
To know for sure if a robot is safe, you need a highly skilled human expert to inspect it. This is like hiring a master mechanic to take apart every single robot. It's accurate, but it takes forever and costs a fortune. You can't afford to check every single one.

Alternatively, you have a fast, cheap robot inspector (an AI judge) that can look at a robot and guess if it's safe in a split second. It's incredibly cheap and fast, but it makes mistakes. It might think a dangerous robot is safe, or vice versa. If you only trust this fast robot, your final report will be wrong.

The Old Way: Pick One or the Other
Traditionally, companies had to choose:

Check everything with humans: Accurate, but you go bankrupt.
Check everything with the fast robot: Cheap, but your data is biased and unreliable.

The New Solution: GLIDE (The "Smart Mix" Library)
This paper introduces a tool called GLIDE. Think of GLIDE as a smart recipe book that teaches you how to mix the two inspectors together to get the best of both worlds.

Here is how it works, using simple analogies:

1. The "De-Biasing" Magic

GLIDE doesn't just ignore the fast robot's mistakes. Instead, it uses a small team of human experts to check a tiny sample of robots (say, 100 out of 10,000).

It compares what the human experts said vs. what the fast robot guessed for those 100 robots.
It calculates exactly how the fast robot is wrong (e.g., "It thinks robots are 10% safer than they really are").
It then takes the fast robot's guess for the other 9,900 robots and mathematically "corrects" that bias using the human data.

The result? You get an answer that is just as accurate as if you had hired humans to check all 10,000 robots, but you only paid for 100 human checks.

2. The "Smart Sampling" Strategies

GLIDE isn't just one method; it's a toolbox with different ways to pick which robots the humans should inspect. The paper describes four main strategies:

The Random Picker (Uniform): You close your eyes and pick 100 robots at random. Good if you know nothing else.
The Grouped Picker (Stratified): Imagine your robots come in five different colors (Red, Blue, Green, etc.), and you know the "Blue" ones are harder to judge. GLIDE ensures you pick a specific number of Blue, Red, and Green robots so no group is ignored. This gives you a sharper picture.
The "Gut Feeling" Picker (Active): Sometimes the fast robot says, "I'm not sure about this one!" GLIDE listens to that uncertainty. If the robot is confused, GLIDE sends a human expert to check that specific robot immediately. This focuses your expensive human time exactly where it's needed most.
The Budget Picker (Cost-Optimal): If checking a "Red" robot costs $10 and a "Blue" robot costs $1, GLIDE figures out the perfect mix to stay within your budget while getting the most accurate answer.

3. The "Confidence Interval" (The Safety Net)

One of the most important things GLIDE does is tell you how sure it is.

If you just use the fast robot, you might get a number like "52% are safe," but you have no idea if that's right.
GLIDE gives you a range, like "We are 95% sure the real number is between 50% and 54%."
Crucially, the paper proves that even if the fast robot is terrible, this safety net never breaks. If the robot is bad, the range just gets wider (like saying "It's between 10% and 90%"), but it will always include the true answer. It never gives you a false sense of security.

4. Real-World Test: The "R-Judge" Case Study

The authors tested GLIDE on a real dataset of 568 conversations between users and AI agents.

They used a real AI (Claude) as the "fast robot" and human experts as the "inspectors."
The AI was biased (it thought things were safer than they actually were).
Using GLIDE with only 100 human checks (instead of checking all 568), they were able to produce a result that was statistically equivalent to checking 157 human reviews.
They saved about 30% of the human effort while keeping the accuracy high.

Summary

GLIDE is a software library that helps companies evaluate AI systems without breaking the bank. It combines a few human experts with a massive army of AI guesses, mathematically correcting the AI's mistakes. It tells you exactly how much you can trust the result and shows you how to spend your money (or time) most efficiently.

The paper's main takeaway is simple: Better AI judges don't replace human experts; they multiply the value of the human experts you already have.

Technical Summary: Industrializing Prediction-Powered Inference with GLIDE

Problem Statement
Reliable evaluation of agentic systems (e.g., clinical decision support, financial advisory) faces a fundamental trade-off. Full human annotation provides unbiased, reliable ground truth but is prohibitively slow and expensive, especially for multi-step trajectories. Conversely, using Large Language Models (LLMs) as judges is fast and cheap but introduces significant bias, particularly in knowledge-intensive or domain-specific tasks. Existing literature on Prediction-Powered Inference (PPI)—a framework that combines a small set of human labels with a large set of proxy predictions to produce unbiased estimates with valid confidence intervals—has matured rapidly. However, these methods (including PPI++, Stratified PPI, Predict-Then-Debias, and Active Statistical Inference) remain scattered across academic papers with heterogeneous notation and partial implementations. Practitioners lack a unified tool to apply these methods to agentic evaluation, where cost asymmetry is extreme, natural stratification exists, and valid uncertainty quantification is critical for deployment.

Methodology: The GLIDE Framework
The authors introduce GLIDE (Generated Label Inference and Debiasing Engine), an open-source Python library designed to unify state-of-the-art PPI estimators and samplers under a scipy-style API specialized for mean estimation. The framework decomposes evaluation into three sequential steps:

Sampling: GLIDE implements four distinct sampling strategies to select a subset of trajectories for human annotation from a larger pool of proxy-labeled data:
- UniformSampler: Standard random sampling (baseline).
- StratifiedSampler: Allocates samples across user-defined strata (e.g., query type, tools used) using proportional or Neyman allocation to handle heterogeneous proxy quality.
- ActiveSampler: Samples based on per-sample proxy uncertainty (e.g., verbalized confidence), concentrating human effort where the proxy is least reliable.
- CostOptimalSampler: Optimizes sampling probabilities based on known proxy-to-annotation cost ratios, minimizing total evaluation cost.
Annotation: Domain experts provide ground-truth labels for the selected subset. GLIDE treats this as an external, task-specific step, requiring only machine-readable output.
Estimation: The library combines the human labels and proxy predictions to produce a debiased point estimate and a confidence interval. It implements several estimators tailored to different statistical regimes:
- PPI++ & Stratified PPI++: Utilize power tuning to minimize asymptotic variance, guaranteeing performance no worse than the classical labeled-only estimator.
- Predict-Then-Debias (PTD) & Stratified PTD: Provide bootstrap-based confidence intervals, specifically designed for small-sample regimes where the Central Limit Theorem (CLT) approximation degrades.
- Active Statistical Inference (ASI): Uses Inverse Propensity Weighting (IPW) to debias samples drawn via active sampling.
- Classical Baselines: Include standard sample means (with and without stratification) to serve as references for effective sample size calculations.

A decision tree guides users in selecting the appropriate sampler and estimator based on the availability of cost estimates, proxy uncertainty signals, natural strata, and the number of available human labels (specifically, whether $n \ge 50$ for CLT-based methods).

Key Contributions

Unified Library: GLIDE is the first library to integrate a comprehensive suite of PPI estimators (PPI++, Stratified PPI, PTD, ASI) and samplers (uniform, stratified, active, cost-optimal) under a single, consistent API.
Reproducible Validation Suite: The package includes a Monte Carlo validation suite that empirically verifies three properties across all estimators: (i) empirical coverage matches the nominal level across varying proxy qualities, (ii) intervals are no wider than the labeled-only baseline, and (iii) effective sample size grows with proxy/ground-truth correlation.
Agentic Evaluation Case Study: The authors demonstrate the library's utility on the R-Judge benchmark (568 agentic trajectories). They show that GLIDE achieves equivalent precision to classical sampling with substantially fewer human annotations. Specifically, combining 100 human labels with proxy signals yielded effective sample sizes of ~143–157 (a 1.4x–1.5x gain) compared to using 100 human labels alone, while maintaining valid coverage despite a ~13% bias in the LLM-as-judge proxy.

Results
Empirical validation confirms that PPI methods in GLIDE provide unconditional validity regarding the proxy: even with a biased or weakly correlated proxy, the resulting confidence intervals remain valid (though potentially wider). The "Effective Sample Size" ( $n_{eff}$ ) serves as the primary metric of return on investment, quantifying how many fully human-labeled samples a PPI estimator is statistically equivalent to. In the R-Judge case study, stratified and active sampling protocols reduced confidence interval widths by 16%–20% compared to uniform sampling with the same human budget. The study highlights that while active sampling is competitive, stratification on semantically meaningful axes (like application domains) can yield comparable gains with zero engineering cost.

Significance and Claims
The paper positions GLIDE as a practical bridge between theoretical PPI advances and industrial application. Its primary claim is that investing in a higher-quality LLM-as-judge translates directly into a larger effective annotation budget. The authors emphasize that better proxies are not a substitute for human annotation but a multiplier on it. By providing a unified, validated, and easy-to-use interface, GLIDE enables practitioners to move beyond the binary choice of "expensive human-only" or "biased proxy-only" evaluation, facilitating the deployment of auditable, statistically rigorous evaluation pipelines for consequential AI systems. The library explicitly limits its scope to mean estimation (covering accuracy, toxicity rates, etc.), deferring broader M-estimation and quantile estimation to the foundational ppi_py library.