🤖 machine learning

Benchmarked Yet Not Measured -- Generative AI Should be Evaluated Against Real-World Utility

This paper argues that the disconnect between generative AI's strong benchmark performance and its limited real-world utility stems from flawed evaluation practices, and proposes the SCU-GenEval framework to shift assessment toward measuring sustained improvements in human outcomes within specific deployment contexts.

Original authors: Ishani Mondal, Shweta Bhardwaj

Published 2026-05-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ishani Mondal, Shweta Bhardwaj

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Video Game Score" vs. Real Life

Imagine you are training a robot to drive a car. You test it in a perfect, empty video game simulation where the weather is always sunny, the roads are straight, and there are no other cars. The robot gets a perfect score of 100/100.

You are thrilled! You buy the robot and put it on a real highway. Crash.

This paper argues that Generative AI (like the chatbots and code writers we use today) is exactly like that robot. It gets perfect scores on standard tests (benchmarks), but when we actually use it in real life (schools, hospitals, law offices), it often fails to help people or even causes harm.

The authors looked at 28 real-world stories where AI looked great on paper but failed in practice. They found three main reasons why this happens:

1. The "Fake Proxy" Problem (Proxy Displacement)

The Analogy: Imagine a restaurant judge who only rates food based on how loudly the chef claps when the dish is served. The chef learns to clap very loudly and gets a 10/10 rating. But the food is actually burnt.
The Reality: AI is often graded on easy-to-measure things like "fluency" (how smooth the sentences sound) or "pass rate" (does the code run?). But in the real world, we care about hard-to-measure things like "is this medical advice actually safe?" or "did this student actually learn the concept?"

Example: A coding AI might write code that passes all the tests (high score) but contains a security hole that lets hackers in. The test didn't measure safety; it only measured if the code ran.

2. The "Snapshot" Problem (Temporal Collapse)

The Analogy: Imagine a student who uses a calculator to solve a math problem instantly. They get an A on the test. But if you take the calculator away a month later, they can't do basic math at all. The calculator helped them in the moment, but it didn't help them learn.
The Reality: Current AI tests are "snapshots." They ask, "Can the AI do this task right now?" They don't ask, "Does using this AI help the human get better over time, or does it make them lazy and forgetful?"

Example: In education, AI might help a student finish homework quickly, but the student might forget how to write or think critically on their own later.

3. The "Average" Problem (Distributional Concealment)

The Analogy: Imagine a doctor says, "This new medicine works great! The average patient feels better." But they don't tell you that it works perfectly for men but makes women sick. The "average" hides the fact that half the people are getting hurt.
The Reality: AI systems often look good when you look at the "average" score. But that average hides the fact that the AI fails miserably for specific groups of people (like minorities, beginners, or people in rural areas).

Example: A medical AI might work well for white patients but fail to detect sickness in Black patients, or a legal AI might be great for big law firms but give terrible advice to regular people.

The Solution: A New Way to Measure Success

The authors say we need to stop asking "How good is the AI's output?" and start asking "How much did the AI change the human's ability to achieve their goals?"

They call this "Utility." It's not about the AI's score; it's about the human's progress.

To measure this, they propose a new framework called SCU-GenEval. Think of it as a four-step recipe for testing AI before you let it loose on the world:

Who and What? (Stakeholder-Goal Mapping)
- Don't just say "The Developer." Say "The Junior Developer," "The Security Expert," and "The End User." What does success look like for each of them?
What Matters? (Construct-Indicator Specification)
- Don't just measure "speed." Measure "did they learn?" "Is the code secure?" "Did the patient get better?"
How Does it Change? (Mechanism Modeling)
- Predict the future. Will using this AI make the junior developer lazy? Will it make the doctor trust the machine too much?
Measure Over Time (Longitudinal Utility)
- Don't just test once. Test today, test next week, and test next month. Did the human get better, or did they get worse?

The Tools to Make It Happen

The authors know this sounds expensive and hard to do. So, they suggest three tools to make it practical:

The "Pre-Flight Checklist" (Structured Protocols): Before launching an AI, you must write down exactly what you are testing, who you are testing it on, and what you expect to happen. This stops you from changing the rules after you see the results.
The "Digital Twin" (User Simulators): Instead of waiting months to see how real humans react, use computer simulations of specific types of people (e.g., "a tired junior coder") to predict how they will perform over time.
The "Specialized Ruler" (Persona-Conditioned Metrics): Instead of using one ruler for everyone, use different rulers for different groups. Measure the "Junior Developer" separately from the "Senior Expert."

The Bottom Line

The paper concludes that a high benchmark score is not enough. Just because an AI wins a video game doesn't mean it's ready for the real world.

We need to shift our focus from "How smart is the machine?" to "How much did the machine help the human become more capable?" If we don't make this shift, we risk deploying AI systems that look impressive on paper but fail to help, or even harm, the people who need them most.

Technical Summary: Benchmarked Yet Not Measured - Generative AI Should Be Evaluated Against Real-World Utility

1. Problem Statement

The paper identifies a critical disconnect between the high performance of Generative AI (GenAI) systems on standard benchmarks and their failure to deliver reliable utility in real-world deployments. While benchmarks like ImageNet, GLUE, HumanEval, and VBench have driven model comparison and progress, they increasingly fail to predict success in high-stakes domains such as education, healthcare, software engineering, and law.

The authors analyze 28 documented deployment cases to demonstrate that systems achieving strong benchmark scores often fail to improve human capability, decision-making, or learning over time. Instead, they frequently introduce security vulnerabilities, miss critical clinical cases, or degrade long-term skills. The paper argues that current evaluation pipelines suffer from a fundamental lack of construct validity: they measure tractable proxy metrics (e.g., BLEU, pass@k, fluency) that are not valid measures of the complex, real-world constructs they are assumed to represent.

This "benchmark–utility gap" is driven by three recurring structural failures:

Proxy Displacement: Treating tractable metrics as valid measures of complex constructs without evidence of construct validity (e.g., equating fluency with correctness or cost reduction with patient need).
Temporal Collapse: Mistaking short-term, AI-assisted performance for durable capability, ignoring long-term effects like skill atrophy, over-reliance, and lack of knowledge transfer.
Distributional Concealment: Allowing aggregate scores to mask systematic failures across specific populations, under-represented groups, or unseen deployment settings.

These failures are reinforced by benchmark saturation, data contamination, and Goodhart-style optimization, where systems optimize the proxy rather than the underlying goal.

2. Methodology and Framework: SCU-GenEval

To address these gaps, the authors propose a paradigm shift from static, artifact-centered evaluation to Stakeholder-Centered Utility Evaluation (SCU-GenEval). This framework defines utility not as the quality of an output artifact, but as the change in a stakeholder's capability to achieve their goals over time due to interaction with an AI system.

2.1 Formalizing Utility

Utility ( $U$ ) is quantified as the counterfactual change in a stakeholder's capability vector ( $C$ ) between a state before interaction and a state after interaction within a specific deployment context ( $D$ ) and time horizon ( $t$ ):
$U(u, G, y, D, t) = C_{after}(u, G, y, D, t) - C_{before}(u, G, D, t)$
Where:

$u$ is the stakeholder.
$G$ is the goal.
$y$ is the AI system.
$C$ is a vector of measurable indicators (e.g., decision quality, retention, safety, collaboration).

This formulation ensures evaluation is user-outcome centered (R1), temporal (R2), and disaggregated (R3), directly addressing the three structural failures.

2.2 The Four-Stage Evaluation Process

SCU-GenEval operationalizes utility through four sequential, connected stages:

Stakeholder–Goal Mapping: Explicitly identifying affected users ( $u$ ) and their goals ( $G$ ) to prevent distributional concealment and ensure the evaluation targets the correct human outcomes.
Construct–Indicator Specification: Defining the real-world capability constructs and selecting valid indicators ( $I_k$ ) to measure them, serving as a defense against proxy displacement.
Mechanism Modeling: Identifying the causal mechanisms (e.g., skill atrophy, automation bias, expertise reversal) through which AI interaction changes capability over time, addressing temporal collapse.
Longitudinal Utility Measurement: Estimating $U$ $U$ using specific protocols:
- Condition Comparison: Comparing human-alone, AI-alone, and human+AI performance.
- Longitudinal Tracking: Measuring durability across time horizons.
- Subgroup Disaggregation: Reporting outcomes per $(u, G, D)$ tuple.
- Mechanism Verification: Testing if predicted mechanisms manifest in interaction traces.
- Auditability: Logging data for independent replay.

2.3 Supporting Instruments

To make this framework practical at scale, the authors propose three instruments:

Structured Deployment Data Collection Protocol (SDDCP): A pre-registered, machine-readable contract (analogous to CONSORT-AI) specifying settings, constructs, mechanisms, and disaggregation keys before deployment.
Context-Conditioned User Simulators: Simulators instantiated with specific personas and goals, calibrated against SDDCP ground truth, to estimate long-term utility effects (e.g., skill decay) without requiring months of fieldwork.
Persona–Goal-Conditioned Proxy Metrics (PGCPMs): Fast, cheap metrics that approximate utility by filtering evaluation sets to match specific stakeholder personas and validated constructs, calibrated against SDDCP data.

3. Key Contributions

The paper makes four primary contributions:

Systematic Analysis: An analysis of 28 deployment cases across education, healthcare, software engineering, and law, demonstrating that the benchmark–utility gap is systematic, cross-domain, and recurring, not anecdotal.
Structural Diagnosis: The identification of three structural causes of the gap: proxy displacement, temporal collapse, and distributional concealment.
SCU-GenEval Framework: A novel framework for measuring utility as changes in human capability over time, shifting the unit of evaluation from artifact quality to human outcome trajectories.
Domain-Specific Calls to Action: Concrete recommendations for researchers, industry, and policymakers to adopt utility measurement, including calls for disaggregated reporting, longitudinal studies, and pre-registered deployment protocols.

4. Results and Evidence

The paper does not present new experimental results from a single model but synthesizes evidence from 28 existing cases (detailed in Table 1) to validate its claims:

Software Engineering: Code generation systems with high pass@k scores introduced security vulnerabilities in ~40% of solutions; AI-assisted programming tools were linked to metacognitive harm and skill atrophy in junior developers.
Healthcare: The Epic Sepsis Model, despite strong internal validation, missed 67% of sepsis cases and falsely alerted 18% of patients in external validation. Other models showed bias against specific racial groups or failed to generalize across hospital sites.
Education: Systems improving short-term task completion often failed to improve long-term retention or transfer, and AI summaries were shown to potentially degrade the "effortful retrieval" necessary for learning.
Law: Legal AI tools produced fluent but fabricated citations and hallucinated case law, leading to sanctions, while risk assessment tools (e.g., COMPAS) provided opaque scores that could not be contested.

The application of the SCU-GenEval framework to a hypothetical "DevAssist" scenario (Figure 2) illustrates how the framework would reveal negative utility for junior engineers and security reviewers that standard benchmarks would obscure.

5. Significance and Claims

The paper argues that for systems with meaningful human consequences, benchmark performance is necessary but not sufficient evidence of utility. The significance of the work lies in its call for a measurement science centered on stakeholder-conditioned utility transparency rather than benchmark performance alone.

The authors position SCU-GenEval not as a replacement for benchmarks, but as a necessary complement for high-stakes deployment. They emphasize that:

Construct validity is the missing anchor in current evaluation; metrics must be validated against the deployment outcomes they claim to represent.
Utility is temporal and disaggregated: Evaluation must track how capability changes over time and across specific user subgroups, rather than relying on static, aggregate scores.
Practicality is achievable: Through the proposed instruments (SDDCP, Simulators, PGCPMs), rigorous utility evaluation can be made tractable and scalable, preventing the "cost" of deployment harms (e.g., patient injury, security breaches) from outweighing the cost of pre-deployment evaluation.

The paper concludes that progress in Generative AI must be evaluated by measurable improvements in human outcomes, urging the community to adopt reporting standards analogous to CONSORT and TRIPOD for socio-technical AI deployment.