The science and practice of proportionality in AI risk evaluations

This paper explores how the EU AI Act's requirement for general-purpose AI providers to evaluate systemic risks can be balanced with innovation through the principle of proportionality, aiming to develop scientific methods that ensure meaningful risk assessments without imposing excessive burdens.

The Gist

Imagine you are the head of a massive construction company building a new type of super-brick that can build houses, bridges, and even entire cities. But there's a catch: if you get the recipe wrong, these bricks could also be used to build bombs or collapse buildings.

The European Union (EU) has stepped in and said, "You must test these bricks before you sell them to make sure they aren't dangerous." This is the AI Act.

However, the regulators face a tricky problem: How do you make sure the tests are good enough to keep everyone safe, without forcing the builders to spend so much time and money testing that they never finish building anything?

This paper is about finding the perfect balance. It uses a legal concept called "Proportionality." Think of proportionality like a Goldilocks principle: the testing shouldn't be too easy (useless), too hard (impossible), but just right.

Here is how the authors break this down using simple analogies:

1. The Three Rules of "Just Right"

The paper says a test is only "proportionate" if it passes three checks:

• Suitability (Does it work?): The test must actually tell you something useful.
  • Analogy: If you want to know if a car can stop in the rain, testing it on a sunny day is useless. The test must match the real-world danger.
• Necessity (Is it the least painful way?): If you can get the same safety information with a cheaper, easier test, you shouldn't force the expensive one.
  • Analogy: If you want to check if a door is locked, you don't need to hire a SWAT team to break it down. A simple "jiggle the handle" test is enough. Using a SWAT team would be "using a sledgehammer to crack a nut."
• Balancing (Is the cost worth the safety?): The trouble and cost of the test shouldn't be way bigger than the risk you are trying to stop.
  • Analogy: It doesn't make sense to spend $1 million to install a security system for a shed that only contains a single garden hose.

2. The "Pareto Frontier" (The Efficiency Curve)

The paper includes a graph that looks like a curve. Imagine this as a menu of tests.

• On one side, you have Low Cost / Low Safety Info (like a quick quiz).
• On the other side, you have High Cost / High Safety Info (like a full-blown simulation).

The "Pareto Frontier" is the best possible deals on that menu. It represents the tests where you get the maximum amount of safety information for the minimum amount of money. If a test is below this line, it's a bad deal (too expensive for the info it gives). If it's above the line, it's impossible (you can't get that much info that cheaply).

The goal of regulators is to pick a point on this line that matches the danger level of the specific AI model.

3. A Real-World Example: The "Hacker" Test

To explain this, the authors look at a specific risk: What if an AI helps a hacker find holes in computer code? They compare three ways to test this:

• Method A (The "Hinted" Test): The test gives the AI a hint like, "Look at line 50, there's a bug there."
  • Pros: Very cheap and fast.
  • Cons: Not very realistic. If the AI fails, it might just mean it didn't read the hint, not that it's safe.
  • Verdict: Good for a quick "screening" to see if the AI is totally useless.
• Method B (The "Realistic" Test): The AI has to find bugs in a realistic environment without hints, like a real hacker would.
  • Pros: Much more accurate.
  • Cons: Takes more computer power and time.
  • Verdict: Good if the first test was scary or inconclusive.
• Method C (The "Super-Simulation" Test): The AI is put in a massive, complex digital world with thousands of scenarios to find bugs.
  • Pros: Extremely rigorous and detailed.
  • Cons: Very expensive and slow.
  • Verdict: Only necessary if the AI is huge, widely used, and the first two tests suggest it might be dangerous.

The Strategy: The paper suggests an iterative approach. Start with the cheap, easy test (Method A). If the AI passes easily, great! Stop there. If it fails or looks suspicious, then move to the harder test (Method B). Only if it still looks dangerous do you spend the fortune on the super-test (Method C). This saves everyone time and money.

4. Why This Matters

Currently, regulators often feel like they are guessing. They might demand a "Super-Simulation" for a small, harmless AI, which kills innovation because the company goes bankrupt testing it. Or, they might accept a "Hinted" test for a dangerous AI, which leaves the public unsafe.

This paper argues that we need science to fix this. We need better tools to measure exactly how much "safety info" a test gives versus how much it costs.

The Bottom Line:
The goal isn't to stop AI progress. It's to make sure the safety checks are smart. We want to use the right tool for the job, ensuring that the burden on companies is fair, but the protection for society is solid. It's about moving from "guessing" to "measuring" so that we can build a safer future without breaking the bank.

Technical Summary

Based on the article "The science and practice of proportionality in AI risk evaluations" published in Science (February 2026), here is a detailed technical summary covering the problem, methodology, key contributions, results, and significance.

1. Problem Statement

The rapid advancement of Artificial Intelligence, particularly General-Purpose AI (GPAI) models, presents a regulatory dilemma: how to manage systemic risks effectively without stifling innovation or imposing excessive burdens on providers.

• Regulatory Context: The European Union's AI Act (effective August 2025 for GPAI providers) mandates risk evaluations for advanced models.
• The Core Challenge: Regulators lack scientifically grounded methodologies to determine when a specific evaluation is legally justifiable. Without clear criteria, there is a risk of either:
  • Insufficient Protection: Evaluations that fail to capture meaningful risk information.
  • Undue Burden: Evaluations that are overly intrusive, costly, or technically unnecessary (a "sledgehammer cracking a nut").
• The Gap: There is a disconnect between legal principles (specifically the EU principle of proportionality) and the practical science of AI evaluation. Regulators need a framework to calibrate evaluation requirements to their intended objectives.

2. Methodology: The Principle of Proportionality

The authors propose operationalizing the EU legal principle of proportionality into a three-step technical framework for AI risk evaluations. This framework moves beyond binary compliance to a nuanced assessment of Effectiveness (Informational Value) versus Burden.

The methodology is structured around three legal criteria:

A. Suitability (Effectiveness)

An evaluation is "suitable" if it provides meaningful information about a specific risk. The authors define four technical criteria for suitability:

1. Realistic: Must reflect real-world constraints (e.g., noisy inputs, human factors, incomplete information).
2. Sensitive: Must detect meaningful performance changes; it should not be saturated (too easy) or impossible (too hard to measure progress).
3. Specific: Must be tightly coupled to the specific risk pathway (e.g., evaluating capabilities relevant to a specific attack chain).
4. Rigorous: Must adhere to measurement science standards (validity, reliability, alignment with technical standards).

B. Necessity (Inter-evaluation Comparison)

An evaluation is "necessary" only if no less burdensome alternative exists that achieves equal or greater effectiveness.

• Burden Metrics: Includes Intrusiveness (interference with business/infrastructure) and Resources (development costs, compute, time, human effort).
• Pareto Frontier: The authors introduce the concept of the Pareto frontier in the burden-effectiveness space. An evaluation is necessary if it lies on this frontier (i.e., you cannot improve effectiveness without increasing burden, or reduce burden without losing effectiveness).
• Multidimensionality: Effectiveness is not a single metric; it requires a coherent set of metrics capturing different risk dimensions, accounting for measurement uncertainty.

C. Balancing (Intra-evaluation Comparison)

This step weighs the informational value against the burden to ensure no "manifest imbalance."

• Dynamic Risk Profiling: Evaluations must adapt to the model's life cycle, considering updated risk estimates and confidence levels.
• Relative Burden: Burden is weighed relative to the provider's capacity and the model's deployment scale.
• Iterative Approach: The authors propose a tiered methodology:
  1. Start with low-cost, minimally intrusive methods (e.g., static prompting).
  2. If results are inconclusive or indicate high risk, escalate to more demanding methods (e.g., realistic tasks with tool access).
  3. Continue until sufficient confidence in the risk estimate is achieved, aligning with the Precautionary Principle.

3. Key Contributions

• Operationalizing Proportionality: The paper translates abstract EU legal requirements into concrete, scientific criteria for AI risk evaluation.
• The "Suitability-Necessity-Balancing" Framework: A structured approach to evaluate whether a specific test is the right one for a specific model and risk.
• Case Study Application (Cyber Vulnerability Discovery): The authors apply their framework to a specific scenario: AI models identifying vulnerabilities in open-source code. They compare three evaluation methods:
  • HonestCyberEval: Low burden, high guidance (simplified), good for screening.
  • BountyBench: Medium burden, high realism, good for distinguishing failure modes.
  • CyberGym: High burden, high rigor/sensitivity, best for end-to-end realistic exploitation.
  • Finding: All three methods have points on the Pareto frontier; the "necessary" choice depends on the specific risk context and the provider's capacity.
• Research Agenda: The paper identifies three critical research challenges:
  1. Defining minimum informational value for suitability.
  2. Developing methods to identify "equally effective" evaluations to find less burdensome alternatives.
  3. Reducing resource demands/intrusiveness without sacrificing informational value.

4. Results and Findings

• Context-Dependence: There is no "one-size-fits-all" evaluation. The appropriateness of an evaluation depends on the model's specific risk profile, the deployment context, and the provider's capacity.
• Iterative Necessity: A rigid requirement for high-cost evaluations for all models is disproportionate. An iterative approach (light-touch first, escalating only if needed) is more legally and scientifically sound.
• Trade-off Management: The case study demonstrates that different evaluation methods occupy different positions on the burden-effectiveness curve. Regulators must select the specific point on the curve that matches the required confidence level for the specific risk.
• Scientific Judgment: Determining proportionality requires a "scientifically grounded normative judgment" rather than a simple checklist. It requires evidence-based case-by-case assessment.

5. Significance

• Regulatory Clarity: Provides the first scientific framework for applying the EU AI Act's proportionality principle, reducing regulatory uncertainty for providers and regulators.
• Innovation Protection: By preventing excessive burdens, the framework aims to preserve technical progress and competitiveness while maintaining safety.
• Evidence-Based Policy: Shifts AI regulation from normative guesswork to evidence-based decision-making, requiring the scientific community to document and compare the effectiveness and burden of evaluation methods.
• Global Relevance: While focused on the EU, the principles of proportionality and risk-based calibration offer a model for global AI governance, addressing the tension between safety and innovation.

In summary, the paper argues that effective AI regulation requires a scientific calibration of evaluation requirements. By treating evaluations as a resource allocation problem governed by the principles of suitability, necessity, and balancing, regulators can ensure that risk assessments are both meaningful and efficient.