Is your AI Model Accurate Enough? The Difficult Choices Behind Rigorous AI Development and the EU AI Act

1. Problem Statement

The paper addresses a critical gap in the implementation of the 2024 European Union AI Act, specifically regarding the requirement for high-risk AI systems to achieve an "appropriate level of accuracy."

• The Core Misconception: Technical and legal debates often treat "accuracy" as an objective, purely technical metric (e.g., a single percentage score).
• The Reality: The authors argue that evaluating AI performance is fundamentally dependent on context-dependent normative decisions. A high accuracy score (e.g., 99.8%) can be misleading if the underlying metrics, data, or thresholds do not account for specific risks (e.g., in a melanoma detection app where missing a cancer diagnosis is far more harmful than a false alarm).
• The Regulatory Challenge: The EU AI Act mandates "appropriate accuracy" but does not define specific metrics or numerical thresholds. This creates a risk that compliance will default to superficial, "learner-centric" evaluations (comparing models to each other) rather than "application-centric" evaluations (assessing safety and harm in real-world deployment). Without clarifying the techno-normative choices (decisions that blend technical implementation with value judgments), the Act's safety goals may not be met.

2. Methodology

The authors employ a legal-technical analysis to deconstruct the performance evaluation process.

• Case Study: The paper uses a high-risk AI-based melanoma detection system as a recurring illustrative example. This domain was chosen because it involves severe consequences for errors (false negatives vs. false positives), class imbalance (rare disease), and strict regulatory oversight (Medical Device Regulation).
• Framework: The analysis focuses on four specific techno-normative choices that shape how accuracy is defined, measured, and assessed. These choices are mapped against the requirements of the EU AI Act (specifically Article 15 and associated documentation obligations in Annex IV).
• Interdisciplinary Approach: The paper bridges Machine Learning (ML) engineering practices (metrics, sampling, thresholds) with Legal Theory (risk allocation, fundamental rights, regulatory compliance).

3. Key Contributions: The Four Techno-Normative Choices

The paper identifies and analyzes four critical decision points in performance evaluation, demonstrating how each embeds normative assumptions about risk and harm:

Choice 1: Selecting Metrics

• Technical Aspect: Choosing which statistical metric to use (e.g., Accuracy, Precision, Recall, F1-Score, AUROC).
• Normative Implication: Different metrics prioritize different types of errors.
  • Example: In melanoma detection, Accuracy treats false positives (unnecessary anxiety) and false negatives (missed cancer) equally. However, the Recall metric prioritizes minimizing false negatives (safety), while Precision prioritizes minimizing false positives (efficiency/trust).
• AI Act Relevance: Providers must justify their metric choices based on the specific risks of their system. The choice of metric effectively encodes a value judgment on which errors are "acceptable."

Choice 2: Balancing Multiple Metrics

• Technical Aspect: Deciding how to aggregate multiple metrics (e.g., using a weighted sum, an F-score, or reporting them separately).
• Normative Implication: Aggregation (like the F-score) hides the trade-offs between competing objectives. A single aggregated score can obscure whether a system is failing in a critical area (e.g., high precision but dangerously low recall).
• AI Act Relevance: The paper argues for disaggregation in technical documentation. Reporting metrics independently increases transparency regarding normative trade-offs, allowing auditors to see exactly how risks are balanced.

Choice 3: Measuring Metrics (Data & Uncertainty)

• Technical Aspect: Selecting the test set and the method for uncertainty estimation (e.g., stratified sampling, bootstrapping, conformal prediction).
• Normative Implication: The test set must be representative of the real-world deployment population.
  • Challenge: In melanoma detection, failing to stratify by intersectional groups (e.g., "Black females") can lead to performance scores that mask discrimination.
  • Uncertainty: How confidence intervals are calculated determines the reliability of the performance claim.
• AI Act Relevance: Article 10 (Data Governance) and Article 15 require that test data be representative. The choice of how to sample data determines who is protected by the system's accuracy claims.

Choice 4: Determining Acceptance Thresholds

• Technical Aspect: Setting the numerical cutoff for "passing" the evaluation (e.g., "Must exceed human performance by 5%").
• Normative Implication: This is the most explicit value judgment. It defines the tolerable level of harm.
  • Context: A threshold for an AI replacing a dermatologist must be higher than for an AI acting as a triage tool for general practitioners.
  • Uncertainty: Thresholds must account for measurement uncertainty (confidence intervals).
• AI Act Relevance: The "appropriate level of accuracy" is defined by this threshold. The paper argues that setting this threshold requires a deliberate assessment of the specific deployment context and the severity of potential harms.

4. Results and Findings

• Accuracy is Contextual: There is no universal "appropriate level of accuracy." It is entirely dependent on the intended purpose, the specific risks of the domain, and the normative choices made during evaluation.
• The "Black Box" of Evaluation: Current ML practices often treat these choices as purely technical optimizations, hiding the normative judgments (e.g., "we chose F1-score because it balances things") from regulators and auditors.
• Documentation Gap: While the AI Act requires documentation, current technical standards (like ISO) and harmonized standards (prEN 18229-2) focus on procedural compliance rather than guiding the substantive normative choices.
• Risk of Superficial Compliance: Without addressing these choices, providers may comply with the letter of the law (reporting a high accuracy score) while failing the spirit of the law (deploying a system that is unsafe for specific subgroups or risk scenarios).

5. Significance and Implications

• For Regulators and Auditors: The paper provides a framework for auditing AI systems. Auditors cannot simply check a score; they must scrutinize the four choices to understand the underlying risk assumptions.
• For Developers: Developers are urged to move beyond "model-centric" evaluation. They must explicitly document the normative reasoning behind their metric selection, balancing, sampling, and thresholding.
• For the EU AI Act: The paper suggests that the Act's open-ended formulation is necessary but insufficient without interdisciplinary guidance. It calls for:
  • Moving beyond "one-size-fits-all" benchmarks.
  • Developing institutional capacity to evaluate techno-normative choices.
  • Encouraging transparency in how trade-offs between safety, efficiency, and fairness are encoded into technical systems.

Conclusion: The paper concludes that rigorous AI development under the EU AI Act requires making the techno-normative dimensions of accuracy explicit. Only by acknowledging that performance evaluation is a series of value-laden choices can the "appropriate level of accuracy" be meaningfully defined and enforced to ensure safe and trustworthy AI.