HLE-Verified: A Systematic Verification and Structured Revision of Humanity's Last Exam

1. Problem Statement

Humanity's Last Exam (HLE) is a widely adopted benchmark designed to evaluate frontier Large Language Models (LLMs) on challenging, multi-domain questions. However, community audits and independent analyses have revealed that the dataset contains a non-trivial number of noisy items. These flaws include:

• Ambiguous or underspecified problem statements.
• Incorrect reference answers or answer keys.
• Inconsistencies between the provided rationale (solution) and the final answer.
• Reliance on contested facts or unstated assumptions.

These annotation errors introduce systematic bias, distorting cross-model comparisons and leading to inaccurate measurements of model capabilities (e.g., penalizing models for correct reasoning that contradicts an erroneous answer key). The paper argues that high-difficulty benchmarks require rigorous post-deployment verification and structured revision protocols, which have been largely absent in current practices.

2. Methodology: The HLE-Verified Framework

The authors introduce HLE-Verified, a rigorously validated and partially revised version of HLE. The construction follows a two-stage validation-and-repair workflow involving a transparent, component-wise verification protocol.

A. Component Decomposition

Each item is decomposed into three distinct components for granular analysis:

1. Problem: The statement and any associated images.
2. Answer: The final reference solution.
3. Rationale: The reference solution/reasoning chain (used as an auxiliary consistency signal).

B. Stage I: Binary Validation (Gold Subset Construction)

• Goal: Identify items that are correct as originally released.
• Process:
  • Expert Screening: Independent domain experts review problem, answer, and rationale.
  • Model-Assisted Checks: Frontier multimodal solvers attempt the problems (pass@8 sampling) to detect human-model disagreement.
  • Adjudication: Internal experts synthesize evidence to make conservative inclusion decisions.
• Outcome: 668 items verified as "Gold" (no modification needed).

C. Stage II: Systematic Revision (Revised Subset Construction)

• Goal: Repair items identified as flawed but fixable, preserving the original evaluation intent.
• Process:
  • Dual Independent Repairs: Two expert teams propose corrections (Problem Fix → Solution Fix → Answer Fix).
  • Model-Assisted Auditing: Models generate additional repair candidates and consistency checks.
  • Final Adjudication: Experts select a canonical version ensuring correctness without altering the core capability being tested.
• Outcome: 1,143 items systematically revised and re-certified.

D. Uncertain Subset

• Definition: Items where validity cannot be conclusively determined due to ambiguous assumptions, lack of authoritative references, or unresolved expert disagreement.
• Outcome: 689 items released with explicit "uncertainty source" labels and "required expertise" tags to facilitate future community refinement.

E. Error Taxonomy

The authors developed a fine-grained taxonomy of 19 defect categories across the three components:

• Problem-level (5 types): Semantic errors, knowledge errors, missing info, theoretical invalidity, format errors.
• Rationale-level (10 types): Circular reasoning, empirical soundness violations, step inconsistencies, missing prerequisites, etc.
• Answer-level (4 types): Incorrect answers, incomplete answers, ambiguous answers, format errors.

3. Key Contributions

1. HLE-Verified Dataset: A unified, certified benchmark comprising 668 gold items, 1,143 revised items, and 689 documented uncertain items.
2. Verification Protocol: A systematic, two-stage framework for post-release benchmark auditing that treats problem statements, answers, and rationales as separate entities.
3. Error Taxonomy & Metadata: A structured schema for recording defect types, revision traces, and epistemic uncertainty, enabling transparent auditing.
4. Empirical Analysis: A comprehensive benchmarking of 8 state-of-the-art models (including GPT-5.2, Claude-Opus4.6, Qwen3.5, etc.) comparing performance on raw HLE vs. HLE-Verified.

4. Experimental Results

The authors evaluated 8 frontier models on both the original HLE and HLE-Verified.

A. Accuracy Improvements

• Overall Gain: Models achieved an average absolute accuracy gain of 7–10 percentage points on the full HLE-Verified set compared to raw HLE.
• Revised Subset Gain: On the subset of items that were corrected (the "Revised Subset"), the accuracy gain was dramatic, ranging from 30% to 40 percentage points.
  • Example: GPT-5.2 improved from 33.35% (Raw) to 52.48% (Revised Subset).
  • Example: DeepSeek-V3.2 improved from 24.90% (Raw) to 47.45% (Revised Subset).
• Interpretation: A significant portion of "failures" on the original HLE were due to benchmark errors, not model incompetence.

B. Calibration and Confidence

• Calibration Error: Calibration error decreased significantly after verification (e.g., GPT-5.2 dropped from 63 to 28 on the revised subset), indicating that flawed items previously induced overconfident but incorrect responses.
• Confidence as a Diagnostic: The study found a strong correlation between model confidence and item quality. On items with problem statement errors, models expressed significantly lower confidence on the raw version. After repair, confidence increased by +1.83 to +11.08 points, suggesting that model confidence can serve as a diagnostic signal for noisy or ambiguous benchmark items.

C. Domain-Specific Findings

• Physics and Biology/Medicine: Showed the largest accuracy jumps, indicating these domains had the highest concentration of answer-key and problem-statement errors.
• Defect Patterns:
  • Answer-level: Dominated by "Incorrect Answer" (deterministic errors).
  • Rationale-level: Dominated by structural issues (missing steps, format semantic errors) rather than pure logical contradictions.
  • Problem-level: Varied by domain (e.g., Format errors in CS/Math, Knowledge errors in Biology).

5. Significance and Impact

• Reliability of Evaluation: HLE-Verified demonstrates that benchmark noise is not uniform; it systematically biases results. Correcting these errors provides a more faithful measurement of true model reasoning capabilities.
• Methodological Shift: The paper advocates for a shift from treating benchmarks as static datasets to viewing them as living infrastructure requiring continuous verification, structured auditing, and transparent revision.
• Community Engagement: By releasing the "Uncertain Subset" with explicit tags for required expertise, the authors provide a roadmap for the community to further refine the benchmark.
• Scientific Rigor: The work highlights that without component-wise verification, scientific claims regarding model reasoning depth and generalization may be based on artifacts rather than genuine capability.

In summary, HLE-Verified transforms a noisy, high-stakes benchmark into a reliable, auditable standard, revealing that current frontier models are significantly more capable than raw HLE scores suggest, provided the evaluation data is free from annotation artifacts.