🤖 AI

EuroExec: Frontier Language Models Fall Short of Expert Judgment on European Executive Decision Tasks

This paper introduces EuroExec, a benchmark of 413 real-world European executive tasks evaluated by human experts, revealing that even the strongest frontier large language models fall significantly short of professional human standards, solving only 56.9% of tasks and being consistently outperformed by expert-written references.

Original authors: Pau Arnal, Khaled Denfir, Danylo Smahliuk, Amrut Avhad, Marcus A. Castro

Published 2026-08-06

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Pau Arnal, Khaled Denfir, Danylo Smahliuk, Amrut Avhad, Marcus A. Castro

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a new assistant to help run a massive, complex company. You don't just need someone who can recite facts from a textbook; you need a strategist who can look at a messy, real-world problem, understand the local rules, and write a brilliant plan that actually works. This is the world of "Large Language Models" (LLMs)—super-smart computer programs trained on vast amounts of text that can write stories, answer questions, and solve puzzles. For a while, we've been testing these AI assistants with simple, multiple-choice quizzes, like asking them to pick the right answer from a list of four. But in the real world, problems rarely come with a list of choices. They are open-ended, messy, and require deep judgment. The big question on everyone's mind is: Can these AI "geniuses" actually handle the high-stakes, open-ended decisions that real executives make every day, or are they just really good at taking tests?

A team of researchers from Sovrano AI decided to find out by creating a brand-new, ultra-difficult challenge called EuroExec. Instead of a simple quiz, they built a massive exam consisting of 413 complex, long-form questions based on real business scenarios in Europe. They didn't just ask the AI to guess; they hired 47 real human experts—people with decades of experience in finance, marketing, business, and product management—to write the questions and grade the answers. These experts created a strict checklist of requirements for every answer, acting like a rigorous boss checking a junior employee's work. They then asked six of the world's most advanced AI models to solve these problems and compared the AI's performance against the human experts.

The results were a bit of a reality check. Even the smartest AI models in the world struggled to keep up with the professionals. The very best AI model managed to "solve" only about 56.9% of the tasks, meaning it failed to meet the professional standard on nearly half the questions. In contrast, the human experts, when asked to write their own ideal answers, solved the problems at a near-perfect rate of 92.4%. When the researchers asked the human experts to rank the answers blindly, they preferred the human-written responses over the AI's answers 74% of the time.

The study also looked at how the AI failed. While the models were decent at writing well-structured text and communicating clearly, they fell flat when it came to actual reasoning and logic. They often missed the specific local rules of European markets or failed to create realistic, actionable plans. Interestingly, the researchers tried using other AI programs to grade the answers automatically, hoping to save time and money. However, these "AI judges" were unreliable; they tended to overestimate the performance of the best models and missed the subtle nuances that human experts caught.

In the end, the paper suggests that while AI is getting incredibly good at generating text, it is not yet ready to replace human experts in high-level decision-making. The "Solve Rate" for the top AI barely cracked 50% on the easiest questions, and even the most capable models were far from the professional standard required for executive work. The researchers conclude that for these kinds of complex, real-world tasks, human judgment is still the gold standard, and we cannot yet rely on automatic measurements or AI judges to tell us if a computer is truly "thinking" like an expert.

Technical Summary: EuroExec – Frontier Language Models Fall Short of Expert Judgment on European Executive Decision Tasks

1. Problem Statement

The evaluation of generative AI has traditionally relied on closed-ended tasks with deterministic metrics (e.g., multiple-choice, code execution, or translation with gold-standard references). However, these benchmarks fail to capture the nature of real-world applications where Large Language Models (LLMs) are increasingly deployed: open-ended, long-form text generation requiring professional judgment, such as executive decision-making. In these scenarios, there is no single "correct" answer, but rather a professional standard that human experts can recognize.

Current evaluation methods face two primary challenges:

Data Contamination: Models may have been trained on public benchmarks with definitive answers, inflating performance metrics through regurgitation rather than genuine reasoning.
Evaluation Fidelity: Automatic metrics (e.g., BLEU, ROUGE) and AI judges (LLMs evaluating LLMs) often fail to replicate the nuance, subjectivity, and consistency of human expert judgment, particularly in complex, domain-specific tasks.

This paper addresses the gap by introducing EuroExec, a benchmark designed to evaluate frontier LLMs on open-ended European executive decision tasks, and rigorously comparing human expert evaluations against automatic metrics.

2. Methodology

2.1 The EuroExec Benchmark

The authors constructed a dataset of 413 open-ended, long-form questions authored by 47 vetted domain experts with certified professional experience in Europe. The questions are divided into four domains:

Finance (74 questions)
Marketing (85 questions)
Business (113 questions)
Product (141 questions)

Question Formulation:
Each item is a self-contained, goal-oriented scenario (approx. 200 words) describing a real business situation with specific friction points (e.g., regulatory constraints, labor costs, market fragmentation). Crucially, each question is paired with a checklist of 5–10 specific criteria (ground truth) that a high-quality response must satisfy.

Validation Pipeline:
To ensure the questions challenge current models and are not trivially solvable by automated systems, a two-stage validation pipeline was employed:

Automated QA: An LLM (Claude Sonnet 4.6) rejected items based on criteria such as hint leakage, checklist relevance, and safety.
Difficulty Gate: Two different LLMs (Claude Haiku 4.5 as solver, Gemini Flash 3.5 as grader) attempted to solve the questions against the checklist. Questions were iteratively rewritten until the automatic fulfillment rate dropped below a 60% threshold, ensuring the tasks required genuine reasoning rather than pattern matching.

2.2 Model Selection and Response Generation

Six frontier LLMs were evaluated via their provider APIs using default decoding settings, without additional context or tools:

Fable 5 (Anthropic)
Claude Opus 4.8 (Anthropic)
GPT-5.5 (OpenAI)
Gemini 3.1 Pro (Google)
GLM-5.2 (Zhipu AI)
Mistral Large (Mistral AI)

2.3 Human Evaluation Protocol

The core of the study is a massive human evaluation effort involving >4,000 expert hours. Responses were evaluated by two independent domain-specific experts per question using three instruments:

Rubric Score: A 5-point Likert scale across five attributes: Domain (factual accuracy), Localization (European market context), Reasoning (logic), Communication (structure for executives), and Actionability (concrete plans).
Checklist Fulfillment: A binary/partial assessment of whether the response met the specific criteria defined in the question's checklist.
Preference Ranking: An explicit ranking of all six model responses for a given question.

Aggregate Metric: The authors introduced the "Solve Rate" (SR). A response is considered "solved" if it achieves a mean rubric score $\ge$ 3.0 AND a checklist fulfillment rate $\ge$ 60%.

2.4 Comparative Analysis

The study also evaluated:

Human Baseline: For a subset of 33 questions, experts wrote ideal answers, which were evaluated blindly as a seventh "model."
Automatic Metrics: ROUGE-Lsum and BLEU scores were computed against expert-written answers.
AI Judges: An independent LLM (DeepSeek v4-Pro) was used to evaluate the full dataset using the same three instruments to test the reliability of "LLM-as-a-Judge."

3. Key Results

3.1 Performance Gap

The results demonstrate a significant performance gap between frontier models and human experts:

Human Experts: Achieved a 92.4% Solve Rate and a 94.9% checklist fulfillment rate.
Best Frontier Model (Fable 5): Achieved only a 56.9% Solve Rate and 61.9% checklist fulfillment.
Preference Rankings: In blind preference rankings, human-written answers were preferred over every model response in 74.24% of cases.

Even the strongest models barely surpassed a 50% Solve Rate on the easiest subset of questions, indicating they are far from the professional standards required for executive decision-making.

3.2 Dimensional Analysis

Reasoning vs. Communication: Models generally performed better on Communication and Localization (text structure and local customs) but showed the weakest performance in Reasoning.
Domain Correlation: Performance was highly correlated across domains, with the exception of Finance, where GPT-5.5 underperformed relative to its standing in other domains, suggesting limitations in numerical reasoning.
Statistical Significance: Pairwise t-tests confirmed that the ranking of models is statistically significant ( $p < 0.01$ ) with as few as 100 samples, and highly significant ( $p \approx 1.26 \times 10^{-6}$ ) with the full 413 items.

3.3 Evaluation Consistency

Human Consistency: Inter-annotator agreement was high, particularly for checklist items. While subjective metrics (rubric, preference) correlated well with each other, the checklist provided a more objective signal that was consistent across different evaluators.
AI Judges vs. Humans: While the AI judge (DeepSeek v4-Pro) showed a strong correlation with human rankings, it failed to replicate the diversity and nuance of human evaluation. The AI judge tended to over-estimate top models and under-estimate weaker ones, producing "crisp" evaluations that lacked the heterogeneity of human opinion.
Automatic Metrics: Traditional metrics (BLEU, ROUGE) showed weak correlation with human rubric scores and failed to predict model rankings accurately.

4. Key Contributions

EuroExec Benchmark: A novel, human-authored benchmark of 413 complex, open-ended European executive tasks, designed to test professional judgment rather than knowledge retrieval.
Rigorous Human Evaluation: A large-scale study utilizing >4,000 expert hours to evaluate models across multiple dimensions, establishing a new standard for evaluating open-ended generation.
Empirical Evidence of Limitations: Quantitative proof that even the most capable frontier LLMs fall short of professional standards in executive decision-making, solving fewer than 60% of tasks compared to >90% for humans.
Evaluation Methodology Critique: A demonstration that automatic metrics and AI judges are insufficient substitutes for human evaluation in tasks with subjective ground truths, as they fail to capture the necessary nuance and diversity of expert judgment.

5. Significance and Claims

The paper claims that for open-ended, professional-grade tasks, human evaluation remains unmatched by current automatic methods. The authors argue that the field must move beyond closed-ended benchmarks that are susceptible to data contamination and rely on rigorous, human-centric evaluation to truly assess the capabilities of generative models in real-world scenarios.

The study concludes that while frontier models are improving, they are not yet ready to replace human experts in high-stakes executive decision-making. Furthermore, the reliance on AI judges or deterministic metrics to evaluate such tasks leads to misleading conclusions about model capabilities. The authors emphasize that the "Solve Rate" metric, derived from human judgment, provides a more honest and reliable measure of progress in this domain.

Limitations Acknowledged:

The study is resource-intensive, limiting the number of expert-written ideal answers (only 33 out of 413).
Reproducibility is challenging due to the cost and coordination required for human evaluation.
The analysis focuses on self-contained, factual scenarios and does not yet address tasks involving low-quality or uncertain data (e.g., healthcare).
Only one LLM was tested as an AI judge, though the authors believe the findings are representative of the class of AI judges.