Evaluating Ill-Defined Tasks in Large Language Models

1. Problem Statement

The paper addresses a critical gap in the evaluation of Large Language Models (LLMs): the assessment of ill-defined tasks. Unlike well-defined tasks (e.g., math problems or code generation with a single correct output), ill-defined tasks have ambiguous goals, open-ended constraints, and subjective success criteria. Examples include Complex Instruction Following (CIF) and Natural Language to Mermaid Sequence Diagrams (NL2Mermaid).

The authors argue that current evaluation benchmarks and metrics fail to provide reliable or diagnostic signals for these tasks due to three primary failure sources:

1. Data Limitations: Benchmarks lack real-world coverage and constraint diversity, often leading to overfitting on specific benchmark formats rather than general capability.
2. Metric Collapse: Evaluation metrics often aggregate multiple diagnostic dimensions into a single score, conflating distinct failure modes (e.g., confusing a formatting error with a logical failure).
3. LLM-as-a-Judge (LLMaJ) Instability: The reliance on LLMs to judge other LLMs introduces significant variance, bias, and sensitivity to prompt phrasing, making scores unstable and non-reproducible.

2. Methodology

The authors employ a case study approach to analyze the structural failures of current evaluation practices. They examine two distinct ill-defined tasks:

A. Complex Instruction Following (CIF)

The authors analyze several popular benchmarks (IFEval, ComplexBench, FollowBench, MT-Bench, HELM, StructFlow) to identify their limitations:

• IFEval: Critiqued for relying on unnatural, rule-based constraints (e.g., "avoid the letter 'i'") that allow degenerate strategies (e.g., outputting fixed tokens) to achieve non-trivial scores without understanding semantics.
• ComplexBench & FollowBench: Critiqued for using hybrid evaluation (rules + LLMaJ) where the LLMaJ component introduces stochasticity (up to 2% score variance with different seeds) and translation bias (2-4% variance between English/Chinese versions).
• Sensitivity Analysis: The paper demonstrates that CIF scores are highly sensitive to surface-level prompt variations. For instance, reformulating IFEval prompts caused scores to drop by up to 61.8%, suggesting the metrics measure prompt robustness rather than true instruction-following capability.

B. Natural Language to Mermaid Sequence Diagrams (NL2Mermaid)

This case study focuses on generating sequence diagrams from text. The authors:

• Decomposed Evaluation: Instead of a single aggregate score, they evaluated outputs across four distinct criteria: Syntax, Logic, Completeness, and Error/Status Tracking.
• Multi-Judge Setup: They utilized two independent LLM judges (DeepSeek-V3 and GPT-OSS-120B) to mitigate single-judge bias.
• Intervention Testing: They applied in-context learning and prompt adjustments to specific models (Llama 3.1-8B and Granite 3.3-8B) to observe how changes affected specific criteria versus the aggregate score.

3. Key Contributions

1. Diagnosis of Evaluation Flaws: The paper systematically identifies that existing benchmarks conflate distinct failure modes, rendering aggregate scores "unstable, non-diagnostic, and difficult to act upon."
2. Demonstration of Metric Decomposition: Through the NL2Mermaid case study, the authors prove that decomposed evaluation reveals actionable insights that aggregate scores hide.
   • Example: Prompt adjustments improved the Logic score for Llama 3.1-8B from 67.71 to 80.58, but the overall aggregate score decreased. This indicated a trade-off (improved logic at the cost of completeness), a nuance lost in a single metric.
3. Exposure of LLMaJ Variance: The study quantifies the instability of LLM judges, showing that scores can vary significantly based on random seeds, model choice, and prompt phrasing, undermining the reliability of current benchmarks.
4. Proposed Design Principles: The authors propose a framework for robust evaluation of ill-defined tasks, moving away from "black box" scoring toward transparent, diagnostic pipelines.

4. Results and Findings

• Benchmark Overfitting: Models trained on datasets similar to benchmark test samples (e.g., Tulu3 on IFEval-like data) perform well on the benchmark but fail on novel constraint types, indicating a lack of generalizability.
• Degenerate Strategies: In IFEval, a trivial baseline model ignoring semantic content achieved a score of 17.55, nearly matching GPT-2 (19.72), proving that surface-form constraints can be satisfied without true instruction following.
• The "Aggregate Score" Trap: In the NL2Mermaid task, relying on a single combined score masked specific improvements. Decomposed metrics allowed researchers to pinpoint that a model's logic had improved even if its overall score dropped due to a trade-off in another dimension.
• Judge Inconsistency: The use of LLMaJ without strict calibration leads to scores that are not reproducible, making it difficult to attribute failures to the model's capabilities versus the judge's noise.

5. Significance and Recommendations

The paper argues that current evaluation practices for ill-defined tasks are insufficient for guiding model development. To address this, the authors propose a Diagnostic Evaluation Suite with the following principles:

1. Explicit Scoping: Benchmarks must clearly define which aspects of behavior are measured (e.g., constraint compliance vs. semantic adequacy) and exclude others to prevent metric conflation.
2. Realistic Variability: Data should include paraphrasing, reordering, and perturbations to reduce overfitting and expose brittle model behaviors.
3. Hybrid Evaluation Architecture:
   • Use Rule-Based Checks for decidable properties (syntax, formatting, length).
   • Use LLMaJ only for subjective dimensions (logic, tone, completeness), but strictly constrained to well-scoped criteria.
   • This separation isolates judge noise from genuine model errors.
4. Transparency and Calibration: Researchers must report inter-judge agreement, exact prompt templates, judge model versions, and hyperparameters (temperature, seed) to ensure reproducibility.

Conclusion: The paper concludes that while current benchmarks can rank models, they fail as diagnostic instruments. Future evaluation designs must prioritize robustness, interpretability, and failure-mode analysis to truly understand and improve LLM capabilities in complex, real-world scenarios.