💬 NLP

Qworld: Question-Specific Evaluation Criteria for LLMs

Qworld introduces a novel evaluation framework that generates question-specific criteria through a recursive expansion tree, enabling more granular and context-aware assessment of large language models on open-ended tasks compared to traditional static rubrics.

Original authors: Shanghua Gao, Yuchang Su, Pengwei Sui, Curtis Ginder, Marinka Zitnik

Published 2026-03-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Shanghua Gao, Yuchang Su, Pengwei Sui, Curtis Ginder, Marinka Zitnik

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a teacher grading essays.

The Old Way (Current LLM Evaluation):
Right now, when we test AI models, we often give them a single, rigid checklist that applies to every question. It's like giving a student a rubric that says, "Always use big words, always have three paragraphs, and always end with a moral lesson."

If a student is asked to write a poem, this checklist is useless. If they are asked to solve a math problem, it's confusing. If they are asked to give medical advice, it's dangerous. The current methods try to force a "one-size-fits-all" rulebook onto questions that are as different as a recipe, a legal contract, and a joke. This leads to scores that look good on paper but miss the real point: Did the AI actually answer this specific question well?

The New Way (Qworld):
The paper introduces Qworld (Question-Specific World). Think of Qworld not as a teacher, but as a master architect who builds a custom house for every single question asked.

Here is how it works, using a simple analogy:

1. The "One Question, One World" Concept

Imagine every question you ask an AI is a unique planet.

Question A: "How do I fix a leaky faucet?" (A planet of plumbing, tools, and water pressure).
Question B: "How do I comfort a grieving friend?" (A planet of empathy, tone, and emotional safety).

The old methods try to use the same map for both planets. Qworld says, "No. Let's build a custom map for each planet."

2. The Recursive Expansion Tree (The Architect's Blueprint)

How does Qworld build these custom maps? It uses a process called a Recursive Expansion Tree.

Imagine you are exploring a cave (the question).

Step 1: Scenarios (The Rooms): You don't just walk in; you look for different rooms. Maybe there's a "Room for Beginners," a "Room for Experts," and a "Room for Emergencies." Qworld breaks the question down into these different contexts.
Step 2: Perspectives (The Flashlights): In each room, you shine a flashlight from different angles. One light checks for Safety (Is this dangerous?). Another checks for Clarity (Can I understand this?). Another checks for Empathy (Is this kind?).
Step 3: Criteria (The Treasure Chests): Finally, you look for specific treasures. "Did the answer mention turning off the water main?" (A specific safety check). "Did the answer avoid medical jargon?" (A specific clarity check).

Qworld doesn't just guess these; it recursively expands, asking "What else?" over and over again, digging deeper until it finds every possible angle that makes a good answer for that specific question.

3. Why This Matters (The "Aha!" Moment)

The paper tested this on medical questions (HealthBench) and hard reasoning puzzles (Humanity's Last Exam).

The Result:

The Old Checklists missed subtle but critical things. For a medical question, they might check if the AI gave the right medicine, but they might miss that the AI failed to warn the user about a dangerous side effect for their specific situation.
Qworld found these hidden traps. It generated criteria that said, "Wait, this user is elderly; the answer must warn about dizziness," or "This user is in a hot climate; the answer must mention hydration."

The Analogy of the "Hidden Dimensions":
Think of the old evaluation as a black-and-white photo. It sees the shape of the answer, but it's flat.
Qworld turns the photo into a 3D hologram. It reveals dimensions that were invisible before:

Equity: Is the advice affordable for everyone?
Long-term Impact: Will this solution work in 10 years?
Error Handling: Did the AI admit what it doesn't know?

4. The Outcome

When the researchers used Qworld to grade 11 top AI models, the rankings changed.

Some models that looked "smart" on standard tests dropped because they were too rigid or missed safety nuances.
Some models that looked "average" rose to the top because they were better at adapting to the specific "world" of the question.

Summary

Qworld is a tool that stops treating every question like a multiple-choice test. Instead, it treats every question like a unique story that needs its own set of rules.

Old Way: "Here is a generic ruler. Measure everything with it."
Qworld: "Here is a custom-made laser scanner that understands the shape, texture, and purpose of this specific object before it measures it."

By building a unique "world" for every question, Qworld helps us see which AI models are truly intelligent and adaptable, and which ones are just memorizing a script.

1. Problem Statement

Evaluating Large Language Models (LLMs) on open-ended questions is a significant challenge because response quality is highly context-dependent.

Limitations of Current Methods: Standard metrics (e.g., accuracy, BLEU) fail to capture nuanced, context-specific requirements. Existing evaluation frameworks rely on static, task-level rubrics (e.g., a single set of criteria for an entire dataset) or single-pass generation of checklists.
The Gap: These approaches assume all questions within a task share the same evaluation needs. However, a medical diagnosis question requires different checks (safety, uncertainty) than a scientific explanation, even if both fall under a "medical" task label. Manual curation of question-level criteria is expensive and often misses subtle, implicit dimensions.
Goal: The authors aim to generate question-specific evaluation criteria that dynamically adapt to the intent, context, and implied constraints of each individual question, rather than applying a fixed rubric.

2. Methodology: Qworld

The authors propose Qworld (One-Question-One-World), a framework that generates tailored evaluation criteria using a Recursive Expansion Tree (RET) algorithm.

Core Concept: The "World" of a Question

Qworld treats each question as having a unique "world" defined by three hierarchical layers:

Scenarios: Contextual variations or implicit constraints (e.g., target audience, stakes, background knowledge).
Perspectives: Evaluation axes relevant to the scenario (e.g., factual correctness, safety, empathy, practicality).
Criteria: Fine-grained, binary, verifiable statements derived from the perspectives.

The Recursive Expansion Tree (RET) Algorithm

Unlike single-pass prompting or linear Chain-of-Thought (CoT), Qworld constructs a tree structure to explore the evaluation space:

Level 1 (Scenarios): The input question is decomposed into distinct scenarios ( $U_1$ ).
Level 2 (Perspectives): Each scenario is expanded into specific evaluation perspectives ( $U_2$ ).
Level 3 (Criteria): Each perspective is instantiated into concrete, measurable criteria ( $U_3$ ).

Expansion Operators:
The tree grows using two complementary operators implemented via LLMs:

Hierarchical Expansion ( $R_h$ ): Decomposes a node at level $\ell$ into child nodes at level $\ell+1$ (e.g., breaking a scenario into perspectives).
Horizontal Expansion ( $R_w$ ): Identifies missing aspects at the current level and generates sibling nodes to improve coverage (e.g., adding overlooked perspectives).

The process alternates between Coverage Expansion (repeated horizontal expansion for $w_\ell$ rounds) and Hierarchical Decomposition until the criteria level is reached. This ensures both breadth (covering diverse dimensions) and depth (fine-grained details).

Scoring Mechanism

Once criteria are generated, an evaluator (e.g., LLM-as-a-Judge) assesses a candidate answer $A$ against the criteria set $C_i$ for question $Q_i$ .

Each criterion $c$ has an importance weight $\alpha_c$ .
The score is binary: $\alpha_c$ if satisfied, $0$ otherwise.
The final score is the normalized sum of criterion-level scores.

3. Key Contributions

Question-Specific Evaluation Framework: Introduces a method to generate unique evaluation rubrics for every question, moving away from static, dataset-level criteria.
Recursive Expansion Tree (RET): Proposes a novel algorithmic structure that combines hierarchical decomposition and horizontal expansion to systematically explore the "evaluation space" implied by a question.
High-Quality Criteria Generation: Demonstrates that Qworld can generate criteria that match expert coverage while surfacing novel, high-value dimensions that experts and other automated methods miss.
Granular Model Benchmarking: Shows that using Qworld criteria reveals capability differences in LLMs (e.g., in equity, long-term impact, error handling) that are collapsed by coarse, standard benchmarks.

4. Experimental Results

The authors evaluated Qworld on HealthBench (medical domain) and Humanity's Last Exam (HLE) (abstract reasoning).

Criteria Quality (vs. State-of-the-Art)

Compared against methods like TICK, RocketEval, OpenRubrics, and EvalAgent:

Coverage: Qworld achieved 0.89 (covering 89% of expert-authored criteria), outperforming the next best (EvalAgent at 0.83).
Uniqueness: Qworld achieved 0.79, meaning 79% of its generated criteria were not present in expert references, indicating it discovers novel evaluation dimensions.
Human Expert Ratings: Experts rated Qworld criteria significantly higher in Insight (0.83 vs. ~0.40 for baselines) and Granularity (0.85), confirming the criteria are less generic and more actionable.

Impact on LLM Benchmarking

When applied to 11 frontier LLMs (including GPT-5, Claude Sonnet 4.5, Qwen3, etc.):

Ranking Shifts: Qworld altered model rankings. For example, on HealthBench, Qwen3-30B jumped from 6th to 2nd place, while Grok-4.1-Fast dropped. This was because Qwen3 excelled in dimensions like "Clarity," "Empathy," and "Health Equity," which coarse rubrics ignored.
Reduced Score Saturation: Qworld scores were ~20% lower than official benchmark scores, indicating that question-specific criteria introduce stricter, more realistic checks (e.g., missing caveats, incomplete risk management) that prevent models from achieving artificially high scores.
Dimensional Insights: The method revealed specific strengths and weaknesses. For instance, GPT-5 showed strength in "Safety & Risk Management," while Gemini 3 Flash excelled in "Visual & Geometric Reasoning" on HLE.

5. Significance and Conclusion

Adaptive Evaluation: Qworld shifts the paradigm from "one-size-fits-all" rubrics to dynamic, question-aware evaluation, acknowledging that the definition of a "good answer" changes based on context.
Discovery of Latent Dimensions: By recursively expanding scenarios and perspectives, Qworld uncovers implicit requirements (e.g., sustainability, equity, specific safety warnings) that standard benchmarks overlook.
Scalability: The method automates the creation of high-quality, expert-level criteria at scale, reducing the cost and effort of manual curation while maintaining or exceeding human quality.
Future Direction: The paper suggests that future LLM evaluation should move beyond scalar accuracy scores toward structured, multi-dimensional assessments that reflect the complexity of real-world open-ended tasks.

In summary, Qworld provides a robust, automated framework for generating fine-grained, context-aware evaluation criteria, significantly improving the fidelity and discriminative power of LLM benchmarks.