Query Disambiguation via Answer-Free Context: Doubling Performance on Humanity's Last Exam

This paper demonstrates that rewriting user queries to reduce ambiguity using answer-free grounding context, rather than simply prepending that context, significantly boosts language model accuracy on challenging benchmarks like Humanity's Last Exam, with gains that cannot be replicated through inference-time prompting alone.

The Gist

Here is an explanation of the paper using simple language and creative analogies.

The Big Idea: Fixing the "Missing Context" Problem

Imagine you are trying to ask a very smart friend (an AI) a question. But you forget to tell them the most important part of the story.

The Scenario:
You ask: "What kind of lasers are these crystals used in?"
Your friend (the AI) looks at you blankly. They don't know which crystals you are talking about. They might guess "ruby," "diamond," or "glass." They are guessing based on what they read in their training books, not on the specific situation you are in.

The Paper's Solution:
The authors found a clever trick. Instead of just giving the AI the answer, they give the AI a clue sheet (called "Answer-Free Context") that explains the background but doesn't give away the answer. Then, they ask the AI to rewrite the question using those clues before trying to answer it.

It's like giving your friend a map and saying, "Okay, now that you know we are in a desert with sand dunes, rewrite your question to be more specific."

---

The Three-Step Magic Trick

The paper describes a process that happens in three distinct phases. Think of it like a detective solving a mystery:

1. The Clue Sheet (Answer-Free Context)

The AI is given a paragraph of text about the topic.

• The Catch: This text is carefully edited to remove the actual answer. It's like a detective being given a file on a suspect's habits, but the file has the suspect's name redacted.
• Example: The text says, "Zinc Sulfide is used in glowing clock hands and X-ray screens." It doesn't say, "Zinc Sulfide is used in infrared lasers."

2. The Rewrite (The "Aha!" Moment)

The AI reads the clue sheet and realizes, "Oh! The user is asking about Zinc Sulfide crystals!"
The AI then rewrites the user's vague question into a super-clear one:

• Old Question: "What kind of lasers are crystals used in?"
• New Question: "What part of the light spectrum do lasers using Zinc Sulfide crystals operate in?"

3. The Answer

Now, the AI (or a second AI) answers this new, clear question. Because the question is no longer vague, the answer is much more likely to be correct.

---

Why This is a Big Deal (The Surprising Results)

The researchers tested this on a very hard exam called "Humanity's Last Exam." Here is what they found:

1. Just giving the clues isn't enough.
If you just paste the clue sheet before the question (like handing the detective the file but not letting them read it first), the AI doesn't get much smarter. It's like handing someone a map but not letting them look at it before they start walking.

2. The "Rewrite" is the secret sauce.
The AI must take the time to rewrite the question first. When they did this, the AI's accuracy on the hard exam more than doubled (going from 14% correct to 37% correct).

3. You can't do it all at once.
The researchers tried to make the AI rewrite the question and answer it in one single thought process (like thinking and talking at the same time). It failed.

• Analogy: It's like trying to write a recipe and cook the meal simultaneously. You end up burning the dinner. The AI needs to stop, write the new question, and then answer it. These need to be two separate steps.

The "Why" Behind the Magic

Why does rewriting the question help so much?

Imagine the AI's brain is a giant library of ideas. When you ask a vague question, the AI wanders into the wrong aisle.

• The Clue Sheet: Tells the AI which aisle to look in.
• The Rewrite: Physically moves the AI's attention to the right shelf.

The paper measured this using "cosine similarity" (a fancy math way of measuring how close two things are). They found that after rewriting, the question was mathematically much closer to the correct answer in the AI's brain.

The Takeaway for Everyone

This paper teaches us that how we ask questions matters more than we think.

Even the smartest AI can get confused if the question is vague. But if we give the AI some background information and let it "think" about how to ask the question better, it becomes a genius.

In short: Don't just ask the AI for the answer. Give it the context, let it clarify the question, and then ask for the answer. It's the difference between a confused guess and a brilliant solution.

Technical Summary

Here is a detailed technical summary of the paper "Query Disambiguation via Answer-Free Context: Doubling Performance on Humanity's Last Exam."

1. Problem Statement

The paper addresses a critical gap in Large Language Model (LM) performance: the mismatch between a user's implicit mental model (background knowledge, context, and intent) and the explicit text provided in a query.

• The Issue: Users often formulate queries assuming the LM shares their background knowledge. When this context is missing, the query becomes ambiguous or underspecified. The LM then relies on training data patterns, potentially leading to factually correct answers under a wrong interpretation, which the user perceives as incorrect.
• The Limitation of Current RAG: Standard Retrieval-Augmented Generation (RAG) systems often fail to improve performance if the retrieved context does not contain the direct answer. Furthermore, simply prepending retrieved "Answer-Free Context" (AFC) to a query often fails to resolve ambiguity effectively, as the model may not know how to integrate the background information to refine the question.
• The Core Hypothesis: The paper posits that using AFC to rewrite and disambiguate the query before the answering phase yields significantly higher accuracy than simply providing the context alongside the original query.

2. Methodology

The authors propose a two-phase pipeline: Rewrite followed by Answer.

A. Data and Models

• Datasets: The study utilizes a diverse set of benchmarks, including Humanity's Last Exam (HLE) (specifically a FutureHouse-validated subset of chemistry/biology questions), SQuADv2, HotpotQA, and various generative benchmarks. Crucially, HLE and some generative benchmarks are post-dated relative to the models' training cutoffs to prevent data contamination.
• Models:
  • Rewriting Models: gpt-oss-20b, gpt-oss-120b, and Qwen3-235B.
  • Evaluation Models: A wide range of models (gpt-5, gpt-oss, Llama, Gemma, Qwen, Phi) with varying sizes and knowledge cutoffs.

B. The Pipeline

1. Answer-Free Context (AFC) Construction:
   • Original context containing the answer is processed by an LM to remove the specific answer and any "giveaway" information, leaving only relevant background knowledge.
   • This ensures the rewriting phase relies on disambiguation, not answer extraction.
2. Query Rewriting:
   • An LM is prompted with the Original Question, the Correct Answer (for validation only), and the AFC.
   • The LM rewrites the question to be self-contained, unambiguous, and grounded in the AFC, explicitly stating assumptions without revealing the answer.
   • Validation: An LM-judge filters rewritten questions to ensure semantic equivalence to the original and checks that the answer is not "given away" in the new question text.
3. Evaluation:
   • The evaluation models are tested in three configurations:
     1. Original Question (Orig_Q): Baseline.
     2. Original Question + AFC (Orig_Q+AFC): Standard RAG baseline (context prepended).
     3. Rewritten Question (Rewrite_Q): The disambiguated query without the AFC (the AFC was used only to generate the question).

C. Analysis Mechanisms

• Embedding Analysis: The authors use an embedding model (e5-mistral-7b-instruct) to measure the cosine similarity between the question and the context. They hypothesize that rewriting reduces the semantic distance between the query and the relevant latent space of the context.
• Task Separation Test: The authors test if rewriting can be done "in situ" (via Chain-of-Thought within the same prompt) versus a distinct preprocessing phase.

3. Key Contributions

1. Answer-Free Context (AFC) Disambiguation: Introduces a method where background information without the answer is used to rewrite queries, significantly boosting accuracy.
2. Separation of Concerns: Demonstrates that a distinct Rewrite-then-Answer paradigm is superior to a single-pass approach. The cognitive load of rewriting and answering simultaneously negates the benefits of disambiguation.
3. RAG Beyond Answer Retrieval: Proves that RAG systems provide value even when they fail to retrieve the direct answer; the retrieved background information is sufficient to improve query formulation.
4. Semantic Alignment Correlation: Establishes a strong positive correlation between the reduction in cosine distance (between query and context) and the increase in benchmark accuracy.

4. Key Results

• Significant Accuracy Gains:
  • On the Humanity's Last Exam (HLE) subset, rewriting using AFC improved gpt-5-mini accuracy from 0.14 to 0.37 (a >2.5x improvement).
  • Across all models and datasets, the average accuracy improvement of Rewrite_Q over Orig_Q was 13.03%.
• Superiority over Standard RAG:
  • The Rewrite_Q approach (question rewritten, no context in prompt) outperformed Orig_Q+AFC (original question with context prepended) by an average of 13.46%. This proves that the act of rewriting is the driver of performance, not just the presence of context.
• Task Separation Necessity:
  • When rewriting was attempted "in situ" (within the same prompt as the answer generation), the accuracy gains disappeared. This confirms that the rewriting phase requires distinct cognitive resources or context window dynamics that are lost when combined with the answering phase.
• Embedding Alignment:
  • 91% of data points showed that rewritten questions had higher cosine similarity to the context and higher accuracy than the original questions, confirming the "closer to the right frame of mind" hypothesis.
• Dataset Nuance:
  • Factual/Extractive QA: Rewriting alone provides massive gains.
  • Reasoning-Intensive QA (e.g., HLE, Generative Benchmarks): Rewriting alone helps, but the best performance is achieved when the model is given both the rewritten question and the raw context (Rewrite_Q + AFC). This suggests a hybrid strategy is optimal for complex reasoning.

5. Significance and Implications

• RAG Paradigm Shift: The paper argues that dynamic context construction should not be viewed merely as an "evidence collector" for direct answers. Instead, retrieved context should be treated as a collaborative partner in query formulation.
• Optimization Strategy: For systems where RAG retrieval is imperfect (i.e., it finds relevant background but not the answer), the optimal strategy is to use that background to refine the user's intent before passing it to the LLM.
• Cognitive Load Management: The findings suggest that LLMs have limited context attention. Splitting the complex task of "understanding context and refining the question" from "generating the answer" allows the model to perform both tasks more effectively.
• Future Directions: The authors suggest future work could explore multi-turn processes (search -> rewrite -> search again) and "Best-of-N" rewriting strategies to further align user intent with the document corpus.

In summary, this work demonstrates that query disambiguation via answer-free context is a powerful, under-explored lever for improving LLM performance, effectively doubling results on difficult benchmarks by separating the tasks of query refinement and answer generation.