💬 NLP

Multi-Faceted Self-Consistent Preference Alignment for Query Rewriting in Conversational Search

This paper introduces MSPA-CQR, a novel framework for conversational query rewriting that enhances performance by constructing multi-dimensional preference data and employing prefix-guided direct preference optimization to align rewriting with retrieval and response generation feedback.

Original authors: Zhiyu Cao, Peifeng Li, Qiaoming Zhu

Published 2026-04-09

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zhiyu Cao, Peifeng Li, Qiaoming Zhu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are talking to a very smart, but slightly literal, librarian. You are having a long conversation about a specific topic, like the history of a famous singer.

The Problem:
In the middle of your chat, you ask, "When was the song released?"

Because the librarian only hears that one sentence and doesn't remember the previous 10 minutes of conversation, they get confused. They don't know which song you mean. They might pull up a book about a completely different artist's song, or they might say, "I don't know what song you are talking about."

This is the core problem of Conversational Search. Computers are great at finding things, but they are terrible at remembering the context of a chat.

The Old Solution:
Previous methods tried to fix this by teaching the computer to "rewrite" your question into a full sentence before searching.

Your Question: "When was the song released?"
Old Rewrite: "When was the song Ruin by Cat Power released?"

This helped, but it was like a translator who only knows how to speak one dialect. They made the sentence clear for humans, but they didn't always make it perfect for the search engine. Sometimes the rewrite was too long, sometimes it missed a crucial keyword, and sometimes it didn't help the computer find the right answer.

The New Solution: MSPA-CQR (The "Three-Headed Coach")
The authors of this paper propose a new system called MSPA-CQR. Instead of just one rewrite, imagine the system has three different expert coaches working together to perfect your question before it goes to the search engine.

Here is how they work, using a Sports Team Analogy:

1. The "Grammar Coach" (Rewriting Preference)

Goal: Make sure the sentence makes sense on its own.
Analogy: This coach is like a strict editor. They say, "You can't just say 'the song.' You need to say 'Cat Power's song Ruin' so anyone reading it knows exactly what you mean, even if they weren't in the room."
Result: A clear, self-contained sentence.

2. The "Scout" (Retrieval Preference)

Goal: Give the search engine exactly the keywords it needs to find the file.
Analogy: This coach is like a scout looking for a specific player in a massive stadium. They say, "Don't give me a long story! Just give me the jersey number and the team name. Cut out all the fluff so the search engine doesn't get distracted."
Result: A short, punchy query packed with the most important keywords.

3. The "Answer Key" (Response Preference)

Goal: Make sure the search results actually lead to a good answer.
Analogy: This coach is like a teacher checking the answer key. They say, "If we search for this, will the document we find actually tell us the release date? If the search brings up a biography that doesn't mention the date, we need to tweak the question to force the search engine to find the specific paragraph we need."
Result: A query designed to pull up the specific document that contains the answer.

How They Work Together (The Magic)

In the past, researchers tried to pick just one of these coaches. But the paper shows that all three are needed.

The system generates many versions of your question. Then, it uses a clever "self-check" method (like a student grading their own homework) to see which version is best for the Grammar Coach, which is best for the Scout, and which is best for the Answer Key.

Finally, it combines the best parts of all three. It takes the clarity of the Grammar Coach, the keywords of the Scout, and the answer-focus of the Teacher, and merges them into one "Super Query."

Why This Matters

No Human Needed: Old methods required humans to write perfect examples, which is slow and expensive. This system teaches itself by simulating these three coaches.
Better Answers: Because the search engine gets a "Super Query," it finds the right documents much faster.
Works Everywhere: Whether you are asking about pop culture, history, or science, this system adapts to make sure the computer understands the intent of your question, not just the words.

In short: Instead of asking the computer to guess what you mean, this system acts like a team of experts that rephrases your question into the perfect form for the computer to understand, ensuring you get the right answer every time.

1. Problem Statement

In Conversational Search, user queries are often ambiguous or context-dependent (e.g., using pronouns like "it" or omitting entities). Conversational Query Rewriting (CQR) aims to decontextualize these queries into self-contained forms to improve passage retrieval.

Limitations of Existing Approaches:

Isolation: Early methods trained CQR models using human-labeled rewritten queries in isolation, ignoring feedback from the actual retrieval and response generation processes.
Readability vs. Retrieval: Models optimized for human readability often fail to retrieve relevant passages because they do not prioritize retrieval-oriented information.
Annotation Costs: Methods that do incorporate retrieval feedback often rely on expensive, labor-intensive human-annotated "gold passages" to construct training signals, limiting scalability.
Single-Dimension Alignment: Previous works typically aligned models only with retrieval preferences, neglecting the nuances of rewriting quality and response relevance.

2. Methodology: MSPA-CQR

The authors propose MSPA-CQR, a framework that aligns query rewriting with three distinct dimensions: Rewriting, Retrieval, and Response. The framework operates in two stages:

Stage 1: Multi-Faceted Preference Data Construction

Instead of relying on human annotations, the authors use Large Language Models (LLMs) to generate diverse candidate rewritten queries and score them using Self-Consistency.

Sampling: Given a dialogue context and current query, an LLM samples $K$ candidate rewritten queries ( $RQ_t$ ).
Feedback Generation: For each candidate:
- Retrieval: The top-100 passages are retrieved (using BM25).
- Response: An LLM generates a response based on the candidate query.
Self-Consistency Scoring: Three scores are calculated to measure the quality of each candidate relative to the others:
- Rewrite Score ($RW$): Measures semantic similarity between candidates using an NLI model. It includes a length penalty to ensure queries are comprehensive enough to stand alone.
- Retrieval Score ($RT$): Measures the intersection size of the retrieved passage sets. High intersection implies the query consistently retrieves relevant content.
- Response Score ($RP$): Measures semantic similarity between the generated responses using an NLI model.
Data Selection: For each dimension, the candidate with the highest score is selected as the Chosen sample, and the lowest as the Rejected sample. This creates a preference dataset $D$ containing pairs $(r_q^+, r_q^-)$ for each of the three dimensions.

Stage 2: Prefix-Guided Multi-Faceted Preference Optimization

The authors introduce Multi-Faceted Direct Preference Optimization (MDPO), an extension of Direct Preference Optimization (DPO).

Prefix Control: To distinguish between the three conflicting or complementary preferences, a specific prefix tag (e.g., [REWRITE], [RETRIEVAL], [RESPONSE]) is prepended to the instruction input.
Optimization Objective: The model is trained to maximize the probability of generating the "Chosen" query over the "Rejected" query, conditioned on the specific prefix. This allows a single model to learn distinct rewriting strategies for different objectives.
Inference Strategy: During inference, the system generates three rewritten queries (one for each preference), concatenates them, and uses this combined query for the final retrieval step.

3. Key Contributions

Multi-Faceted Alignment: First framework to simultaneously align CQR with rewriting, retrieval, and response preferences, recognizing that optimal queries for one dimension may differ from others.
Self-Consistent Preference Data: A novel, annotation-free method to construct high-quality preference data using LLM sampling and self-consistency scoring, eliminating the need for gold-standard passages.
Prefix-Guided MDPO: A technique to train a single model to handle multiple, potentially conflicting objectives by using prefix tags, enabling flexible control over the rewriting style.
Complementarity Analysis: Empirical evidence showing that the three dimensions are complementary; combining them yields significantly better retrieval results than any single dimension alone.

4. Experimental Results

The model was evaluated on TopiOCQA and QReCC datasets using both sparse (BM25) and dense (ANCE) retrieval systems.

Performance: MSPA-CQR outperformed all state-of-the-art baselines (including AdaCQR, RETPO, and CHIQ) across all metrics (MRR, NDCG@3, Recall@10/100).
- Example: On TopiOCQA (Dense), MSPA-CQR achieved an MRR of 41.4, surpassing the previous SOTA (AdaCQR) of 38.5.
Generalization: The model demonstrated strong zero-shot capabilities on out-of-distribution datasets (CAsT-19, CAsT-20, CAsT-21), outperforming baselines even without fine-tuning on those specific datasets.
Ablation Studies:
- Removing any single preference dimension (Rewrite, Retrieval, or Response) led to performance drops, confirming their complementary nature.
- Removing the Prefix Guidance (using vanilla DPO) significantly degraded performance, proving the necessity of distinguishing dimensions.
- Using stronger LLMs (e.g., GPT-4o) for the sampling stage improved final performance, highlighting the importance of high-quality preference data.

5. Significance and Conclusion

MSPA-CQR addresses a critical gap in conversational search by moving beyond isolated rewriting tasks to a holistic optimization process that considers the entire search pipeline (rewriting $\to$ retrieval $\to$ response).

Efficiency: It avoids the high cost of human annotation by leveraging self-consistency scoring.
Robustness: It effectively handles the trade-offs between different query rewriting goals (e.g., conciseness for retrieval vs. detail for response) by learning to adapt based on the specific preference tag.
Impact: The results suggest that future CQR systems should not just aim for "human-readable" queries but must be explicitly aligned with the downstream retrieval and generation tasks to maximize search efficacy.

The paper concludes that while the method introduces some latency due to multi-query generation, the significant gains in retrieval accuracy justify the cost, and future work could explore merging these queries to further optimize efficiency.