💬 NLP

Building Open-Retrieval Conversational Question Answering Systems by Generating Synthetic Data and Decontextualizing User Questions

This paper proposes a pipeline for automatically generating synthetic, annotated open-retrieval conversational question answering (OR-CONVQA) datasets from plain text documents, which are then used to train efficient question rewriters that enable existing retrieval systems and large language models to handle dialog-aware, document-grounded responses.

Original authors: Christos Vlachos, Nikolaos Stylianou, Alexandra Fiotaki, Spiros Methenitis, Elisavet Palogiannidi, Themos Stafylakis, Ion Androutsopoulos

Published 2026-08-07

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Christos Vlachos, Nikolaos Stylianou, Alexandra Fiotaki, Spiros Methenitis, Elisavet Palogiannidi, Themos Stafylakis, Ion Androutsopoulos

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to find a specific fact in a massive, dusty library, but you are talking to a librarian who has never heard of the library's layout. If you ask, "How much is it?" the librarian might look confused because they don't know what "it" is. This is the core problem of Conversational Question Answering: humans speak in shortcuts, using words like "it," "that," or "he" that only make sense if you remember what we just talked about. To fix this, computers use a trick called Retrieval-Augmented Generation (RAG). Think of RAG as giving the librarian a magical magnifying glass that instantly finds the right page in the library before they answer. But here's the catch: the librarian needs to know how to ask the question clearly to the magnifying glass, and they need a huge pile of practice conversations to learn how to do it.

The problem is that for new topics—like a specific insurance company's rules or a new software manual—nobody has written down thousands of practice conversations. Hiring people to write them is slow, expensive, and boring. This paper tackles that exact headache. The authors propose a clever way to build a "training gym" for these AI librarians using only the raw documents the company already has. They don't just copy-paste sentences; they first break the documents down into tiny, crystal-clear "fact cards" (which they call propositions). Then, they use a super-smart AI to imagine what a human would ask about those cards, creating a fake but realistic conversation. They found that this method of turning messy documents into clean fact cards, and then generating synthetic conversations from them, makes the AI librarian much better at finding the right answers than if they had just used the raw, messy sentences directly.

The Story of the Synthetic Conversation Factory

So, how do you teach a robot to be a good conversationalist when you don't have a million real human chats to show it? The authors of this paper built a two-step assembly line that turns boring documents into exciting, realistic practice conversations.

Step 1: The Fact Card Factory
Imagine you have a thick instruction manual for a new video game. It's full of long, complicated sentences like, "If you press the red button while holding the controller, the character will jump, but only if the battery is low." That's a nightmare for a search engine. The authors' pipeline first takes these documents and asks a powerful AI (they used a model called Claude 3.5) to break them down. The AI acts like a meticulous editor, chopping long sentences into short, standalone "fact cards." It removes confusing references like "it" or "that" and ensures every card is a complete, self-contained thought.
Crucially, the AI is told to only make cards for things a human would actually care about asking. It ignores boring bits like "Contact us at 555-0199" unless someone might actually ask about it. From their collection of documents, they generated over 11,000 fact cards for their private company data and over 14,000 for public government documents. These cards become the "truth" the AI librarian will eventually search through.

Step 2: The Conversation Simulator
Now that they have a pile of fact cards, the pipeline asks the AI to imagine a conversation. It picks a small group of these cards and says, "Pretend a human is asking questions about these facts." The AI generates a back-and-forth chat. Here is the magic trick: for every question the "human" asks, the AI writes two versions.

The Real Human Version: This is how a person actually talks, full of shortcuts. "How much does it cost?" (referring to something mentioned earlier).
The Clear Version: This is the same question rewritten to make total sense on its own, like "How much does the insurance policy cost?"
The system also keeps track of exactly which fact cards were used to answer each question. This creates a perfect training dataset where the computer knows the messy question, the clear version, the answer, and the source.

Why "Fact Cards" Beat "Raw Sentences"

The authors didn't just stop at making up conversations; they tested a big hypothesis. They wanted to know: Is it better to train the AI using these clean "fact cards," or should they just chop the original documents into sentences and use those?

Think of it like searching for a needle in a haystack. If you use the raw sentences, the haystack is full of extra straw—irrelevant words, long explanations, and confusing context. If you use the fact cards, you've already pulled out the straw and left just the needles.
When they tested this, the results were clear. The conversations built from the clean fact cards led to much better search results. The AI could find the right information significantly faster and more accurately. In contrast, using the raw sentences was like trying to find a needle in a haystack that had been glued together; the search engine got confused by the noise. The authors found that while the raw sentences were okay for keeping the conversation flowing smoothly, they were terrible for helping the AI actually find the answer.

Teaching the Robot to Think for Itself

Once they had these thousands of synthetic conversations, they used them to train "lightweight" AI models. These are smaller, faster, and cheaper computers compared to the giant super-AIs used to generate the data.
They trained two types of helpers:

The Rewriter: A small model that learns to take a messy, context-heavy question ("How much is it?") and turn it into a clear one ("How much is the insurance?").
The Retriever: A small model that learns to look at the clear question and instantly find the right fact card in the library.

The results were impressive. These small, cheap models performed just as well as using the giant, expensive AI to do the rewriting every single time. In fact, in some tests, the small model trained on their synthetic data was even better than a model trained on massive, human-written datasets that are hard to get.

They also discovered a neat shortcut. They taught the rewriter to check if a question was already clear. If the user asked, "What is the price of the policy?" (which is already clear), the rewriter learned to say, "No rewrite needed," and skip the step. This saved time, cutting the processing time in half for many questions.

Does It Work in the Real World?

To make sure this wasn't just a method that worked on fake data, they tested their system on real-world datasets from government websites (like insurance and student aid). They threw away the human-written training data from these real datasets to prove their method could work from scratch.
The results held up. Their small models, trained only on the synthetic conversations they generated, were able to find the right answers in real-world documents much better than if they had just used the raw, messy questions. While using a giant AI to rewrite questions in real-time was slightly better, it was also much slower and more expensive. Their small, trained models offered a sweet spot: fast, cheap, and highly accurate.

The Bottom Line

The paper suggests that we don't need to wait for humans to write millions of perfect conversations to build smart AI assistants. Instead, we can take the documents we already have, turn them into clean, bite-sized facts, and let a powerful AI imagine the conversations. This synthetic data is good enough to train smaller, faster models that can handle real-world questions effectively.

However, the authors are careful to note that this isn't a magic wand. Their system still relies on a large, expensive AI to generate the initial training data. And while the small models are great at finding answers, they still need the big AI to actually write the final response to the user. But for the hard part of finding the right information in a sea of documents, this "synthetic factory" approach seems to be a game-changer, turning messy manuals into clear, searchable knowledge without needing a team of human annotators.

Technical Summary: Building Open-Retrieval Conversational Question Answering Systems by Generating Synthetic Data and Decontextualizing User Questions

Problem Statement

The paper addresses the challenge of Open-Retrieval Conversational Question Answering (OR-CONVQA), a task where a system must generate responses that are both aware of the dialog history and grounded in retrieved documents. While Retrieval-Augmented Generation (RAG) is effective for grounding Large Language Models (LLMs), OR-CONVQA introduces two specific hurdles:

Contextual Complexity: User questions often contain ellipsis or co-reference (e.g., "What about the price?"), requiring the system to resolve them using dialog history. Relying solely on the last query often yields suboptimal retrieval results.
Data Scarcity: High-quality, domain-specific training datasets with manual annotations are difficult and expensive to obtain. Existing synthetic data generation methods often rely on unrealistic assumptions, such as a one-to-one mapping between document sentences and user questions, or require additional manually annotated data for fine-tuning.

Methodology

The authors propose a two-step pipeline to generate realistic, annotated synthetic OR-CONVQA dialogs from domain-specific documents (e.g., product documentation) without requiring any pre-existing manual annotations.

Step 1: Proposition Generation
Instead of using raw document sentences, the pipeline prompts an LLM (specifically Claude 3.5 Sonnet) to extract propositions.

Constraints: Propositions are standalone, simple sentences that express prominent information from the documents.
Selection: The LLM is instructed to generate propositions only for information likely to be queried by a user, filtering out irrelevant or trivial content.
Structure: Documents are split into chunks of $n$ propositions (e.g., $n=30$ ), which may span multiple documents. This creates a retrieval pool of clean, information-dense units rather than raw text.

Step 2: Synthetic Dialog Generation
Using the sampled propositions, the same LLM generates dialogs through three sub-prompts:

Dialog Generation: Creates a user-system turn pair where the user question is initially generated as decontextualized (self-contained).
Contextualization: The LLM creates a contextualized version of the user question (e.g., adding pronouns based on history) to simulate real conversation.
Ground Truth Verification: The LLM identifies which propositions ground each question-answer pair and generates a token (accepted or not_accepted). Pairs marked as rejected are discarded. To ensure consistency, propositions generated in this step are replaced with their closest matches from Step 1 using BM25.

System Architecture
The generated synthetic data is used to train lightweight models for two components:

Question Rewriter: A light model (e.g., T5) fine-tuned to convert contextualized user questions into decontextualized queries.
Retriever: A lightweight Sentence Transformer (MiniLM) fine-tuned to retrieve relevant propositions from the repository.
Response Generator: A larger LLM (LLaMA-8B) is prompted with the retrieved propositions and the user query to generate the final response. The authors also propose a conditional rewriting mechanism where the rewriter predicts a no_rewrite token if the input question is already self-contained, reducing inference latency.

Key Contributions

Synthetic Data Pipeline: A novel pipeline that generates high-quality, annotated OR-CONVQA dialogs from domain documents without manual annotation, avoiding the one-to-one sentence-to-question assumption of prior work.
Propositions over Sentences: Demonstration that converting documents into "propositions" before dialog generation yields superior retrieval performance compared to using raw document sentences.
Efficient Fine-Tuning: Evidence that synthetic data can effectively fine-tune lightweight models (T5, MiniLM) for question rewriting and retrieval, achieving performance comparable to prompting large LLMs for rewriting.
Public Release: The authors release their source code and a synthetic OR-CONVQA dataset to facilitate future research.

Experimental Results

The authors evaluated their approach on proprietary documents (call center data) and public datasets (DOC2DIAL, MULTIDOC2DIAL).

Propositions vs. Sentences: While dialog coherence scores were similar, proposition-based dialogs significantly outperformed sentence-based dialogs in retrieval metrics (Mean Average Precision and Recall). This suggests propositions provide clearer, more prominent information for retrieval.
Rewriting and Retrieval:
- Fine-tuned lightweight rewriters (T5) achieved retrieval scores comparable to using "ground truth" decontextualized queries generated by the LLM.
- Fine-tuning a retriever (MiniLM) on synthetic data proved more effective than query rewriting alone, matching the performance of ground-truth decontextualized queries.
- Both approaches substantially outperformed using raw contextualized questions (with or without history concatenation).
Real-World Performance: On real-world test sets (DOC2DIAL/MULTIDOC2DIAL), the synthetic-data-trained T5 rewriter and MiniLM retriever significantly improved retrieval performance over baselines.
Response Generation: Using the fine-tuned retriever (MiniLM) with dialog history provided better response generation scores (SacreBLEU, METEOR, BERTScore) than using ground-truth decontextualized queries alone, likely because the history provided additional context to the Response-LLM.
Efficiency: The conditional rewriting mechanism reduced average generation time by approximately 50% (e.g., from 0.19s to 0.09s for proprietary dialogs) without sacrificing performance.

Significance and Claims

The paper claims that its pipeline offers a practical solution to the data scarcity problem in domain-specific OR-CONVQA. By leveraging the abundance of existing organizational documents, it enables the creation of realistic training data without manual effort. The authors emphasize that propositions serve as a superior intermediate representation for synthetic data generation, leading to better retrieval outcomes. Furthermore, the work demonstrates that lightweight models fine-tuned on this synthetic data can replace expensive, real-time LLM prompting for query rewriting and retrieval tasks, offering a cost-effective and efficient alternative for deploying conversational agents in new domains.

The authors remain modest regarding limitations, acknowledging that the pipeline relies on costly LLMs for data generation and that response generation still currently relies on prompting larger LLMs rather than fine-tuning lightweight generators, as preliminary experiments showed the latter underperformed.