KCS: Diversify Multi-hop Question Generation with Knowledge Composition Sampling

1. Problem Statement

Multi-hop Question Answering (MHQA) requires models to integrate multiple pieces of information from a long context to answer complex questions. A critical bottleneck in this field is data sparsity, which leads language models to learn spurious patterns, compromising their robustness and generalization.

While Question Generation (QG) has been proposed to augment MHQA datasets, existing methods suffer from two main limitations:

1. Over-reliance on Simple Questions: Prior approaches focus on diversifying content planning (what to ask) and expression (how to ask) but often generate simple, single-hop questions.
2. Neglect of Knowledge Integration: Existing methods typically assume that the "knowledge composition" (the specific set of sentences required to answer the question) is pre-identified and fixed. They fail to explore different valid combinations of sentences within the context that could lead to the same answer but result in semantically distinct, complex multi-hop questions.

The core problem addressed is: How to generate diverse, high-quality multi-hop questions by sampling varied, logically coherent knowledge compositions from unstructured long contexts, rather than relying on fixed or pre-defined knowledge sets.

2. Methodology: The KCS Framework

The authors propose Knowledge Composition Sampling (KCS), a framework that treats the selection of knowledge pieces as a dynamic, probabilistic process. The framework consists of three main stages:

A. Knowledge Composition Selection (Sentence-Level Conditional Prediction)

Instead of treating sentence selection as a simple classification task, KCS models it as a sequence prediction task.

• Architecture: A hierarchical neural network comprising a Sentence Encoder (BERT-based) and a Sentence-level Sequence Model (Transformer-based Encoder-Decoder).
• Mechanism: The model predicts the next relevant sentence ($f_k$) conditioned on the answer ($a$), the long context ($D$), and previously selected sentences ($f_{0:k-1}$).
• Probabilistic Contrastive Loss ($L_{seq}$): To learn "potential knowledge coherence" and prevent the model from selecting irrelevant sentences, the authors introduce a contrastive loss.
  • It maximizes the Mutual Information (MI) between the latent representation of the current prediction and the true next sentence.
  • It minimizes the MI between the prediction and other sentences in the context (negative samples).
  • Formula: $L_{seq} = -E_c [\log \frac{MI(x_{f_k}, e_{k-1})}{\sum_{s \in D, s \neq f_k} MI(x_s, e_{k-1})}]$

B. Diversifying Knowledge Composition (Stochastic Decoding)

Standard greedy decoding (selecting the highest probability sentence) leads to a lack of diversity. To balance accuracy and diversity, KCS employs a Stochastic Decoding Strategy:

• Nucleus Sampling (Top-p): The model truncates the "unreliable tail" of the probability distribution (sentences with very low probability) and samples the next sentence from a "dynamic nucleus" containing the majority of the probability mass.
• Dynamic Nucleus: The cutoff point ($p$) is determined dynamically at each timestep to ensure the selected sentences are both high-quality and diverse.
• Result: This allows the generation of multiple distinct knowledge compositions ($C = \{c_1, ..., c_{N_q}\}$) for a single answer.

C. Multi-hop Question Generation

Once diverse knowledge compositions are sampled, a standard "vanilla" Multi-hop Question Generation (MHQG) model (fine-tuned MixQG-base) generates the final questions. This decouples the diversification of knowledge from the generation of text, allowing the framework to be compatible with various MHQG backbones.

3. Key Contributions

1. Novel Framework (KCS): Introduced a framework that diversifies multi-hop QG by sampling varied knowledge compositions from unstructured text, moving beyond fixed knowledge graphs or pre-identified sentences.
2. Sentence-Level Conditional Prediction: Formulated knowledge selection as a sequence prediction task and introduced a probabilistic contrastive loss to effectively learn the coherence between sentences without explicit graph structures.
3. Stochastic Decoding Strategy: Validated the use of nucleus sampling to balance the trade-off between selecting accurate knowledge and maintaining high diversity, preventing model degeneration.
4. Performance Gains: Demonstrated that KCS significantly improves the accuracy of knowledge composition selection and enhances downstream MHQA performance through data augmentation.

4. Experimental Results

The method was evaluated on two benchmark datasets: HotpotQA and 2WikiMultihopQA.

• Knowledge Composition Selection Accuracy:

  • KCS achieved state-of-the-art performance, improving the overall accuracy (F1-score) of knowledge composition selection by 3.9% compared to the best competitive baseline (CLS using BERT-base).
  • On 2WikiMultihopQA, KCS reached 84.18% and 89.60% of the theoretical upper bound (MAX) for $K=2$ and $K=3$ respectively.
  • Notably, KCS showed improved accuracy as the knowledge composition length increased, indicating it successfully learned long-range knowledge coherence.

• Downstream MHQA Performance (Data Augmentation):

  • When used to augment training data for LLMs (LLaMA3.1 and Qwen2.5), KCS yielded consistent improvements in Exact Match (EM), Precision, Recall, and F1 scores on both datasets.
  • For example, on HotpotQA with LLaMA3.1, KCS improved EM from 54.00 (Original) to 61.00.
  • Diversity vs. Consistency: KCS achieved high diversity (Pairwise-BLEU: 68.1) while maintaining reasonable consistency, outperforming methods like RAST and Composition which often sacrificed diversity for consistency.

• Ablation Studies:

  • Architecture: The Encoder-Decoder architecture significantly outperformed Decoder-only models, highlighting the importance of global context encoding.
  • Order & Length: The model is sensitive to the order of sentences; training with "document" order (maintaining original context flow) yielded the best results.
  • Loss Function: The probabilistic contrastive loss ($\lambda=1$) was crucial; reducing it to 0.5 caused performance drops.

5. Significance and Impact

• Scalability: Unlike previous methods that rely on expensive, structured knowledge graphs or entity extraction, KCS operates directly on unstructured text, making it more scalable and applicable to diverse domains.
• Data Efficiency: By effectively utilizing the latent diversity within existing long-context datasets, KCS addresses the data sparsity problem in MHQA, reducing the need for manual annotation of complex multi-hop questions.
• Generalizability: The framework is modular; the knowledge sampling component can be paired with any downstream question generation model, offering a flexible approach to improving the robustness of reasoning models.

In conclusion, KCS represents a shift from "generating questions based on fixed facts" to "generating questions by dynamically exploring the space of valid knowledge combinations," thereby creating richer and more robust training data for multi-hop reasoning.