CQ-Transformer: A Query Decomposition Framework with ALBERT-Based Semantic Matching for Complex Extractive Question Answering

Technical Summary: CQ-Transformer

Problem Statement
Existing transformer-based Question Answering (QA) systems, including foundational models like BERT and its derivatives, face significant limitations when processing complex, multi-category queries and long-range contextual dependencies. These models frequently produce ambiguous or incomplete answer extractions in such scenarios. While standard extractive QA approaches excel at single-span extraction, they struggle to decompose multifaceted questions into discrete components or to maintain coherence across extended text sequences, leading to performance degradation in real-world, complex information retrieval tasks.

Methodology
The paper proposes CQ-Transformer, a hybrid QA framework designed to address these limitations through a three-stage pipeline: query decomposition, semantic matching, and span extraction.

1. Query Decomposition (Split Function): The framework introduces a delimiter-based split function that processes complex, multi-category queries. By utilizing delimiters such as commas, "and," and question marks, the system decomposes a single complex query into discrete, single-category sub-queries. This allows the system to handle multi-hop or multi-faceted questions that standard models treat as monolithic inputs.
2. Semantic Similarity Matching (ALBERT): Once decomposed, each sub-query undergoes sentence-level relevance filtering. The framework employs a pre-trained ALBERT (A Lite BERT) model to calculate semantic similarity between each sub-query and the sentences within the context passage. ALBERT is selected for its parameter-sharing mechanisms, which reduce model complexity and memory usage while maintaining strong semantic understanding. The system identifies the most relevant sentence for each sub-query to serve as the focused context for the next stage.
3. Span Extraction (RoBERTa/XLNet): The selected sub-queries and their corresponding relevant sentences are fed into pre-trained encoders—specifically RoBERTa and XLNet. These models tokenize the input, generate embeddings, and predict the start and end boundaries of the answer spans. The framework utilizes an argmax function to select the most probable span boundaries. Finally, the extracted answer spans from all sub-queries are aggregated into a unified response.

The system is evaluated using the SQuAD 2.0 dataset, which includes both answerable and unanswerable questions, utilizing metrics such as Exact Match (EM) and F1 score.

Key Contributions
The authors identify three primary contributions:

• Hybrid Framework: The integration of RoBERTa and XLNet with a query decomposition mechanism and ALBERT-based semantic matching to address ambiguity and long-range dependencies.
• Query Decomposition Mechanism: The introduction of a specific split function that breaks down complex multi-category queries into single-category sub-queries, a step often omitted in prior studies that process queries holistically.
• ALBERT-Based Matching: The substitution of conventional similarity methods (such as cosine similarity or LSTM-based matching) with an ALBERT-based semantic similarity module to more accurately identify the most relevant sentence for each query category.

Results
Experimental evaluation on the SQuAD 2.0 dataset demonstrates that the proposed framework significantly outperforms baseline models:

• RoBERTa-CQ Variant: Achieved an F1 score of 87.95% and an Exact Match (EM) of 84.96%. This represents a substantial improvement over the baseline BERT (F1: 66.36%, EM: 57.85%), standard RoBERTa (F1: 82.58%, EM: 76.58%), and XLNet (F1: 68.14%, EM: 65.45%).
• XLNet-CQ Variant: Attained an F1 score of 77.49% and an EM of 74.79%, also outperforming its respective baselines.
• Qualitative Analysis: The paper notes that while baseline models often fail entirely or provide incomplete answers to complex multi-category queries, the CQ-Transformer framework effectively resolves these uncertainties by addressing each query category individually.

Significance and Claims
The paper claims that the CQ-Transformer framework offers a scalable, high-performance solution for real-world extractive QA tasks, particularly where queries are complex and require multi-step reasoning. By effectively resolving multi-category queries and long-range dependencies, the framework demonstrates potential for deployment in intelligent information systems, such as enterprise search engines, knowledge base querying, and automated customer support platforms.

The authors maintain a modest stance regarding future work, noting that the current study focuses on extractive QA. They propose that future research will explore the framework's generalizability on other datasets (e.g., TriviaQA, Natural Questions), the integration of generative models for free-form answer generation, and extensions to support multilingual QA. The study emphasizes that the proposed improvements are derived from the specific combination of query decomposition and semantic matching, rather than solely from the underlying pre-trained models.