LiCQA : A Lightweight Complex Question Answering System

1. Problem Statement

The paper addresses the challenge of Open-Domain Complex Question Answering (QA). While significant progress has been made in answering simple factoid questions (e.g., "Who won the Turing Award in 1970?"), answering complex questions remains difficult. Complex questions are defined as those involving multiple entities and relationships, requiring the synthesis of evidence from multiple documents (e.g., "Which Nolan films won an Oscar, but missed a Golden Globe?").

Existing solutions face two main limitations:

• Knowledge Graph (KG) based systems: Often suffer from incompleteness and lack of up-to-date coverage.
• Supervised Deep Learning models (e.g., DrQA, BERT-based): Require massive amounts of annotated training data, are computationally expensive to train and deploy (often requiring GPUs), and are generally designed for single-document answers.

The authors propose a system that is unsupervised, lightweight, and capable of handling complex queries without relying on large-scale annotated datasets or heavy computational resources.

2. Methodology: The LiCQA Pipeline

LiCQA is an unsupervised system that operates primarily on corpus evidence. It follows a standard four-stage pipeline (illustrated in Figure 1 of the paper):

A. Question Type Classification

The system first classifies the expected answer type (e.g., PERSON, LOCATION, ORGANIZATION).

• Approach: The authors experimented with two methods:
  1. Traditional Machine Learning: A Support Vector Machine (SVM) using features like Named Entities (NEs), lemmas, and Part-of-Speech (POS) tags.
  2. Neural Classifier: Using the Universal Sentence Encoder (USE) to embed the question, followed by a feed-forward neural network.
• Result: The SVM-based classifier performed slightly better than the neural approach in their experiments.

B. Answer Extraction and Filtering

• Retrieval: The question is used as a query to retrieve a set of documents ($D$) from a corpus (simulated via search engine results).
• Entity Extraction: Named Entities (NEs) are extracted from the retrieved documents using Flair, a state-of-the-art NER tool based on contextualized embeddings.
• Filtering: Entities are filtered based on the predicted answer type from the classification stage. A mapping table (Table 1) aligns the classification labels (from [25]) with OntoNotes 5 tags used by Flair.

C. Answer Scoring

This is the core innovation for handling complex questions. Since the answer entity itself may not appear in the question, the system scores entities based on the semantic similarity between the question and the sentences containing the candidate entity.

• Sentence Embeddings: Two methods were tested: InferSent (Bi-LSTM with GloVe) and Sentence-BERT (fine-tuned BERT/RoBERTa). InferSent performed slightly better.
• Scoring Strategies:
  1. Simple Averaging: Average cosine similarity of all sentences containing the entity.
  2. Max-Average: Average of the best-matching sentence per document.
  3. Max-Score: The single highest similarity score found for the entity.
  • Finding: The Max-Score approach yielded the best results.
• Frequency Filtering: To reduce computational load, entities appearing in fewer than a threshold of documents are filtered, and only the top 100 entities (ranked by Document Frequency) are scored.

D. Answer Ranking

The final ranking combines the semantic similarity score with a statistical factor: Document Frequency (DF).

• Rationale: DF acts as a relevance feedback mechanism; entities appearing frequently in relevant documents are likely correct.
• Combination: The authors tested weighted addition and simple multiplication.
  • Formula: $comb\text{-}score^* = score(e) \times \frac{df(e)}{|D|}$
• Result: Simple multiplication of the semantic score and normalized DF provided the optimal performance.

3. Key Contributions

1. Unsupervised Complex QA: LiCQA addresses complex questions without requiring large-scale human-annotated training datasets, relying instead on corpus evidence and unsupervised techniques.
2. Lightweight Architecture: The system avoids end-to-end deep learning training. It utilizes pre-trained models (like USE, Flair, InferSent) as modules, significantly reducing the computational cost and latency compared to state-of-the-art baselines.
3. Novel Scoring Mechanism: The integration of semantic similarity (via sentence embeddings) with document frequency allows the system to synthesize evidence across multiple documents effectively.
4. Tie-Aware Evaluation: The authors introduce and utilize tie-aware metrics (tMRR, tP@1, tHit@5) to fairly evaluate systems that may return multiple answers with the same rank, a common occurrence in complex QA.

4. Experimental Results

The system was evaluated on two datasets of complex questions: WikiAnswers (CQ-W) and Google Trends (CQ-T). It was compared against:

• QUEST: A state-of-the-art unsupervised complex QA system.
• DrQA: A supervised deep learning QA system.
• BFS & ShortestPaths: Graph-based baselines.

Performance Metrics:

• Effectiveness: LiCQA significantly outperformed all baselines.
  • On the CQ-W dataset (Top10 documents), LiCQA achieved a 21.6% improvement in MRR and a 71.8% improvement in Hit@5 over the next best system (QUEST).
  • On the CQ-T dataset, LiCQA consistently achieved the highest Hit@5 scores.
  • When using tie-aware metrics (which penalize systems that return many incorrect answers at the same rank), LiCQA's superiority over QUEST was even more pronounced, while DrQA's performance dropped significantly relative to LiCQA.
• Efficiency (Latency):
  • LiCQA was 8x faster than the baselines.
  • While QUEST and DrQA took ~120–160 seconds to process a query set, LiCQA took only ~13–18 seconds. This is largely because QUEST spends significant time constructing a quasi-knowledge graph on the fly.

5. Significance

• Practicality: LiCQA demonstrates that high-performance complex QA does not necessarily require massive computational resources or expensive training data. This makes it suitable for real-time applications and environments with limited hardware.
• Robustness: The system handles the "long-tail" nature of complex questions better than supervised models trained on factoid datasets (like SQuAD), which often fail when answers are spread across multiple documents.
• Reproducibility: The authors emphasize the simplicity of the implementation and have made the source code available, facilitating further research in lightweight QA systems.

In conclusion, LiCQA represents a shift towards efficient, unsupervised methods for complex information retrieval, proving that carefully engineered pipeline components using pre-trained embeddings can outperform heavy, end-to-end supervised models in specific, challenging domains.