ColBERT Retrieval and Ensemble Response Scoring for Language Model Question Answering

Technical Summary: ColBERT Retrieval and Ensemble Response Scoring for Language Model Question Answering

Problem Statement
Domain-specific question answering (QA) remains a significant challenge for Large Language Models (LLMs), particularly smaller models like Phi-2 and Falcon-7B. These models lack the parameter capacity to encode deep, specialized technical knowledge required for fields such as telecommunications. Furthermore, standard LLMs often struggle with factual accuracy in specialized domains and may fail to adhere to specific instruction formats or reasoning constraints when presented with multiple-choice options. The "Specializing Large Language Models for Telecom Networks" challenge highlighted these limitations, where baseline solutions achieved only 41.5% accuracy for Phi-2 and 24.5% for Falcon-7B on telecommunication standards questions.

Methodology
The authors developed distinct Retrieval-Augmented Generation (RAG) pipelines for the Phi-2 and Falcon-7B tracks, both leveraging ColBERT for dense retrieval and a domain-specific glossary for lexical enhancement.

• Common Components:

  • Retrieval: Both systems utilize ColBERT [11] to retrieve relevant chunks from 554 3GPP technical documents. The documents are segmented into 150-token chunks to maintain semantic coherence.
  • Lexicon Enhancement: A glossary of technical abbreviations extracted from 3GPP documents is used to dynamically expand abbreviations found in queries and options, addressing the "lexicon mismatch" problem.
  • Vector Search: FAISS [18] is employed for efficient similarity search.

• Phi-2 Track (Fine-tuning and Instruction Alignment):

  • Fine-tuning: A LoRA adapter [12] was fine-tuned on the training set. The training objective included not only predicting the correct answer but also generating explanations to improve reasoning capabilities.
  • Prompting: The system uses a structured prompt containing instructions, retrieved context, expanded abbreviations, the question, and the multiple-choice options.
  • Optimization: Experiments determined that retrieving $k=13$ chunks was optimal within the model's 2048-token context window. The authors found that providing clear, concise instructions outperformed complex few-shot prompting strategies.

• Falcon-7B Track (Ensemble Scoring):

  • Prompting Strategy: Initial experiments revealed that Falcon-7B struggled to reason effectively when options were included in the prompt. Consequently, the system was modified to generate responses based only on the question, context, and expanded abbreviations, excluding the options from the input.
  • Ensemble Scoring Mechanism: Since the model generates a free-form response rather than selecting an option directly, a scoring system was developed to map the response to the most likely option. This system treats the task as textual entailment recognition.
  • Scoring Components:
    1. Semantic Similarity: A General Text Embedding (GTE) v1.5 model [18] was fine-tuned using triplet loss (anchor: explanation, positive: correct option, negative: random option) to maximize entailment similarity.
    2. Lexical Overlap: A Term Frequency-Inverse Document Frequency (TF-IDF) weighted word overlap score was computed between the response and the options.
  • Final Score: The final option selection is based on a weighted sum of the cosine similarity ($\alpha_2 = 0.8$) and the TF-IDF overlap score ($\alpha_1 = 0.2$).

Key Results
The proposed systems achieved leading performance on the private evaluation dataset of the challenge:

• Phi-2: Achieved 81.9% accuracy, a significant improvement over the 41.5% baseline.
• Falcon-7B: Achieved 57.3% accuracy, a substantial improvement over the 24.5% baseline and previously reported results in the domain.

Comparative Analysis:

• Retrieval: An ensemble of ColBERT and BM25 did not outperform ColBERT alone, suggesting that the evaluation dataset lacked significant lexical gaps between queries and documents, making semantic understanding (ColBERT's strength) more critical than keyword matching.
• Context Sensitivity: Phi-2 benefited from increasing context chunks up to the limit of its window, whereas Falcon-7B performance degraded when more than 3 chunks were provided, indicating an inability to filter noise in larger contexts.
• Scaling: When compared to larger, non-fine-tuned models (e.g., Llama 3.1 405B), the authors' Phi-2 system (80.3% on public set) was outperformed by 6.3%, highlighting the continued advantage of larger parameter counts and training data. However, the specialized RAG and fine-tuning approach allowed the smaller models to compete effectively within the constraints of the challenge.

Significance and Claims
The paper claims that its primary contribution is demonstrating that small language models can be effectively specialized for complex, domain-specific QA tasks through a combination of:

1. Model-specific optimizations: Fine-tuning for instruction alignment (Phi-2) and removing options to prevent reasoning failures (Falcon-7B).
2. Specialized Retrieval: The use of ColBERT-based RAG to inject necessary domain knowledge at inference time.
3. Lexical Adaptation: Dynamic abbreviation expansion to bridge the gap between technical terminology and model vocabulary.

The authors note that their fine-tuning objectives primarily enhanced instruction and task alignment rather than knowledge alignment, as evidenced by the Phi-2 model's strong performance on an out-of-domain pharmacology dataset when fine-tuned on telecom data. The work underscores the importance of retrieval pipelines and reasoning constraints in enabling smaller models to handle technical queries, providing a blueprint for deploying LLMs in specialized industrial domains.