WebFAQ 2.0: A Multilingual QA Dataset with Mined Hard Negatives for Dense Retrieval

1. Problem Statement

The development of robust multilingual dense retrieval systems is hindered by a lack of large-scale, diverse, and high-quality training data. While previous resources like the original WebFAQ, CCQA, and PAQ provided valuable FAQ-based question-answer (QA) pairs, they suffered from significant limitations:

• Limited Scale and Diversity: The original WebFAQ relied on preprocessed data dumps (Web Data Commons), restricting its size (96M pairs) and language coverage (75 languages), with a heavy skew toward English.
• Lack of Hard Negatives: The original dataset lacked "hard negatives" (plausible but incorrect answers), which are critical for training dense retrievers to distinguish between semantically similar but irrelevant content.
• Static Nature: Previous datasets were static snapshots, lacking mechanisms for continuous updates or expansion.

2. Methodology

A. Data Collection Strategy (WebFAQ 2.0)

The authors fundamentally revised the data collection pipeline to move away from static dumps toward active web crawling:

• Source: Instead of relying solely on Web Data Commons, the team mined URLs containing FAQPage schema markup directly from Common Crawl (2025 snapshots).
• Crawling Tool: They utilized OWLer, a distributed web crawler, to retrieve full page content.
• Advantages: This approach yields a significantly larger dataset, captures rich metadata (page titles, descriptions) to resolve ambiguity, and extracts hreflang links to improve bilingual alignment.
• Processing:
  • Cleaning: HTML tags are stripped, and malformed entries removed.
  • Language Detection: Performed via FastText.
  • Contextual Filtering: Jina v3 embeddings compute semantic similarity between questions and answers to flag ambiguous pairs (e.g., questions lacking context without the page title).
  • Duplicate Handling: Instead of removing near-duplicates via embedding thresholds (which proved difficult to tune), the test sets are constructed by selecting only one QA pair per domain to ensure independence.

B. Hard Negative Mining

To address the community's need for hard negatives, the authors introduced a two-stage mining pipeline:

1. Stage 1 (Lexical Retrieval): BM25 retrieves the top 200 candidate answers for each query from the monolingual corpus.
2. Stage 2 (Semantic Reranking): A strong multilingual cross-encoder (BGE-m3) reranks these 200 candidates.
3. Output: The result is a dataset of 1.25 million quintuples (query, positive, pos_score, negatives, neg_scores) across 20 languages. The cross-encoder scores are provided to enable denoising (filtering out false negatives) or knowledge distillation.

C. Dataset Statistics & Expansion

• Scale: 198 million QA pairs (up from 96M).
• Languages: 104 languages (up from 75).
• Bilingual Alignments: Increased from 1.5M to 14.3M aligned QA pairs across 3,970 language combinations. Notably, this includes significant growth in non-English pairs (e.g., Marathi–Telugu, Russian–Ukrainian).
• Topic Distribution: A shift toward "Traveling and Hospitality" (59.2%), inferred via an XLM-RoBERTa classifier fine-tuned on GPT-5-mini annotated data.
• Question Types: Categorized using an LLM ensemble (LLaMA 3.1, Gemma 2, Qwen 2.5) to label types like Factoid, Experience, and Instruction.

3. Key Contributions

1. WebFAQ 2.0 Dataset: The largest FAQ-based resource to date, offering 198M QA pairs with significantly improved multilingual balance and richer context (titles/descriptions).
2. Hard Negatives Resource: A dedicated dataset of 1.25M queries with mined hard negatives and cross-encoder scores, specifically designed to train dense retrievers.
3. Continuous Infrastructure: The dataset is part of a long-term effort linked to the Open Web Index, allowing for regular updates and continuous expansion of structured FAQs.
4. Training Frameworks: The paper demonstrates and evaluates two specific fine-tuning strategies enabled by the new data:
   • Contrastive Learning: Using MultipleNegativesRanking (MNR) loss.
   • Knowledge Distillation: Using MarginMSE loss to mimic cross-encoder scores.

4. Results and Evaluation

The authors evaluated the impact of the hard negatives on multilingual dense retrievers using three benchmarks: WebFAQ Retrieval, MIRACL-HN, and Mr. TyDi.

• Contrastive Learning (MNR) Findings:

  • Models trained with random negatives generally outperformed those trained with unfiltered hard negatives.
  • This indicates a high prevalence of false negatives in the mined data (items that look relevant but are actually correct).
  • Applying a "denoising" step (filtering based on cross-encoder scores) improved MNR performance but did not surpass the robustness of random negatives in all configurations.

• Knowledge Distillation (MarginMSE) Findings:

  • Using the cross-encoder scores directly as soft targets via MarginMSE loss yielded the most robust improvements, particularly for non-English languages.
  • Trade-off: This approach resulted in a performance drop for English. Since the base model was pre-trained on English (MS MARCO) and the hard negatives included significant English data, the distillation process shifted the model's focus, slightly degrading English performance to gain gains in other languages.

• Overall Performance:

  • The best-performing configuration (M-MSE) achieved the highest NDCG@10 scores across most non-English languages in the MIRACL and Mr. TyDi benchmarks, demonstrating the value of the mined hard negatives when used with appropriate loss functions.

5. Significance

• Advancement in Multilingual IR: WebFAQ 2.0 provides the necessary scale and diversity to train state-of-the-art multilingual retrieval models, moving beyond English-centric benchmarks.
• Practical Training Insights: The paper offers critical empirical evidence on the challenges of hard negative mining (specifically false negatives) and validates that Knowledge Distillation is often superior to simple Contrastive Learning when high-quality cross-encoder scores are available.
• Community Resource: By releasing the dataset, hard negatives, and training scripts on GitHub and HuggingFace, the authors lower the barrier for researchers to develop and evaluate cross-lingual information retrieval systems.
• Future-Proofing: The integration with the Open Web Index ensures the dataset remains a living resource, capable of adapting to the evolving web landscape.