BETA-Labeling for Multilingual Dataset Construction in Low-Resource IR

1. Problem Statement

The field of Information Retrieval (IR) in low-resource languages faces a critical bottleneck: the scarcity of high-quality, task-specific annotated datasets. The paper identifies two primary challenges in addressing this gap:

• Manual Annotation: Traditional human annotation is prohibitively expensive and lacks scalability, making it difficult to build large-scale benchmarks for numerous low-resource languages.
• Automated Annotation (LLMs): While Large Language Models (LLMs) offer a scalable alternative, their direct use introduces significant risks, including label unreliability, inherent model biases, and questions regarding the validity of evaluations derived from such data.
• Cross-Lingual Reuse: There is an unverified assumption that IR datasets from one low-resource language can be effectively translated (via machine translation) and reused for another. The paper questions the semantic fidelity and task validity of this "one-hop" translation approach.

2. Methodology

The authors propose a hybrid framework called BETA-labeling to construct a Bangla IR dataset, followed by an experimental analysis of cross-lingual dataset reuse.

• BETA-Labeling Framework (Dataset Construction):

  • Multi-Model Consensus: Instead of relying on a single LLM, the framework employs multiple LLM annotators from diverse model families to generate initial labels.
  • Quality Control Mechanisms: The process integrates three specific layers of validation:
    1. Contextual Alignment: Ensuring labels fit the specific retrieval context.
    2. Consistency Checks: Verifying logical coherence across the dataset.
    3. Majority Agreement: Resolving conflicts by selecting the label agreed upon by the majority of the LLMs.
  • Human-in-the-Loop: A final human evaluation step is conducted to verify the quality of the aggregated LLM labels, ensuring the dataset meets benchmark standards.

• Cross-Lingual Reuse Experiment:

  • The study investigates the viability of reusing IR datasets from other low-resource languages by translating them into the target language using LLM-based translation.
  • Evaluation Metrics: The authors analyze "meaning preservation" (semantic fidelity) and "task validity" (whether the translated data still functions correctly for IR tasks) across multiple language pairs.

3. Key Contributions

• Novel Dataset Construction: The creation of a high-quality Bangla IR dataset using the BETA-labeling framework, demonstrating a scalable path for low-resource language data generation.
• Empirical Framework for LLM Annotation: A structured methodology that mitigates the risks of single-model bias and hallucination by leveraging multi-model consensus and human verification.
• Critical Analysis of Translation Reuse: The paper provides empirical evidence challenging the efficacy of one-hop machine translation for dataset adaptation, highlighting that semantic preservation is not guaranteed across all language pairs.
• Risk Identification: It explicitly identifies language-dependent biases and inconsistent semantic preservation as primary failure points in cross-lingual dataset reuse.

4. Results

• Dataset Quality: The BETA-labeling framework successfully produced a reliable Bangla IR dataset, validating that a combination of diverse LLMs and human oversight can overcome the limitations of purely manual or purely automated approaches.
• Translation Variability: The experiments revealed substantial variation in the quality of translated datasets depending on the specific language pair.
• Semantic Degradation: The study found that meaning preservation is often inconsistent. In many cases, the translation process introduced biases or altered the semantic intent of the queries and documents, directly compromising the reliability of the reused dataset for IR tasks.
• Task Validity: The effectiveness of cross-lingual reuse is highly language-dependent; what works for one pair may fail for another due to structural and semantic differences between languages.

5. Significance

This work offers a pragmatic roadmap for the IR community working with low-resource languages:

• Practical Guidance: It moves beyond theoretical discussions to offer a concrete, reproducible framework (BETA-labeling) for constructing benchmarks without relying solely on expensive human annotation.
• Cautionary Evidence: It serves as a critical warning against the "copy-paste" approach to dataset creation via translation. Researchers are advised to rigorously evaluate semantic preservation before reusing translated datasets.
• Future Directions: The study underscores the need for more robust evaluation pipelines that account for language-specific nuances, ensuring that low-resource IR systems are built on reliable, high-fidelity data rather than flawed translations.