From Voting to Agent Collaboration: Answer-Type-Aware LLM Pipelines for BioASQ 14b

Technical Summary: From Voting to Agent Collaboration: Answer-Type-Aware LLM Pipelines for BioASQ 14b

Problem Statement
Biomedical question answering (QA) faces significant challenges due to the rapid expansion of scientific literature, where critical evidence is often distributed across multiple documents, expressed in heterogeneous terminology, or presented in conflicting contexts. While Large Language Models (LLMs) have shown promise in retrieval-augmented settings, their application to biomedical QA is hindered by specific limitations: sensitivity to the order of retrieved snippets, difficulty in resolving conflicting evidence, the need for precise entity normalization, and the risk of hallucination. Furthermore, the BioASQ Task B challenge involves distinct answer formats—yes/no, factoid, and list—each with unique reasoning requirements and evaluation metrics. The authors argue that a single, uniform inference strategy is insufficient to address the diverse reasoning and evaluation criteria across these question types.

Methodology
The paper presents ku_dmis, a question-type-specific LLM framework designed for BioASQ 14b Task B. The system employs a modular architecture that routes questions to specialized inference pipelines based on their type, utilizing multiple LLMs and ensemble strategies to enhance robustness.

1. System Architecture:

   • Input: The system takes a question, its type, and gold-standard relevant snippets.
   • Routing: A router directs the input to one of three specialized branches: Yes/No, Factoid, or List.
   • Ensemble Approach: Multiple LLMs (including GPT-4o, GPT-5 variants, Claude Sonnet/Opus, and Gemini models) are used as complementary reasoners. Their outputs are aggregated to mitigate model-specific biases.

2. Question-Type-Specific Strategies:

   • Yes/No Questions: To address sensitivity to evidence ordering and conflicting signals, the system employs snippet shuffling combined with self-reflection. Each model queries the shuffled snippets four times. A majority vote determines the initial answer. If outputs are inconsistent, a verification agent intervenes, structuring evidence into positive, negative, and uncertain categories to make a final binary decision (Yes, No, or None).
   • Factoid Questions: These require precise identification of biomedical entities (e.g., drugs, genes). The pipeline uses full-snippet input combined with retrieval-based Chain-of-Thought (CoT) in-context learning (ICL). Semantically similar training questions are retrieved to provide demonstrations that guide the model in evidence-to-answer reasoning. Outputs from multiple models are normalized and aggregated via consensus filtering, retaining candidates appearing in at least three model outputs, ranked by vote count.
   • List Questions: To balance high recall and precision, a multi-agent collaboration architecture is used. This involves four sequential agents:
     • Evidence Analyst: Scans snippets, scores relevance, and extracts candidates without committing to a final answer.
     • Reasoning Agent: Generates a candidate list while re-reading evidence to avoid omissions.
     • Verification Agent: Audits the list for grounding, completeness, formatting, and duplicates.
     • Supervisor Agent: Reviews intermediate reports and feedback to finalize the list, either adopting it or revising it based on original snippets.

3. Development and Evaluation:

   • Strategies were selected based on preliminary experiments on the BioASQ 13b test set.
   • The final system was evaluated on the official BioASQ 14b Task B challenge across four batches.
   • Submission variants (ku_dmis1–5) were tested to optimize prompting, normalization, and aggregation thresholds.

Key Contributions

• Unified Framework: The paper unifies two previously distinct approaches—LLM-based ensembles and agentic QA frameworks—into a single, type-aware system.
• Type-Specific Inference: It demonstrates that distinct inference procedures (shuffling/voting for binary, CoT/consensus for factoid, and multi-agent verification for lists) are necessary to handle the specific reasoning and evaluation constraints of each BioASQ question type.
• Robustness Mechanisms: The introduction of snippet shuffling, selective verification agents, and retrieval-based ICL specifically targets the instability and hallucination issues common in biomedical LLM applications.

Results
In the official BioASQ 14b evaluation, the ku_dmis system demonstrated competitive performance across all batches and question types:

• Yes/No: Achieved stable performance with an average macro F1 score of 0.9110 across four batches. The system maintained high accuracy even in later, more challenging batches.
• Factoid: Achieved an average Mean Reciprocal Rank (MRR) of 0.4626. Notably, the system secured first place in the factoid subtask of Batch 4. Performance varied by batch, with the strongest results observed in the final batch, suggesting the effectiveness of full-snippet reasoning and consensus filtering.
• List: Showed significant improvement across batches, with the mean F-measure rising from 0.2734 in Batch 1 to 0.5652 in Batch 4, resulting in an overall average of 0.4439. This improvement is attributed to the collaborative agent pipeline reducing both over-generation and under-generation of entities.

Significance and Claims
The authors claim that their results validate the effectiveness of combining question-type-specific inference, ensemble prediction, and explicit verification mechanisms for reliable biomedical QA. The paper emphasizes that a "one-size-fits-all" prompting strategy is suboptimal for the diverse requirements of biomedical literature. By tailoring the inference procedure to the answer format—using voting for stability in binary decisions, consensus for entity precision, and agent collaboration for list coverage—the system achieves robust performance.

The paper modestly acknowledges limitations, noting that the system relies on proprietary models (limiting reproducibility), focuses on exact answers rather than ideal paragraph generation, and that performance variability suggests a need for better biomedical synonym normalization and adaptive thresholding. However, the primary contribution remains the demonstration that specialized, type-aware pipelines significantly enhance the reliability and accuracy of LLMs in complex biomedical domains.