💬 NLP

RaguTeam at SemEval-2026 Task 8: Meno and Friends in a Judge-Orchestrated LLM Ensemble for Faithful Multi-Turn Response Generation

RaguTeam's winning system for SemEval-2026 Task 8, which employs a heterogeneous ensemble of seven LLMs orchestrated by a GPT-4o-mini judge to select the best response, achieved first place by significantly outperforming the strongest baseline while also introducing the cost-effective Meno-Lite-0.1 model and critiquing the task's annotation limitations.

Original authors: Ivan Bondarenko, Roman Derunets, Oleg Sedukhin, Mikhail Komarov, Ivan Chernov, Mikhail Kulakov

Published 2026-05-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ivan Bondarenko, Roman Derunets, Oleg Sedukhin, Mikhail Komarov, Ivan Chernov, Mikhail Kulakov

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: A "Talent Show" for AI

Imagine you are hosting a talent show to find the best answer to a tricky question. You have a panel of seven different judges (AI models), each with their own unique style, strengths, and weaknesses. Some are huge, super-smart giants; others are smaller, specialized experts.

The team from Novosibirsk State University, called RaguTeam, didn't just pick the "smartest" judge. Instead, they set up a system where all seven judges write an answer, and then a Head Judge (a lightweight AI called GPT-4o-mini) reads all seven answers and picks the single best one for that specific question.

They entered this system into a competition called SemEval-2026 Task 8 (specifically Task B), which tests how well AI can answer questions based only on provided documents, even during a long conversation. Their system won 1st place out of 26 teams.

The Problem: The "Hallucination" Trap

In the real world, AI often tries to be too helpful. If it doesn't know the answer, it might make something up (hallucinate) to sound confident. This is dangerous when the AI is supposed to be a fact-checker.

The competition task was tricky because it involved:

Multi-turn conversations: The AI has to remember what was said earlier in the chat.
Strict rules: The AI must only use the provided "reference passages" (like a closed-book exam).
The "I Don't Know" test: Sometimes the documents don't have the answer. The AI is supposed to say, "I don't know," rather than guessing.

How They Won: The "Swiss Army Knife" Strategy

1. The Diverse Team (The Ensemble)

Instead of relying on one super-computer, they used a team of seven different AI models.

The Giants: They included massive models (like GLM-4.6 with 357 billion parameters) that are like encyclopedias.
The Specialists: They included a custom-built, smaller model called Meno-Lite-0.1. Think of this as a 7-billion-parameter model that was specifically trained to be a "fact-checker" rather than a "creative writer." It's like a small, nimble detective who is very good at spotting when information is missing.
The Mix: They used models from different companies (Google, Meta, Microsoft, etc.) and different sizes. The idea is that if one model makes a mistake, another might get it right.

2. The Two Prompting Styles

They didn't just ask the models the same way. They used two different "instruction styles":

Style A (The Rulebook): A strict set of rules telling the AI to stick only to the text provided.
Style B (The Examples): They gave the AI a few examples of how to handle tricky situations (like when there are no documents or when the conversation history is empty). This is like showing a student a practice test before the real one.

3. The Head Judge (The Selection)

This is the secret sauce. For every single question, all seven models generated an answer. Then, the Head Judge (GPT-4o-mini) read them all and asked: "Which of these answers is the most faithful to the documents?"

If the documents were empty, the system automatically said, "I don't know" (a safe, perfect move).
If the documents had an answer, the Head Judge picked the one that didn't invent facts.

Key Findings (The "Aha!" Moments)

Diversity > Size: The winning team proved that having a mix of different models is better than just having the biggest, most expensive one. The "team" beat the single best giant model (GLM-4.6) by a significant margin. It's like a relay team beating a solo runner because they covered more ground.
Examples Beat Rules: When they gave the AI specific examples of how to handle "I don't know" situations (few-shot prompting), it performed much better than when they just gave it abstract rules. It's like teaching a child by showing them a video of someone sharing, rather than just telling them "be nice."
The Small Model's Role: The custom 7B model (Meno-Lite-0.1) didn't win the most votes, but when it did get picked, it was often the perfect answer. It was a "specialist" that saved the day in niche situations, proving you don't always need a massive brain for every job.

The Critique: The Game Was a Little Rigged

The authors also pointed out a flaw in the competition itself.

The "Empty Box" Shortcut: In the test, whenever a question was "unanswerable," the provided documents were completely empty. This made it too easy for the AI to win: it just had to detect "empty box" and say "I don't know."
Real Life is Harder: In the real world, unanswerable questions usually come with irrelevant documents (distractors). The authors argue that future tests should include these "distractors" to make the AI work harder and prove it actually understands the text, rather than just spotting empty space.

Summary

RaguTeam won by treating AI generation like a talent show. They gathered a diverse cast of characters (different AI models), gave them different ways to prepare (prompts), and hired a smart referee (the Head Judge) to pick the winner for every single round. They showed that a well-coordinated team of diverse AIs is smarter and more reliable than any single AI working alone.

Technical Summary: RaguTeam at SemEval-2026 Task 8

Problem Definition

The paper addresses SemEval-2026 Task 8 (MTRAGEval), specifically Task B, which focuses on faithful multi-turn response generation in Retrieval-Augmented Generation (RAG) settings. The core challenge involves generating a response to a final user query within a multi-turn dialogue, strictly grounded in provided reference passages.

The task requires systems to navigate several complex constraints:

Context Integration: Combining dialogue history with retrieved documents.
Faithfulness: Ensuring all claims are supported by the provided evidence, avoiding hallucinations or reliance on parametric world knowledge.
Abstention: Correctly identifying and declining to answer unanswerable or underspecified questions.
Multi-turn Coherence: Resolving coreferences and tracking user intent across turns.

The evaluation metric is a Conditioned Harmonic Mean (HM3) of three scores: $RB_{alg}$ (BERTScore Recall, BERT-K-Precision, ROUGE-L), $RB_{llm}$ (LLM-judge score), and $RL_F$ (reference-less faithfulness). A critical component is the handling of "unanswerable" instances, where the system must abstain to achieve high scores.

Methodology

The RaguTeam system employs a heterogeneous ensemble approach combined with a judge-based selection mechanism. The pipeline consists of three stages:

1. Prompt Engineering Strategies

The team utilized two distinct prompting strategies to maximize candidate diversity:

System Prompt Only (Group 1): An iteratively refined system prompt ( $P$ ) generated via an LLM-based procedure (using Gemini). This prompt emphasizes strict context adherence, extractive phrasing, depersonalized synthesis, and explicit abstention for unanswerable questions.
Few-Shot Strategy (Group 2): Augmented prompts containing category-aware few-shot examples. Instances were clustered into three categories based on context availability: (1) full context, (2) empty context (unanswerable), and (3) empty history (first turn). Medoids were selected from each category to provide robust behavioral demonstrations.

2. Heterogeneous Ensemble Composition

The system generates candidates from seven diverse LLMs to exploit complementary failure modes across model families, scales, and providers:

Group 1 (System Prompt): Gemini-3-Pro-Preview, GLM-4.6, Llama-3.3-70B-Instruct, and Qwen3-235B-A22B-Instruct-2507.
Group 2 (Few-Shot): Claude 4.5 Haiku, Qwen2.5-32B-Instruct, and Meno-Lite-0.1.
- Meno-Lite-0.1 is a 7B model introduced by the authors, derived from Qwen2.5-7B-Instruct. It underwent continued pretraining on Russian-English educational data and supervised fine-tuning for extraction, multi-hop QA, and RAG-oriented instruction following.

3. Judge-Based Selection

For each instance, a lightweight GPT-4o-mini model acts as a judge. It evaluates all seven candidate responses for faithfulness (checking if claims are supported by the provided passages) and assigns a score from 0 to 1. The candidate with the highest score is selected.

Post-processing: For instances with empty reference contexts, the system bypasses generation and outputs a fixed "I don't have an answer" response to ensure reliable abstention.

Key Contributions

Judge-Selected Ensemble: A system combining seven LLMs with two prompting strategies, selected by a GPT-4o-mini judge, achieving a conditioned HM3 of 0.7827.
Diversity Ablations: Demonstrations that diversity in model families, scales, and prompting strategies yields consistent gains over any single model.
Meno-Lite-0.1: The introduction of a 7B domain-adapted model that achieves a strong cost-performance trade-off, reaching an HM3 of 0.681 on answerable instances and showing superior abstention capabilities on unanswerable instances compared to larger baselines.
Benchmark Analysis: A critical examination of MTRAGEval, highlighting limitations such as "target leakage" via empty contexts and annotation gaps.

Results

The system ranked 1st out of 26 teams in SemEval-2026 Task 8.

Performance: Achieved a conditioned HM3 of 0.7827, significantly outperforming the strongest baseline (gpt-oss-120b, 0.6390) by +0.1437 absolute points (+22.5% relative).
Ensemble vs. Single Models: The ensemble (0.783) outperformed the best individual model, GLM-4.6 (0.748), confirming that judge-based selection adds value beyond raw model scale.
Ablation Insights:
- Diversity: Random selection from the ensemble outperformed most individual models, proving the value of candidate diversity.
- Prompting: Few-shot prompting with category-aware examples outperformed the iteratively refined system prompt, particularly on underspecified questions (+3.7 p.p. gain).
- Meno-Lite-0.1: While rarely selected by the judge (2/424 instances), it provided high-quality niche selections and demonstrated strong standalone performance on unanswerable questions (IDK rate of 0.54 vs. 0.24 for Qwen2.5-32B).

Significance and Claims

The paper posits three primary insights derived from their results:

Diversity over Scale: A heterogeneous ensemble of smaller and larger models can outperform the largest single model (e.g., 357B GLM-4.6) for grounded generation, suggesting that complementary failure modes are more critical than parameter count.
Concrete Demonstrations: Category-aware few-shot examples are more effective than abstract system instructions, especially for handling edge cases like underspecified questions.
Faithfulness vs. Pragmatic Appropriateness: While faithfulness is necessary, it is not sufficient for pragmatic appropriateness. The ensemble underperformed on underspecified questions compared to GLM-4.6, indicating a need for multi-objective selection that balances grounding with completeness.

The authors also note system limitations, including computational costs, latency due to sequential generation/judging, and dependency on the specific judge model (GPT-4o-mini). They conclude that future benchmarks should include distractor passages for unanswerable questions to prevent trivial shortcuts and restrict reference-less metric computation to the evaluation phase to prevent leakage.