uva-irlab-conv at SemEval-2026 Task 8: Multi-Turn RAG with Learned Sparse Retrieval and Listwise Reranking
This paper presents the uva-irlab-conv system for SemEval-2026 Task 8, which employs a multi-turn retrieval-augmented generation pipeline combining learned sparse retrieval with LLM-based conversational query rewriting and listwise reranking to effectively handle diverse domains and unanswerable queries.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are trying to find specific information in a massive library, but you aren't just asking one question; you are having a long, back-and-forth conversation with a librarian. Sometimes you say, "Who is the CEO?" and the next minute you say, "What's their address?" The second question makes no sense without the first one. This is the challenge of Multi-Turn Retrieval and Question Answering.
The paper you shared describes a team from the University of Amsterdam (called uva-irlab-conv) who built a smart system to act as this librarian. They entered a competition called SemEval-2026 Task 8 to see how well their system could handle these conversations across four very different topics: finance, cloud computing, government, and Wikipedia.
Here is how their system works, explained through simple analogies:
1. The Translator (Query Rewriting)
The Problem: Humans speak in shortcuts. If you ask, "Who is the CEO?" and then "How old is he?", a computer doesn't know who "he" is.
The Solution: The system has a "Translator" (an AI) that listens to the whole conversation history. Before it goes to the library, it rewrites your last question into a complete, standalone sentence.
- Analogy: It's like a translator who hears you say, "I want the blue one," and rewrites it for the shopkeeper as, "I want the blue shirt from the rack we were just looking at."
2. The Fast Scout (Learned Sparse Retrieval)
The Problem: The library has millions of documents. You can't read them all.
The Solution: The system uses a "Fast Scout" called LION-SP. This is a special type of search engine that is good at understanding both the meaning of words and the exact words used. It is designed to work well even if the topic changes from finance to government.
- Analogy: Imagine a scout who runs into a huge forest. Instead of reading every tree, they quickly scan for specific keywords and concepts, pulling out the top 1,000 most promising leaves (documents) to bring back to the main office.
3. The Expert Judge (Reranking)
The Problem: The Scout brought back 1,000 leaves, but you only need the best 10.
The Solution: The system uses a two-step judging process:
- Step A (Pointwise): An AI looks at each leaf one by one and gives it a score. This narrows the list down to the top 20.
- Step B (Listwise): A super-smart AI (GPT-4.1) looks at the top 20 leaves all at once, while remembering the entire conversation. It compares them against each other to decide the perfect order.
- Analogy: First, a coach picks the top 20 players from a tryout. Then, a head coach watches those 20 play together in a scrimmage to decide the final starting lineup, ensuring they work well together in the context of the game.
4. The Storyteller (Response Generation)
The Problem: Now you have the top 10 documents. You need a clear answer.
The Solution: The system takes the top 5 documents and the full conversation history and asks a powerful AI to write the final answer.
- Analogy: A writer sits down with the top 5 research notes and the full story of your conversation to write a short, accurate summary for you.
How Did They Do?
The team entered three different challenges in the competition:
- Finding the Documents (Task A): They did amazingly well, coming in 2nd place out of 38 teams. Their system was very good at finding the right documents, even in tricky topics like finance.
- Answering with Perfect Documents (Task B): They were given the "correct" documents by the judges and asked to write answers. They did okay, but not the best.
- Answering with Their Own Search (Task C): They used their own search results to write answers. They ranked 20th out of 29.
The Big Takeaway:
The paper highlights a specific trade-off. Their system was excellent at finding the right information (retrieval). However, when it came to writing the final answer, it struggled a bit with questions that couldn't be answered (like "What is the secret password?" when the password isn't in the documents).
The authors note that their system is very "faithful," meaning it rarely makes things up (hallucinates) if the information isn't there. However, because they didn't explicitly program the system to say "I don't know" when information is missing, it sometimes tried to answer questions it couldn't fully answer, which lowered its score in the generation tasks.
In short: They built a team of AI specialists (a translator, a scout, a judge, and a writer) that works together to handle complex conversations. They are masters at finding the right books in the library, but they are still learning how to admit when a book simply doesn't have the answer.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.