🤖 AI

Seeking Information with RAG-Assistants: Does Model Size Matter in Human-AI Collaborations?

This study evaluates RAG-assisted chatbots in realistic multi-turn information-seeking scenarios, finding that while human-AI collaboration significantly improves performance regardless of underlying model size, user satisfaction and perceived usability remain largely unaffected by the model's scale.

Original authors: Lennard C. Froma, Tom Kouwenhoven, Maaike H. T. de Boer, Catholijn M. Jonker, Max J. van Duijn

Published 2026-05-06

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Lennard C. Froma, Tom Kouwenhoven, Maaike H. T. de Boer, Catholijn M. Jonker, Max J. van Duijn

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a complex puzzle, but the instructions are hidden inside a massive, 400-page manual written in technical jargon. You have two tools to help you: a smart assistant (an AI chatbot) and the manual itself.

This paper is like a report card on how well different sizes of these "smart assistants" work when humans team up with them to solve these puzzles. The researchers wanted to know: Does having a "bigger brain" (a larger AI model) actually make the human-AI team work better, or is a smaller, cheaper brain just as good when a human is in the loop?

Here is the breakdown of their experiment and findings, using simple analogies:

The Setup: Three Types of Assistants

The researchers created three versions of the AI assistant, each with a different "brain size" (model size):

The Compact Brain (3B): Small, fast, and efficient. Think of this as a very smart intern who knows the basics but might miss subtle details.
The Mid-Size Brain (8B): A step up. Think of a junior manager who is quite capable.
The Giant Brain (70B): A massive, powerful model. Think of a senior expert with a huge library of knowledge.

The Twist: None of these assistants were allowed to just "guess" from their memory. They were all connected to the 400-page manual (a technique called RAG, or "Retrieval-Augmented Generation"). This is like giving the assistant a direct phone line to the library so they can look up facts instantly, rather than relying on what they memorized in school.

The Experiment: Humans vs. Machines

The researchers hired 112 people to act as "pilots" (even though they weren't real pilots) and asked them to answer specific questions about the manual. They were split into three groups:

Group A: Used the Compact Brain assistant.
Group B: Used the Mid-Size Brain assistant.
Group C: Used the Giant Brain assistant.

The participants could choose to ask the AI, read the PDF manual directly, or use a mix of both. They were paid extra if they got the answers right, so they were motivated to be accurate.

The Big Findings

1. The Power of the "Human-in-the-Loop"

The most important discovery was that humans working with AI were significantly better than AI working alone.

The Analogy: Imagine a GPS (the AI) that gives you directions. If you just follow the GPS blindly, you might miss a road closure. But if you have a human driver who can look at the map, question the GPS, and say, "Wait, that doesn't look right," the team gets to the destination much faster and more accurately.
The Result: Whether the AI was the "Compact," "Mid-Size," or "Giant" brain, the human-AI team crushed the scores of the AI working by itself.

2. Does a Bigger Brain Matter? (The Accuracy Test)

When the AI works alone: The "Giant Brain" (70B) was the best at finding answers on its own. The "Compact Brain" (3B) struggled more.
When the Human helps: This is where it gets interesting. When humans teamed up with the "Compact Brain," the humans helped fill in the gaps. The team's performance jumped so high that the gap between the small brain and the giant brain disappeared.
The Takeaway: A human partner can help a smaller, cheaper AI perform just as well as a massive, expensive one.

3. What Did the Humans Feel? (The Satisfaction Test)

The researchers asked the participants: "Did you like the assistant? Was it easy to use? Did you trust it?"

The Surprise: Even though the "Giant Brain" was technically the smartest when working alone, the humans didn't feel a huge difference between the three assistants when they were working together.
The Nuance: The only clear preference was that people felt the "Giant Brain" was slightly more reliable. However, for everything else (how easy it was to talk to, how much they liked it, how engaging it was), the "Compact," "Mid-Size," and "Giant" brains were rated almost the same.
The Analogy: It's like driving three different cars (a compact car, a sedan, and a luxury SUV) with a co-pilot. Even if the luxury SUV has a more powerful engine, once you have a co-pilot navigating, you might not feel like the luxury car is a better driving experience than the compact one.

Why This Matters

The paper argues that we often obsess over making AI models bigger and bigger to get higher scores on computer tests (benchmarks). But in the real world, where humans are actually using the tools:

Collaboration is key: Humans + AI is a winning team, regardless of the AI's size.
Bigger isn't always "felt" as better: Users didn't necessarily feel the massive, expensive models were much better to work with than the smaller ones.
Efficiency: Since smaller models are cheaper and faster, and humans can help them perform just as well, we might not need to build giant, energy-hungry models for every job.

In short: You don't always need a supercomputer to solve a problem if you have a smart human to help you use a smaller, simpler tool. The human-AI partnership levels the playing field.

Technical Summary: Seeking Information with RAG-Assistants: Does Model Size Matter in Human-AI Collaborations?

Problem Statement
While extensive research on Large Language Models (LLMs) focuses on increasing benchmark performance, the evaluation of these models in real-world, multi-turn human-AI collaborative workflows remains underdeveloped. Current evaluations often rely on quantitative metrics (accuracy, precision, recall) derived from single-turn interactions, which favor larger commercial models and fail to capture the nuances of professional information-seeking tasks. In workplace scenarios involving sensitive data and strict compliance (e.g., EU AI Act), there is a need for locally deployable, open-weight models. However, it remains unclear how small these models can be when integrated into Retrieval-Augmented Generation (RAG) systems while maintaining adequate performance and usability in a collaborative setting.

Methodology
The authors conducted a user experiment with 112 participants to evaluate a chatbot-style RAG-assistant in a realistic information-seeking scenario.

Task: Participants answered nine detailed questions based on a 400+ page flight manual, a document chosen for its structural complexity and domain specificity.
Conditions: The study compared three model sizes from the Llama family (3B, 8B, and 70B) across two settings:
1. Baseline (LLM+RAG): A single-shot interaction where the RAG-assistant answers the original question directly.
2. Human-AI Collaboration: Participants could interact with the RAG-assistant over multiple turns, choose to query the source document directly, or combine both methods.
System Architecture: The RAG pipeline utilized a sequential approach. A document was chunked (size 1024) and embedded using intfloat/multilingual-e5-large-instruct. Upon receiving a query, the system retrieved the top 5 relevant chunks based on cosine similarity and passed them to the generator LLM with a system prompt.
Evaluation Metrics:
- Objective Performance: Accuracy of answers (correct, partially correct, incorrect) measured via a linear mixed-effects model controlling for participant and question variability.
- Subjective Metrics: Perceived usability and satisfaction were assessed using a 10-item questionnaire based on the ASA framework (5-point Likert scale).
- Efficiency: Task completion time was recorded.

Key Contributions

Systematic Evaluation of Model Sizes: A comparative analysis of RAG-assistants based on small, open-weight LLMs (3B, 8B, 70B) in a hybrid human-AI setting.
Realistic Scenario Simulation: An experimental design mimicking professional information-seeking tasks involving large, complex documents, moving beyond standard benchmark datasets.
Multi-Turn Interaction Analysis: An experiment measuring accuracy, usability, and satisfaction across multiple turns, highlighting the dynamics of human-AI collaboration rather than single-turn performance.

Results

Human-AI Collaboration Value: The human-AI setting significantly outperformed both the LLM-only and the LLM+RAG baseline settings across all model sizes ( $p < .001$ ). This confirms the added value of human interaction in information-seeking tasks.
Impact of Model Size on Accuracy:
- In the baseline (LLM+RAG) setting, performance increased significantly from 3B to 8B and from 3B to 70B.
- In the Human-AI setting, the performance gap between the 8B and 70B models was closed; the 8B model performed comparably to the 70B model when assisted by humans.
- The 3B model benefited most from human collaboration, narrowing the gap with larger models.
Perceived Usability and Satisfaction: Contrary to objective performance trends, perceived usability and satisfaction showed little difference across model sizes.
- The 70B model was rated significantly higher than the 8B model on reliability and overall ASA scores, but the 3B model scored equally well or better on engagement and responsiveness.
- Overall, participants did not perceive the larger models as decisively more capable than the smaller ones in terms of user experience.
Interaction Strategies: Participants predominantly used a "copy-paste" strategy. When this failed, they employed strategies such as "distribution" (splitting questions), "contextualisation" (providing excerpts), and "backtracking."

Significance and Claims
The paper argues that while model size influences baseline performance, human-AI collaboration acts as a mitigating factor that can close performance gaps between smaller and larger models. The study highlights a "nuanced trade-off" where larger models do not necessarily yield higher perceived usability or satisfaction compared to smaller, more efficient models.

The authors claim that their work demonstrates the necessity of evaluating AI applications in actual multi-turn interactions with human users. They suggest that focusing solely on benchmark performance provides an incomplete picture, as it overlooks the synergistic benefits of hybrid systems and the subjective user experience. The findings support the viability of using smaller, locally deployable models in RAG applications, provided they are integrated into collaborative workflows where human oversight can compensate for model limitations.