💬 NLP

On-Device Deep Research at 4B: Exposure Bounds Faithfulness, Retrieval Bounds Coverage

This study demonstrates that for on-device 4B research agents, citation faithfulness is primarily determined by the amount of source text exposed to the model rather than source quality, while the coverage of correct sources remains strictly limited by retrieval recall.

Original authors: Vinay Kumar Chaganti

Published 2026-07-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Vinay Kumar Chaganti

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart, pocket-sized robot librarian named "4B." This robot lives on your personal laptop and its job is to read a bunch of research papers, figure out the answers to tricky questions, and write a short report with citations (like "according to page 5 of this book").

But here's the catch: sometimes the robot lies. It might say, "The book says X," when the book actually says Y. Or, it might pick the wrong book entirely. Researchers wanted to know: How do we make this little robot tell the truth, and what does it cost?

They ran a massive experiment on a laptop with 24 GB of memory to find the secret sauce. Here is what they discovered, using two simple levers.

Lever 1: The "Peek-a-Boo" Effect (Exposure)

Think of the robot's reading glasses. In the first test, the robot was only allowed to peek at the first 400 characters (about a short paragraph) of each paper before writing its report. It was like trying to judge a whole movie by only watching the first 30 seconds.

The result? The robot was shaky. It only got about 45% of its claims right when looking at real search results, and even worse (37%) when looking at perfect, "gold standard" papers. It was guessing too much.

Then, the researchers gave the robot better glasses. They let it read 1,500 characters (about a full page) of each paper. Suddenly, the robot's brain lit up!

On real search results, its accuracy jumped to 58%.
On perfect papers, it also hit 58%.

The Big Surprise: It didn't matter if the paper was perfect or just a random search result. Once the robot could read enough of the text, it supported its claims just as well in both cases. The paper argues that faithfulness is bound by how much the robot sees, not by whether the source is perfect.

However, the paper is careful to say this isn't a magic fix. Even at its best, the robot still got about 42% of its claims wrong. It's better, but it's not perfect yet. Also, this "reading more" trick only worked up to a point; reading even more didn't help much after 1,500 characters.

Lever 2: The "Library Search" Problem (Retrieval)

Now, imagine the robot is reading a perfect book, but the book isn't even on the shelf! The researchers found a second problem: Coverage.

Even when the robot read 1,500 characters, if the search engine didn't find the right book in the first place, the robot couldn't cite it. The "Trustworthy Coverage" (how often the robot cites the right source and gets the facts right) stayed stuck around 0.22 (or 22%) no matter how much text they let the robot read.

Why? Because the search engine was only finding about 40% of the right papers to begin with. The robot can't cite what it can't find. The paper explicitly rules out the idea that "reading more" will fix this. If the robot never sees the right book, reading more of the wrong books won't help.

What the Robot Can't Do (The "Repair" Myth)

You might think, "What if we let the robot write a draft, then check its work and fix the mistakes?" The researchers tested this too. They had the robot try to "gate" (drop bad claims), "revise" (rewrite bad claims), or "reattribute" (move citations).

The result? The robot got slightly better at supporting the claims it did make, but its overall ability to find the right sources didn't improve. In fact, trying to fix things after the fact often made the robot cite fewer sources, which lowered its overall score. The paper suggests that repairing a report after it's written is too late; the damage is done if the right sources weren't found or read enough in the first place.

The "Recipe" for a Better Robot

So, what's the practical takeaway for building a robot that runs on your laptop?

First, give it more to read. Let the robot see 1,500 characters of every source it finds. This is cheap (it only costs about 235 extra tokens of computer memory) and it boosts the robot's truthfulness from 45% to 58%.
Second, fix the search. Once the robot is reading enough, the only thing left to fix is the search engine. You need to make sure the robot actually finds the right books before it starts reading.

The Bottom Line

This study didn't create a perfect robot. It found that a small robot on a laptop can be surprisingly good at telling the truth if it's allowed to read enough of the source material. But it also proved that no amount of reading will help if the robot is searching the wrong library.

The authors are confident about the "reading more" part (it worked consistently across different tests and even with a second, independent judge). They are also confident that "fixing it later" doesn't work well. But they admit that the absolute numbers are still modest—the robot is getting better, but it's not ready to replace a human researcher just yet. It's a map showing us where to look next, not a finished destination.

Technical Summary: On-Device Deep Research at 4B: Exposure Bounds

Problem Statement

Deep research agents are typically evaluated on answer accuracy at frontier model scales, often overlooking two critical constraints for personal device deployment: citation faithfulness and hardware cost. A deployable agent must not only generate accurate reports but also ensure that every cited claim is actually supported by the source it references (faithfulness) and that it cites the correct sources (coverage). Current literature reports citation quality as a single metric, obscuring the distinction between an agent supporting the sources it chooses to cite versus an agent citing the correct sources. Furthermore, the cost of running these agents on personal hardware (e.g., a 24 GB laptop) and the specific design levers that improve faithfulness for small models (below 8B) remain unmeasured.

Methodology

The study isolates the variables affecting citation quality using a fixed, deployable setup:

Model & Hardware: A single 4B parameter generator (Qwen3-4B-Instruct-2507) running on an Apple M4 Pro with 24 GB RAM.
Substrate: A corpus of 1,062 documents derived from 63 recent arXiv papers in ML and language technology. The "gold" evidence for each task is the reference list of the source paper itself.
Pipeline: The agent plans queries, performs hybrid retrieval (BM25 + dense), and drafts a cited brief.
Experimental Design: A $2 \times 4$ $2 \times 4$ grid crossing two factors:
1. Source Quality: Oracle (handed the gold papers directly) vs. Naive (handed retrieved papers).
2. Exposure: The number of characters of each source visible to the generator during drafting (400, 800, 1200, 1500).
Metrics: The study separates Cited Claim Faithfulness (does the cited source support the claim?) from Trustworthy Coverage (faithfulness weighted by citation recall). A DeBERTa-v3 NLI judge scores entailment, with results verified by a second independent judge (HHEM-2.1-Open).
Repair Panel: A secondary experiment tested post-generation repair actions (gating, revising, reattributing) to see if they could recover lost grounding.

Key Results

The study identifies two distinct levers that act on different outcomes:

Exposure Bounds Faithfulness:
- Increasing the exposure of each source from 400 to 1500 characters significantly lifts faithfulness.
- Naive (Retrieved) Sources: Faithfulness rose from 0.45 to 0.58.
- Oracle (Gold) Sources: Faithfulness rose from 0.37 to 0.58.
- Convergence: At high exposure (1500 chars), the faithfulness of the naive arm converges with the oracle arm (difference $\approx 0.001$ ). This indicates that faithfulness is limited by how much of the source the model sees, not by whether the source is the "correct" gold paper.
- Cost: This lift costs approximately 235 additional output tokens per task, a negligible fraction of the budget.
Retrieval Bounds Coverage:
- Trustworthy Coverage remains low and flat across exposure levels for the naive arm (hovering near 0.22).
- This is because citation recall is determined at the retrieval stage (held near 0.40) before the generator writes. Increasing exposure cannot fix the fact that the correct sources were never surfaced.
- The Oracle arm achieves higher coverage (up to 0.33) simply because it bypasses the retrieval bottleneck, not because of exposure.
Ineffectiveness of Post-Generation Repair:
- Repair mechanisms (gating, revising) improved faithfulness by dropping unsupported claims but simultaneously reduced citation recall.
- Consequently, Trustworthy Coverage did not improve; it remained near 0.19–0.20, confirming that repair cannot recover grounding lost during retrieval or initial generation.
Robustness:
- The exposure lift and the convergence of arms were replicated using a second, independent judge (HHEM), confirming the findings are not artifacts of a single scoring model.

Significance and Claims

The paper claims to provide a "map of where reliability must come from" for on-device research agents, rather than a deployable system ready for immediate use. Its primary contributions are:

Decoupling Metrics: It demonstrates that reporting a single citation score is misleading. High faithfulness with low recall indicates an agent confidently citing the wrong sources. The study advocates for reporting the triple: faithfulness, recall, and trustworthy coverage.
The Exposure Lever: For small models (4B), the primary bottleneck for faithfulness is input exposure, not model capacity. A fixed 4B model can support claims on retrieved sources as well as gold sources if given sufficient context (1500 chars).
The Retrieval Lever: Coverage is strictly bound by the retrieval stage. No amount of generation tuning or exposure can compensate for missed retrieval recall.
Practical Recipe: The study proposes a specific build order for deployable agents:
1. First: Increase per-source exposure (cheap, effective for faithfulness).
2. Second: Improve retrieval recall (the only remaining lever for trustworthy coverage).
Modesty: The authors explicitly state that the absolute performance levels are modest (faithfulness ~0.58, coverage ~0.22–0.33). The 4B agent is not yet a "reliable citer," but the study clarifies that the path to reliability lies in exposure and retrieval, not in simply scaling model size or adding repair loops.

The paper concludes that for a 4B agent on a laptop, citation quality is not a monolithic problem but a split one: exposure solves the "support" problem, while retrieval solves the "correctness" problem.