💬 NLP

Iterate Until Retrieved: Factual Nugget Optimization for Discoverable Continual Corrections in Agentic RAG

This paper introduces Iterative Nugget Optimization (INO), an index-time method that transforms free-form factual corrections into optimized "factual nuggets" by iteratively refining them through production RAG agent testing to ensure their discoverability and effective usage in B2B knowledge-assistance systems.

Original authors: Moshe Hazoom, Gal Patel, Alon Talmor, Tom Hope

Published 2026-05-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Moshe Hazoom, Gal Patel, Alon Talmor, Tom Hope

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you work for a very smart, helpful robot assistant that answers questions about a company's products. Sometimes, the robot gets a fact wrong. A human user might say, "Actually, that's not right. The password reset only works for managers, not regular employees."

In the past, fixing this was tricky. You couldn't just edit the robot's massive instruction manual because that might break other things, and the robot might not "hear" the new rule if it's buried in a huge document.

This paper introduces a new way to teach the robot: Iterative Nugget Optimization (INO). Think of it as a "try, fail, and tweak" system that turns a user's correction into a tiny, super-durable fact card called a "factual nugget."

Here is how it works, using simple analogies:

1. The Problem: The "Needle in a Haystack"

Imagine the robot's knowledge base is a giant library with millions of books (documents). When a user asks a question, the robot has to find the right book instantly.

The Issue: If you just write a new fact card and drop it into the library, the robot might miss it. The user might ask, "How do I reset a password?" but the new card is titled "SSO Reset Policy for Managers." The robot might not connect the two because the words don't match perfectly.

2. The Solution: The "Factual Nugget"

Instead of rewriting the whole library, the system creates a factual nugget.

What is it? A tiny, self-contained note (like a sticky note) that says exactly one corrected fact.
The Goal: This note needs to be so clear and well-labeled that no matter how a user asks the question later, the robot will find it immediately.

3. The Magic Trick: "The Test Drive" (INO)

This is the core innovation. Instead of just writing the note and hoping for the best, the system puts the note through a test drive before it goes live.

Think of INO as a rehearsal room:

Create a Draft: The system takes the user's correction and writes a draft nugget.
The Test Run: It puts this draft into the robot's library and asks the robot a series of questions. It doesn't just ask the original question; it asks many different versions (e.g., "How do I change my password?", "Can I reset my login?", "Who can reset passwords?").
The "Fail" Check: If the robot doesn't find the nugget, or finds the wrong book instead, the system doesn't give up.
The Reflection: An AI "coach" looks at why the robot failed. Did the title sound too technical? Did it miss a keyword?
The Rewrite: The coach rewrites the nugget to make it easier to find. Maybe it adds a synonym or clarifies the title.
Repeat: The system tests the new version again. It keeps doing this loop (up to 3 times) until the robot always finds the nugget when asked.

4. The Results: Why It Matters

The researchers tested this on real-world business robots (one that answers customer questions, another that helps support engineers).

The Baseline: Without this test-drive loop, the robot only found the new fact about 50-60% of the time.
With INO: The robot found the new fact 77-97% of the time.
The Impact: Because the robot found the fact, it gave the correct answer much more often. It didn't just memorize the exact question; it learned to recognize the idea behind the question, even if the words were different.

5. Real-World Status

This isn't just a theory. The paper states that this system is already running in production for their customers. It processes hundreds of these "nuggets" every week, fixing errors automatically without needing a human to manually edit the robot's brain.

In short: The paper teaches us how to turn a single user's correction into a "super-fact" that is guaranteed to be found by the robot, by letting the robot practice finding it over and over until it gets it right.

Technical Summary: Iterative Nugget Optimization for Discoverable Continual Corrections in Agentic RAG

1. Problem Statement

In complex B2B (business-to-business) settings, Agentic Retrieval-Augmented Generation (RAG) systems frequently receive free-form user feedback. While many feedback signals are vague (e.g., "not helpful") or subjective (e.g., tone), a critical subset consists of actionable factual corrections. These corrections address specific product behaviors, role restrictions, naming clarifications, or policy details that should be available to future users.

The core challenge is converting these corrections into factual nuggets: short, self-contained knowledge-base entries that are:

Conservative: Encoding the corrected fact without introducing new errors.
Standalone: Including sufficient context for reuse without relying on the original conversation.
Discoverable: Matching future queries that may share little lexical overlap with the original feedback.
Precise: Avoiding over-broadness that leads to retrieval for unrelated traffic.

Standard approaches that simply summarize feedback or inherit the original interaction's wording often fail because future users ask from different roles, mention different issues, or use distinct vocabulary. Furthermore, directly editing existing customer documents is often infeasible due to validation difficulties and the risk of disrupting multi-step procedures.

2. Methodology: Iterative Nugget Optimization (INO)

The paper proposes Iterative Nugget Optimization (INO), an index-time optimization method that treats nugget construction as a measurable problem solved by using the production Agentic RAG system itself as a test harness.

The INO Loop

Extraction: An LLM-based filter identifies actionable factual corrections from user feedback streams.
Initialization: A candidate nugget is created (typically including the correction and a paraphrase of the original query).
Probing: The candidate nugget is inserted into a frozen copy of the customer corpus. The system then runs a "probe set" consisting of the original query ( $q_0$ ) and 3–5 held-out paraphrases generated specifically for evaluation.
Reflection & Revision: If the nugget fails to be retrieved or cited for any probe query, an LLM reflector analyzes the failure trace. It inspects:
- The failed query.
- The retrieved competing documents.
- The generated answer.
- The current nugget representation.
  The reflector then revises the nugget's title, body wording, and retrieval anchors to improve discoverability, strictly preserving the factual correction without adding new claims.
Iteration: This process repeats for a maximum of $T=3$ iterations until the nugget is successfully retrieved and cited.

Key Distinctions

Unlike prompt optimization methods that adjust system instructions to amortize changes across many tasks, INO amortizes work across future queries regarding one specific factual correction by optimizing the retrievable artifact itself. This keeps the production agent configuration fixed, making the optimization local, auditable, and reversible.

3. Experimental Setup

The authors evaluated INO on real production data from two distinct B2B agents across multiple companies:

Chat Agent: Answers free-form support questions over customer-specific knowledge bases (KBs).
Support Ticket Agent: Reads support ticket transcripts and suggests responses.

Evaluation Metrics:

Retrieval: Whether the nugget passes the re-ranker and reaches the generator.
Citation: Whether the nugget is cited in the final answer's sources.
Answer Compliance: An LLM judge (calibrated against human annotators) checks if the generated answer addresses the original correction, adheres to the nugget's facts, avoids regression, and maintains groundedness.

Baselines:
The study compared INO against four other strategies:

Standard: Indexing the nugget as-is.
Trigger-Query Anchor: Appending a paraphrase of the original query.
Synthetic Anchors: Appending LLM-generated future-user queries.
Combined: Appending both trigger and synthetic anchors.

4. Key Results

INO consistently outperformed all baselines across both agents and evaluation types.

Discoverability: On held-out queries for the Chat Agent, INO improved retrieval by 25–29 percentage points over the standard nugget baseline (77.3% vs. 52.3%).
Usage: Citation rates also saw significant gains, with INO reaching 68.1% compared to 41.3% for the baseline.
Generalization: The performance gap widened on "real" historical queries (which are harder than synthetic LLM paraphrases), indicating INO generalizes well to unseen phrasings rather than merely memorizing the original query.
Support Tickets: The gains were even more pronounced for the Support Ticket Agent (where inputs are noisy transcripts), with INO improving retrieval by 43.1 points (78.2% vs. 35.1%) and citation by 40.7 points over the baseline.
Answer Quality: In answer-level evaluations, INO increased the share of answers that addressed the requested correction from 52.2% to 73.4%. It also reduced major regression errors and hallucinated groundedness labels.
Precision: A negative control test on 1,400 random queries showed a low false-positive rate (1.4% retrieval), with manual inspection confirming these were semantically equivalent paraphrases rather than noise.

5. Significance and Claims

The paper claims that INO provides a practical, scalable solution for continual correction in production RAG systems. Its significance lies in:

Operational Viability: The optimization is offline, costing approximately $0.31 per nugget and taking under two minutes, with zero added latency for end-users after indexing.
Production Deployment: The system is currently live, generating hundreds of retrieved nuggets weekly across customer deployments.
Local Optimization: By optimizing the artifact rather than the agent's global configuration, the method ensures that corrections are atomic and do not inadvertently degrade performance on unrelated tasks.

The authors maintain modesty regarding limitations, noting that results are currently English-only and dependent on the specific hybrid agentic RAG architecture used. They acknowledge that different retrieval architectures (pure dense or pure sparse) might respond differently to the proposed optimization.