🤖 AI

Search Beyond What Can Be Taught: Evolving the Knowledge Boundary in Agentic Visual Generation

This paper addresses the structural world-knowledge bottleneck in visual generation by introducing the SearchGen benchmark and corpus, demonstrating that naive search fails due to indiscriminate retrieval, and proposing a teach-then-search co-training framework that dynamically discovers a generator's knowledge boundary to enable effective agentic visual generation.

Original authors: Haozhe Wang, Weijia Feng, Jinpeng Yu, Che Liu, Ping Nie, Fangzhen Lin, Jiaming Liu, Ruihua Huang, Jimmy Lin, Wenhu Chen, Cong Wei

Published 2026-07-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Haozhe Wang, Weijia Feng, Jinpeng Yu, Che Liu, Ping Nie, Fangzhen Lin, Jiaming Liu, Ruihua Huang, Jimmy Lin, Wenhu Chen, Cong Wei

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Core Problem: The "Confident Faker" Artist

Imagine you hire a brilliant artist who has memorized every book in a massive library. They can draw anything from that library perfectly. But, if you ask them to draw something that happened yesterday (like a new celebrity) or something very specific that never made it into a book (like a local town festival), they don't say, "I don't know." Instead, they confidently make up a fake version that looks real but is completely wrong.

This is the problem with current AI image generators. They are great at rendering (drawing the picture), but they are terrible at knowledge (knowing the facts). They are stuck with a "cutoff date" in their memory. If you ask for the 2025 World Cup standings, they can't know them because their training data stopped before 2025.

The Solution: Giving the Artist a Search Engine

The obvious fix seems simple: "Let's give the artist a search engine so they can look up the facts before drawing."

However, the authors found that naive searching makes things worse.

The Analogy: Imagine asking a chef to cook a steak. If you just dump a pile of random ingredients (search results) onto their counter, they might get confused. They might use a tomato when they didn't need one, or copy the entire photo of a steak you showed them instead of just using it as a reference for the color.
The Result: The AI gets "noise." It starts copying the wrong things or getting confused by information it already knew, leading to worse pictures than if it had just tried to remember on its own.

The Discovery: The "Knowledge Boundary"

The paper introduces a clever concept called the Knowledge Boundary. Think of the AI's brain as a house with two rooms:

The Memory Room (Internal): Things the AI has learned and can draw from memory (e.g., what a cat looks like).
The Reference Room (External): Things the AI must look up because they are too new, too rare, or too specific (e.g., the specific outfit of a character from a game released last week).

The problem is: The AI doesn't know where the wall between these two rooms is.

If it searches for things it already knows, it gets confused (noise).
If it doesn't search for things it doesn't know, it fakes the answer.

The Fix: A Two-Step "Teach-Then-Search" Team

The authors created a system with two parts working together:

The Reasoner (The Manager): An AI agent that decides when to search and what to search for.
The Generator (The Artist): The AI that actually draws the picture.

They used a special training method called Co-Training to teach them how to work together. It happens in two phases:

Phase 1: Teach the Artist (Expand the Memory)
First, they show the Artist examples of things it didn't know, along with the correct search results. The Artist learns to memorize these new things.

Result: The "Memory Room" gets bigger. The Artist now knows more things without needing to search.

Phase 2: Train the Manager (Refine the Search)
Now that the Artist knows more, the Manager needs to learn to stop searching for those new things. If the Manager keeps searching for things the Artist already knows, it causes confusion.

Result: The Manager learns to say, "I know the Artist can handle this one, no need to search!" and only searches for the truly hard stuff.

The Analogy: The Apprentice and the Master

Imagine a Master Painter (the Generator) and a Research Assistant (the Reasoner).

Before: The Assistant blindly hands the Master a stack of 50 reference photos for every single request. The Master gets overwhelmed and paints a mess.
After Co-Training:
1. The Master studies the photos and learns to paint them from memory.
2. The Assistant learns to look at the Master's new skills. If the Master can paint a "red dragon" from memory, the Assistant doesn't hand over a photo.
3. The Assistant only hands over a photo if the Master is asked to paint a "dragon from a video game released yesterday."

The Results: Why This Matters

The paper tested this on a massive new dataset called SEARCHGEN-20K, which contains 20,000 difficult requests about new events, specific characters, and cultural symbols.

The Gap: Standard AI models scored very low (around 20–28 out of 100) on these new, knowledge-heavy requests. They failed because they tried to fake the facts.
The Improvement: By using this "Teach-Then-Search" method, the AI's score improved significantly.
The Key Finding: The system learned to be selective. It stopped searching when it didn't need to (preventing confusion) and searched when it did (preventing faking).

Summary

The paper argues that we shouldn't just build bigger AI models that try to memorize everything. Instead, we should build smart teams where the AI learns what it can remember and what it must look up. By training the "searcher" and the "drawer" together, the system learns its own limits, stops making up facts, and only uses the internet when it truly needs to.

In short: Don't just give the AI a search engine; teach it when to use it.

Technical Summary: Search Beyond What Can Be Taught: Evolving the Knowledge Boundary in Agentic Visual Generation

1. Problem Statement: The World-Knowledge Bottleneck

Visual generators excel at rendering known concepts but fail catastrophically on requests requiring unbounded, evolving, and long-tailed world knowledge (e.g., recent events, niche cultural symbols, or specific IP characters). This is a structural bottleneck: generators are trained on fixed corpora with inherent knowledge cutoffs, while user requests are open-ended.

The paper identifies that current generators lack a mechanism to recognize what they do not know. Consequently, they "confidently fabricate" answers rather than admitting ignorance or seeking external information. Existing benchmarks fail to capture this dynamic, as they primarily test rendering within known concepts. When tested on search-intensive prompts, frontier open-weight generators collapse, scoring only 21–28 out of 100, a drop of up to 40 points compared to their performance on standard prompts.

A naive remedy—equipping generators with search tools—fails because it introduces search noise. Blindly retrieving information for every prompt degrades quality by injecting irrelevant visual details, stylistic contamination, or spurious structures into the generator's input. The core challenge is not just retrieving knowledge, but defining a generator-specific, evolving knowledge boundary: the divide between what a generator can internalize through training and what must remain in external context.

2. Methodology

The paper proposes SearchGen, a framework for agentic visual generation that employs a noise-resistant agentic reasoner and a co-training framework to discover and respect the knowledge boundary.

2.1 Dataset and Benchmark Construction

To systematically study this problem, the authors constructed:

SEARCHGEN-20K: A dataset of 20,839 world-knowledge-grounded prompts spanning 12 failure categories and 22 domains (e.g., temporal events, entity/IP, cultural specificity). It includes 34,694 visual reference slots and 16,345 textual knowledge slots.
SEARCHGEN-BENCH: A benchmark derived from the dataset to evaluate world-knowledge grounding.
SEARCHGEN-CORPUS-1M: A pre-executed, frozen multimodal corpus containing 145,642 archived search sessions and 370,733 cached downloads. This enables reproducible, offline research without reliance on live APIs or drifting search results.

2.2 Noise-Resistant Agentic Reasoner

To prevent search noise from corrupting generation, the authors introduce a three-stage Gate–Filter–Integrate protocol:

Gate: The reasoner (a VLM) analyzes the prompt to identify knowledge gaps. It classifies gaps by type and severity, triggering search only for "critical" or "important" gaps. It decides the modality (image vs. web text) for each query.
Filter: Upon retrieving results, the reasoner selects references that specifically fill the identified gap while minimizing extraneous content (e.g., filtering out full images to extract only necessary visual features).
Integrate: Instead of passing raw images directly, the reasoner converts visual references into grounded natural language citations (e.g., "render the character in a teal-and-gold robe following Image I"). This preserves the missing knowledge while discarding pixel-level noise that causes style leakage or identity blending.

2.3 Co-Training Framework: "Teach, Then Search"

The authors define the Knowledge Boundary $B(\theta)$ as the partition between internalizable knowledge ( $K_{int}$ ) and contextual knowledge ( $K_{ctx}$ ). This boundary is generator-specific and shifts as the model improves. They propose a two-phase co-training loop to discover this boundary:

Phase 1: Teach (Online Iterative DPO)
The visual generator is trained on search-augmented inputs using Direct Preference Optimization (DPO). The generator samples multiple images per prompt, and preference pairs are constructed based on quality scores. This phase serves two purposes:
1. Internalization: It pushes the boundary outward, allowing the generator to internalize stable visual knowledge that previously required search.
2. Noise Robustness: By training on imperfect search-augmented inputs, the generator learns to incorporate references without being dominated by their noise.
Phase 2: Search (Rejection-Sampling Finetuning)
Once the generator's boundary has shifted, the reasoner is recalibrated. Using Rejection Sampling Finetuning (RFT), the reasoner is trained on trajectories where the search policy improved the output (positive advantage) versus those where it degraded it. The reasoner learns to abstain from searching for concepts the generator has now internalized, effectively "moving the search policy inward" to match the new boundary.

3. Key Results

The paper validates the framework on two open-weight generators (Flux.2-Klein-4B and Bagel-7B) using the SEARCHGEN-BENCH.

Monotonic Improvement: The progression from No Search $\to$ Blind Search (SFT reasoner) $\to$ Generator-Adaptive Search (RFT reasoner) yields consistent, monotonic gains across all difficulty tiers.
Selectivity: The calibrated reasoner successfully recovers performance on prompts that do not require search (NoSearch stratum), where naive search previously caused degradation. For example, on the NoSearch set, the adaptive reasoner improved scores from 49.9 (No Search baseline) to 56.9, whereas Blind Search dropped scores to 60.4.
Generator-Specificity: The optimal search policy is specific to the generator's current capability. A reasoner calibrated for a DPO-strengthened generator performs significantly worse when paired with the base generator, confirming that the boundary is a joint property of the pair.
Performance Gaps:
- Open generators with naive search collapse to ~22–28/100 on search-intensive prompts.
- The co-trained system (8B reasoner + 4B generator) achieves 31.8/100, slightly exceeding a frontier oracle (Gemini-3-Flash) paired with the same generator (31.2/100).
- While still below commercial systems with integrated search (e.g., GPT-Image-2 at 71.2), the framework demonstrates that a small, co-trained agent can approach the performance of much larger, generic reasoners.

4. Significance and Claims

The paper claims that the central challenge in agentic visual generation is not merely retrieving information, but knowing what one does not know.

Discoverable Boundary: The knowledge boundary is not a fixed property of the prompt distribution but a dynamic, discoverable quantity that shifts with generator training.
Co-Training Necessity: The authors argue that search policies cannot be optimized against a fixed generator; the generator must be taught to internalize knowledge first, and the reasoner must then be recalibrated to the new boundary.
Minimalist Recipe: The framework validates that a "minimalist recipe" (one DPO pass, one RFT pass, an 8B reasoner, and a 4B generator) is sufficient to establish this self-improving loop.
Scalability: The work suggests that while larger models may internalize more knowledge, the boundary will never disappear due to the infinite nature of the real world. Co-training provides a mechanism to continuously adapt the search policy to the generator's evolving capacity.

The authors release the full dataset, co-training corpus, and search corpus to serve as a "replayable harness" for future research into tool-augmented, world-knowledge-grounded visual generation.