💻 computer science

Caesar: Deep Agentic Web Exploration for Creative Answer Synthesis

Caesar is a novel agentic LLM architecture that leverages knowledge graphs for associative reasoning and an adversarial refinement loop to transition from passive information retrieval to the creative synthesis of novel insights, significantly outperforming existing agents in generating original and coherent answers.

Original authors: Jason Liang, Elliot Meyerson, Risto Miikkulainen

Published 2026-04-24

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jason Liang, Elliot Meyerson, Risto Miikkulainen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a mystery, like figuring out how to build a better coffee machine.

The Old Way (Current AI Agents):
Most AI agents today act like a very fast, very obedient librarian. You ask them, "How do I make better coffee?" They run to the shelves, grab the top three books that say "coffee," read the first page of each, and hand you a summary. They are fast and accurate, but they only give you what they already know is popular. They don't go into the basement to find a dusty book about fluid dynamics that might revolutionize how water flows through a filter. They stick to the "safe" answer.

The New Way (Caesar):
Caesar is different. It doesn't just act like a librarian; it acts like a detective with a super-powered map.

Here is how Caesar works, broken down into simple steps:

1. The Detective's Map (The Knowledge Graph)

Instead of just reading one book at a time, Caesar draws a giant, living map as it explores the internet.

The Old Way: Reads Book A, then Book B. They are separate.
Caesar's Way: Reads Book A, sees a link to Book B, and draws a line connecting them. Then it sees Book C mentions something in Book A, so it draws a line there too.
The Magic: Soon, Caesar has a web of connections. It can see that "Coffee" is connected to "Fluid Dynamics" and "Music Theory" (because of rhythm in brewing). It finds the hidden bridges between ideas that no one else noticed. This is how it finds creative answers, not just summaries.

2. The Two-Phase Mission

Caesar has a strict two-step process to ensure it doesn't just ramble.

Phase 1: The Deep Dive (Exploration)
Caesar puts on its detective hat and starts wandering the web.

It has a "Memory Stack": Imagine a stack of plates. When Caesar goes down a rabbit hole (a specific topic), it puts a plate on the stack. If it hits a dead end, it takes the plate off and goes back to the last interesting spot to try a different path. This prevents it from getting lost or going in circles.
It changes its personality: If you ask about cooking, Caesar becomes a chef. If you ask about coding, it becomes a programmer. It adapts its brain to fit the job.
The Goal: It keeps digging until it finds those weird, non-obvious connections (like the coffee/fluid dynamics link).

Phase 2: The Devil's Advocate (Synthesis)
Once Caesar has all its clues, it doesn't just write a report. It plays a game of "Devil's Advocate" with itself.

Draft 1: It writes a first answer.
The Attack: It then acts as a harsh critic. "Wait, this part is boring. This part contradicts that other fact. Why didn't we think of this angle?"
The Fix: It asks new, tricky questions to force itself to find better answers.
The Merge: Finally, it takes all the different drafts—the good parts of the first one, the wild ideas from the second, and the corrections from the third—and smashes them together into one perfect, coherent story.

3. The Result: "Surprising" Answers

Because Caesar does this deep digging and self-critique, its answers are New, Useful, and Surprising.

New: It doesn't just repeat what's on the first page of Google. It finds the "long-tail" ideas.
Useful: It checks its work so the answer actually makes sense.
Surprising: It connects dots that seem unrelated, like explaining cooking using the math of calculus.

A Real-Life Example from the Paper

The paper tested Caesar by asking: "Apply the math of calculus to cooking."

A Normal AI would say: "Cooking is like math because you add ingredients (addition) and heat things up (multiplication)." (Boring, obvious).
Caesar looked at its map, saw connections between "flow rates" in pipes and "heat transfer" in pans. It built a complex, brilliant model where a recipe is a machine with "knobs" (temperature, time) and "gauges" (taste, texture). It explained how changing a knob slightly (a derivative) affects the whole dish, and how the total flavor is the sum of every second of cooking (an integral).

In short:
Current AI is a photocopier that makes perfect copies of existing ideas.
Caesar is a combinatorial artist that takes pieces of different ideas, glues them together in new ways, and creates something that never existed before. It trades speed for depth, turning a simple search into a creative discovery.

1. Problem Statement

Current autonomous web agents and Large Language Model (LLM) research frameworks are primarily designed for passive retrieval and convergent search. They typically rely on linear interaction histories (e.g., ReAct loops) or standard Retrieval-Augmented Generation (RAG), treating the internet as a flat sequence of disconnected documents.

Limitations: These approaches suffer from "navigational amnesia," leading to redundant loops and generic summaries. They optimize for precision in answering known questions but fail at open-ended inquiry and creative discovery.
The Gap: They lack the ability to perform deep, associative reasoning required to bridge disparate conceptual islands, resulting in derivative outputs that lack novelty, structural coherence, and the "surprise" element essential to true creativity.

2. Methodology: The Caesar Architecture

Caesar is an agentic LLM architecture designed to shift the objective from mechanical retrieval to graph-based discovery. It operates through a two-phase cognitive cycle:

Phase 1: Deep Web Exploration (Knowledge Collection)

Instead of linear scraping, Caesar treats exploration as a stateful graph traversal problem.

Dynamic Knowledge Graph ( $G$ ): The agent constructs a topological graph where nodes represent visited URLs and edges represent navigational transitions. This graph serves as a "topological memory" to track the exploration path.
Perceive-Think-Act Loop:
- Perceive: Extracts content from the current URL, filtering noise and handling anti-bot measures.
- Think (Graph-Augmented Insight Generation): Before summarizing a page, the agent retrieves the semantic state of the local neighborhood (predecessors and neighbors) from the graph. It uses this context to identify how new information builds upon or contradicts existing insights, fostering associative reasoning.
- Act (Knowledge-Guided Exploration): A dynamic policy ( $\pi$ $π$ ) determines the next action based on a composite context of the knowledge base and episodic memory. It selects between three meta-strategies:
  - EXPLORE: Deepen current topic (Depth-First).
  - BACKTRACK: Return to parent nodes to escape stagnation or loops.
  - WEB_SEARCH: Pivot to new areas if local information is saturated.
Domain-Specific Role Adaptation: Before exploration, the agent dynamically rewrites its system prompt to adopt a persona specifically tuned to the query's domain, overcoming safety-biased generic responses.

Phase 2: Adversarial Artifact Synthesis (Answer Generation)

This phase moves beyond single-pass RAG to emulate a recursive human research process.

Recursive Insight Discovery: The agent generates a chain of inquiries ( $q_t \to a_t \to q_{t+1}$ ) to identify gaps and contradictions in the retrieved knowledge, ensuring a logical chain of reasoning rather than a bag of disjoint facts.
Adversarial Refinement Loop:
- The agent generates an initial draft ( $A_k$ ).
- It then analyzes the draft to identify narrative weaknesses and formulates adversarial queries specifically designed to target these gaps or contradictions.
- This process repeats for $N$ rounds, pushing the agent out of the "basin of attraction" of generic LLM outputs.
Generative Merge: All draft versions are consolidated into a single cohesive artifact. The LLM performs a high-level synthesis to integrate the most relevant insights, resolving tensions between divergent perspectives.
Post-Processing (ELI5): The final artifact is distilled into an "Explain Like I'm 5" (ELI5) summary, decoupled from the core synthesis to ensure readability without compromising citation integrity.

3. Key Technical Innovations

Domain-Specific Role Adaptation: Dynamically rewrites system prompts to align with domain constraints, bypassing RLHF-induced genericity.
Graph-Augmented Insight Generation: Conditions information extraction on the local topology of the exploration graph, enabling the detection of non-obvious connections between disparate concepts.
Knowledge-Guided Exploration Policy: Uses a dynamic policy to switch between depth-first expansion and strategic backtracking, preventing infinite loops and maximizing information gain.
Adversarial Artifact Synthesis: Replaces static summarization with an active gap-analysis loop that formulates orthogonal queries to refine drafts, ensuring the final output is novel and structurally sound.

4. Experimental Results

Caesar was evaluated against state-of-the-art agents (including deep research modes of GPT-5.2, Claude Sonnet 4.5, and Gemini 3) using a blinded LLM-as-a-Judge framework.

Evaluation Metrics: Answers were scored on the NUS criteria: New (Novelty/Rarity), Useful (Viability), and Surprising (Non-obvious connections).
Performance:
- Caesar significantly outperformed all baselines across all settings (Full Answers, ELI5, and Length-Constrained ELI5).
- In the unconstrained Full Answer setting, Caesar achieved a total score of 25.29, surpassing the runner-up (Gemini 3 Deep) by 3.02 points.
- It showed the highest gains in New (8.64) and Surprising (8.27) metrics, demonstrating superior creative capability.
Ablation Studies:
- Exploration Depth: Reducing the exploration budget (iterations) from 1000 to 250 caused a significant drop in "Surprising" and "New" scores, confirming that deep topological traversal is necessary for high-value insights.
- Synthesis Rounds: Adversarial refinement increased "Surprising" scores but temporarily lowered "Useful" scores; the final Generative Merge successfully recovered utility while retaining novelty.

5. Significance and Conclusion

Paradigm Shift: Caesar demonstrates that moving from linear retrieval to graph-based associative reasoning allows agents to discover non-obvious connections, effectively automating the combinatorial process of creative hypothesis generation.
Creativity as a System: The paper argues that creativity in AI requires System 2 thinking (deliberate, structured reasoning) rather than stochastic sampling. By explicitly seeking gaps and bridges between distant insights, Caesar aligns with theories of creativity as the discovery of non-obvious semantic connections.
Future Impact: The architecture provides a framework for transforming web search agents from passive summarizers into active researchers capable of generating artifacts that are not just informative, but genuinely innovative. Future work envisions extending this to multi-agent swarms for solving complex benchmarks like ARC-AGI.

In summary, Caesar bridges the gap between information gathering and the synthesis of new insights by leveraging a dynamic knowledge graph for exploration and an adversarial loop for refinement, setting a new standard for creative AI agents.