KARL: Knowledge Agents via Reinforcement Learning

The paper introduces KARL, a system that leverages multi-task reinforcement learning and an agentic synthesis pipeline to train enterprise search agents achieving state-of-the-art, Pareto-optimal performance across diverse, hard-to-verify search tasks while surpassing leading closed models with sufficient compute.

The Gist

Imagine you have a brilliant but very literal librarian named KARL. Before this paper, librarians like KARL were great at reading books they already knew, but if you asked them to find a specific fact hidden inside a massive, messy warehouse of millions of documents (like a company's internal notes or a century of medical journals), they would often get lost, give up, or make things up.

This paper introduces KARL, a new kind of "Knowledge Agent" that has been trained to be the ultimate detective in a library. Here is how they did it, explained simply:

1. The Problem: The "Lost in the Stacks" Librarian

Most AI models are like students who memorized a textbook. If you ask a question about the textbook, they ace it. But if you ask them to find a needle in a haystack of new information they've never seen, they struggle. They might:

• Give up too early: "I can't find it, I'm done."
• Search blindly: Read the same 10 pages over and over.
• Get overwhelmed: The library is too big, and they forget what they read 10 minutes ago.

2. The Solution: KARL's Training Camp

The researchers at Databricks didn't just tell KARL to "try harder." They built a special training camp with four main ingredients:

A. The "Gym" (KARLBench)

Instead of just practicing one type of puzzle, KARL trained in a massive gym with six different obstacle courses:

• The Constraint Course: Find one specific person who fits 5 different weird rules (e.g., "born in a city with a tall tower, likes cats, and wrote a book in 1972").
• The Synthesis Course: Read 50 different medical papers and write one clear report.
• The Math Course: Find numbers in a 100-page financial report and do the math.
• The "Needle" Course: Find every mention of a specific topic in a huge encyclopedia.
• The "How-To" Course: Read technical manuals to fix a broken computer code.
• The "Messy Notes" Course: Find facts hidden in informal, messy meeting notes.

The Lesson: By training on all these different types of puzzles, KARL learned general search skills, not just how to solve one specific riddle.

B. The "Self-Playing Video Game" (Agentic Synthesis)

Usually, humans have to write thousands of practice questions for AI. That's slow and expensive.
Instead, the researchers built a robot teacher. This robot:

1. Went into the library and found interesting documents.
2. Created its own difficult questions based on those documents.
3. Asked the "student" (KARL) to solve them.
4. If the student got it right, the robot kept the question. If the student failed or the question was too easy, the robot tossed it.
5. The Magic: As KARL got smarter, the robot teacher got smarter, creating even harder questions. It was like a video game where the levels get harder automatically as you level up.

C. The "Coach" (Reinforcement Learning)

This is the secret sauce. KARL didn't just read answers; it played a game of trial and error.

• The Rule: Every time KARL found the right answer, it got a "point." Every time it wasted time or got lost, it lost a point.
• The Result: KARL learned to be efficient. It learned to stop searching when it had enough info, to summarize what it read so it wouldn't forget, and to try different search strategies when one failed. It learned to "think before it speaks."

D. The "Parallel Brain" (Test-Time Compute)

Sometimes, even the best detective needs a second opinion.
When KARL faces a really hard question, the researchers let it run 10 different versions of itself at the same time.

• Version A searches for the answer using Strategy 1.
• Version B uses Strategy 2.
• Version C uses Strategy 3.
• Then, a "Manager" reads all 10 answers, picks the best parts of each, and combines them into one perfect answer.
• Analogy: It's like asking 10 different experts to solve a mystery, then having a moderator sit down and write the final report using the best clues from all 10.

3. The Results: The Pareto Frontier

The paper compares KARL to the most famous, expensive AI models (like GPT-5 and Claude Opus).

• Cost: KARL is much cheaper. It's like buying a high-performance sports car that gets 50 miles per gallon, while the others are gas-guzzling luxury cars.
• Speed: KARL is faster.
• Quality: With enough "parallel thinking" (running multiple versions), KARL actually beats the most expensive, closed-source models.

The Big Takeaway

The paper proves that you don't need a bigger, more expensive brain to be smarter. You need a better training method.

By teaching an AI to:

1. Practice on diverse, hard problems,
2. Generate its own practice tests,
3. Learn from its mistakes (Reinforcement Learning), and
4. Think in parallel when it's stuck,

...you can create an agent that is cheaper, faster, and smarter than the current giants, specifically for tasks that require digging through real-world data.

In short: KARL isn't just a smart librarian; it's a librarian who knows how to study, how to manage a team, and how to never give up until the book is found.

Technical Summary

1. Problem Statement

Modern enterprise applications require Knowledge Agents capable of "grounded reasoning"—the ability to iteratively query, retrieve, and reason over large, proprietary data collections (e.g., internal notes, financial reports, medical records) that are not present in the model's pre-training weights.

Current challenges include:

• Task Fragmentation: Existing benchmarks (e.g., HotpotQA, FinanceBench) only capture narrow slices of agent behavior. A model optimized for one type of search (e.g., entity lookup) often fails at others (e.g., cross-document synthesis).
• Data Scarcity: High-quality, difficult, and grounded training data is hard to generate using static prompting or simple synthesis.
• Training Instability: Training large-scale Mixture-of-Experts (MoE) models with online Reinforcement Learning (RL) is computationally expensive and unstable, often requiring complex heuristics to handle discrepancies between training and inference engines.
• Cost and Latency: State-of-the-art closed models (e.g., GPT-5, Claude Opus) are often too expensive or slow for enterprise-scale deployment.

2. Methodology

The authors propose KARL, a system that combines specialized synthetic data creation, multi-task reinforcement learning, and test-time compute scaling.

A. KARLBench: A Multi-Capability Evaluation Suite

To rigorously evaluate grounded reasoning, the authors introduced KARLBench, a suite spanning six distinct search regimes:

1. Constraint-driven entity search: Finding a single entity satisfying multiple distributed attributes (e.g., BrowseComp-Plus).
2. Cross-document report synthesis: Integrating dispersed findings into a coherent report (e.g., TREC-Biogen).
3. Tabular numerical reasoning: Navigating long financial documents to extract and calculate numbers (e.g., FinanceBench).
4. Exhaustive entity retrieval: Finding all entities satisfying a condition (e.g., QAMPARI).
5. Procedural reasoning: Deriving step-by-step solutions from technical documentation (e.g., FreshStack).
6. Fact aggregation over internal notes: Synthesizing information from noisy, unstructured enterprise notes (PMBench).

B. Agentic Synthesis Pipeline

To overcome data scarcity, the authors developed an iterative, self-improving data synthesis pipeline:

• Stage I (Question-Answer Synthesis): An agent explores the corpus using vector search to generate diverse, difficult, and grounded question-answer pairs.
• Stage II (Solution Synthesis & Filtering): Multiple "Solver Agents" attempt to answer the synthetic questions. The pipeline filters out questions that are too easy (all attempts correct) or too hard/ambiguous (all attempts incorrect). A Quality Filter Agent removes factual errors and ambiguities.
• Bootstrapping: The process iterates, using the improved KARL model to synthesize higher-quality data for subsequent training rounds.

C. OAPL: Iterative Large-Batch Off-Policy RL

The core training algorithm is OAPL (Optimal Advantage-based Policy Optimization with Lagged Inference).

• Off-Policy Design: Unlike online RL (e.g., GRPO) which requires constant interaction, OAPL uses large-batch off-policy training. It generates a massive dataset of rollouts from a reference model ($\pi_{ref}$) and trains the policy ($\pi$) to minimize a least-squares regression loss against the optimal advantage.
• Robustness: This approach is robust to discrepancies between the trainer and the inference engine (e.g., vLLM), eliminating the need for heuristics like clipped importance weighting or router replay.
• Multi-Task Training: The framework naturally extends to multi-task training by combining losses from different regimes (e.g., BrowseComp-Plus and TREC-Biogen), fostering out-of-distribution (OOD) generalization.

D. Test-Time Compute (TTC) Scaling

KARL leverages compute at inference time to boost performance:

• Parallel Thinking: Generates $N$ independent rollouts in parallel and uses a generative aggregator to synthesize a final answer, often outperforming simple voting.
• Value-Guided Search (VGS): Trains a value model to predict the success probability of partial rollouts, guiding a tree search to select the most promising branches.

E. Agent Infrastructure

The system uses a custom harness ("aroll") featuring:

• Embedded Vector Search: An in-process vector database to achieve high throughput (>500 QPS) and eliminate network I/O latency during data generation.
• Context Compression: An RL-trained mechanism where the agent summarizes its own interaction history when context limits are reached, enabling long-horizon reasoning without losing salient information.

3. Key Contributions

1. KARLBench: A comprehensive benchmark covering six diverse search regimes, demonstrating that training on heterogeneous behaviors yields better generalization than single-task optimization.
2. Agentic Synthesis Pipeline: A method for generating diverse, grounded, and high-quality training data via iterative bootstrapping from increasingly capable models.
3. OAPL Algorithm: A new post-training paradigm based on iterative large-batch off-policy RL that is sample-efficient, stable for large MoE models, and naturally supports multi-task training.
4. Pareto-Optimal Performance: The resulting agent achieves state-of-the-art performance on grounded reasoning tasks while significantly outperforming competitors in cost and latency trade-offs.

4. Results

Evaluated against top-tier proprietary models (Claude 4.6, GPT-5.2) and open-source baselines (GLM 4.5 Air, Qwen 3.5):

• Performance: KARL achieves Pareto-optimal results on KARLBench.
  • Cost: KARL achieves competitive scores at <$0.10 per query, significantly cheaper than closed models. With parallel sampling (10 traces), it matches Claude Opus 4.6 quality at roughly 33% lower cost.
  • Latency: KARL is the fastest model among those scoring >55 points. With 10 parallel traces, it matches Opus 4.6 quality at 47% lower latency.
  • Quality: KARL surpasses Claude Sonnet 4.6 with 3 parallel rollouts and matches Opus 4.6 with 10 rollouts. It also generalizes well to OOD tasks (e.g., FreshStack, PMBench) not seen during training.
• Generalization: Multi-task RL training leads to better OOD generalization compared to Multi-Expert Distillation (SFT). While distillation improves in-distribution performance, it fails to scale on OOD tasks, whereas KARL improves consistently across both.
• Behavioral Shifts: RL training transforms agent behavior:
  • Efficiency: KARL solves tasks in fewer steps and with less token overhead.
  • Diversity: It retrieves a wider variety of documents (37% more unique docs on BrowseComp-Plus).
  • Commitment: KARL learns to stop searching once sufficient evidence is gathered, avoiding the "exhaustive search, no convergence" behavior seen in base models.
  • Capability Expansion: Analysis of max@k curves shows RL expands the model's problem-solving coverage (solving previously unsolvable tasks) rather than just sharpening existing distributions.

5. Significance

• Enterprise Viability: KARL demonstrates that tailored synthetic data combined with multi-task RL can create cost-efficient, high-performing knowledge agents suitable for real-world enterprise deployment, where data is proprietary and tasks are complex.
• Algorithmic Advancement: The success of OAPL suggests that off-policy RL is a viable and superior alternative to online RL for training large-scale agentic systems, offering stability and scalability without complex infrastructure heuristics.
• Beyond Sharpening: The paper provides strong evidence that RL post-training on agentic tasks genuinely expands model capabilities (learning new search strategies and reasoning patterns) rather than merely increasing the probability of correct answers the model already knew.
• Scalability: The results show that with sufficient test-time compute, open-source agents trained via this methodology can surpass the strongest closed models, challenging the notion that only massive proprietary models can handle complex grounded reasoning.