💬 NLP

MARS: Co-evolving Dual-System Deep Research via Multi-Agent Reinforcement Learning

The paper introduces MARS, a novel multi-agent reinforcement learning framework that co-evolves dual cognitive systems (fast intuition and deliberate reasoning) to overcome the token inefficiency and knowledge limitations of Large Reasoning Models, achieving state-of-the-art performance on knowledge-intensive tasks without supervised fine-tuning.

Original authors: Guoxin Chen, Zile Qiao, Wenqing Wang, Donglei Yu, Xuanzhong Chen, Hao Sun, Minpeng Liao, Kai Fan, Yong Jiang, Penguin Xie, Wayne Xin Zhao, Ruihua Song, Fei Huang

Published 2026-02-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Guoxin Chen, Zile Qiao, Wenqing Wang, Donglei Yu, Xuanzhong Chen, Hao Sun, Minpeng Liao, Kai Fan, Yong Jiang, Penguin Xie, Wayne Xin Zhao, Ruihua Song, Fei Huang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Over-Thinker" and the "Out-of-Date" Library

Imagine you have a brilliant, super-smart assistant (an AI) who is great at solving hard puzzles. However, this assistant has two major flaws:

The Over-Thinker: When you ask them to read a simple news article or summarize a webpage, they sometimes get stuck in "deep thought" mode. They analyze every single word, wasting time and energy on things that don't need deep analysis. It's like using a sledgehammer to crack a nut.
The Out-of-Date Library: This assistant's knowledge stops at the day they were trained. If you ask them about something that happened yesterday, they are clueless. They can't look things up on their own without getting confused by the sheer volume of new information.

The Solution: MARS (The Two-Brain Team)

The researchers created a new system called MARS. Instead of one brain trying to do everything, they built a team of two distinct "personalities" working inside the same AI, inspired by how human brains work:

System 1 (The Fast Scout): This is the "intuitive" brain. It's fast, efficient, and good at scanning. Its job is to look at a mountain of new information (like 10 different web pages or research papers) and quickly pull out only the most important facts. It's like a scout who runs ahead, grabs the useful supplies, and leaves the junk behind.
System 2 (The Deep Thinker): This is the "deliberate" brain. It's slow, careful, and great at solving complex logic puzzles. It takes the clean, distilled facts from System 1 and uses them to solve the actual problem.

The Magic Trick:
In previous systems, the "Scout" and the "Thinker" were trained separately. The Scout didn't know what the Thinker needed, so it might summarize things the Thinker didn't care about, or miss the details the Thinker desperately needed.

In MARS, they co-evolve. They train together. The Scout learns exactly what kind of information helps the Thinker solve the puzzle, and the Thinker learns how to ask the Scout for the right things. They share the same "score" (reward), so they work as a perfect team rather than two strangers.

How They Train: The "Group Game"

To teach these two systems to work together without a human teacher (a method called "Zero RL," meaning they start from scratch with no pre-written examples), the researchers used a clever training game based on Group Relative Policy Optimization (GRPO).

Think of it like a sports team practicing for a championship:

The Team Huddle (Bin-Packing): When the team goes out to gather information, they might find 10 different web pages. That's too much to read at once. The system uses a "Bin-Packing" strategy (like fitting groceries into bags). It organizes these messy, different-sized pages into neat, manageable chunks so the Scout can read them all at the same time without getting overwhelmed.
The Scoreboard (Shared Rewards): The team goes out and tries to solve a problem. If they get the answer right, both the Scout and the Thinker get a point. If they fail, neither gets a point. This forces them to cooperate.
Fair Play (Decoupled Gradients): Here is the tricky part. Even though they share the score, the researchers made sure the Scout gets credit for summarizing well and the Thinker gets credit for reasoning well. It's like a relay race: if the team wins, both runners get a medal, but the coach knows exactly who ran the first leg and who ran the second. This ensures the Scout learns to be a better Scout, not just a better Thinker.
Balancing the Team (Balanced Sampling): Sometimes the Scout has to read 5 pages, and other times only 1. This creates an imbalance in training data. The system uses a special sampling method to make sure the Scout and the Thinker get equal practice time, so one doesn't dominate the learning process.

The Results: Small Team, Big Wins

The researchers tested MARS on a very difficult exam called HLE (Humanity's Last Exam), which covers advanced topics in science, math, and history.

The Underdog: MARS used a relatively small model (8 billion parameters).
The Giants: They compared it to much larger models (32 billion or 72 billion parameters) and even some expensive, proprietary "super-intelligence" models from big tech companies.
The Outcome: MARS beat the larger models that had been pre-trained with human examples (Supervised Fine-Tuning). It even came very close to the performance of the most advanced commercial models available, despite starting with zero human guidance and using a much smaller brain.

Why This Matters (According to the Paper)

The paper claims that the secret sauce isn't just having more tools (like search engines or calculators); it's the partnership.

Without the "Scout" (System 1), the "Thinker" gets overwhelmed by too much raw data and makes mistakes.
Without the "Thinker" (System 2), the "Scout" just summarizes things without solving the actual problem.
Together, they create a system that can read a massive amount of up-to-date information and use it to solve complex, multi-step reasoning problems efficiently.

In short, MARS teaches an AI to know when to "scan fast" and when to "think deep," and how to do both together without getting confused.

Technical Summary: MARS – Co-evolving Dual-System Deep Research via Multi-Agent Reinforcement Learning

1. Problem Statement

Large Reasoning Models (LRMs) face two fundamental limitations when tackling complex, knowledge-intensive tasks:

Inefficiency in Information Processing: When faced with simpler information processing tasks (e.g., analyzing web pages or summarizing papers), LRMs tend to "overanalyze," consuming excessive reasoning tokens. This limits their capacity to process large volumes of external information required for deep research.
Static Knowledge: LRMs are confined to the knowledge cutoff of their training data and cannot access up-to-date information without external tools.

Existing approaches that combine reasoning models with external tools (e.g., WebThinker, Search-R1) typically employ fixed or independently-trained information processors. These processors are not optimized for the specific downstream reasoning task, leading to a mismatch where the processor either provides too much irrelevant information (overload) or misses critical details (loss). Furthermore, these systems often rely on Supervised Fine-Tuning (SFT) on curated trajectories, which may not generalize well or scale efficiently.

2. Methodology

The authors propose MARS (Multi-Agent System for Deep ReSearch), a framework that enables the co-evolution of two cognitive systems within a single Large Language Model (LLM) via Multi-Agent Reinforcement Learning (MARL).

2.1 Dual-System Collaborative Framework

Inspired by the dual-process theory of human cognition, MARS integrates two systems:

System 1 (Fast, Intuitive): Responsible for high-throughput information processing. It distills key information from raw tool outputs (e.g., search results, papers) based on the specific "purpose" provided by System 2.
System 2 (Slow, Deliberate): Responsible for strategic reasoning, planning, and tool invocation. It maintains the reasoning context and decides when to call tools and what information is needed.

Workflow:

System 2 analyzes the current context and generates a reasoning step, optionally requesting a tool with a specific purpose ( $p_i$ ).
The external environment executes the tool, returning raw outputs (which can be multiple, variable-length documents).
Bin-Packing: To handle variable-length outputs efficiently, a First Fit Decreasing (FFD) algorithm organizes these outputs into optimally-sized chunks.
System 1 processes these chunks in parallel, conditioned on the purpose $p_i$ , to extract distilled information ( $\tilde{o}_i$ ).
The distilled information is concatenated into the context for System 2's next turn.

2.2 Multi-Agent Optimization Strategies

The authors extend Group Relative Policy Optimization (GRPO) to the multi-agent setting with three key innovations to ensure stable and effective joint training:

Decoupled Gradient Computation: Although System 1 and System 2 share the same trajectory-level reward, their gradient updates are decoupled.
- System 2's loss is computed only over reasoning and planning tokens, masking System 1's output tokens.
- System 1's loss is computed only over its distillation outputs, conditioned on the bin-packed input.
- This ensures proper credit assignment: System 2 learns better planning/purposes, while System 1 learns better extraction strategies specific to those purposes.
Bin-Packing Optimization: To enable parallel processing by System 1, variable-length tool outputs are sorted by length and packed into bins that fit within the model's context window. This reduces the number of generation steps required for System 1.
Advantage-Weighted Balanced Sampling: In a rollout, System 2 generates a fixed number of samples (one per trajectory), while System 1 generates a variable number (dependent on tool calls and bin counts). To prevent one system from dominating the learning process:
- Advantages are pre-computed for all samples within a group.
- System 1 samples are then downsampled or upsampled (via random duplication) to match the number of System 2 samples, ensuring balanced optimization while preserving statistical integrity.

2.3 Training Setting: "Zero RL"

A critical aspect of MARS is its training regime. It operates under a "Zero RL" setting:

The model is trained directly from the base model (e.g., Qwen2.5-7B or Qwen3-8B).
No Supervised Fine-Tuning (SFT) or policy distillation from stronger models is used.
This contrasts with baselines like WebThinker, which rely on SFT warmups, allowing for a direct evaluation of the multi-agent RL framework's efficacy.

3. Key Contributions

Co-evolution Framework: Introduction of MARS, a dual-system framework where System 1 (information processing) and System 2 (reasoning) are jointly optimized through shared trajectory rewards. This allows System 1 to learn task-relevant information distillation rather than generic summarization.
Adapted GRPO for Multi-Agent RL: Development of three specific techniques to enable stable joint training: decoupled gradients for credit assignment, bin-packing for parallel processing, and advantage-weighted balanced sampling.
Zero RL Performance: Demonstration that an 8B parameter model trained without SFT can outperform larger, SFT-trained baselines and approach proprietary models on challenging benchmarks.

4. Experimental Results

The authors evaluated MARS on several benchmarks, including Humanity's Last Exam (HLE) and 7 knowledge-intensive QA tasks.

Humanity's Last Exam (HLE):
- MARS (8B) achieved 8.17% accuracy.
- This outperforms WebThinker (32B with SFT, 6.87%) and C-3PO (72B with SFT, 5.79%).
- It narrows the gap with proprietary models like Claude 3.7 Sonnet (7.89%) and o1 (7.75%).
- The improvement is broad across disciplines (Bio/Med, CS/AI, Math, etc.), not isolated to a single domain.
Knowledge-Intensive QA Tasks:
- MARS achieved an average improvement of 8.9% over the C-3PO baseline across 7 tasks (including NQ, TriviaQA, HotpotQA, etc.).
- Gains were particularly pronounced in multi-hop reasoning (+12.2% over C-3PO), indicating the framework's ability to handle complex reasoning chains requiring multiple retrieval-reasoning iterations.
Ablation Studies:
- System 1 Removal: Removing System 1 (forcing System 2 to process raw tool outputs) caused a 1.91% drop in HLE accuracy (7.38% $\to$ 5.47%). This confirms that System 1 actively co-evolves to extract task-relevant information, rather than acting as a passive summarizer.
- Tool Usage: The model learns to utilize multiple tools (Search, Scholar, Python) effectively. Removing any single tool (especially Search or Python) significantly degraded performance in specific domains.
- Training Dynamics: Analysis shows that as training progresses, the model shifts from conservative "single-shot" retrieval to iterative evidence gathering, with System 1 effectively filtering noise.

5. Significance and Claims

The paper claims that MARS demonstrates the viability of co-evolving dual-system architectures through multi-agent reinforcement learning without relying on supervised fine-tuning.

Efficiency vs. Capacity: The framework successfully addresses the trade-off between processing large volumes of external information and maintaining reasoning quality. System 1 handles the "reading load," allowing System 2 to focus on complex reasoning without context dilution.
Learned Specialization: The results suggest that the "optimal information distillation strategy" is not static but depends on the reasoning task. Through co-evolution, System 1 learns to extract insights specifically useful for System 2, a capability that fixed prompts or independent training cannot achieve.
Scalability: The "Zero RL" success indicates that the multi-agent RL framework itself is a powerful driver of performance, potentially reducing the reliance on massive model scales or expensive SFT datasets.

The authors emphasize that this approach improves the reliability of deep research by enabling models to synthesize up-to-date information while maintaining strong multi-step reasoning, with potential applications in education and scientific literature review. They also note potential risks regarding misinformation and the need for human oversight in high-stakes settings.