Technical Summary: Hybrid Open-Ended Tri-Evolution (HOTE)

Problem Statement

Deep research agents are increasingly critical for real-world applications aiming toward Artificial General Intelligence (AGI), tasked with autonomously retrieving and integrating information in open-ended, long-cycle environments. However, current deep research systems face two primary limitations:

1. Static Capabilities: Existing agents rely on fixed training sets and strategies, bounding their parametric research capabilities.
2. Verification Gap: While agent evolution via self-play has shown promise in domains with verifiable, standard answers (e.g., math, coding), it struggles with open-ended research tasks (e.g., long-form report generation) where deterministic answers are absent. Previous attempts to bridge this gap (e.g., SPICE, Dr. Zero) remain limited to tasks with verifiable outcomes or fail to address the specific challenges of open-ended environments.

There is a critical need for a framework that enables autonomous agent evolution specifically for open-ended research tasks without relying on standard ground-truth answers.

Methodology: Hybrid Open-Ended Tri-Evolution (HOTE)

The authors propose HOTE, a framework designed to facilitate the collaborative evolution of three distinct modules: a Proposer, a Solver, and a Judge. The system leverages hybrid-mode reinforcement learning (RL) and web-scale knowledge to drive autonomous improvement.

Core Components

1. Solver ($\pi_{\theta_s}$):

   • Role: Receives a research query, generates a plan, performs multi-turn information seeking (using tools like search engines), and produces a referenced long-form research report.
   • Evolution: Optimized using Group Relative Policy Optimization (GRPO). The solver aims to align its outputs with research report requirements defined by the Judge's rubrics.
   • Modes: Operates in both tool-use (utilizing search tools) and no-tool (relying on parametric knowledge) modes.

2. Judge ($\pi_{\theta_j}$):

   • Role: Dynamically generates and updates evaluation rubrics to assess the Solver's responses. It compares multiple responses to the same query to identify strengths and weaknesses.
   • Mechanism: Instead of relying on fixed answers, the Judge assigns rewards (0 or $\pm1$) based on a set of rubrics ($R_{s_0}$). Crucially, the Judge evolves the rubric set itself by identifying active rubrics (new strengths or common failure modes like reward hacking) and persistent rubrics.
   • Meta-Rubrics: To handle the complexity of evaluating the Proposer, the Judge summarizes assessments into "meta-rubrics" that capture common model weaknesses across responses.

3. Proposer ($\pi_{\theta_p}$):

   • Role: Generates challenging yet learnable research tasks based on the weaknesses identified by the Judge. It acts as an adversary to push the Solver's boundaries.
   • Evolution: Also optimized via GRPO. The Proposer's reward is calculated based on how effectively its generated tasks expose the Solver's weaknesses (i.e., tasks where the Solver fails to meet the meta-rubrics).
   • Diversity: The framework employs "Diverse Proposing," where the Proposer generates tasks based on various combinations of previous tasks and assessments to prevent topic concentration and ensure data diversity.

Training Strategy: Dual-Mode Hybrid Training

HOTE introduces a Dual-mode Hybrid Training Strategy to balance efficiency and capability:

• Hybrid Data: Each training batch consists of 50% real tasks from the original dataset and 50% synthetic tasks generated by the Proposer.
• Hybrid Modes: Tasks are randomly assigned to either tool-use or no-tool modes (50/50 split).
  • Rationale: Training solely in no-tool mode improves report generation techniques but leads to over-reliance on parametric knowledge. Training solely in tool-use mode introduces noise. The hybrid approach allows the model to learn report generation without tools while simultaneously mastering tool usage, preventing over-reliance on either.
• Efficiency: The framework theoretically proves that hybrid mode results in a lower expected maximum generation time compared to pure tool-use strategies.

Key Contributions

1. First Open-Ended Evolution Framework: HOTE is the first deep researcher evolution framework explicitly designed for open-ended environments and tasks, bridging the gap between deep research and agent evolution.
2. Tri-Module Co-Evolution: The paper designs a co-evolution strategy where the Proposer, Solver, and Judge mutually reinforce each other. The Judge prevents reward hacking by evolving rubrics, the Proposer ensures task difficulty remains high, and the Solver improves response quality.
3. Dual-Mode Efficiency: The introduction of a hybrid training strategy (mixing tool-use and no-tool modes) significantly improves training efficiency and model robustness compared to single-mode approaches.

Experimental Results

The authors evaluated HOTE on three long-form deep research benchmarks: HealthBench (healthcare), ResearchQA (scientific literature synthesis), and DeepResearchBench (DRB) (general domain).

• Performance: An 8B model trained with HOTE outperformed the strongest open-source 8-32B models (including Qwen3-32B, WebThinker-32B) and state-of-the-art deep research training methods (e.g., DR Tulu-8B-RL, GRPO, GSPO).
• Efficiency: HOTE achieved these results with less time overhead. Specifically, the hybrid mode reduced the total number of tool-use operations required during training compared to baselines like DR Tulu.
• Ablation Studies:
  • Removing any of the three modules (Proposer, Solver, or Judge evolution) resulted in performance degradation, confirming their indispensability.
  • Models trained with HOTE showed a sustained upward trend in performance over 1500 training steps, whereas baselines converged earlier.
  • The hybrid mode successfully facilitated collaborative progress in both tool-use and no-tool capabilities, avoiding the performance drop in tool-use tasks often seen in models trained exclusively in no-tool mode.
• Generalization: The framework demonstrated effectiveness across different base models (Qwen3-8B and Llama3.1-8B).

Significance and Claims

The paper claims that HOTE represents a significant step toward autonomous evolving agents capable of handling open-ended tasks without human intervention or standard answers. By successfully integrating a Proposer-Solver-Judge loop with hybrid training, HOTE addresses the "verification gap" that has previously limited agent evolution to deterministic domains.

The authors position this work as a move toward AGI by enabling agents to autonomously interact with the environment, gain experience, and evolve their capabilities in complex, real-world research scenarios. They acknowledge limitations, noting that evolution slows as training progresses and that the upper bound may still be constrained by model scale, but assert that transcending the limitations of initial training data through evolution is inherently valuable. Future work is proposed to explore scaling to larger Mixture-of-Experts (MoE) models and handling noise in real-world search tools.