Technical Summary: PROBE – Measuring Proactive Problem Solving in LLM Agents

1. Problem Definition

Current Large Language Model (LLM) agents are predominantly reactive, operating only after receiving explicit user instructions. While significant progress has been made in reactive domains like coding and web navigation, the transition to proactive agents—those that anticipate user needs, identify issues autonomously, and execute solutions without prompting—remains under-evaluated.

Existing benchmarks for proactivity are limited by:

• Localized Context: They often rely on narrow, immediate temporal contexts, failing to capture insights that require long-term analysis across multiple documents.
• Lack of Search: They do not adequately test an agent's ability to search through a "user datastore" to find unspecified issues.
• Synthetic Constraints: Real-world data collection for long-horizon proactivity is hindered by privacy concerns and annotation overhead.

The core challenge is to evaluate whether agents can: (1) search a personalized datastore for unspecified issues, (2) identify specific "bottlenecks" (critical, actionable problems), and (3) execute the correct resolution among plausible alternatives.

2. Methodology: The PROBE Benchmark

To address these gaps, the authors introduce PROBE (Proactive Resolution of Bottlenecks), a benchmark designed to operationalize proactivity as a three-stage pipeline.

2.1 Data Generation Pipeline

The benchmark utilizes a synthetic data generation pipeline to create 1,000 diverse test samples without privacy risks:

1. World Model Construction: Starting from real-world LinkedIn profiles, the system constructs comprehensive "world models" for simulated users. These models encode workplace hierarchies, communication styles, pain points, and available action spaces.
2. Bottleneck Generation: For each user, the system generates a specific "bottleneck"—a critical, actionable issue (e.g., a missed deadline, missing documentation) that is hidden within the user's document history.
3. Document Synthesis: A datastore ($D$) is created for each sample, consisting of:
   • True Positives ($T$): Documents containing evidence of the bottleneck. Evidence is often distributed across multiple documents, requiring synthesis and temporal reasoning.
   • Distractors ($K$): Realistic, irrelevant documents (emails, calendar events) of comparable length and relevance to introduce noise.
4. Action Space: A set of candidate actions ($A$) is defined, where exactly one action ($a^*$) correctly resolves the bottleneck, while others are plausible but suboptimal.

2.2 Task Formulation

An agent receives the datastore $D$ and the set of available actions $A$. It must output a tuple $\hat{O} = (\hat{T}, \hat{b}, \hat{a}, \hat{p})$:

• $\hat{T}$: The subset of documents identified as relevant.
• $\hat{b}$: The identified bottleneck description.
• $\hat{a}$: The selected action.
• $\hat{p}$: The parameters for the selected action.

2.3 Evaluation Metrics

Performance is measured across three hierarchical stages:

1. Search: Precision, Recall, and F1 score for retrieving the correct documents ($T$).
2. Bottleneck Identification: Evaluated via an LLM-as-a-judge framework, scoring the accuracy of essential details (root cause, blocker) and non-essential details.
3. Task Execution: Scoring the correctness of the selected action and the completeness of its parameters.

Human evaluation on a "short-context" subset (removing distractors) showed annotators achieving ~71% identification and ~67% execution scores, validating that the primary difficulty of the full benchmark stems from long-context reasoning and search rather than task ambiguity.

3. Key Contributions

• PROBE Benchmark: A dataset of 1,000 samples systematically evaluating proactive capabilities through a unified framework of search, identification, and execution across long-horizon, multi-document contexts.
• Comprehensive Evaluation: A rigorous assessment of frontier closed-source models (GPT-5, Claude Opus 4.1, etc.) and open-source models (DeepSeek-R1, Kimi K-2) alongside leading agentic frameworks (ReACT, Reflexion, ReWOO).
• Failure Mode Analysis: An in-depth breakdown of where systems fail, distinguishing between search errors, reasoning errors (root cause identification), and execution errors (parameter specification).

4. Results

The evaluation reveals a significant capability ceiling for current AI systems in proactive tasks:

• Low End-to-End Performance: Even the state-of-the-art models struggle to achieve high success rates. The best end-to-end performance (Task Execution score) is 40%, achieved by both GPT-5 and Claude Opus 4.1.
• Pipeline Bottlenecks:
  • Search: GPT-5 leads with an F1 of 0.65, while other models drop significantly (e.g., GPT-OSS-20b at 0.04).
  • Identification: Claude models (Opus 4.1, Sonnet 4) show a slight edge in reasoning (0.43 score) compared to GPT-5 (0.42), suggesting a trade-off between search strength and reasoning depth.
  • Execution: Performance drops sharply from identification to execution. Models often identify the problem correctly but fail to select the correct action or provide complete parameters.
• Agentic Frameworks: Standard frameworks (ReACT, Reflexion, ReWOO) perform significantly worse than "out-of-the-box" LLMs, with execution scores ranging from 0.01 to 0.11. This is attributed to their reliance on iterative retrieval tools which struggle in the absence of a well-defined search target within a large, unstructured datastore.
• Failure Modes:
  • Root Cause Identification: The dominant failure mode, accounting for 73.8% of identification errors. Even top models frequently misidentify the root cause.
  • Interpersonal Reasoning: Models struggle to reason about relationships and roles (46.9%–78.0% failure rates), often failing to attribute the bottleneck to the correct person.
  • Parameter Specification: High rates of missing critical parameters (65%–79%) or incorrect parameter filling, indicating a gap between diagnosing a problem and formulating a precise solution.

5. Significance and Claims

The paper claims that PROBE exposes a fundamental limitation in current autonomous agentic systems: the inability to reliably translate long-horizon context into actionable, parameterized solutions.

• Beyond Reactivity: The results highlight that simply scaling models or using standard agentic loops (like ReACT) is insufficient for true proactivity. The gap between "reactive" tools and "proactive" assistants remains substantial.
• Faithful Evidence Use: The authors argue that future progress depends on "faithful evidence use"—the ability to synthesize information from multiple sources to identify bottlenecks accurately before attempting resolution. Current models often "shortcut" by guessing the solution without sufficient evidence, leading to high identification scores but low execution scores.
• Research Directions: The paper identifies specific areas for future work, including:
  • Developing benchmarks that incorporate temporal dynamics and evolving user models (moving beyond fixed world models).
  • Extending evaluation to multi-step workflows where actions modify the agent's state.
  • Improving interpersonal reasoning and root cause analysis capabilities.

In summary, the paper posits that while LLMs have made immense progress in reactive tasks, the leap to proactive autonomy requires solving complex challenges in long-context search, multi-document synthesis, and precise action planning, which current frontier models have not yet mastered.