Technical Summary: ProactBench

Problem Definition

Current Large Language Model (LLM) benchmarks predominantly evaluate reactive competence: how well a model answers explicit user requests. While models have improved significantly at instruction following, a critical conversational capability remains unmeasured: conversational proactivity. This is defined as the ability to notice and act on needs that a user has implied but not explicitly stated.

Existing evaluations often fail to distinguish between genuine proactive assistance and reactive behaviors such as:

• Instruction-following: Polished compliance with obvious requests.
• Clarification-seeking: Asking for information the user has already supplied.
• Generic helpfulness: Boilerplate closings (e.g., "Let me know if you need anything") that add no grounded value.
• Sycophancy: Agreeable but epistemically weak behavior.

The paper argues that true proactivity requires a model to infer unstated implications from disclosed details and introduce useful forward-looking value without being prompted.

Methodology

ProactBench introduces a novel evaluation framework designed to measure proactivity across three distinct conversation phases, utilizing a three-agent architecture to ensure validity.

1. The Three Trigger Types

The benchmark decomposes proactivity into phase-tied trigger types, each with specific rubrics:

• EMERGENT (Turns 1–3): Inferring an unstated need from a single disclosed anchor (factual detail) early in the conversation.
• CRITICAL (Turns 4–7): Synthesizing multiple disclosed anchors into a new conclusion mid-dialogue.
• RECOVERY (Turns 8–10): Offering grounded, forward-looking value after the user signals task completion. This phase is critical because it is where models often sound helpful while saying nothing useful.

2. Three-Agent Evaluation Architecture

To defend against style-confounded scoring, rubric leakage, external-context contamination, and information dumps, the benchmark employs:

• The Planner: Authors the dialogue strategy, decides when a trigger occurs, and commits to a prospective rubric before the model answers. It sees the persona and scenario but not the communication style.
• The User Agent: Renders the Planner's tactical orders into natural user messages based on a specific persona and one of 24 communication styles derived from the Communication Styles Inventory (CSI).
• The Assistant Model: Receives only the natural conversation history. It is blind to the persona, the rubric, the trigger schedule, and the hidden goals.
• The Offline Judge: Reruns only the trigger turns of a curated dialogue to score the response against the fixed rubric, ensuring identical contexts for all evaluated models.

3. Dataset and Protocol

• Corpus: 198 curated dialogues containing 624 trigger points.
• Variability: Dialogues span 24 communication styles (combinations of 6 CSI traits: expressiveness, preciseness, verbal aggressiveness, questioningness, emotionality, and impression manipulativeness) and 5 life domains (Professional, Sports, Arts, Travel, Culinary).
• Curation: Dialogues were generated using a "cold-start" isolation where the curation model (Gemini-2.5-Pro) had no access to the rubric or hidden goals, ensuring the conversation history contains only information a real assistant would naturally possess.
• Scoring: Responses are rated Pass, Partial, or Fail based on whether they meet the proactive principle (e.g., single-anchor inference, multi-anchor synthesis, or grounded forward value).

Key Contributions

1. Definition of Conversational Proactivity: A formal definition separating proactivity from instruction-following, clarification, and generic helpfulness, structured into three phase-tied trigger types.
2. Evaluation Architecture: A three-agent system (Planner, User Agent, Assistant) with information asymmetries that mitigates four specific validity threats common in LLM-as-judge evaluations.
3. The ProactBench Corpus: A release of 198 dialogues with 624 triggers, grounded in synthetic US personas and varied across 24 communication styles, audited by an independent LLM judge.
4. Empirical Evaluation: A comprehensive evaluation of 16 frontier and open-weight models, revealing that RECOVERY is a distinct capability that decorrelates from standard benchmarks.
5. Human Validation: A pairwise human study demonstrating that rubric-conditioned proactive responses are preferred over vanilla generations in 80% of non-tie comparisons, confirming that the benchmark measures a capability humans value.

Results

The evaluation of 16 models (including GPT-5.5, Claude-Opus-4.7, Qwen3.5, and Llama variants) yields several key findings:

• RECOVERY is Difficult and Distinct: RECOVERY is the hardest trigger type. The best model (GPT-5.5) passes only 37.2% of RECOVERY triggers, while 14 of 16 models pass fewer than 20%.
• Decorrelation from Standard Benchmarks: While EMERGENT and CRITICAL scores correlate strongly with existing benchmarks (GPQA, MMLU, SWE-bench, etc.), RECOVERY does not. Its mean correlation with six standard benchmarks is $\bar{r} = 0.51$ (95% CI [0.29, 0.71]), compared to $\bar{r} \approx 0.83$ for the other types.
  • Example: Kimi-K2.6 outperforms GPT-5.5 on LiveCodeBench and SWE-bench but scores 5x lower on RECOVERY (7.4% vs. 37.2%).
• Policy vs. Capability: The gap in RECOVERY performance appears to reflect a policy difference (e.g., post-training data mix and user-session feedback) rather than a pure capability ceiling. GPT-5.5 exhibits a "style-uniform" policy, maintaining high RECOVERY performance across difficult user styles, whereas other strong models concentrate their advantage on easier styles.
• Failure Modes:
  • Frontier closed-source models (excluding GPT-5.5) predominantly fail via generic closings (polished but empty sign-offs).
  • GPT-5.5 and smaller open-weight models predominantly fail via deliverable repetition (re-stating the completed task without adding new value).
• Human Preference: In a controlled study, humans preferred responses generated with the RECOVERY rubric as a system instruction over vanilla generations in 80% of cases, even when the vanilla response had already passed the rubric.

Significance and Claims

The paper claims that ProactBench identifies a "blind spot" in current LLM evaluation. While models are becoming increasingly capable of following instructions, they struggle to provide grounded forward-looking value after a task is ostensibly complete.

The authors assert that:

1. RECOVERY is a new evaluation signal: It is weakly predicted by standard reasoning and coding benchmarks, suggesting it measures a distinct behavioral capability.
2. Proactivity is preferred by humans: The specific proactive behaviors defined and measured by ProactBench are not just "nanny-style overreach" but are genuinely preferred by human raters when they are grounded in specific disclosed details.
3. The gap is behavioral, not just cognitive: The variation in RECOVERY performance across models suggests that current frontier models differ in their "policy" regarding initiative, likely driven by alignment data, rather than a fundamental lack of reasoning ability.

The paper concludes that ProactBench serves as a capability probe for this specific behavior, offering a substrate for reward modeling and preference learning to teach models when initiative is welcome, rather than training them to be maximally proactive in all contexts.