The PROPER Approach to Proactivity: Benchmarking and Advancing Knowledge Gap Navigation

1. Problem Formulation

Current proactive conversational agents often fail because they rely on either clarification (asking users for missing details, addressing "known unknowns") or contextual extrapolation (guessing based on environment/history). These approaches suffer from two main issues:

1. Mistimed/Misaligned Interventions: They often burden users with unnecessary questions or provide irrelevant information, degrading trust.
2. Inability to Model "Unknown Unknowns": They fail to identify knowledge gaps—task-relevant considerations that the user is unaware of and cannot articulate (e.g., a patient not knowing to mention a specific comorbidity, or a coder not considering edge cases).

The authors frame proactive assistance not as a simple response expansion, but as a calibration problem: determining when and which implicit, task-relevant dimensions should be introduced to a user without disrupting their intent.

2. Methodology: The PROPER Framework

PROPER (Proactivity Driven Personalized Agents) is a modular architecture designed to explicitly model and navigate user knowledge gaps. It operates through three core stages:

A. Dimension Generating Agent (DGA)

The DGA is a fine-tuned Large Language Model (LLM) responsible for inferring implicit dimensions.

• Input: User query ($q$), interaction history ($h$), and persona attributes ($p$).
• Mechanism: The DGA is trained on successful interaction trajectories annotated with "dimensions" (structured, task-relevant factors like safety constraints, input validation, or risk factors).
• Output: It generates a set of candidate implicit dimensions ($D_{imp}$) representing potential knowledge gaps, alongside explicit dimensions ($D_{exp}$) extracted from the user's input.
• Goal: To internalize the distribution of critical considerations for specific domains (Medical, Coding, Shopping) and surface what the user didn't say but should have considered.

B. Post-hoc Calibrated Reranker

This module acts as a control mechanism to prevent over-proactivity. It selects a budgeted subset of dimensions to activate.

• Selection Objective: It optimizes a utility function (Eq. 7) that balances three factors:
  1. Quality: Confidence score from the DGA.
  2. Alignment: Semantic similarity to unmet explicit user needs.
  3. Diversity: Penalizing redundancy among selected dimensions.
• Calibration: A parameter $k$ (budget) limits the number of dimensions activated, ensuring the agent remains focused and non-intrusive.

C. Response Generating Agent (RGA)

The RGA generates the final response by selectively integrating the calibrated set of activated dimensions into a baseline response.

• Strategy: It treats the baseline response as a draft and performs "repair" rather than rewriting.
• Constraints:
  • Explicit gaps are addressed by default.
  • Implicit gaps are included only if the interaction context warrants it.
  • It limits clarifying questions to at most one per turn to avoid interrogation.
  • It prioritizes concise additions over full rewrites.

3. Key Contributions

1. Formalization of Proactivity as Calibration: The paper reframes proactive assistance as a problem of calibrating intervention against user knowledge gaps, rather than simple response generation.
2. Dimension-Based Representation: Introduces "dimensions" as an intermediate representation to explicitly model task-relevant considerations beyond the user's query, enabling the system to reason about "unknown unknowns."
3. Modular Architecture (PROPER): Proposes a decoupled framework where knowledge gap discovery (DGA) is separated from response generation (RGA), allowing for controlled, personalized proactivity.
4. Gap-Aware Evaluation: Develops a structured rubric that evaluates agents on coverage of knowledge gaps, initiative appropriateness, and intent alignment, moving beyond simple correctness metrics.

4. Experimental Results

The authors evaluated PROPER across three distinct domains: Medical (MD), Code-Contests, and Personalized Shopping (PWAB).

• Performance Gains:
  • PROPER significantly outperformed base LLMs (LLaMA-8B, Qwen-8B) and Chain-of-Thought (CoT) baselines.
  • Win Rates: Achieved up to 84% win rates in single-turn evaluations against base models.
  • Multi-turn: Demonstrated consistent dominance in multi-turn interactions, maintaining appropriate initiative over time.
• Domain Specificity:
  • Medical: Showed the largest gains. PROPER successfully surfaced critical latent risks (e.g., comorbidities, vaccination context) that base models missed, which is crucial for safety.
  • Coding: Gains were present but more modest, as coding tasks often have well-defined solutions where proactive expansion offers diminishing returns.
  • Shopping: Improved personalization by balancing user preferences with latent constraints (e.g., budget, compatibility).
• Ablation Studies:
  • Removing the DGA caused the most significant performance drop, proving that identifying the gaps is more critical than the generation capacity alone.
  • Removing the Reranker led to consistent but smaller degradation, highlighting the importance of calibrating how much to intervene.

5. Significance and Future Directions

• Epistemic Proactivity: PROPER shifts the paradigm from "answering questions" to "navigating knowledge gaps." It helps users understand what they don't know, supporting sense-making rather than just goal execution.
• Safety and Trust: By calibrating proactivity, the system avoids the "annoying assistant" problem, intervening only when the risk of omission is high (e.g., medical safety) or when the user is clearly uncertain.
• Limitations & Future Work:
  • Current evaluation relies on LLM judges; future work requires human-in-the-loop validation for trust and intrusiveness.
  • The system currently uses free-form text for dimensions; future iterations could integrate ontologies or concept bottlenecks for better auditability.
  • High-stakes deployment (e.g., clinical settings) requires further safety validation beyond benchmark accuracy.

In conclusion, PROPER demonstrates that explicitly modeling and selectively activating implicit knowledge dimensions allows AI agents to provide significantly more helpful, safe, and personalized assistance than traditional reactive or uncalibrated proactive systems.