Technical Summary: From Profiling to Synthesis: Benchmarking Implicit Behavioral Alignment in Personalized LLM Agents

1. Problem Definition: The Knowledge-to-Action Gap

Current research in Large Language Model (LLM) agents has shifted from conversational chatbots to autonomous systems capable of executing complex real-world tasks. However, a critical bottleneck remains in personalization. Existing benchmarks and evaluation frameworks largely rely on static preference snapshots, fixed interaction logs, or explicit Question Answering (QA) over predefined user profiles.

The authors identify a fundamental limitation termed the knowledge-to-action gap. This gap describes the discrepancy where an agent may successfully retrieve or infer crucial user knowledge (e.g., a user recently had a tooth extraction) but fails to apply this knowledge to satisfy implicit constraints during task execution (e.g., ordering soft food instead of spicy food based on a static "likes spicy" profile). Current evaluations predominantly measure an agent's ability to extract attributes via QA, failing to assess whether an agent can integrate implicit, evolving, and potentially conflicting user signals into concrete, complex actions and decisions.

2. Methodology

2.1 IBA-BENCH: A New Benchmark

To address the lack of evaluation for implicit behavioral alignment, the authors introduce IBA-BENCH. Unlike prior benchmarks that focus on profiling or static matching, IBA-BENCH is constructed from longitudinal interaction histories containing noise, implicit cues, and temporal inconsistencies.

• Construction Pipeline: The benchmark is generated using a multi-agent framework. It starts with 400 seed personas defined by long-term attributes (background, personality) and short-term states (transient stress, current role).
• Data Characteristics: The system generates interaction histories where informative preference evidence is buried within large volumes of noisy, task-irrelevant content (e.g., casual small talk). Preferences are revealed implicitly through behavioral patterns (e.g., repeatedly shortening drafts) rather than explicit statements.
• Task Scope: The benchmark comprises 6,962 task instances across 9 domains (Writing, Work, Daily Consumption, Planning, Health, Transportation, Medical, Leisure, Information Management) and 66 scenarios.
• Evaluation: Tasks require agents to execute concrete actions (generating text or calling APIs with parameters) based on the history. A behavioral evaluator assesses correctness against preference-specific criteria, moving beyond simple accuracy to "behavioral success."

2.2 IBA-Agent: A Framework for Behavioral Alignment

To bridge the gap between historical preference understanding and active task execution, the authors propose IBA-Agent, an LLM-driven framework consisting of two core modules:

1. Deep Retrieval:

   • Query Expansion: Instead of a single query, the agent generates diverse sub-queries targeting different preference facets (e.g., tone, structure, decision trade-offs, corrections, habits).
   • Retrieval & Refinement: Using a dense retriever (BGE-M3), the system retrieves semantically relevant passages. It employs redundancy filtering to remove overlapping snippets and salience re-ranking (using an LLM scorer) to prioritize high-confidence preference signals, ensuring reliable evidence is weighted over noisy implicit patterns.

2. Broad Thinking & Deep Alignment:

   • Synthesis: This module transforms the retrieved evidence into a task-specific personalization plan.
   • Process: It performs compact evidence organization, preference extraction (identifying what was asked, corrected, rejected, or updated), and task-preference alignment.
   • Output: The agent produces an alignment plan that dictates response style/structure for generation tasks or constraints/priorities for API-action tasks, effectively reconciling conflicting priorities before execution.

3. Key Contributions

The paper makes three primary contributions:

1. Conceptual Shift: It advocates for moving personalized agent research from static preference profiling to dynamic preference synthesis, identifying the knowledge-to-action gap as a key bottleneck.
2. Benchmark Introduction: The release of IBA-BENCH, the first benchmark to evaluate implicit behavioral alignment through concrete task execution in realistic, dynamic, and noisy settings. It covers all dimensions of history, dynamics, implicitness, task execution, and behavioral evaluation.
3. Framework Proposal: The introduction of IBA-Agent, a framework combining deep retrieval with trajectory-level synthesis, providing a strong empirical baseline for agents capable of personalized reasoning.

4. Experimental Results

Experiments were conducted in an inference-only setting (no fine-tuning) using a unified RAG pipeline with bge-m3 for retrieval. The study evaluated ten modern LLMs (Qwen, DeepSeek, GPT, ChatGLM, QwQ families) and three general-purpose agent systems (Claude Code, Hermes Agent, nanobot).

• Baseline Performance: Standard RAG approaches and standalone LLMs (including strong models like GPT-5.1 and Claude Code) struggle significantly on these synthesis-heavy execution tasks. Performance is not strictly monotonic with model scale, indicating that stronger backbones alone are insufficient.
• IBA-Agent Performance: The proposed framework substantially improves behavioral alignment.
  • Using DeepSeek-V3.2 as the backbone, IBA-Agent improved the overall score from 66.9 to 78.8.
  • Using Qwen3-4B-Instruct, the score rose from 67.6 to 78.1.
  • Adding Function Calling (FC) further boosted performance to 81.9 and 78.7 respectively.
• Ablation Studies:
  • Deep Retrieval had the largest impact; removing it caused an 8.3-point drop in overall performance, highlighting the difficulty of accessing relevant evidence from long histories.
  • Broad Thinking & Deep Alignment contributed a 2.5-point gain, essential for multi-constraint alignment.
  • Trajectory-level Synthesis was critical; replacing it with standard RAG-style synthesis resulted in a 5.5-point drop.
• Gap Analysis: The study confirms a clear knowledge-to-action gap: models perform significantly better on QA tasks (stating preferences) than on task-oriented applications (enforcing preferences). IBA-Agent demonstrates robustness to preference shifts, outperforming the strongest baseline by +12.7 points in dynamic preference settings.

5. Significance and Claims

The paper claims that effective personalization requires agents to synthesize implicit signals from longitudinal histories rather than merely retrieving explicit tags. The authors argue that closing the knowledge-to-action gap is a prerequisite for personalized agents to be practically useful in the wild.

The significance of this work lies in:

• Revealing Limitations: Empirically validating that current state-of-the-art agents fail to translate historical context into behavioral constraints, even when they possess the necessary knowledge.
• Providing a Solution: Demonstrating that explicit trajectory-level synthesis and deep retrieval can substantially improve alignment in complex, noisy scenarios.
• Establishing a Standard: Offering IBA-BENCH as a rigorous evaluation tool that moves beyond static profiling to assess the dynamic, conflicting, and implicit nature of real-world user preferences.

The authors acknowledge limitations, noting that the benchmark relies on LLM-generated synthetic data and currently focuses on offline, history-conditioned execution rather than online interactive adaptation. However, they position this work as a necessary step toward building agents that can truly understand and act upon evolving user needs.