Technical Summary: Personalizing Large Language Model Agents with Small Policy Models

Problem Formulation

The paper addresses the challenge of personalizing Large Language Model (LLM) agents for individual users without modifying the underlying host model. While LLM agents can retrieve memory, invoke tools, and adjust response styles, adapting these execution decisions to specific user preferences remains difficult. Existing approaches face significant limitations:

• Fine-tuning: Adapting model parameters per user is computationally costly and often impossible for proprietary, black-box systems.
• Prompts and Memory: These methods expose user information to the agent but do not adapt the agent's execution policy (e.g., when to ask questions, when to use tools) based on feedback.
• Single-Mechanism Adaptation: Recent methods that adapt only one component (e.g., memory retrieval or tool selection) fail to attribute a single scalar feedback signal (the user's reaction to the whole interaction) across the joint space of execution decisions.

The core difficulty is sparse feedback: the system observes a scalar outcome only for the specific combination of execution decisions chosen, while the performance of alternative combinations remains unobserved. Furthermore, the system must respect hard constraints (e.g., permissions, safety rules) and treat onboarding information as revisable prior evidence rather than fixed rules.

The authors formulate this as an online learning problem for a per-user execution policy over a factorized action space, constrained by external feasibility sets, using only scalar feedback observed for the selected action.

Methodology: FABLE

The authors propose FABLE (Factorized Adaptive Bandit Layer for Execution), a lightweight policy layer that operates outside a potentially black-box host agent. FABLE does not require access to host weights, gradients, or hidden states.

Key Components

1. Factorized Action Space: An execution decision $a$ is defined as a tuple $(m, \tau, s)$ representing memory mode, information-acquisition mode, and response behavior. Instead of learning a policy over the full joint space (which is combinatorially large), FABLE uses a factorized feature map $\phi(c, a)$. This map includes main effects for each component and pairwise interactions, allowing feedback from one joint action to update preferences for related components (e.g., a dislike for unnecessary clarification updates the "ask clarification" preference regardless of the memory setting).

2. Residual Preference Model: The expected feedback is modeled as:
   $$E[y_t | c, a] = \bar{b}(c, a) + \phi(c, a)^\top \theta^*$$
   where $\bar{b}(c, a)$ is a known, prespecified default score minus an operational cost, and $\theta^*$ represents the user-specific residual preference. This approach allows the system to retain the host's useful generic behavior while learning only the user's deviation from it.

3. Revisable Onboarding: User preferences provided during onboarding are treated as finite-precision pseudo-observations. They initialize the Gaussian posterior over $\theta$ but do not fix the policy. As online feedback arrives, the posterior updates, allowing the system to revise initial assumptions.

4. Feasibility-Constrained Thompson Sampling:

   • Filtering: Before selection, an external filter removes inadmissible actions based on hard constraints (permissions, tool availability).
   • Sampling: FABLE samples a residual parameter $\tilde{\theta}_t$ from the current Gaussian posterior (scaled by a calibrated factor $\nu_t$) and selects the action $a_t$ that maximizes the expected score within the feasible set $A_t$.
   • Update: After execution, the residual feedback is used to update the Gaussian posterior via conjugate updates.

5. Promotion Rule: Learned preferences can be written back to the host as persistent rules only if an anytime-valid confidence sequence excludes zero, ensuring that only statistically significant preferences are promoted.

Theoretical Results

The paper establishes a feasible-oracle regret guarantee for FABLE under a linear residual-feedback model.

• Regret Bound: With a theoretically calibrated Gaussian sampling scale $\nu_t = \sqrt{9d \log(t/\delta)}$, FABLE achieves a high-probability regret of $\tilde{O}(d^{3/2}\sqrt{n})$ against the best feasible action in each round.
• Identifiability: The authors characterize which preference directions remain identifiable when feasibility constraints persistently exclude certain actions, noting that preferences requiring comparisons of excluded components cannot be learned from online data alone.

Experimental Evaluation

FABLE was evaluated on a four-domain tau2-bench protocol (Airline, Retail, Telecom, Banking Knowledge) involving executable tool-use tasks.

Key Findings

• Personalized Reward and Alignment: The full FABLE policy achieved the highest observed personalized reward and synthetic verbosity alignment among all tested policies. Compared to the host baseline, it showed significant gains in personalized reward (+0.077) and alignment (+0.281) with positive 95% confidence intervals.
• Task Success: FABLE tied with the frozen host for the highest mean task success. However, the improvement over the host was not statistically significant (95% CI included zero). The paper notes that task success varied by domain (lower on Airline, tied on Retail, higher on Telecom and Banking).
• Ablation Studies:
  • Factorization: Comparing FABLE to a flat complete-action LinTS showed that factorization significantly improved alignment, supporting the transfer of information across action components.
  • Onboarding: Removing onboarding resulted in the largest performance drop, confirming the value of initializing the residual coordinates with prior evidence.
  • Online Adaptation: Comparing full FABLE to a "frozen" version (no online updates) showed that online adaptation improved alignment, though task success remained comparable.
• Additional Benchmarks: Experiments on PAHF (Embodied Manipulation and Online Shopping) and Math500 (personalized reasoning) showed mixed results. In Embodied Manipulation, FABLE (full) achieved lower reward (0.6214 vs. 0.6470) but higher success (0.078 vs. 0.052) than FABLE (no onboarding), indicating that the relative benefit of components varies by environment. In Online Shopping, FABLE (full) outperformed FABLE (no onboarding) in both reward and success. On Math500, FABLE (full) and FABLE (no onboarding) tied for the highest accuracy, though FABLE (full) had lower alignment than the "Known Preference" baseline.

Significance and Claims

The paper claims that FABLE provides a principled framework for personalizing frozen, black-box agents by treating execution control as a constrained contextual bandit problem. Its significance lies in:

1. Decoupling Personalization from Model Tuning: It enables per-user adaptation without fine-tuning the host LLM, making it applicable to proprietary systems.
2. Handling Sparse Joint Feedback: By factorizing the action space and modeling residual preferences, it efficiently learns from sparse scalar feedback that evaluates the entire interaction.
3. Safety and Constraints: It rigorously enforces external feasibility constraints before exploration, ensuring safety and compliance are never compromised by the learning process.
4. Modest Claims: The authors explicitly state that while FABLE improves preference-sensitive metrics (reward and alignment) in specific contexts, the experiments do not establish a statistically significant improvement in general task success or uniform domain-level gains. The results support controlled synthetic verbosity adaptation and executable integration but do not claim universal superiority over all metrics or benchmarks; notably, performance rankings on PAHF varied significantly between environments and ablation conditions.

The work demonstrates that a small, external policy layer can effectively learn and adapt to user preferences, bridging the gap between static agent capabilities and dynamic user needs.