💬 NLP

PEARL: Self-Evolving Assistant for Time Management with Reinforcement Learning

This paper introduces PEARL, a reinforcement learning framework equipped with an external preference memory that significantly improves large language models' performance in resolving long-horizon calendar conflicts by dynamically learning and adapting to user preferences, as demonstrated on the newly proposed CalConflictBench.

Original authors: Bingxuan Li, Jeonghwan Kim, Cheng Qian, Xiusi Chen, Eitan Anzenberg, Niran Kundapur, Heng Ji

Published 2026-04-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Bingxuan Li, Jeonghwan Kim, Cheng Qian, Xiusi Chen, Eitan Anzenberg, Niran Kundapur, Heng Ji

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a busy CEO or a research lab director. Your calendar is a chaotic battlefield. Every day, you get hit with overlapping invitations: a critical board meeting clashes with a team brainstorming session, which also overlaps with a doctor's appointment.

You have to make a split-second decision: Go to the meeting? Reschedule the doctor? Decline the team?

Doing this manually is exhausting. It drains hours of your week. You might hire a human assistant to help, but even they get overwhelmed. They don't know your deep, personal preferences (like "I hate back-to-back meetings" or "I always prioritize my boss over my peers") unless you tell them every single time. Over months, they start guessing wrong, and you end up missing important things.

This paper asks a big question: Can we teach an AI (a Large Language Model) to be the perfect, super-smart assistant that learns exactly how you make these decisions and gets better at it over time?

Here is the breakdown of their solution, using some everyday analogies.

1. The Problem: The "Amnesiac" Assistant

The researchers first tested existing AI models to see if they could handle this job. They created a test called CALCONFLICTBENCH.

Think of this test like a video game where the AI has to manage a calendar for an entire year. Every week, the AI faces a new scheduling conflict.

The Result: The AI failed miserably. It was like a student taking a final exam who forgot everything they learned in the first week.
Why? The AI was "amnesiac." It could read the current meeting details, but it couldn't remember why it chose a specific meeting last month. It couldn't learn patterns like, "Oh, whenever James is in the meeting, I should attend," or "I always skip the 3 PM meetings unless it's an emergency."
The Stat: The best AI models made mistakes about 35% of the time. That's like flipping a coin and hoping for the right answer.

2. The Solution: PEARL (The "Smart Notebook" Assistant)

To fix this, the team built PEARL. The name stands for Preference Evolving Agent with Reinforcement Learning.

Think of PEARL not just as a robot, but as an assistant with two superpowers:

A. The "Strategy Hub" (The External Notebook)

Standard AI tries to remember everything by stuffing it all into its "brain" (context window). But the brain gets foggy and forgets details when the list gets too long.

PEARL uses an external notebook called the Strategy Hub.

How it works: Every time the AI makes a decision, it doesn't just say "Yes" or "No." It writes a note in its notebook: "Ah, I see. The user prioritizes 'Urgent Deadlines' over 'Internal Syncs'."
The Magic: Before making a new decision, the AI opens this notebook, reads the notes it wrote last week, and updates them if the user's habits have changed. It's like a human assistant who keeps a "User Preferences" binder and constantly rewrites it to stay accurate.

B. Reinforcement Learning (The "Coach")

Usually, AI learns by reading a book once (Supervised Learning). But PEARL learns by playing the game and getting coached.

The Coach: The system acts like a strict coach. After every decision round, the coach says:
- "Good job picking the right meeting!" (Reward)
- "Bad job! You picked the wrong one because you forgot the user hates morning meetings." (Penalty)
Curriculum Learning: The coach is smart. In the beginning, the coach focuses on helping the AI learn what the user likes. Later, the coach focuses on applying those likes perfectly. It's like teaching a kid to drive: first, you teach them the rules; later, you teach them how to handle traffic.

3. The Results: From "Clueless" to "Expert"

When they tested PEARL on the same difficult calendar game:

The Improvement: PEARL didn't just get slightly better; it got 55% better than the next best AI.
The Learning Curve: While other AIs kept making the same mistakes over and over, PEARL started with some errors but quickly learned the user's style. By the end of the "year," it was making almost perfect decisions.
The Error Reduction: It reduced the mistake rate by 76%.

The Big Picture Analogy

Imagine you are teaching a new intern how to manage your schedule.

Old AI: You give the intern a list of rules. They try to follow them, but when a new, weird situation comes up, they panic and guess. They don't remember what you did last time.
PEARL: You give the intern a notebook and a mentor.
1. Every time they make a choice, they write down why they chose it in the notebook.
2. The mentor (the reward system) checks their work. If they guessed wrong, the mentor says, "No, remember, you always skip Tuesday meetings."
3. The intern updates the notebook.
4. Next week, they open the notebook, see the pattern, and make the right choice automatically.

Why This Matters

This isn't just about calendars. It proves that AI can learn long-term habits and personal preferences if we give it the right tools (a memory notebook) and the right training method (learning from mistakes over time).

Instead of just being a "chatbot" that answers questions, PEARL shows we can build agents that evolve, adapt, and truly understand you over months and years, saving us from the headache of managing our own time.

1. Problem Definition: Calendar Conflict Resolution

The paper addresses the challenge of Calendar Conflict Resolution, a long-horizon, preference-driven decision-making task faced by busy professionals (e.g., CEOs, PIs). When overlapping meeting invitations occur, users must decide which events to attend, reschedule, or decline based on evolving personal and organizational priorities.

Key Challenges Identified:

Long-Horizon Dependencies: Decisions are not isolated; they depend on a history of past choices to infer user preferences (e.g., prioritizing leadership duties over routine meetings).
Preference Drift: Human assistants often fail at scale because they cannot reliably infer or update a user's implicit preference model over time, leading to errors that compound.
LLM Limitations: Current Large Language Models (LLMs) struggle with this task. They exhibit high error rates and fail to "evolve" their decision-making principles as they receive more context, often treating long contexts as static rather than dynamic learning signals.

2. Methodology: PEARL Framework

To address these gaps, the authors propose PEARL (Preference Evolving Agent with Reinforcement Learning), a framework designed to train agents to infer and adapt to user preferences online.

A. Benchmark: CALCONFLICTBENCH

Before proposing the solution, the authors introduced CALCONFLICTBENCH, a benchmark for systematic evaluation:

Synthetic Data Engine: Generates realistic, role-specific calendars (e.g., PI, PhD student, CEO) with ground-truth preference principles (e.g., "leadership duties > routine syncs").
Conflict Injection: Systematically creates overlapping events with varying difficulty (single-factor vs. multi-factor trade-offs).
Evaluation Protocol: Agents face a sequence of decision rounds ( $N$ ) over a simulated year. They must infer preferences progressively without explicit ground-truth rules, receiving feedback only after each decision.

B. The PEARL Architecture

PEARL combines an external memory module with a curriculum-based Reinforcement Learning (RL) approach.

Strategy Hub (External Memory):
- Instead of relying solely on long context windows (which LLMs struggle to utilize effectively), PEARL equips the agent with a persistent, structured external memory called the Strategy Hub.
- This hub stores a compact set of interpretable decision strategies (e.g., "Prioritize external stakeholder meetings over internal syncs").
- Agent Actions: At each round, the agent can:
  - Query (list): Retrieve current strategies.
  - Update (update): Refine or add new strategies based on the current conflict and history.
  - Decide: Accept or decline specific events.
Curriculum-Based Reward Modeling:
- To stabilize training over long horizons, PEARL uses a Round-Wise Reward system that shifts focus as the agent learns:
  - Format Reward ( $r_f$ ): Ensures syntactic validity.
  - Decision Reward ( $r_a$ ): Rewards correct acceptance/decline decisions (ground truth).
  - Ranking Reward ( $r_r$ ): Rewards placing the correct event high in the priority list (denser signal).
  - Interaction Reward ( $r_i$ ): Rewards valid interactions with the Strategy Hub.
- Curriculum Scheduling: The weights for Ranking and Interaction rewards are linearly interpolated based on the round index ( $t/N$ $t / N$ ).
  - Early Rounds: Emphasis on Preference Inference (Strategy Hub interaction and ranking).
  - Later Rounds: Emphasis on Preference Execution (Decision correctness).
Training Algorithm:
- Uses Group Relative Policy Optimization (GRPO).
- Computes advantages separately for each round position to prevent later rounds from dominating the gradient updates, ensuring stable learning across the entire trajectory.

3. Key Contributions

Task Formulation: Defined calendar conflict resolution as a long-horizon, preference-evolving task, moving beyond static, one-shot scheduling.
CALCONFLICTBENCH: Created a rigorous benchmark with synthetic data generation and evaluation metrics (Average Error Rate, Optimal Rank Distance, Error Reduction Rate) to expose the failure modes of current LLMs.
PEARL Framework: Introduced a novel RL framework integrating an explicit Strategy Hub for memory and a curriculum-based reward mechanism, enabling agents to progressively learn and adapt to user preferences.

4. Experimental Results

The authors evaluated PEARL against various baselines (Zero-shot, SFT, ReAct, Memory-Augmented ReAct) using models like Qwen3, OLMo3, and GPT-5.

Baseline Performance: Current LLMs perform poorly. For instance, Qwen3-30B-Think has an average error rate of 35%. Crucially, they show no preference-evolving capability; error rates remain flat or increase as the number of rounds ( $N$ ) increases, indicating they cannot learn from history.
PEARL Performance:
- Error Reduction: PEARL achieved an Error Reduction Rate (ERR) of 0.76, meaning it significantly improved its decision quality from the first quarter to the last quarter of the evaluation trajectory.
- Accuracy: PEARL reduced the Average Error Rate (AER) to 0.12, a 55% improvement over the strongest baseline (SFT, which had an AER of 0.27).
- Ablation: Simply adding the Strategy Hub without RL (Zero-shot + Hub) yielded minimal gains, proving that learning (via RL) is essential to effectively utilize the memory module.

5. Significance and Implications

Trust in AI Agents: The paper demonstrates that while current LLMs are unreliable for long-horizon time management, they can be made trustworthy through specific architectural changes (external memory) and training paradigms (RL with curriculum).
Beyond Static Prompts: It challenges the notion that larger context windows solve long-horizon reasoning. Instead, it shows that structured, explicit memory combined with online preference learning is superior.
Real-World Impact: A reliable calendar agent could save professionals hours of administrative work weekly, reducing productivity loss caused by scheduling logistics.
Future Directions: The work highlights the need for agents that can handle "preference evolution" and suggests that RL is a necessary component for agents operating in dynamic, multi-turn environments where user intent is implicit and evolving.

In conclusion, PEARL represents a significant step forward in making LLM agents capable of acting as reliable, self-improving personal assistants for complex, long-term decision-making tasks.