🤖 AI

ProEvent: An Event-centric Benchmark for Proactive Agents

The paper introduces ProEvent, the first event-centric benchmark for evaluating proactive agents' ability to autonomously track user events from instant messaging chats, revealing that current large language models struggle significantly with timing, event cancellation, and implicit reasoning.

Original authors: Guanzhen Li, Liangming Pan, Leye Wang

Published 2026-07-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Guanzhen Li, Liangming Pan, Leye Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart digital assistant that doesn't just wait for you to ask for help, but actually watches your life like a hawk, guessing what you need before you even say a word. This is the dream of "proactive agents." Unlike a regular robot that sits idle until you press a button, a proactive agent is like a personal chef who starts chopping vegetables the moment they smell the garlic, or a friend who hands you an umbrella the second the sky turns gray. The big challenge in computer science right now is teaching these digital brains to do this without getting confused by the messy, chaotic reality of human life. We know computers are great at following strict rules, but real life is full of hidden hints, overlapping conversations, and people changing their minds without saying "cancel." If we want these agents to be truly helpful, they need to be able to track our future plans—like a hiking trip or a meeting—just by listening to our text messages, and they need to know exactly when to speak up and when to stay quiet.

Enter PROEVENT, a new "training ground" designed by researchers to test if these digital assistants are actually ready for the real world. Think of PROEVENT as a giant, high-stakes video game level where the player is an AI trying to manage a human's calendar. Instead of giving the AI a neat list of appointments, the researchers feed it a stream of messy, realistic text chats. The AI has to listen to conversations between friends, colleagues, and family members, figure out what events are being planned, and then update the calendar accordingly. But here's the twist: the chats are full of distractions. There are side conversations about nothing, people changing plans halfway through, and messages that sound like cancellations but aren't. The AI has to be a detective, a scribe, and a time-manager all at once.

The researchers built this test by creating thousands of fake but very realistic chat logs. They programmed "characters" with different personalities and goals, then simulated complex scenarios where people negotiate meeting times, cancel plans because they feel sick, or try to schedule things that never actually happen. They even added "noise"—random messages that have nothing to do with the main event—to see if the AI would get distracted. The goal was simple: can the AI correctly build and update a timetable just by reading these chats?

When they put eight different powerful AI models through this test, the results were a mix of "not bad" and "oh no." The biggest surprise was that the AIs tend to be over-eager. Imagine a dog that barks at every leaf that moves; these AIs were constantly trying to update the calendar even when nothing had changed. In fact, for some models like DeepSeek-V3.2, they tried to make changes more than 96% of the time when they should have just stayed silent. They were so afraid of missing a chance to help that they ended up creating chaos.

Another major hurdle was cancellation. If a user says, "I'm not feeling well, maybe I can't make it," a human knows that's a cancellation. The AIs were actually quite good at spotting that a deletion might be needed (catching about half of the actual cancellations), but they struggled to be sure about it. They frequently deleted events that were still happening or removed the wrong plans entirely. The paper suggests that even the most advanced models, like the one named GPT-5.1, only got the right answer in about 26.7% of the tricky scenarios. That means in roughly three out of four cases, the AI messed up the schedule.

The study also found that these digital brains have a weird blind spot: they don't really understand who they are helping. When a user says, "I can't go to the party," the AI often just removes the user's name from the guest list but leaves the party on the calendar. It's like a waiter who, when you say you're full, just takes your name off the reservation but leaves the table set for you. The AI is looking at the event from the outside, like a third-party observer, instead of stepping into the user's shoes and realizing, "Oh, I (the user) am not going, so this event is over for me."

In short, while these AI agents are getting smarter, they aren't quite ready to be your proactive life manager just yet. They are too jumpy, they get confused by complex conversations, and they struggle to understand the difference between a "maybe" and a "no." The researchers hope that by using PROEVENT to show exactly where these AIs fail, future versions will learn to be more patient, more accurate, and finally, more like a helpful human friend who knows exactly when to hand you that umbrella.

Technical Summary: PROEVENT – An Event-centric Benchmark for Proactive Agents

Problem Definition

Current Large Language Model (LLM) based agents are predominantly reactive, relying on explicit user instructions to perform tasks. This limitation leads to repetitive human intervention and the risk of missing time-sensitive information. While "proactive agents" are expected to anticipate user needs by perceiving environmental context, existing benchmarks largely overlook event-centric assistance—the ability to identify, track, and maintain a user's upcoming events based on ongoing interactions.

Furthermore, the open-ended nature of proactive assistance poses significant challenges for reliable evaluation. Existing benchmarks often rely on semantic similarity or LLM-simulated human judges, which lack objective grounding. There is a critical gap in evaluating an agent's ability to:

Continuously track users' upcoming events from unstructured, noisy instant messaging (IM) chats.
Correctly determine the timing of assistance (when to act vs. when to wait).
Accurately perform event management operations (Insert, Update, Delete) in complex, multi-threaded environments.

Methodology: PROEVENT Benchmark

To address these gaps, the authors introduce PROEVENT, the first event-centric benchmark designed to assess proactive agents' ability to maintain a user's timetable derived from ongoing IM chats.

1. Task Formulation

The task is formalized as a discrete-time sequence where an agent receives a set of chats $D_t$ within a time interval $[t-1, t]$ . The goal is to update the user's timetable state $S_{t-1}$ to a new state $S_t$ by generating a list of explicit operations:

Insert: Add a new event with specific attributes (time, location, participants, description).
Update: Modify attributes of an existing event.
Delete: Remove an event (crucial for handling cancellations).

The agent must process multiple concurrent chat threads and distinguish relevant information from noise.

2. Data Construction Pipeline

PROEVENT utilizes a rigorous four-step pipeline to synthesize realistic, high-quality data:

Contact Pool Construction: 56 distinct contact profiles (e.g., colleagues, friends) with specific roles and event types are defined to ensure diversity.
Chat Synthesis: A "scheduling trajectory" is generated first, recording the ground-truth evolution of events (e.g., time changes, cancellations). This trajectory guides the generation of multi-turn dialogues using an LLM (GPT-OSS-120B), ensuring alignment with the underlying event timeline.
Noise Injection: To simulate real-world conditions, three types of noise are injected:
- Message-level: Irrelevant messages interleaved with relevant ones.
- Chat-level: Entire chats unrelated to the target event.
- Event-level: Distracting historical or tentative events.
Scenario Construction: Multiple chats are merged and segmented into fixed time windows to simulate concurrent threads and periodic context collection.

The final dataset comprises 1,249 high-quality chats, generating 2,463 single-step questions and 1,279 multi-step cases covering 7 event categories and 56 subcategories.

3. Evaluation Suite

The benchmark evaluates agents across three dimensions:

Response Timing: Measured by False Detection Rate (FDR) (triggering action when unnecessary) and Missed Need Rate (MNR) (failing to act when needed).
Single-Step Correctness: Precision and Recall of operations at each time step.
Multi-Step Correctness: Event Success Rate (ESR) and Timetable Success Rate (TSR), which measure the proportion of events and entire timetables that are perfectly correct after multiple accumulation steps. This resolves ambiguities where different operation sequences (e.g., Update vs. Delete+Insert) lead to the same final state.

Key Results

The authors evaluated eight LLMs and pipelines (including Qwen-3, DeepSeek-V3.2, DeepSeek-R1, GPT-5.1, and specialized pipelines like ProCoT) on PROEVENT.

Performance Gap: Even the strongest models struggle. GPT-5.1 achieves a Timetable Success Rate (TSR) of only 26.7%, indicating that current LLMs are far from delivering dependable proactive assistance.
Overreaction Bias: Models exhibit a systemic bias toward over-responsiveness. For instance, DeepSeek-V3.2 has a False Detection Rate (FDR) of 96.5% versus a Missed Need Rate (MNR) of 21.9%. They frequently trigger actions when no intervention is required.
Difficulty with Cancellations: Identifying implicit event cancellations is a major failure point. While some models achieve high recall for deletions, their precision is low, leading to the removal of events without sufficient evidence.
Impact of Complexity: Performance degrades significantly as the number of negotiation turns and concurrent chat threads increases.
Reasoning Improves Performance: Models with stronger reasoning capabilities (e.g., DeepSeek-R1, ProCoT) show better precision in distinguishing genuine cancellations from temporary changes. The ProCoT pipeline, which explicitly prompts the model to evaluate the necessity of an action, reduced the FDR of DeepSeek-V3.2 by 70%.

Qualitative Analysis and Limitations of Current LLMs

The paper identifies three fundamental deficiencies in current LLMs acting as proactive agents:

Perspective Misalignment: LLMs often reason from a third-person perspective rather than the user's first-person viewpoint. For example, when a user states they cannot attend an event, the model often updates the participant list (removing the user) rather than deleting the event from the user's personal timetable.
Implicit Instruction Failure: Models struggle to detect operational requirements embedded in narrative styles (e.g., "invite the same people") without explicit structural cues.
Reasoning Deficits: Weaker models fail at temporal reasoning (converting dates to weekdays) and memory-enhanced reasoning (retrieving the correct event ID among multiple similar events).

Significance and Claims

The paper claims that PROEVENT is the first benchmark to specifically evaluate the ability of agents to proactively maintain a user's timetable from instant messaging chats. Its significance lies in:

Bridging the Evaluation Gap: It provides an objective, ground-truth-based evaluation protocol for proactive agents, moving beyond semantic similarity metrics.
Realistic Complexity: It introduces a data generation pipeline that captures dynamic interactions, concurrent threads, and pervasive noise, which are critical for real-world deployment but often ignored in synthetic datasets.
Diagnostic Value: The benchmark reveals that current LLMs, even state-of-the-art models like GPT-5.1, are prone to over-anticipation and struggle with the nuance of event cancellation and user-centric reasoning.

The authors conclude that PROEVENT serves as a challenging benchmark to facilitate future research on enhancing LLM capabilities for proactive event tracking in real-world settings, highlighting that significant improvements in reasoning and perspective alignment are required before these agents can be reliably deployed.