💻 computer science

PEARL: Personalized Streaming Video Understanding Model

This paper introduces the novel task of Personalized Streaming Video Understanding (PSVU), accompanied by the PEARL-Bench benchmark and a training-free PEARL strategy, to enable real-time, interactive personalized responses in vision-language models by bridging the gap between continuous visual input and instant feedback.

Original authors: Yuanhong Zheng, Ruichuan An, Xiaopeng Lin, Yuxing Liu, Sihan Yang, Huanyu Zhang, Haodong Li, Qintong Zhang, Renrui Zhang, Guopeng Li, Yifan Zhang, Yuheng Li, Wentao Zhang

Published 2026-03-24

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yuanhong Zheng, Ruichuan An, Xiaopeng Lin, Yuxing Liu, Sihan Yang, Huanyu Zhang, Haodong Li, Qintong Zhang, Renrui Zhang, Guopeng Li, Yifan Zhang, Yuheng Li, Wentao Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a long, continuous movie with a friend. In the middle of the movie, your friend points at a character and says, "That's Bob." A few minutes later, they point at a specific dance move and say, "That's the Moonwalk."

Now, imagine your friend asks you questions about Bob and the Moonwalk while the movie is still playing. They might ask, "Is Bob wearing a hat right now?" (Real-time) or "What was Bob doing 10 minutes ago?" (Past-time).

The Problem:
Current AI assistants are like people who only have a short-term memory or who only watch the movie after it's finished.

Static AI: They can look at a single photo and say, "That's Bob." But if the movie keeps playing, they forget who Bob is.
Offline Video AI: They can watch the whole movie and then answer questions, but they can't talk to you while it's happening. They can't say, "Oh, I see Bob just walked in!" in real-time.
The Gap: Real life is a streaming video. We need an AI that can watch a live stream, learn new names and actions on the fly, and answer questions about them instantly, just like a human does.

The Solution: PEARL
The paper introduces a new task called Personalized Streaming Video Understanding (PSVU) and a new AI framework called PEARL to solve this.

Here is how PEARL works, using a simple analogy:

1. The Two-Notebook System (Dual-grained Memory)

Imagine PEARL has two special notebooks to help it remember things while the movie plays:

Notebook A: The "Who's Who" List (Concept Memory)
When your friend says, "This is Bob," PEARL writes down a quick description in this notebook: "Bob is a guy with messy blonde hair and blue eyes." It doesn't just write the name "Bob"; it writes what Bob looks like so it can recognize him later, even if he changes his shirt.
Notebook B: The Movie Log (Streaming Memory)
As the movie plays, PEARL constantly takes short video clips and writes a "summary" of what happened in each clip. It's like a librarian who keeps a running log of every scene in the library, organized by time.

2. The Smart Search Engine (Concept-aware Retrieval)

When your friend asks a question, PEARL doesn't just guess. It uses a clever search strategy:

The Question: "What is Bob doing?"
The Translation: PEARL looks at its "Who's Who" list, sees that Bob is "the guy with messy blonde hair," and rewrites the question in its own internal language: "What is the guy with messy blonde hair doing?"
The Search: It then flips through its "Movie Log" (Notebook B) to find the exact moment where a guy with messy blonde hair appears. It pulls up that specific clip and shows it to the main AI brain.
The Answer: The AI brain looks at the clip and says, "He is holding a coffee cup."

3. Why This is a Big Deal

The authors also built a test track called PEARL-Bench. Think of this as a driving test for AI.

The Track: It has 132 long videos (like movies and anime) with 2,000+ specific questions.
The Challenge: The questions require the AI to remember specific people or actions defined during the video and answer questions about them later.
The Result: When they tested PEARL against other top AI models, PEARL won easily. It was the only one that could handle the "live stream" nature of the task without getting confused or forgetting things.

The Takeaway

Before this paper, AI was like a student who could only study a textbook (static images) or watch a recorded lecture (offline video) and then take a test.

PEARL is like a student who can sit in a live lecture hall, take notes on new concepts the professor introduces, and answer questions from the audience while the lecture is still happening.

This is a crucial step toward building real AI assistants for the future—robots or apps that can watch your workout, remember your specific form, and correct you in real-time, or a robot that can watch your kids playing and remember who is who, even if they run in and out of the room.

1. Problem Definition: Personalized Streaming Video Understanding (PSVU)

Current Vision-Language Models (VLMs) face a critical disconnect between human cognition and AI capabilities. Humans naturally process the world as a continuous visual stream, recognizing new objects/identities dynamically and updating memories in real-time. In contrast, existing personalization methods are largely limited to:

Static Images: Focusing on single-frame recognition (e.g., Yo'LLaVA, MC-LLaVA).
Offline Videos: Processing pre-recorded, short clips with single-turn interactions (e.g., PVChat).

These approaches fail to support real-time, multi-turn interactions where concepts are defined dynamically on-the-fly within a continuous video stream. The paper formally defines the new task of Personalized Streaming Video Understanding (PSVU), which requires models to:

Process continuous streaming video inputs.
Dynamically define user-specific concepts (names for people or actions) at arbitrary timestamps.
Perform multi-turn reasoning involving both real-time queries (current state) and past-time queries (historical evidence retrieval).

2. Methodology: The PEARL Framework

To address the challenges of maintaining unbounded stream history and instant concept registration without computationally expensive retraining, the authors propose PEARL, a training-free, plug-and-play framework.

A. Dual-Grained Memory System

PEARL decouples concept-centric knowledge from stream-centric observations using two distinct memory modules:

Streaming Memory:
- Continuously archives segmented video clips (detected via scene boundaries) into a sequence.
- Each clip is encoded into a compact multimodal embedding ( $e_i$ ) using a pre-trained embedding model.
- This allows for efficient retrieval of historical visual evidence without storing raw video indefinitely.
Concept Memory:
- Stores structured representations of user-defined concepts.
- When a user defines a concept (e.g., "This is "), the system extracts visual evidence (a frame for static entities, a clip for actions) and generates a textual description of salient features.
- This creates a mapping between the user-defined name (e.g., <Bob>) and its visual/semantic description.

B. Concept-Aware Retrieval Algorithm

When a user query arrives, PEARL executes a retrieval pipeline to construct the context for the VLM:

Concept Retrieval: Identifies concept names in the query and retrieves their corresponding descriptions from the Concept Memory.
Query Rewriting: Replaces the abstract concept names in the user's query with their specific visual descriptions (e.g., replacing <Bob> with "a man with a red hat"). This bridges the gap between user-defined tokens and the embedding model's vocabulary.
Visual Evidence Retrieval: The rewritten query is embedded and compared against the Streaming Memory embeddings via cosine similarity.
- Top- $K$ relevant historical clips are retrieved.
- Adjacent clips ( $N$ ) are expanded to capture temporal context.
Inference: The VLM receives the current clip, the retrieved historical clips, the retrieved concept descriptions, and the original query to generate the final response.

3. Key Contributions

A. New Task and Benchmark: PEARL-Bench

The authors introduce PEARL-Bench, the first comprehensive benchmark for PSVU.

Scale: 132 unique videos (anime, movies, reality shows, digital humans) with 2,173 fine-grained annotations.
Granularity:
- Frame-level: Recognizing static entities (people/objects) across discrete frames.
- Video-level: Recognizing dynamic actions unfolding across continuous frames (a novel capability).
Query Types:
- Concept-Definition QA: Registering new concepts.
- Real-Time QA: Querying the current state.
- Past-Time QA: Retrieving historical evidence to answer questions about prior states.
Data Quality: Ensured via automated filtering (ablation-based) and rigorous human verification by experts.

B. Training-Free Framework

PEARL requires no parameter updates or fine-tuning. It acts as a wrapper around existing VLMs, making it highly adaptable and efficient for deployment.

C. State-of-the-Art Performance

PEARL was evaluated against 8 baseline models (including offline and online SOTA models like LLaVA-OV, Qwen2/3-VL, and proprietary models like Gemini).

4. Experimental Results

Performance Gains: PEARL significantly outperforms all baselines across different architectures.
- Frame-level: Average improvement of 13.79% over base models.
- Video-level: Average improvement of 12.80%.
- Specifics: Qwen3-VL-8B + PEARL achieved 52.24% average accuracy, surpassing the best online baseline (StreamForest-7B) by 17.22% and the proprietary Gemini3-pro-preview by 4%+.
Robustness: The framework consistently improves performance across 3 distinct model architectures (LLaVA, Qwen2, Qwen3), proving its architecture-agnostic nature.
Ablation Studies:
- Concept Memory is critical: Without it, models cannot link names to entities (Real-Time accuracy drops to ~11%).
- Streaming Memory is essential for Past-Time QA: Without historical retrieval, accuracy on past events is near random.
- Query Rewriting: Improves retrieval accuracy by translating names into descriptive semantics.
Efficiency: While introducing a minor latency overhead (due to retrieval and rewriting), PEARL maintains real-time responsiveness. The core retrieval modules add negligible latency compared to the LLM inference bottleneck.

5. Significance and Impact

Bridging the Gap: This work addresses the fundamental limitation of current AI assistants: the inability to handle continuous, evolving visual streams with dynamic personalization.
Practical Application: The framework enables real-world use cases such as customized fitness coaching (monitoring specific user actions in real-time), social wearable AI, and personalized robotics.
Future Direction: By defining PSVU and providing a robust baseline (PEARL) and benchmark (PEARL-Bench), the paper sets the stage for the next generation of interactive, memory-augmented AI assistants that can truly "learn" from users in real-time without retraining.

In summary, PEARL transforms static VLMs into dynamic, personalized streaming agents through a clever memory-augmented architecture, achieving state-of-the-art results without the cost of retraining.