💬 NLP

Benchmarking Living-Screen-Native GUI Agents on Short-Video Platforms

This paper introduces LivingScreen, the first benchmark for "Living-Screen-Native" GUI agents that operate on dynamic short-video platforms, revealing that current frontier models fail to match human performance due to an inability to effectively control observation timing amidst continuously changing content.

Original authors: Jiashu Yao, Heyan Huang, Daiqing Wu, Wangke Chen, Huaxi Ai, Haoyu Wen, Zeming Liu, Yuhang Guo

Published 2026-06-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jiashu Yao, Heyan Huang, Daiqing Wu, Wangke Chen, Huaxi Ai, Haoyu Wen, Zeming Liu, Yuhang Guo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to use a smartphone. Most current tests for these robots are like giving them a frozen photograph of a screen. The robot looks at the picture, decides what to click, and then the picture changes. It's a bit like playing a game of "Simon Says" where the board never moves unless you touch it.

But real life isn't frozen. Think of TikTok, Instagram Reels, or YouTube Shorts. The videos play automatically, comments scroll by, and new content loads while you are still watching. The screen is "alive."

This paper introduces a new way to test robots (called GUI Agents) on these "living" screens. Here is the breakdown in simple terms:

1. The Problem: The "Frozen Screen" vs. The "Living Screen"

Current AI models are great at looking at a static image or a pre-recorded video file. But they struggle when the screen is constantly changing on its own.

The Old Way: The AI gets a snapshot, thinks, clicks, and then gets a new snapshot.
The New Way (Living-Screen-Native): The screen keeps playing. The AI has to decide: "Should I watch this whole video? Should I just glance at the first 3 seconds? Should I skip this one entirely?"

The authors call this "Living-Screen-Native" because the agent has to navigate a screen that evolves in real-time, just like a human does.

2. The Solution: "LIVINGSCREEN" (The New Test)

To test this, the researchers built a special playground called LIVINGSCREEN.

The Environment: It's a realistic fake version of a short-video app inside a web browser. It plays real videos, shows comments, and has buttons for "like," "share," and "follow."
The Rules: The AI can't just "download" the video file. It has to interact with the screen exactly like a human: swiping up, clicking buttons, and deciding how long to watch a clip.
The Tasks: The test has three levels of difficulty:
- Level 1 (Basic): Can you click the "like" button or scroll down?
- Level 2 (Understanding): Can you watch a few videos, read the comments, and figure out if two videos tell the same story?
- Level 3 (The Boss Fight): Can you act like a human user? For example, "Find videos that break the rules and report them," or "Find videos about cooking and save the best ones."

3. The Results: The Robots Are Clumsy

The researchers tested the smartest AI models available today on this new test. The results were surprising:

Humans Win: Humans are much better at balancing speed and accuracy. We know exactly how long to watch a video to get the answer.
AI Struggles: Even the best AI models failed to match human performance. They were either too slow (watching too much) or too hasty (missing important details).

4. The Big Discovery: "Over-Observing" and "Under-Observing"

The paper found that the main reason AI fails isn't that it can't see or click; it's that it doesn't know how to watch. The authors call this "Over- and Under-Observation."

Think of it like a student taking a test:

Under-Observation: The student glances at a question, guesses the answer immediately, and moves on, missing the crucial detail in the middle of the paragraph.
Over-Observation: The student reads the same paragraph five times, even after they already understood it, wasting time and energy.

The AI's Problem:

Some AIs glance too little. They skip videos they should have watched, leading to wrong answers.
Some AIs watch too much. They stare at irrelevant videos for minutes, wasting time, without actually getting smarter about the task.
The Human Difference: Humans are "smart scanners." We do a quick 2-second "glance" to see if a video is relevant. If it is, we watch longer. If not, we swipe away. The AI models mostly lack this "quick glance" filter. They tend to either ignore a video completely or binge-watch it.

5. Can We Just Tell Them to Do Better?

The researchers tried giving the AI models simple instructions, like "Watch less" or "Watch more," or even telling them to "Copy how humans watch."

The Result: It didn't work well. Telling the AI to "watch less" made it skip too many important videos. Telling it to "watch more" made it waste time.
The Conclusion: This isn't just a matter of giving better instructions. The AI models genuinely lack the capability to control their own attention. They don't know how to decide when to look and when to stop looking.

Summary

This paper says that to build truly helpful AI assistants for apps like TikTok or YouTube, we need to stop testing them on frozen screens. We need to teach them how to manage their attention in a world that never stops moving. Currently, the best AI models are like a person who either never looks at the menu or reads every single word on it for an hour—they just haven't learned how to "glance" effectively yet.

Technical Summary: Benchmarking Living-Screen-Native GUI Agents on Short-Video Platforms

1. Problem Definition

Current Graphical User Interface (GUI) agents operate under a static assumption: the screen is frozen between two actions, and the world only evolves when the agent acts. This paradigm fails to capture real-world interfaces, particularly short-video platforms, where content plays autonomously in continuous time. In these environments, a competent user must actively decide what to watch, how long to watch, and when to stop, balancing information acquisition against time costs.

Existing benchmarks either treat video as a passive input (Video QA) or treat the GUI as a sequence of static screenshots (standard GUI agents), missing the "living screen" quadrant where the environment evolves independently while the agent proactively selects observation granularity. The authors formalize this gap as Living-Screen-Native GUI Agents, defined as agents operating on screens evolving in continuous time that actively determine which visual slices of that evolution to observe.

2. Methodology: The LIVINGSCREEN Benchmark

To instantiate and evaluate this setting, the authors introduce LIVINGSCREEN, the first benchmark for living-screen-native agents on short-video platforms.

2.1 Environment Construction

Platform: A high-fidelity, browser-based replica of a modern short-video application (modeled after Chinese platforms), built using Playwright.
Interaction: Agents interact via a low-level action API (pixel-coordinate clicks, swipes, typing) and observe exclusively through rendered screenshots or screen recordings, without access to underlying DOM or video files.
Content: The corpus consists of real short videos from public datasets, augmented with metadata (titles, hashtags, comments) and a small fraction (<1%) of synthesized videos for long-tail safety risks.
Feed Generation: Feeds are constructed by clustering videos by keywords and mixing them with random distractors to simulate a realistic recommendation stream.

2.2 Task Taxonomy

The benchmark comprises 499 tasks across 1,528 unique videos, organized into a three-tier hierarchy:

L1 (Atomic GUI Action): Tests elementary primitives like liking, collecting, commenting, reporting, and navigating (swiping/seeking).
L2 (Cross-Source Understanding): Requires integrating evidence across videos, comments, and UI cues. Sub-tasks include contextual association, event comparison, feature analysis, spatiotemporal aggregation, and robust evaluation (handling unanswerable questions).
L3 (Closed-Loop Application): Demands a full loop of perception, reasoning, and action to achieve environment-graded outcomes. Examples include fact-checking contested events, content moderation (policy violation detection), and preference simulation.

2.3 Evaluation Metrics

The benchmark scores agents on two dimensions:

Accuracy: Measured by Success Rate (SR), determined by comparing the final environment state against ground truth (e.g., specific videos liked/reported) or option matching.
Efficiency:
- Number of Steps (NS): Average tool calls per episode (operational cost).
- Watch Ratio (WR): The fraction of total feed runtime actually recorded by the agent (observational cost).
- Goal: High SR with low NS and WR.

2.4 Agent Formalization

The authors propose a Living-Screen-Native POMDP that modifies the standard Markov Decision Process:

Continuous Evolving State ( $\Delta_1$ ): The state evolves autonomously ( $\Phi$ ) even when the agent is idle (e.g., playhead advancing), distinct from the agent-driven state transitions in standard GUI POMDPs.
Agent-Initiated Observation ( $\Delta_2$ ): Observations are not pointwise samples but are parameterized by the agent's action. The agent decides the time window $W(a_k)$ to observe, choosing between a single screenshot or a video clip of duration $\Delta t$ .

3. Experimental Setup

The authors evaluated a representative set of frontier Multimodal Large Language Models (MLLMs), including native video models (Gemini-3.5, Doubao-Seed series, Qwen-3.6, etc.) and image-based models (Claude, GPT-5 variants). All models were wrapped in a unified agent architecture using a "think-then-tool-call" format, with a maximum episode length of 30 steps.

4. Key Results

Performance Gap: No evaluated model approaches human performance on the cost-accuracy tradeoff. The best models achieve significantly lower Success Rates (SR) than humans (e.g., top models ~69% vs. Human 94% overall) while often incurring higher observation costs.
Divergent Paradigms: Models exhibit inconsistent strategies. Some favor high exploration (high NS, fragmented info gathering), while others favor heavy observation (high WR, watching long clips).
Dominant Failure Mode: The primary failure mode identified is Over- and Under-Observation:
- Under-Observation: Agents skip videos or glance too briefly to gather sufficient evidence (affecting 39–61% of trajectories).
- Over-Observation: Agents watch irrelevant videos or continue watching after the necessary information is obtained (affecting 25–43% of trajectories).
- Co-occurrence: 9–24% of trajectories exhibit both errors simultaneously.
Human Comparison: Humans employ an "information foraging" strategy, spending a small amount of attention to "glance" at videos (52% of observations) to filter relevance before committing to deep consumption. Current models lack this cheap scouting step, leading to inefficient attention allocation.

5. Analysis of Observation Control

The authors investigated whether these failures are due to a lack of awareness (prompting) or capability deficits.

Prompting Experiments: Variants instructing agents to "Watch Less," "Watch More," or "Mimic Human" strategies successfully shifted the Watch Ratio (WR) but failed to improve Success Rate (SR).
Conclusion: Over- and under-observation are not merely awareness failures solvable by prompting; they represent a genuine capability gap in current MLLMs regarding observation control.

6. Significance and Contributions

The paper claims three primary contributions:

Formalization: It identifies and formalizes the "Living-Screen-Native" setting, distinguishing it from static GUI and passive video QA tasks.
Benchmark Release: It introduces LIVINGSCREEN, providing the first faithful environment, task suite, and efficiency metrics for this setting.
Capability Gap Identification: It uncovers observation control (specifically the management of over- and under-observation) as a critical, previously under-studied axis for future GUI agent research.

The authors position LIVINGSCREEN as a stepping stone toward agents capable of inhabiting truly living interfaces, emphasizing that future progress requires models to master not just what to click, but how and how long to observe.

7. Limitations

The authors acknowledge that the benchmark is a controlled browser replica, lacking live recommendation engines or social graph effects. The data is exclusively in Chinese, and the scale (499 tasks) is intermediate, reflecting the high cost of annotation required for dynamic, multi-step tasks. Ethical considerations regarding the dual-use potential of autonomous agents for misinformation or evasion are noted, recommending human-in-the-loop oversight for real-world deployment.