🤖 AI

Activity Frames: Deterministic Screen-Activity Compilation for Agent Memory and Replay

This paper introduces "Activity Frames," a deterministic, zero-model pipeline that compiles raw screen activity into compact, auditable memory blocks, enabling agents to answer questions about past user actions with significantly higher accuracy than LLM summaries while simultaneously providing the first empirical measurements of routine overhead and recurrence to optimize agent delegation costs.

Original authors: Nossa Iyamu

Published 2026-08-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Nossa Iyamu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a super-smart robot assistant how to help you with your day. You might think the robot just needs to listen to your voice commands, like "send an email" or "check my calendar." But there's a big problem: the robot can only hear what you say, not what you actually do. If you spend three hours staring at a spreadsheet, switching between twenty different websites, and typing furiously, the robot has no idea that happened unless you tell it. This is the world of "AI agents"—computer programs designed to act like humans. For these agents to be truly helpful, they need "episodic memory," which is just a fancy way of saying a record of what happened, when it happened, and what you were looking at. Without this, the robot is like a friend who only remembers your conversations but has amnesia about your actual life.

The big question researchers are asking is: How do we give these robots a memory of your screen activity without making them confused, expensive, or unreliable? Currently, some methods try to summarize your day using other AI models, but those summaries can be messy, change every time you ask, or even make things up. Others just dump a massive list of raw data, which is too huge for the robot to read. We need a middle ground: a way to turn your chaotic screen time into a clean, trustworthy, and tiny summary that the robot can actually use.

This paper introduces a clever solution called "Activity Frames." Think of it as a super-organized librarian who watches your screen, but instead of using a brain to guess what you're doing, they use a strict, unchangeable set of rules to sort your day. The researchers built a system that takes a day's worth of screen snapshots and compiles them into a neat, structured story. It doesn't use any AI to "think" about what happened; it just follows a recipe. Because it's a recipe, the result is always exactly the same if you run it twice. This makes the memory safe to store, easy to check, and tiny enough to fit in the robot's pocket.

The team tested this on their own computer over 51 days. They found that this "deterministic" method (meaning no guessing involved) shrinks a day's worth of raw data down to a size that is 86 times smaller than the original, and it does this in just 68 milliseconds—faster than you can blink. When they asked an AI agent to answer questions about the day using this new "Activity Frame" memory, the agent got it right 98.4% of the time. In comparison, when the agent tried to read a summary written by another AI, it only got it right between 66% and 80% of the time. The new method was so good that a smaller, cheaper AI model could answer questions just as well as a giant, expensive one when given this clean memory.

The paper also uses this system to measure something called the "Routine Overhead Ratio." Imagine you ask a robot to do a task it has done a hundred times before. If the robot has to re-figure out every single click and keystroke from scratch every time, it wastes a huge amount of energy and money. The researchers found that by using their compiled memory to replay these routines, the robot could save a massive amount of effort. They calculated that re-deriving a routine from scratch costs about 60 to 343 times more than just replaying the recorded script. They also measured that about 9% of the actions a person takes on their computer are part of these repeatable routines that could be handed off to a robot to handle automatically.

However, the paper is careful to point out what this isn't. It doesn't claim to know why you did something (your intent), only what you did. It also admits that the data comes from just one person's computer, so it's a proof of concept rather than a final rule for everyone. The "replay" part was tested in a controlled way, showing it works perfectly when the screen looks exactly the same, but it might stumble if the website changes its layout. The researchers emphasize that this isn't a magic wand that solves all AI problems, but rather a solid, boring, and reliable tool that turns messy screen data into something an AI can actually trust and use.

Technical Summary: Activity Frames: Deterministic Screen-Activity Compilation for Agent Memory and Replay

1. Problem Statement

Current Large Language Model (LLM) agents suffer from a critical memory deficit: they retain what users say (conversation history) but lack a record of what users do (behavioral episodic memory). While screen capture technologies (e.g., Microsoft Recall, OpenAI's Chronicle) have made raw data collection commodity, the consumption of this data remains inefficient. Existing strategies rely on either:

Raw Search: Feeding flat lists of snapshot rows to the agent, which incurs massive token costs (e.g., ~126k tokens for a single day) and forces the model to perform sessionization and duration accounting at inference time.
LLM Summarization: Compressing data via generative models, which introduces non-determinism, per-run costs, reliability degradation over long contexts, and the risk of hallucinating activities that never occurred.

Neither approach yields memory that is cacheable, mechanically auditable, or trustworthy. The paper posits that the missing component is deterministic compilation: a pipeline that transforms raw capture streams into structured, typed activity frames without any model in the loop.

2. Methodology: The Activity Frames Compiler

The proposed system is a zero-model, deterministic pipeline that compiles passive screen capture data into a structured schema.

2.1 The Two-Tier Schema

The system employs a schema that strictly separates measured facts from inferred interpretations:

Tier 1 (Measured): Produced entirely by deterministic code reading the capture database. Fields include application, site (URL host), start/end times, dwell-based active time, input volume (keystrokes/clicks), and evidence pointers back to raw rows. This tier contains no intent labels.
Tier 2 (Inferred): An optional extension for tools to add interpretation (e.g., task labels). This tier is strictly quarantined: inferred content must be namespaced, confidence-tagged (high/medium/speculative), and linked to specific measured evidence. Consumers can strip this tier to leave a valid, purely measured document.

2.2 Deterministic Compilation Rules

The compiler processes event-driven capture streams (snapshots on interaction, screen changes, and periodic heartbeats) using specific rules to segment and enrich data:

Sessionization: Frames are bounded stretches of attention keyed by (application, site).
Dwell Crediting: Active time is calculated as min(∆t, 90s), where ∆t is the gap to the next frame. This accounts for reading/watching time but caps credit to prevent inflation during capture stalls.
Flicker Merging: Short detours (A → B → A where B ≤ 20s) are merged into a single frame, with B recorded as an interruption rather than a new session.
Enrichment: The system repairs reliability defects in raw data, such as re-attributing stale events to the nearest snapshot, resolving anonymous clicks against element trees, and handling keyboard layout mismatches via explicit operator maps.
Entity Typing: URLs are mapped to typed references (e.g., pull_request, profile, email) using deterministic parsers. Unmatched URLs fall back to a generic reference with the domain, ensuring no data loss.

2.3 Implementation

The reference implementation is a dependency-free Python package featuring:

A local capture engine (recording focus, titles, URLs, accessibility trees, and inputs to a local SQLite DB).
A compiler that emits JSON, YAML, Markdown, and compact context blocks.
An MCP (Model Context Protocol) server exposing tools for context retrieval, activity summaries, and pattern detection.

3. Key Results

The system was evaluated on a single-user corpus of 61 calendar days (46 active days), comprising ~110k snapshot rows and 214k input events.

3.1 Token Cost and Efficiency

Compression: Compiling a full day of raw capture (2,066 rows) into a prompt-ready context block reduces token count by 86× (from 126,812 tokens to 1,469 tokens).
Latency: Full-day compilation takes a median of 68 ms on a consumer laptop.
Reproducibility: The output is byte-identical across runs, enabling caching and mechanical auditing.

3.2 Downstream Question Answering (QA)

An independent SQL oracle generated ground-truth answers for 64 questions across 8 days. Two models (Claude Sonnet 4.5 and Opus 4.5) were tested on three representations: raw rows, LLM summaries, and the compiled activity frames.

Accuracy: The compiled block achieved 98.4% accuracy at both model tiers.
- Raw rows: 82.1% (Sonnet) / 91.1% (Opus).
- LLM Summaries: 66.1% (Sonnet) / 80.4% (Opus).
Hallucination: Both models achieved 0% hallucination on "absent-fact" probes (denying non-existent apps/sites) across all representations.
Magnitude Errors: LLM summaries failed significantly on duration magnitudes (e.g., misstating active minutes by 135% on average for Sonnet), whereas the compiled block's error was negligible (7.3%).
Model Tier Independence: The compiled block allowed a mid-tier model (Sonnet) to match the performance of a frontier model (Opus), whereas raw data and summaries showed a significant performance gap between tiers.

3.3 Routine Overhead Ratio ( $R$ ) and Recurrence ( $h$ )

The compiler acts as a demand-side cost instrument, measuring parameters previously assumed but unmeasured in agent cost models:

Routine Overhead Ratio ( $R$ ): The ratio of tokens required for an agent to re-derive a routine versus replaying a compiled script.
- Operational $R$ (Guarded Plan): Median 60× (injecting a guarded skill plan costs ~1/60th of re-deriving).
- Ceiling $R$ (Minimal Script): Median 343× (bare information content).
Routine Recurrence ( $h$ ): The fraction of action steps falling into recurring, delegable routines.
- Raw Rate: 83.1% (dominated by generic input micro-structure).
- Delegable Rate ( $h_{specific}$ ): 9.0% in-sample and 7.7% out-of-sample (temporal holdout). This represents the fraction of work that is truly repeatable and delegable.

4. Significance and Claims

The paper argues that deterministic compilation transforms episodic memory from an inference-heavy, unreliable process into a piece of infrastructure.

Trust and Auditability: By separating measured facts from inferred labels and providing evidence pointers to raw rows, the system creates memory that is mechanically auditable and resistant to poisoning or hallucination.
Cost Efficiency: The system enables agents to answer questions about user activity with high accuracy using a context block that fits within standard token limits, eliminating the need for expensive, non-deterministic summarization or raw data dumps.
Demand-Side Measurement: The paper provides the first empirical measurements of the Routine Overhead Ratio ( $R$ ) and routine recurrence ( $h$ ) derived from passive human activity rather than agent rollouts. These metrics suggest that while a significant portion of desktop work is repetitive (9–13% delegable), the cost of re-deriving these routines via screen-based agents is orders of magnitude higher than deterministic replay.
Modesty of Scope: The author explicitly notes that these results are based on a single-user corpus and a specific capture engine. They do not claim the metrics are universal population constants but rather a proof-of-concept for a new class of demand-side instruments. The determinism claim is scoped to the compilation path over stored capture, acknowledging that the initial OCR step (perception) involves a model.

In conclusion, the paper proposes that "dull" deterministic code is the necessary foundation for agent memory, enabling caching, auditing, and cost-effective replay, while reserving generative models for the optional, quarantined layer of interpretation.