🤖 AI

FileGram: Grounding Agent Personalization in File-System Behavioral Traces

To address the limitations of data constraints and interaction-centric approaches in personalizing coworking AI agents, the paper proposes FileGram, a comprehensive framework featuring a scalable data engine, a diagnostic benchmark, and a bottom-up memory architecture that grounds agent personalization in fine-grained file-system behavioral traces.

Original authors: Shuai Liu, Shulin Tian, Kairui Hu, Yuhao Dong, Zhe Yang, Bo Li, Jingkang Yang, Chen Change Loy, Ziwei Liu

Published 2026-04-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Shuai Liu, Shulin Tian, Kairui Hu, Yuhao Dong, Zhe Yang, Bo Li, Jingkang Yang, Chen Change Loy, Ziwei Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a digital assistant that lives inside your computer's hard drive. You want this assistant to be the ultimate coworker: someone who knows exactly how you work, remembers your habits, and helps you organize your life without you having to explain everything from scratch.

The problem? Most current assistants are like amnesiac tourists. They can chat with you, but they don't really "know" you. They forget what you did five minutes ago, and they certainly don't understand the subtle ways you organize your files, rename your documents, or edit your spreadsheets.

This paper introduces FileGram, a new system designed to give your AI assistant a real memory of how you work, not just what you say.

Here is the breakdown of FileGram using simple analogies:

1. The Problem: The "Amnesiac Tourist"

Currently, AI assistants rely mostly on conversation. If you say, "I like my files in deep folders," they remember that. But if you actually spend 20 minutes renaming files and moving them around, the AI often ignores those actions because it's only listening to your voice, not watching your hands.

It's like having a personal chef who only listens to your orders but never watches how you actually cook. They might know you like "spicy food," but they won't notice that you always chop onions first, then garlic, and never use a wooden spoon.

2. The Solution: FileGram (The "Super-Observer")

FileGram changes the game by teaching the AI to watch behavioral traces. Instead of just listening to your voice, it watches every click, every file rename, and every edit you make.

To make this work, the researchers built a three-part toolkit:

A. The Training Ground: FileGramEngine

You can't train a memory system on real people's private data because of privacy laws (and it's too hard to get). So, they built a virtual simulator.

The Analogy: Imagine a flight simulator for pilots. The researchers created 20 different "virtual pilots" (user profiles) with very specific habits. One is a messy hoarder who never deletes anything; another is a minimalist who deletes files the second they are done.
The system simulates thousands of hours of these virtual people working on their computers. This creates a massive library of "how-to-be-a-person" data without ever spying on a real human.

B. The Test: FileGramBench

How do you know if the AI actually learned? You need a test.

The Analogy: Think of this as a driver's license exam for AI. Instead of asking, "What is 2+2?", the test asks: "Based on the fact that this user always creates three backup copies before editing a file, what will they do next?"
It tests the AI on three things:
1. Understanding: Can it guess the user's personality just by watching them work?
2. Reasoning: Can it figure out why a user did something weird?
3. Detection: Can it notice if a user suddenly changed their habits (e.g., a neat freak suddenly starts making a mess)?

C. The Brain: FileGramOS

This is the actual memory system that runs the AI. Most AI memory systems are like diaries: they summarize your day into a paragraph ("Today I worked on a report").

The Analogy: FileGramOS is more like a forensic lab. It doesn't just summarize; it breaks your actions down into tiny, atomic pieces.
- Procedural Memory: It counts things. "This user creates 5 folders deep." "This user edits files 3 times before saving."
- Semantic Memory: It understands the content. "This user writes in a formal tone."
- Episodic Memory: It remembers the timeline. "On Tuesday, they did X. On Friday, they did Y."
By keeping these separate, the AI doesn't get confused. It knows the difference between "what you did" (procedure) and "what you wrote" (content).

3. Why This Matters

The researchers tested FileGramOS against 12 other top-tier AI memory systems.

The Result: The old systems (the "diaries") got about 50% of the answers right. They were good at remembering facts but bad at understanding habits.
FileGramOS: Got nearly 60% right. It was much better at spotting patterns, like realizing a user prefers deep folder structures or always searches for files before reading them.

The Big Takeaway

FileGram proves that to build a truly helpful AI coworker, we can't just listen to what people say. We have to watch what they do.

By treating your file system as a diary of your habits rather than just a storage locker, FileGram allows AI to finally say: "I know you usually organize your project files by date, so I've already set up that folder for you before you even asked."

It's the difference between a robot that follows orders and a partner who truly knows you.

1. Problem Statement

The paper addresses the critical gap in personalizing AI agents that operate within local file systems. While AI agents are evolving from conversational interfaces to "file-system coworkers," current methods fail to adapt to diverse user workflows due to three main bottlenecks:

Data Scarcity: Collecting real-world, multimodal, long-trajectory file-system data is prohibitively difficult due to strict privacy constraints and the lack of scalable collection strategies.
Evaluation Limitations: Existing benchmarks focus on conversational recall or isolated GUI task success, overlooking memory-centric, personalized behavior understanding tasks grounded in file operations.
Methodological Flaws: Current memory architectures are interaction-centric (relying on top-down dialogue summaries) or document-centric (treating files as static knowledge). They fail to capture bottom-up behavioral traces (atomic actions and content deltas) which contain stronger personal signatures than dialogue.

2. Methodology: The FileGram Framework

To bridge these gaps, the authors propose FileGram, a unified framework comprising three core components:

A. FileGramEngine (Data Generation)

A scalable, persona-driven data engine designed to synthesize realistic file-system behavioral traces.

Profile Design: Defines 20 diverse user profiles based on 19 attributes, including 6 core behavioral dimensions (Consumption Pattern, Production Style, Organization Preference, Iteration Strategy, Curation, Cross-Modal Behavior). Each dimension is discretized into three tiers (Left/Middle/Right).
Task Design: 32 tasks (16 text, 16 multimodal) ranging from "Understand" to "Maintain," initialized with real-world file collections.
Simulation: Agents execute tasks in isolated sandboxes using a Think-Act-Observe loop.
Data Processing: Raw tool traces are filtered to remove simulation artifacts (e.g., LLM thoughts) and canonicalized into Atomic Actions (e.g., read, write, mkdir) paired with Content Deltas (full snapshots for new files, patch diffs for edits).
Output: Generates 640 trajectories containing ~20k atomic actions and ~10k multimodal files (PDFs, images, audio).

B. FileGramBench (Evaluation Benchmark)

The first benchmark dedicated to memory-centric personalization based on file-system operations.

Scale: 4.6K QA pairs across 9 sub-tasks.
Four Tracks:
1. Understanding: Attribute recognition, behavioral fingerprinting, and profile reconstruction.
2. Reasoning: Behavioral inference on unseen tasks and trace disentanglement (distinguishing interleaved user streams).
3. Detection: Anomaly detection and shift analysis (identifying when a user's behavior drifts).
4. Multimodal Grounding: Evaluating agents on rendered documents (PDFs/Images) and real-world screen recordings.
Channels: Evaluates Procedural (operation patterns), Semantic (content understanding), and Episodic (temporal consistency) memory.

C. FileGramOS (Memory Architecture)

A bottom-up memory framework that constructs user profiles directly from atomic actions and content deltas, avoiding premature summarization.

Stage 1: Per-Trajectory Encoding (Engram Generation):
- Procedural Extraction: Counts actions and computes metrics (e.g., search_ratio, avg_output_length) to create a 17-dimensional behavioral fingerprint.
- Semantic Parsing: Uses Vision-Language Models (VLMs) to generate captions and behavioral descriptors from file content and diffs.
- Episodic Segmentation: Clusters event timelines into logical episodes (e.g., "Survey" $\to$ "Creation").
Stage 2: Cross-Engram Consolidation: Routes Engrams into three specialized channels:
- Procedural Channel: Aggregates statistical fingerprints to identify stable traits (e.g., "Deeply nested organization").
- Semantic Channel: Embeds content chunks and summarizes stylistic preferences across sessions.
- Episodic Channel: Clusters sessions to detect temporal consistency and uses an Anomaly Judge (LLM) to distinguish task-dependent variations from genuine behavioral shifts.
Stage 3: Query-Adaptive Retrieval: Dynamically retrieves clues from the three channels based on the query to compose grounded answers.

3. Key Contributions

FileGramEngine: A novel data engine that generates scalable, fine-grained, multimodal file-system traces with controlled persona attributes, solving the data scarcity problem.
FileGramBench: A comprehensive diagnostic benchmark that evaluates agents on procedural, semantic, and episodic memory, specifically targeting personalization and behavioral drift detection.
FileGramOS: A state-of-the-art bottom-up memory architecture that preserves atomic action statistics and content deltas, proving that structural behavioral signals are more critical for personalization than narrative summaries.
Open Source: The framework, data, and code are open-sourced to facilitate future research.

4. Results

Extensive experiments on FileGramBench reveal significant limitations in current methods and the superiority of FileGramOS:

Performance: FileGramOS achieves 59.6% accuracy, significantly outperforming the strongest narrative baseline (EverMemOS at 49.9%) and context-based methods (Full Context at 40.5%).
Channel Importance:
- Procedural Channel: The most critical component. Removing it caused a 11.1% drop in accuracy. It is essential for tasks like Trace Disentanglement (dropping from 80.9% to 53.1% without it).
- Semantic Channel: Important for attribute recognition but less critical for structural reasoning.
- Episodic Channel: Crucial for detecting behavioral shifts and anomalies.
Failure of Existing Methods:
- Narrative Summarization: Methods that summarize trajectories early (e.g., Mem0, Zep) lose critical discriminators like directory depth and edit granularity, leading to "flattened" profiles.
- Multimodal Blindness: Vision-based methods (VisRAG, MMA) fail to capture operation-level statistics (e.g., file counts, edit frequencies), performing poorly on procedural tasks.
- Sim-to-Real Gap: While FileGramOS is robust, all methods struggle with real-world screen recordings (accuracy drops to single digits), highlighting the difficulty of translating structured trace analysis to noisy video inputs.

5. Significance

Paradigm Shift: Moves the field from dialogue-centric memory to file-system behavioral memory, recognizing that how a user interacts with files (procedural traces) is a stronger signal of identity than what they say.
Measurable Personalization: Provides the first rigorous, reproducible framework to measure and improve agent personalization in local environments.
Future Directions: Identifies shift attribution (explaining why behavior changed) and sim-to-real transfer as the next major frontiers for memory-centric AI agents.
Practical Application: Lays the groundwork for truly adaptive AI coworkers that can proactively synchronize workspaces, predict user intent, and maintain long-term context without relying on explicit user prompts.