Technical Summary: SERUM – State Extraction and Refinement for User Modeling

Problem Statement

Proactive, personalized AI assistants require structured models of user intent and workflow. However, constructing these models from raw, unstructured egocentric video remains a significant challenge. Existing approaches face two primary limitations:

1. Dependence on Predefined Taxonomies: Traditional activity recognition methods (e.g., on EPIC-KITCHENS or Ego4D) rely on fixed, hand-crafted taxonomies, failing to handle open-ended, unstructured video.
2. Limitations of Single-Pass Annotation: While Vision-Language Models (VLMs) can annotate video frames, single-pass inference suffers from hallucination and temporal conflation. Without surrounding context, VLMs often collapse semantically distinct activities into generic labels or fail to infer the underlying intent of an action, as the meaning of a specific frame often only becomes clear retrospectively.

Process mining offers structured behavioral models but assumes the existence of clean event logs with known event types, which does not apply to raw video.

Methodology: The SERUM Framework

SERUM (State Extraction and Refinement for User Modeling) is a multi-pass framework designed to extract interpretable, structured behavioral models directly from unstructured egocentric screen video without manual annotation or predefined ontologies.

Core Architecture

The framework operates on two complementary levels:

• Actions ($a_t$): Mid-level, observable behaviors (e.g., "pulling a git repository").
• Intents ($i_t$): Latent, goal-directed states inferred from action sequences (e.g., "setting up a dev environment").

The system utilizes an alternating multi-pass pipeline using the Qwen3-VL-8B-Instruct model:

1. Alternating Passes: The system runs odd-numbered passes to annotate actions and even-numbered passes to infer intents.
2. Contextual Grounding:
   • Temporal Context: A sliding window of $w=20$ neighboring frames is encoded as a Run-Length Encoding (RLE) to provide dense local context without exceeding token limits.
   • Inter-Pass Summary: A natural-language summary of the entire video (generated via map-reduce for long videos) propagates global narrative context forward, acting as compressed episodic memory.
3. Iterative Refinement: Each pass conditions its output on the labels generated in the previous pass. This feedback loop allows the model to revise earlier judgments as context accumulates, driving the system toward a self-consistent vocabulary.

Label Normalization

To address the "open-vocabulary" problem where free-form annotation produces surface synonyms (e.g., "rinsing produce" vs. "cleaning vegetables"), SERUM employs a label normalization step:

• Semantic Merging: Pairwise cosine similarity is computed using Sentence-BERT embeddings.
• Thresholding: A human-calibrated threshold ($t^*=0.43$) merges synonymous labels. This step reduces vocabulary size by approximately 46% on average and is treated as a core design component rather than a post-hoc fix.

Output: User Models

The final output consists of two First-Order Markov Models (User Models):

• One model over the canonicalized action states.
• One model over the canonicalized intent states.
  These models are directed weighted graphs where edges represent transition probabilities, capturing the probabilistic structure of user behavior to support next-state prediction.

Key Contributions and Results

The authors evaluated SERUM on 61 egocentric videos across four domains: coding, cooking, physical activities, and daily life (totaling 11,125 frames).

1. Schematic Equilibrium (RQ1)

The iterative annotation process converges to a stable state vocabulary.

• Convergence: The raw vocabulary size stabilizes by Pass 8, a phenomenon the authors term schematic equilibrium.
• Normalization Impact: Label normalization further compresses the vocabulary to a stable size of ~10 states, significantly reducing sparsity in the transition matrices.

2. Predictive Performance (RQ2)

The resulting Markov models were evaluated on next-state prediction (training on the first 60% of frames, testing on the remaining 40%).

• Superiority over Baselines: Normalized Markov models significantly outperform frequency-based baselines (Majority, Weighted Random, Uniform) in both Top-1 accuracy and Perplexity.
  • Action Prediction: Normalized Markov achieved 48.5% Top-1 accuracy vs. 37.6% for the Majority baseline.
  • Intent Prediction: Normalized Markov achieved 58.2% Top-1 accuracy vs. 29.9% for the Majority baseline.
• Domain Variance: The largest gains were observed in structured tasks like coding (76.4% normalized action accuracy), reflecting the repetitive transition structures inherent in such workflows.

3. Human Assessment (RQ3)

Human annotators with domain expertise evaluated the quality of the labels.

• Accuracy: 88.3% of final-pass labels were rated as accurate (Krippendorff's $\alpha = 0.40$).
• Preference: Annotators preferred final-pass labels over first-pass labels 82.8% of the time, indicating that iterative refinement produces meaningfully better labels.
• Intent vs. Action: Intent labels were slightly more accurate (90%) than action labels (86.7%), suggesting the context-aware inference effectively disambiguates goals.

4. Ablation Studies (RQ4)

• Label Normalization: Merging synonyms improved Markov Top-1 accuracy by +18.2 percentage points and reduced perplexity by 14.9 points.
• Intent Passes: Removing intent inference (activity-only pipeline) caused activity labels to collapse into uninformative dominant states (e.g., "typing_on_keyboard" for 92–96% of coding frames). Annotators preferred the full pipeline (activity + intent) 73% of the time.
• Temporal Window: Increasing the context window size ($w=20$) improved the gap between Markov and Majority baselines, confirming that temporal context is critical for model quality.

Significance and Claims

The paper claims that SERUM is, to their knowledge, the first system to produce interpretable process models (both action and intent) directly from unstructured egocentric screen video without:

• Manual annotation.
• Predefined taxonomies.
• Existing event logs.

The authors position SERUM as an early validation for downstream applications in proactive agentic assistance and personalization, offering a scalable pathway for user modeling "in the wild." They acknowledge limitations, such as the computational cost of the multi-pass pipeline (though front-loaded) and the challenge of split-half evaluation on linear videos, but emphasize the system's ability to self-correct hallucinations through iterative context and consensus.

The work demonstrates that iterative, context-aware VLM annotation can suppress hallucinations and temporal conflation, converging toward a stable, human-validated behavioral taxonomy that enables predictive modeling of user workflows.