💬 NLP

"What Are You Really Trying to Do?": Co-Creating Life Goals from Everyday Computer Use

This paper introduces "striving co-creation," a system grounded in Activity Theory that infers and collaboratively refines users' long-term life goals from their everyday computer use through an interactive editing interface, thereby overcoming the limitations of existing systems that only capture surface-level actions.

Original authors: Shardul Sapkota, Matthew Jörke, Zane Sabbagh, Omar Shaikh, Grace Wang, James A. Landay

Published 2026-09-10

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Shardul Sapkota, Matthew Jörke, Zane Sabbagh, Omar Shaikh, Grace Wang, James A. Landay

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a computer that does not just watch what you click or type, but understands why you are doing it. For decades, the dream of human-computer interaction has been to build systems that grasp the deeper purpose behind our daily actions. We have long accepted that a machine can tell us we are looking at a spreadsheet or reading an email, but it has struggled to see that these fragmented moments are actually part of a larger, long-term effort, such as planning a move to a new city or building a stable life for a growing family. This gap exists because current technology focuses on the immediate task rather than the underlying motivation. It sees the steps but misses the journey. Researchers at Stanford University have now taken a significant step toward bridging this divide by creating a system that does not just record behavior, but collaborates with people to uncover the long-term goals that drive their everyday computer use.

The team behind this work, led by Shardul Sapkota and colleagues, developed a system called Tempo. The core idea is a process they call "striving co-creation." Instead of a computer silently guessing what a user wants, or a user having to manually type out a list of their life goals, the system and the person work together to build a picture of those goals. The system starts by taking screenshots of the user's screen throughout the day, capturing everything from browsing the web to messaging friends. It then uses advanced artificial intelligence to organize these raw images into a hierarchy. At the bottom are simple operations, like a single click or a scroll. These group into actions, like searching for a specific topic. Actions combine into activities, which are recurring patterns of behavior. Finally, these activities are synthesized into "strivings," which are the ongoing, long-term pursuits that give meaning to the daily work, such as "preserving cultural identity" or "establishing a professional reputation."

However, the researchers knew that a computer looking at a screen cannot always know the full story. The same action, like researching schools, could mean a family is moving for a better job, or it could mean they are fleeing a difficult situation. To solve this, Tempo includes a second part where the user can review and edit the system's guesses. If the computer thinks a user is "building social capital" but the user knows they are actually just doing administrative paperwork, they can correct the system. This correction is not just a one-time fix; it becomes a rule the computer must follow in the future. The system learns from these edits, refining its understanding of the person's life goals over time. This creates a loop where the machine proposes an interpretation, the human refines it, and the machine incorporates that new understanding into its next round of analysis.

To test if this approach actually works, the researchers deployed Tempo with fourteen volunteers over the course of one week. The participants installed the software on their personal computers, which captured thousands of screenshots while they went about their normal lives. The system processed this data, built the hierarchy of actions and goals, and then invited the participants to review the results in a lab session. The findings were encouraging. When the system included the user's own description of their life—such as their current job, stressors, and priorities—the goals it generated were rated as significantly more accurate and aligned with the person's true priorities. Without this personal context, the system could still identify what people were doing, but it often missed the deeper "why."

The study also showed that the hierarchical structure was essential. When the researchers tested a version of the system that skipped the middle steps and tried to guess long-term goals directly from the screenshots, the results were less effective. The participants felt those direct guesses were too tied to specific, momentary tasks and failed to capture the broader purpose of their efforts. The step-by-step hierarchy allowed the system to connect the dots between unrelated activities, revealing that a search for health monitoring services and a message to a parent about a family tradition were both part of a single, larger striving to care for loved ones.

Perhaps the most surprising discovery was how the process of reviewing these goals changed the participants' own thinking. Even before they made any edits, simply seeing the system's interpretation of their week prompted many to reflect on their own behavior. Some realized they were spending time on tasks that did not align with their stated values, while others saw patterns in their work that they had not noticed before. One participant described the experience as feeling like a therapist analyzing their behavior, offering insights they had not considered. The system did not just record data; it acted as a mirror, helping people see the connection between their daily digital habits and their long-term life aspirations.

The researchers were careful to note that this system is not perfect and is not a finished product. The study lasted only one week, which is enough to see recurring patterns but not long enough to prove that these inferred goals will remain stable over months or years. There were also moments where the system made confident guesses that were slightly off, such as assuming a user was pivoting their career when they were not. The participants found that the ability to edit the hierarchy was crucial; it gave them control and allowed them to correct the system's misunderstandings. Without the ability to intervene, the system's guesses could feel like a rigid label rather than a helpful interpretation.

Ultimately, this work suggests a new way for technology to interact with us. Instead of acting as a passive tool that waits for commands, or an intrusive observer that tries to guess our minds, the computer becomes a partner in understanding our lives. It proposes a story about what we are trying to achieve, and we have the power to rewrite that story until it rings true. The researchers found that when people and machines collaborate in this way, the result is a representation of human goals that is far more accurate and meaningful than either could achieve alone. This approach treats our life goals not as fixed facts to be extracted, but as ongoing narratives that we shape together with the tools we use every day.

Technical Summary: "What Are You Really Trying to Do?" Co-Creating Life Goals from Everyday Computer Use

Problem Statement

Current user modeling systems and context-aware computing applications are limited by their focus on immediate, surface-level actions. While recent advances allow systems to infer tasks and states from cross-application behavior, these inferences remain situational and do not capture the why behind user actions. Existing approaches either restrict goals to explicit user specification, infer only immediate tasks, or rely on pre-defined activity structures. Consequently, systems fail to synthesize behavior across domains and time to understand a person's long-term life goals (strivings). Furthermore, the mapping from behavior to goals is often ambiguous; the same action can serve multiple, conflicting long-term pursuits. Purely automated inference risks imposing incorrect interpretations on users, while relying solely on user articulation misses goals the user has not yet recognized or verbalized.

Methodology: The Tempo System

The authors introduce Tempo, a system that operationalizes striving co-creation. This process involves a person and a system jointly constructing a hierarchical representation of long-term goals from unstructured computer use. Tempo is grounded in Activity Theory (hierarchical decomposition of action) and Emmons' personal strivings framework (ongoing pursuits that organize activities).

The system architecture consists of two primary modules and a privacy audit layer:

1. The Induce Module

This module progressively abstracts raw screen observations into a four-level hierarchy using a property graph and Large Language Models (LLMs), specifically Gemini 3 Flash. The hierarchy is conditioned on user-provided context.

Operations: Atomic interactions (e.g., clicks, scrolls) transcribed from screenshots.
Actions: Goal-directed sequences of operations grouped by intent, regardless of application boundaries.
Activities: Recurring patterns of actions sharing a common motive, synthesized across sessions and applications.
Strivings: Ongoing personal pursuits that organize multiple activities, representing long-term goals.
Inference Process: The system uses prompt chaining. Each level's structured output serves as the input and context for the level above. A "propose-and-reconcile" (for activities) and "synthesize-and-self-refine" (for strivings) process ensures the hierarchy is consistent and theoretically grounded.

2. The Edit Module

To resolve ambiguity and ensure user agency, the Edit module surfaces the inferred hierarchy to the user for refinement.

Interface: Users can trace a striving down through activities and actions to the underlying screenshots.
Operations: Users can perform four types of edits: Inline Edit (rewriting descriptions), Reassign (moving an activity to a different striving), Remove (discarding unrepresentative nodes), and Merge (combining separate nodes).
Feedback Loop: Edits are not merely corrections; they are persisted as constraints in the property graph. Flags (e.g., [user-edited]) are injected into subsequent induction prompts, ensuring future inferences respect the user's self-understanding.

3. Privacy and Security

Tempo incorporates a Contextual Integrity Audit module. It uses local OCR to filter Personally Identifiable Information (PII) and applies a contextual integrity check to block observations that violate the user's disclosure norms (e.g., blocking a banking login if the user has never shared financial data). All data is stored locally, and inference calls are routed through enterprise cloud agreements that prohibit model training on user data.

Key Contributions

Striving Co-Creation: A novel computational process where users and systems collaboratively construct long-term goal representations, treating goals as negotiable interpretations rather than objective facts.
The Tempo System: A working instantiation of this process that integrates hierarchical induction with an editable interface, bridging the gap between raw behavioral traces and high-level life goals.
Hierarchical Abstraction: A four-level model (Operations $\to$ Actions $\to$ Activities $\to$ Strivings) that synthesizes cross-application behavior into a coherent narrative of purpose.
Constraint-Based Editing: A mechanism where user edits act as hard constraints for future inferences, ensuring the system's model evolves to align with the user's self-perception.

Evaluation and Results

The system was evaluated in a week-long field deployment with 14 participants ( $N=14$ ). Participants used Tempo on their personal MacBooks, generating a median of 1,498 processed screenshots.

Ablation Studies

The authors compared the full system against two ablations:

Tempo (Full): Hierarchical pipeline + User Context.
Tempo -{User Context}: Hierarchical pipeline only.
Tempo -{Hierarchy, User Context}: Direct inference from observations (no hierarchy, no context).

Findings:

User-Provided Context: Significantly improved individual striving quality ( $p < 0.05$ ), particularly in terms of alignment (reflecting the person's priorities).
Hierarchical Structure: Significantly improved set-level representativeness ( $p < 0.05$ ). The hierarchy produced a more representative set of goals and fewer, broader goals that better captured "what matters" rather than just "what happened."
Accuracy: Across all conditions, 80% of strivings were rated as accurate ( $\ge 1$ on a -3 to +3 scale), indicating that even without hierarchy or context, the system could infer factually grounded behaviors.

Editing Experience

Participants compared editing via the Hierarchical View (Tempo) versus a Screenshot View (ablation).

The hierarchical view was rated significantly higher ( $p < 0.001$ ) across all dimensions, including transparency (93% vs. 43% found it sufficient) and evidence utility (100% vs. 71%).
Users reported that the hierarchy allowed them to understand the system's reasoning and locate exactly where the interpretation diverged from their own, whereas raw screenshots were difficult to connect to high-level goals.

Qualitative Feedback

Participants recognized the inferred strivings as representative of their long-term lives, with many noting the system surfaced goals they had not explicitly articulated. However, some noted the system could be over-confident when context was over-weighted, and the "observer effect" (participants altering behavior to appear more productive) was observed in some cases.

Significance and Claims

The paper claims that modeling what people are working toward over the long term requires both hierarchical inference and co-creative mechanisms.

Shift in User Modeling: The work argues for a shift from capturing "what a person is doing" to interpreting "what they are working toward."
Agency and Interpretation: By treating strivings as "interpretive artifacts" rather than ground truth, the system empowers users to shape how they are represented. This addresses the limitations of purely automated inference, which cannot resolve the ambiguity of human motivation.
Reflection: The co-creation process itself serves as a reflection tool. The tension between the system's inference and the user's self-perception prompts users to reconsider their daily behaviors and their connection to broader life goals.

The authors modestly conclude that while a week-long deployment successfully surfaces pursuits extending beyond the observation window, the stability of these strivings over months requires longer-term study. They emphasize that the system is a tool for negotiation and self-understanding, not an objective extractor of facts.