From Logs to Agents: Reconstructing High-Level Creative Workflows from Low-Level Raw System Traces

This paper proposes a method to reconstruct high-level creative workflows from low-level system logs by parsing raw data into structured behavioral graphs, thereby enabling future process-aware AI agents to better understand user intent and assist in creative tasks.

The Gist

Imagine you are watching a master chef cook a complex dish. You want to understand their creative process: Why did they add salt? Why did they switch from a knife to a whisk? How did they decide to change the recipe halfway through?

Now, imagine the only way you can see what happened is by looking at a massive, chaotic spreadsheet of the chef's hand movements. It lists every single muscle twitch, every time they blinked, every time they adjusted their grip, and every time the kitchen timer beeped.

This is the problem with current AI design tools.

The paper "From Logs to Agents" by Tae Hee Jo and Kyung Hoon Hyun argues that while we have tons of data about how people use creative AI tools (like node-based generators), it's all "noise." It's like that messy spreadsheet. It tells us that a button was clicked, but not why or what the creative idea was.

Here is the paper's solution, broken down into simple concepts:

1. The Problem: The "Black Box" of Creativity

Current AI tools are like magic boxes. You put a prompt in, and an image comes out. But inside, the computer is doing thousands of tiny, invisible things (connecting wires, cleaning up memory, updating settings).

• The Analogy: If you try to understand a movie by looking at the raw electrical signals flickering through the projector bulb, you'll miss the story. You need the movie, not the electricity.
• The Issue: Researchers and AI agents (smart assistants) can't understand the user's "creative intent" because the data is too messy and low-level.

2. The Solution: Turning Noise into a Story

The authors built a "translator" pipeline that turns those messy raw logs into a clear, high-level story of the design process. They do this in three steps:

Step A: The Noise Filter (Cleaning the Kitchen)

First, they ignore the boring stuff. They filter out the "system noise"—like the computer cleaning up temporary files or syncing data.

• The Analogy: Imagine a detective ignoring the sound of the wind and the ticking of a clock to focus only on the footsteps and the voice of the suspect.
• The Result: They strip away 40% of the data, leaving only the "real" creative moves: Inserting a new idea, Modifying an existing one, Generating a new result, or Removing something that didn't work.

Step B: The Family Tree (Visualizing the Journey)

Creativity isn't a straight line; it's a tree. You might try a blue sky, then branch off to try a purple sky, decide you hate it, and go back to the blue sky to make it brighter.

• The Analogy: Think of a "Choose Your Own Adventure" book. The authors build a map (a graph) that shows every branch the user took.
• The Result: Instead of a long list of timestamps, they create a visual map showing how one idea led to another, revealing where the user was "exploring" (trying many branches) and where they were "refining" (deepening one branch).

Step C: The Secret Code (Tokenization)

Finally, they turn these visual steps into a simple language (tokens) that computers can easily read.

• The Analogy: Instead of saying "The user clicked the red button on the left panel to change the brightness of the image," they just say: MODIFY_IMAGE.
• The Result: This creates a universal language. It doesn't matter what specific software the user used; the AI can now understand the action.

3. Why This Matters: From "Dumb" Tools to "Smart" Partners

The ultimate goal of this research is to build Process-Aware Agents.

• The Old Way (Passive Tool): You ask the AI, "Make this image brighter." The AI does it. It has no memory of what you did five minutes ago. It's like a waiter who only hears your current order and forgets everything else.
• The New Way (Smart Partner): Because the AI now has the "Family Tree" of your workflow, it can predict what you need next.
  • The Analogy: Imagine a sous-chef who watches you chop onions, then realize you've done it five times in a row. Instead of waiting for you to ask, the sous-chef says, "I see you're prepping a lot of onions. Should I start chopping the next batch for you?"
  • Real Example: If the AI sees you are constantly resizing images after importing them, it might say, "I notice you resize every image you import. Would you like me to do that automatically next time?"

The Big Takeaway

This paper provides the missing link between raw data and human understanding. By cleaning up the messy logs and turning them into a clear story of how a human thinks and creates, we can teach AI agents to stop just "executing commands" and start "collaborating" with us.

It's the difference between a robot that just follows orders and a creative partner who understands your journey.

Technical Summary

Here is a detailed technical summary of the paper "From Logs to Agents: Reconstructing High-Level Creative Workflows from Low-Level Raw System Traces" by Tae Hee Jo and Kyung Hoon Hyun.

1. Problem Statement

The rise of complex, node-based Creativity Support Tools (CSTs) like ComfyUI, Flora AI, and GENPRESSO has generated massive amounts of low-level system logs (e.g., CSV/JSON files containing clicks, parameter tweaks, and metadata updates). However, a critical gap exists between this raw data and the understanding of creative intent.

• The "Black Box" Issue: While researchers possess rich raw data, they lack a machine-readable, semantically meaningful representation of the user's creative journey.
• Noise vs. Intent: Raw logs are extremely noisy. A single high-level creative decision (e.g., "tweak a prompt to change lighting") can trigger dozens of low-level system events, including connection re-routing, cleanup routines, and redundant metadata updates.
• Limitation of Current Agents: Current AI agents in CSTs operate only on the current state (the canvas) and lack the process history (how the user arrived there). Without understanding the workflow lineage, agents cannot effectively collaborate, predict next steps, or explain rationales.

2. Methodology

The authors propose a three-stage pipeline to transform raw system traces into structured, high-level User Behavioral Workflow Graphs. This method was applied to generative design logs from the node-based CST, GENPRESSO.

Stage 1: Semantic Filtering and Classification (De-noising)

The goal is to distinguish explicit user intent from system noise.

• Heuristic Filtering: The pipeline filters out events related to system maintenance (e.g., purging temporary IDs, backend graph re-routing, asynchronous state updates) that do not reflect new creative decisions.
• Classification: Remaining events are classified into four high-level "Creative Design Moves" based on rule-based mapping:
  1. INSERT: Introducing new content (importing images, creating new nodes).
  2. MODIFY: Refining existing assets (altering prompts, adjusting parameters like aspect ratio or AI models).
  3. GENERATION: Executing the generative AI model to produce a new artifact (committing parameters to create an image/video).
  4. REMOVE: Explicit deletion of assets (distinguishing user rejection from system cache clearing).
• Result: In a pilot dataset, this reduced event volume by ~40% (from 927 raw logs to 563 meaningful events), successfully isolating tangible creative changes.

Stage 2: Design Sequence Reconstruction (Workflow Visualization)

To capture the non-linear, branching nature of creativity, the authors construct a Directed Acyclic Graph (DAG).

• Structure:
  • Nodes: Represent creative assets (Prompts, Image Processes, Video Outputs).
  • Edges: Represent data flow and parent-child relationships extracted from metadata (e.g., connected_from fields).
  • Layout: Uses a depth-based algorithm where the X-axis represents generation depth and the Y-axis represents chronological order.
• Visual Encoding:
  • Black numbers: Global sequence order.
  • Blue numbers: Sequence of MODIFY moves.
  • Red numbers: Sequence of REMOVE moves.
  • Shapes: Squares for AI-generated outputs; Circles for manually created assets.
• Outcome: This visualization distinguishes between "wide exploration" (many branches) and "deep refinement" (long branches), revealing the overall shape of the design process.

Stage 3: Tokenization

To enable computational analysis and cross-platform comparison, the graph nodes are converted into a standardized language of interaction.

• Format: Concatenation of User_Action + Node_Asset_Type (e.g., INSERT_prompt, MODIFY_image, GENERATION_video).
• Purpose: This abstraction removes tool-specific interface details, allowing for sequence mining and probabilistic modeling.

3. Key Results and Analysis

Using the reconstructed workflow data, the authors performed sequence mining and Markov Chain analysis on the pilot dataset (563 events).

• Probabilistic Signatures:
  • Blind Re-rolling: The most common bigram (2-step sequence) was GENERATION_image followed by GENERATION_image (19.1% frequency), indicating users often rely on the stochastic nature of models rather than refining inputs.
  • Setup vs. Exploration:
    • After INSERT_image, there is a 69.6% probability the next action is MODIFY_image (indicating a "setup phase").
    • After GENERATION_image, there is a 66.1% probability of immediate re-generation (indicating an "exploration phase").
• Transition Modeling: These transition probabilities provide a mathematical basis for agents to distinguish between different phases of the creative process.

4. Key Contributions

1. Workflow Reconstruction Pipeline: A novel method to parse noisy, low-level CSV/JSON logs into structured, high-level behavioral workflow graphs.
2. Semantic Abstraction: The introduction of a classification system (INSERT, MODIFY, GENERATION, REMOVE) that translates technical system events into human-readable creative intents.
3. Process-Aware Agent Foundation: The paper establishes that structured workflow history is a prerequisite for "Process-Aware Agents." It moves the field from analyzing static states to analyzing dynamic processes.
4. Quantitative Behavioral Insights: The demonstration that Markov Chain analysis on these logs can reveal distinct behavioral patterns (e.g., distinguishing between refinement loops and exploration loops).

5. Significance and Future Implications

The paper argues that for AI agents to evolve from passive command executors to collaborative partners, they must possess a "memory" of the user's behavioral design process.

• Scaffolding Agentic Decisions: By feeding workflow graphs into the context window of LLM-driven assistants, agents can infer intent based on historical patterns.
  • Example: If a user repeatedly performs INSERT_image followed by MODIFY_metadata (resizing), a process-aware agent can proactively suggest: "I see you are importing and resizing references. Shall I automatically apply this scale factor to future imports?"
• Rationale Generation: Instead of generic suggestions, agents can provide rationale grounded in the provenance of the creative process (e.g., "Users who followed a similar generation path found success by...").
• Future of CSTs: This work lays the groundwork for the next generation of CSTs where agents understand the journey, not just the destination, enabling more intuitive and supportive creative workflows.