"When to Hand Off, When to Work Together": Expanding Human-Agent Co-Creative Collaboration through Concurrent Interaction

This paper introduces CLEO, a system enabling real-time collaborative context awareness for human-agent co-creation, and presents a decision model derived from two user studies that clarifies when designers should delegate, direct, or work concurrently with AI agents based on process visibility and interaction patterns.

The Gist

Imagine you are a chef cooking a complex meal with a very talented, but slightly literal-minded, robot assistant.

In the old way of working with AI (the "Turn-Taking" model), you would tell the robot, "Make me a lasagna," and then you'd have to sit on your hands, staring at the wall, waiting for it to finish the entire dish before you could taste it or say, "Hey, too much garlic!" If you tried to touch the food while the robot was chopping, the robot would get confused, maybe drop the knife, or ignore you completely.

This paper asks: What if we could cook together at the same time? What if you could taste the sauce while the robot is still stirring, and the robot could understand that you're adjusting the salt because you want it that way, not because it made a mistake?

Here is the story of how the researchers at KAIST and their colleagues figured out how to make that happen.

The Problem: The "Black Box" Robot

First, the researchers tried a system where the robot showed its work step-by-step. Instead of just saying "Here is the lasagna," the robot would show you: Chopping onions... Sautéing... Adding sauce.

The Good News: This was great! You could see what the robot was doing. If it started adding too much cheese, you could stop it early. You felt like a partner, not just a boss.

The Bad News: The robot was still too rigid. If you tried to tweak the sauce while it was stirring, the robot didn't know if you were helping it or if you were just doing your own thing. It would get confused, sometimes undoing your work or getting stuck. It lacked "Collaborative Context Awareness." It didn't know the difference between "I'm fixing your mistake" and "I'm making my own salad."

The Solution: Meet "Cleo"

To fix this, they built a new robot assistant named Cleo (Collaborative Linked Executive Operator).

Think of Cleo as a sous-chef who is incredibly perceptive.

• The Magic: When you reach in to adjust the robot's work, Cleo doesn't panic. It asks itself: "Is this person correcting me? Are they taking over this part to finish it faster? Or are they just working on a side dish?"
• The Result: You can now work side-by-side. You can grab the spoon and stir while the robot chops, or you can take a half-finished dish and finish it yourself while the robot starts on the next course.

The Big Discovery: The "Dance" of Collaboration

The researchers watched 10 professional designers work with Cleo for two days. They recorded 214 moments of interaction. They found that humans don't just stick to one mode; they dance between five different moves:

1. The "Hands-Off" (70% of the time): "You handle this, I'm busy." You trust the robot and go do your own work.
2. The "Observer" (69% of the time): "I'm watching closely." You aren't touching anything, but you're keeping an eye on the robot to see how it's doing.
3. The "Concurrent" (32% of the time): "Let's do this together!" This is the new stuff. You and the robot are working on the same thing at the same time.
   • Example: The robot is drawing a button, and you are changing its color. The robot sees your change and says, "Oh, blue is the new color! I'll use blue for the next button too."
4. The "Director" (29% of the time): "Wait, stop! Do it this way instead." You give a verbal command to change the plan.
5. The "Stopper" (9% of the time): "Cancel!" You realize the robot is going down the wrong path and kill the task immediately.

The "Traffic Light" System

The researchers realized that people switch between these modes based on a few simple signals, like a traffic light system:

• The Spark: If the robot does something cool that gives you a new idea, you jump in and work with it (Concurrent).
• The Mistake: If the robot is going the wrong way, you either yell "Stop!" (Director) or just take over the task yourself (Concurrent).
• The Busy Signal: If you are super busy with your own work, you just let the robot go (Hands-Off).
• The Learning Curve: If you don't know how good the robot is yet, you watch it closely (Observer) until you trust it.

Why This Matters

This paper changes the game for how we use AI.

• Old Way: You are the boss, the AI is the worker. You give an order, wait, and get a result.
• New Way: You are a co-pilot. You and the AI are flying the plane together. You can grab the controls, adjust the altitude, or let the autopilot take over, and the plane (the AI) understands what you're doing instantly.

The Takeaway

The future of AI isn't about building robots that are smarter than us; it's about building robots that understand us. It's about creating systems that know when to step back, when to step in, and how to work in the messy, chaotic, beautiful rhythm of human creativity.

Instead of waiting for the robot to finish the painting, you can now pick up a brush and add a few strokes while it's still mixing the paint, and the robot will know exactly what you meant. That is the power of Concurrent Interaction.

Technical Summary

1. Problem Statement

Current Human-Agent collaboration in creative domains (e.g., design, writing) is largely constrained by a sequential turn-taking paradigm. In this model, users delegate a task, wait for the agent to execute (often with hidden or read-only reasoning processes), and then review the final output. This approach suffers from several limitations:

• Lack of Context Awareness: Agents cannot distinguish between a user's independent work and their feedback on the agent's ongoing execution.
• Loss of Agency: Users often feel forced to wait passively or fear that intervening mid-execution will disrupt the agent's workflow.
• Inefficiency: Creative work is non-linear and often requires synchronous, interleaved workflows (similar to human-human collaboration in tools like Figma), which current agents cannot support.

The core research question is: Can AI agents achieve "collaborative context awareness" to interpret concurrent user actions on shared artifacts and adapt their execution in real-time, enabling fluid, interleaved collaboration?

2. Methodology

The authors conducted a two-phase empirical study involving professional UI/UX designers.

Study 1: Initial Technology Probe (N=10)

• System: A Figma plugin prototype featuring Process Transparency (step-by-step visual execution), Voice Interaction, and Presence Awareness (agent cursor).
• Goal: To test if making the agent's execution visible improves collaboration.
• Findings: While visibility allowed users to anticipate actions, the system failed to distinguish user feedback from independent work. When users tried to intervene concurrently, the agent often reverted changes or misinterpreted them, highlighting the need for collaborative context awareness.

Study 2: The "Cleo" Probe (N=10, Two-Day Study)

• System: Cleo (Collaborative Linked Executive Operator), a refined agent built on the ReAct architecture.
  • Key Innovation: Cleo incorporates three new modules to handle concurrency:
    1. User Change Detection: Tracks canvas changes while the agent is inactive.
    2. Attribution Change: Distinguishes between agent actions and user modifications.
    3. Plan Update: Dynamically adjusts the agent's execution plan based on user intent (feedback vs. independent work).
• Procedure:
  • Day 1: Participants performed two design tasks (closed-ended and open-ended) using Cleo. All interactions were logged and screen-recorded.
  • Day 2: Stimulated Recall Interviews were conducted. Participants reviewed visualized interaction logs (timelines of agent vs. user actions) to explain their decision-making processes.
• Data Analysis:
  • Analyzed 214 turn-level interactions.
  • Used qualitative coding to identify action patterns, triggers for intervention, and enabling factors.
  • Developed a Decision Model to characterize the dynamics of human-agent co-creation.

3. Key Contributions

1. Characterization of Concurrent Interaction: Identified five distinct user action patterns that enable fluid collaboration during agent execution:
   • Hands-off: Full delegation.
   • Observational: Monitoring without intervening.
   • Terminating: Stopping the agent.
   • Directive: Providing verbal instructions or switching tasks.
   • Concurrent: Direct manipulation of the agent's work (e.g., In-situ co-editing, Opportunistic takeover, Artifact takeover).
2. Decision Model with Six Interaction Loops: Synthesized findings into a model explaining how users navigate between delegation, direction, and concurrent work. The model is driven by:
   • Six Triggers: Events prompting intervention (e.g., "Idea Spark," "Misaligned Task Interpretation," "Execution Quality Drop").
   • Four Enabling Factors: Contextual conditions determining the specific action (e.g., User's Mental Model of the agent, Task Priority, Intervention Preference, and Expectation of Success).
3. Dataset: Released an annotated dataset of 214 interaction turns, including logged actions, triggers, and designer rationales.
4. Design Implications: Provided concrete guidelines for building adaptive collaborative agents.

4. Key Results

• Interaction Distribution: Across 214 turns, users engaged in:
  • Full Delegation (Hands-off): 70.1%
  • Observational Monitoring: 68.7% (often preceding other actions)
  • Concurrent Work: 31.8%
  • Directive Intervention: 28.5%
  • Termination: 8.9%
• The Role of Process Transparency: Visibility allowed users to build mental models of the agent's capabilities. Once a mental model was established, users shifted from passive observation to strategic, fluid transitions between working independently and collaborating concurrently.
• Context Awareness is Critical: The ability of Cleo to distinguish between "feedback" (user modifying the agent's work to correct it) and "independent work" (user working on a separate part of the canvas) was the primary enabler of successful concurrent collaboration.
• Low Social Cost: Users frequently terminated or overwrote agent work without hesitation, treating the agent as a disposable experimental tool rather than a social partner, which enabled more rapid iteration.

5. Significance and Design Implications

This work shifts the paradigm of Human-Agent interaction from sequential turn-taking to synchronous co-creation.

• Temporal Coordination: Agents should use execution milestones (step-by-step progress) to help users synchronize their own work with the agent's progress.
• Granularity of Intervention: Systems must interpret the type of concurrent action as a signal:
  • Fine-grained edits = Immediate course correction.
  • Mid-level takeovers = Task redistribution.
  • Coarse-grained appropriation = Validation of direction.
• Proactive Behavior: Agents should anticipate user intent by monitoring viewport focus and cursor proximity. For example, if a user's cursor moves toward the agent's work area, the agent should pause to allow intervention rather than completing the full sequence.
• Explicit Signals: Future systems should allow users to manipulate the agent's "presence" (e.g., moving the agent's cursor) to explicitly signal whether they want the agent to work autonomously or collaborate closely.
• Cross-Domain Applicability: While tested in design, the principles of reifying intermediate outputs into manipulable objects (e.g., code blocks, text paragraphs) can extend concurrent interaction to coding, writing, and web navigation.

In conclusion, the paper demonstrates that collaborative context awareness is the missing link for true human-agent partnership. By enabling agents to interpret and adapt to concurrent user actions in real-time, we can move beyond rigid automation toward dynamic, co-creative workflows that mirror human-human collaboration.