SWE-chat: Coding Agent Interactions From Real Users in the Wild

1. Problem Statement

Despite the rapid adoption of AI coding agents (e.g., Claude Code, Cursor) and their reported success on curated benchmarks, there is a critical lack of empirical evidence regarding how these agents function in real-world developer workflows. Current research relies heavily on:

• Curated Benchmarks: Tasks with well-defined, verifiable solutions (e.g., SWE-bench) that do not capture the iterative, ambiguous nature of real development.
• Anecdotal Evidence: Limited understanding of how developers prompt, steer, override, or discard agent output.
• Missing Data: No public dataset exists that links agent interaction logs with actual code commits and provides line-level attribution of human vs. agent authorship.

The authors pose two primary Research Questions (RQs):

1. RQ1: How do users interact with coding agents in real-world coding tasks?
2. RQ2: How do coding agents fail in practice, and how do users respond?

2. Methodology

Data Collection: The SWE-chat Dataset

The authors introduce SWE-chat, the first large-scale dataset of real coding agent sessions collected from open-source developers "in the wild."

• Collection Mechanism: Developers opt-in via Entire.io, an open-source CLI tool. This tool installs git hooks that automatically log coding agent sessions (transcripts, tool calls, token usage) to a dedicated branch and links them to code commits.
• Attribution: The system performs line-level code attribution, distinguishing between code written by the human and code written by the agent.
• Scale: As of April 2026, the dataset contains 6,000 sessions from 205 public GitHub repositories, comprising 63,000 user prompts and 355,000 agent tool calls.
• Agents Covered: Primarily Claude Code (~85%), along with OpenCode, Gemini CLI, Cursor, and Factory AI Droid.

Analysis Framework

The authors employ a multi-faceted analysis approach:

• Annotation: They use LLMs (validated against human expert gold labels) to classify sessions and prompts. Key annotations include:
  • Session Success: A 0–100 rating based on goal completion and code quality.
  • User Personas: Expert Nitpicker, Vague Requester, Mind Changer, or Other.
  • Prompt Intent: Create, Refactor, Debug, Understand, Connect, Git, Test, or Other.
  • Pushback Types: Correction, Rejection, Failure Report, or Non-pushback.
• Efficiency Metrics:
  • Code Survival Rate: The fraction of agent-produced code that survives into user commits.
  • Coding Efficiency: The fraction of total agent effort (including self-overwrites) that ends up in the final commit.
  • Cost/Time Efficiency: Tokens, dollars, and time spent per 100 committed lines.
• Security Analysis: The authors run Semgrep (static analysis) on pre- and post-commit snapshots to count security vulnerabilities introduced or fixed by the commits.

3. Key Contributions

1. SWE-chat Dataset: A "living dataset" that bridges the gap between agent trajectories and real-world code commits, offering the first large-scale view of human-agent collaboration with granular authorship attribution.
2. Empirical Characterization of "Vibe Coding": Identification of a bimodal usage pattern where agents either write almost nothing or almost everything, with the "vibe coding" mode (agent writes >99% of code) rapidly increasing.
3. Failure Mode Analysis: Quantification of agent inefficiency and security risks, revealing that agents often fail silently and require significant human oversight.
4. Benchmarking Roadmap: A proposal for moving beyond static benchmarks to dynamic, workflow-based evaluation using real interaction traces.

4. Key Results

Interaction Patterns (RQ1)

• Task Diversity: The most common user intent is understanding existing code (19%), followed by creating new code (13.4%). Agents spend ~33% of tool calls executing bash commands (mostly git operations) rather than editing files.
• Bimodal Coding Modes:
  • Human-only (23%): Humans write all code; agents assist with comprehension/debugging.
  • Collaborative (36.5%): Humans and agents co-author code.
  • Vibe Coding (41%): Agents author >99% of the code. This mode has doubled in prevalence over a three-month period.
• User Behavior: Users act primarily as "Expert Nitpickers" (47% of vibe coding sessions), providing precise corrections rather than vague requests.

Failure Modes and Efficiency (RQ2)

• Low Code Survival: Only 44.3% of all agent-produced code survives into user commits. In "vibe coding" sessions, this rises to 59%, but the cost is significantly higher.
• Inefficiency of Vibe Coding:
  • Cost: Vibe coding costs ~3x more tokens and ~2x more dollars per 100 committed lines compared to collaborative coding.
  • Time: Vibe coding takes ~12.6 minutes per 100 lines, compared to 4.8 minutes for collaborative coding.
• Security Vulnerabilities:
  • Vibe-coded commits introduce security vulnerabilities at a rate of 0.76 per 1,000 lines, which is 9x higher than human-only coding and 5x higher than collaborative coding.
  • Common vulnerabilities include path traversal, command injection, and unsafe format strings.
• Oversight Asymmetry:
  • User Pushback: Users interrupt or push back against agent output in 44% of turns (5% hard interruptions, 39% soft pushbacks/corrections).
  • Agent Silence: Agents rarely ask for clarification (<2% of turns), even as their autonomous runtime increases (99.9th percentile turn duration >100 minutes).

5. Significance and Implications

• Reality Check on Autonomy: While agents are becoming more autonomous, this does not equate to higher efficiency or safety. Full autonomy ("vibe coding") currently leads to higher costs, slower delivery, and significantly more security vulnerabilities.
• The "Collaborative" Sweet Spot: The data suggests that collaborative coding (where humans and agents co-author) is the most efficient and safe mode, challenging the industry trend toward fully autonomous one-shot problem solving.
• New Evaluation Paradigm: SWE-chat provides the foundation for "in-the-wild" evaluation, enabling the creation of realistic benchmarks, user simulators, and interaction designs that account for the iterative, corrective nature of real software engineering.
• Security Risks: The findings highlight a critical gap in current agent safety; as agents write more code autonomously without signaling uncertainty, the burden of catching security flaws shifts entirely to the human, who may not be equipped to review massive volumes of AI-generated code effectively.

In conclusion, SWE-chat reveals that while AI coding agents are powerful, their current deployment in the wild is often inefficient and risky. The path forward lies not in full autonomy, but in designing agents that better support human oversight, clarify intent, and collaborate effectively within iterative workflows.