Technical Summary: SousCoder – Cross-Application Confusion Detection for Proactive Developer Assistance

1. Problem Statement

Modern software development requires constant context switching between an Integrated Development Environment (IDE), web browsers for documentation/search, and terminals for build/runtime monitoring. Current AI-powered coding assistants (e.g., GitHub Copilot, Cursor) are fundamentally reactive: they only provide assistance when a developer explicitly formulates a query. This process is cognitively expensive and temporally wasteful, as developers must recognize their "stuck" state, switch contexts, and articulate the problem.

While prior work like CodingGenie (FSE 2025) introduced proactive suggestions within VS Code, it operates on a single data source (IDE context), triggers after every code change regardless of whether the developer is actually stuck, and lacks persistent memory across sessions. There is a need for a system that can detect genuine "stuck" states by synthesizing behavioral signals across multiple applications and intervene proactively without user prompting, while preserving developer privacy.

2. Methodology and System Design

2.1 Architecture Overview

SousCoder is a local-first proactive developer assistant built as a native desktop application (Tauri 2.0 + React 18) with a Python asyncio backend. It operates on three simultaneous data sources:

1. VS Code Extension (TypeScript): Monitors file saves, error appearances/resolutions, and large deletions via the VS Code API.
2. Chrome Browser Extension (MV3): Tracks search queries and repeated visits to development domains (e.g., Stack Overflow, MDN) via content scripts.
3. Terminal Watcher (Python): Hooks into .zshrc/.bashrc to capture commands and exit codes via a named pipe.

These sources feed a unified event pipeline that normalizes data into a SousCoderEvent schema, deduplicates events within a 30-second window, and maintains a 500-event rolling history.

2.2 Confusion Detection and Intervention Modes

The core innovation is a weighted confusion score derived from cross-application behavioral signals. The system operates in two distinct modes:

• Active Mode: Fires after significant code events (errors, saves) with a 30-second cooldown per error template. It provides continuous, lower-threshold suggestions.
• Stealth Mode: Fires only when a weighted confusion score exceeds a threshold of 0.65. This score is calculated from four specific rules:
  1. Error Loop: Same VS Code error occurs twice in 10 minutes (Weight: 0.80).
  2. Terminal Error Loop: Same command fails twice in 10 minutes (Weight: 0.75).
  3. Idle After Stack: Stack trace followed by 3 minutes of no progress (Weight: 0.70).
  4. Repeated Search: Same browser search query performed three times in 15 minutes (Weight: 0.65).

Stealth Mode includes a 7-minute cooldown to prevent interruption fatigue.

2.3 Context Modeling and Memory

• Intent Graph (Kuzu): A local embedded graph database models the developer's current goal, active task, and primary blocker in real-time. Every 60 seconds or upon significant events, a local LLM infers these states from recent activity and updates the graph.
• Vector Memory (Qdrant): Session summaries are embedded using nomic-embed-text and stored locally. During intervention, the system retrieves the three most similar past sessions (cosine similarity > 0.72) to provide personalized context.
• Inference Engine: All inference runs locally via Ollama using mistral:7b-instruct. A cloud fallback (Anthropic Claude) is available but optional and privacy-conscious (transmitting only structured blocker descriptions, not raw code).

2.4 Intervention Generation

The system constructs structured prompts combining the intent graph, memory context, recent errors, and active file content. A validator filters outputs to ensure they are concise, actionable, and avoid common LLM pitfalls (e.g., starting with "I", ending with questions). Interventions are rendered in a non-modal, floating Tauri panel.

3. Key Contributions

The paper claims five primary contributions:

1. Cross-Application Confusion Detection: A model integrating IDE, browser, and terminal signals to identify stuck states without user input.
2. Dual-Mode Architecture: A controlled comparison between continuous event-triggered assistance (Active) and confusion-triggered assistance (Stealth).
3. Local-First Implementation: Full on-device inference, storage, and retrieval, ensuring privacy and offline capability.
4. Live Intent Graph: A real-time graph database modeling developer goals and blockers to structure LLM context.
5. Empirical Evaluation: A combination of latency benchmarks and a two-week field deployment with human-labeled ground truth and LLM-as-judge relevance assessment.

4. Results

4.1 Latency and Diagnostic Correctness

• Latency: Both modes met the sub-8-second interactivity target.
  • Stealth Mode: Median latency of 3.31 seconds.
  • Active Mode: Median latency of 4.90 seconds.
• Correctness: In controlled scenarios (SQL injection, missing import, TypeScript middleware misconfiguration), the local mistral:7b-instruct model correctly diagnosed all issues without cloud assistance.

4.2 Field Deployment (Two-Week Study)

A single-subject deployment (100 labeled sessions) yielded the following metrics for the confusion detector:

• Precision: 78.1%
• Recall: 69.4%
• F1 Score: 0.735
• False Positive Rate: 10.9% (Target was <20%)

Threshold Sensitivity: Lowering the threshold to 0.22 (single-rule trigger) increased recall to 100% and F1 to 0.911 without increasing the false positive rate in this specific dataset, though the authors caution this may not hold for broader, more varied datasets.

4.3 Intervention Relevance

Using an LLM-as-judge protocol (Claude), 88 interventions were evaluated for relevance:

• Active Mode: 59.6% relevant.
• Stealth Mode: 58.3% relevant.
• Frequency: Stealth Mode fired roughly two orders of magnitude less often than Active Mode, demonstrating its ability to reduce interruption while maintaining high relevance.

5. Significance and Claims

The paper positions SousCoder as an extension of CodingGenie, addressing its limitations by adding cross-application perception, confusion-state detection, and persistent memory.

• Differentiation: The system uniquely captures behavioral signals invisible to IDE-only tools (e.g., repeated browser searches), which serve as strong indicators of failed resolution attempts.
• Interruption Cost: The results suggest that confusion-gated intervention (Stealth Mode) preserves usefulness while substantially reducing interruption frequency compared to continuous event-triggered assistance.
• Privacy: The system demonstrates that capable proactive assistance can be achieved entirely on-device using quantized local models, avoiding the transmission of code or sensitive data to cloud providers.
• Future Work: The authors note that the optimal confusion threshold is likely individual and task-dependent, motivating future work on adaptive, per-developer threshold learning.

The study acknowledges limitations, including its single-subject nature, the use of an LLM judge rather than human acceptance metrics for relevance, and the heuristic nature of the rule weights. However, it establishes a foundational framework for proactive, privacy-preserving developer assistance that moves beyond reactive query-response models.