In-IDE Toolkit for Developers of AI-Based Features

Technical Summary: In-IDE Toolkit for Developers of AI-Based Features

Problem Statement
The paper addresses the difficulty software engineers face in testing, debugging, and reproducing AI-enabled features built on Large Language Models (LLMs) and agentic workflows. Unlike traditional software, these systems are often "black boxes" where behavior depends on non-deterministic sampling, data context, and model versions. This makes failures hard to localize and successes hard to reproduce.

The authors identify a specific gap: while robust AI-native tooling exists (e.g., LangSmith, Weights & Biases, MLflow), these tools are optimized for ML practitioners and typically operate as external dashboards. Product-focused software engineers, who lack ML backgrounds but are increasingly building LLM features, often lack the intuition for systematic evaluation and find the user journey in external tools unclear. Consequently, they rely on ad-hoc manual testing or production feedback rather than structured, repeatable evaluation pipelines.

Methodology
The research employs a mixed-methods approach to define requirements and validate the solution:

1. Survey Analysis: Data was drawn from a JetBrains Strategic Research & Market Intelligence survey of AI/ML practitioners and general developers. Key findings indicated that 56% of respondents struggle with testing non-deterministic behavior, and 36% find building robust evaluation pipelines challenging. Most developers (58%) rely on manual testing rather than automated pipelines.
2. Semi-Structured Interviews: Eight engineers building AI applications were interviewed. The core insight was that non-ML engineers often skip pre-release evaluation entirely, shipping prototypes after manual self-checks and relying on production signals for quality assessment.
3. Design Requirements: Based on these insights, the authors established three design requirements:
   • DR1: Prioritize tracing and evaluation as high-priority capabilities.
   • DR2: Enable a "datasetless start," allowing evaluation to incorporate data from manual testing without requiring pre-existing datasets.
   • DR3: Provide education through lightweight guidance and templates.

Key Contributions: The AI Toolkit Plugin
The authors present the AI Toolkit, a plugin for JetBrains IDEs (initially released for PyCharm) that integrates observability and evaluation directly into the developer's primary environment. The toolkit consists of two main components:

1. AI Agents Debugger (Tracing & Observability):

   • Mechanism: Uses a client-server architecture. A lightweight Python wrapper intercepts framework events (specifically LangChain and LangGraph) via callbacks, serializes them, and streams them to a local IDE-side server over TCP.
   • Workflow: Tracing is automatically triggered when the user clicks "Run" or "Debug." The plugin detects AI projects by indexing imports of langchain or langgraph.
   • Visualization: It renders a hierarchical trace view and an execution graph. Unlike verbose full-callback payloads, it defaults to a "Pretty view" showing only signal-bearing fields (e.g., invoked tools, last LLM messages) to maintain a readable narrative. It also displays execution time and token usage for profiling.

2. AI Evaluation (Unit-Test-Like Workflow):

   • Concept: Models evaluation after unit tests. Developers declare data points as test cases and run them against the current code version without modifying the code itself.
   • Dataset Curation: A key feature is the "Add to Dataset" capability. Developers can promote a captured trace directly into a dataset. Inputs are auto-extracted, and reference outputs are editable.
   • Execution: The toolkit instruments the Python process at launch (via monkey-patching I/O boundaries) to propagate dataset inputs into the system. It captures the run trace and extracts generated outputs based on configured JSON paths.
   • Scoring: Evaluations use configurable metrics, including standard metrics, "LLM-as-a-judge," and custom code. Results are presented in a tool window analogous to a unit-test runner, showing aggregated statistics (average score, token usage, wall-clock time).
   • Persistence: Datasets and artifacts are stored as project files (YAML), enabling version control, diffing, and sharing.

Results and Early Adoption Signals
The paper reports initial telemetry from a four-week period following the PyCharm 2025.2 release:

• Reach: The plugin acquired 970 new users.
• Conversion: A popup prompt appearing after clicking "Run" in LangGraph projects achieved a 58% installation rate, generating one-third of all new users.
• Activation: 40% of users who installed via the popup captured their first trace. However, activation was lower (12%) for users arriving via the "What's New" tab, primarily because the initial release only supported LangGraph, and many users were not using that framework.
• Retention: By week four, 26% of all installers remained active. Among developers who actually ran code in PyCharm, retention rose to 40%. Churn (uninstall/disable) remained low at 10–15%.
• Key Takeaway: The data suggests that IDE-native, run-triggered workflows are legible and valuable. The primary barrier to adoption is currently framework coverage rather than developer intent.

Significance and Claims
The paper claims that integrating AI observability and evaluation into the IDE lowers the "activation energy" required for developers to adopt disciplined AI development practices. By meeting developers in their existing workflow (the Run/Debug loop) rather than exporting them to external consoles, the toolkit makes modern AI development accessible to non-ML specialists while preserving software engineering practices.

The authors argue that this approach transforms the informal "try–look into" routine into a disciplined loop of trace → inspect → curate to dataset → evaluate. They posit that standardizing trace schemas and treating evaluations as first-class tests within the IDE can bridge the gap between AI-native tooling and the broader software engineering community.

Limitations
The authors acknowledge several limitations:

• Narrow Coverage: Initial results are based solely on LangGraph support, likely underestimating true demand.
• Short Window: Metrics cover only the first four weeks of a single release with a sample of ~1,000 users.
• IDE Bias: The study is limited to PyCharm users, excluding a large portion of the developer base using other IDEs like VS Code.

Future Directions
The authors outline plans to broaden framework coverage to include Kotlin's Koog, PydanticAI, and the OpenAI Python SDK. They also intend to monitor the usage of AI Evaluation features to guide further UX improvements and tighter integration between the debugger and evaluation modules.