Technical Summary: Control Models for In-IDE Code Completion

Problem Statement

The integration of Large Language Models (LLMs) into Integrated Development Environments (IDEs) has introduced significant challenges regarding user flow and resource efficiency. While LLMs offer powerful code completion capabilities, they frequently generate suggestions that are incorrect, poorly timed, or misaligned with the developer's current state of mind and experience level. This misalignment leads to distracting interactions, unnecessary context switches, and wasteful inference costs.

Current data indicates that only approximately 31% of LLM inferences result in a shown completion, and of those shown, only 31% are accepted by the user. Furthermore, many shown completions no longer match the code context by the time they are generated. The authors argue that simply improving the content generation is insufficient; the interaction dynamics between the developer and the AI must be managed to prevent disruption to the developer's flow.

Methodology

The paper introduces Control Models: lightweight machine learning classifiers designed to intervene in the code completion pipeline. These models do not generate code but rather decide when to trigger an inference and whether to display a generated suggestion.

Model Architectures

The authors evaluate two primary architectural approaches:

1. Gradient-Boosting Models (CatBoost): These models operate on tabular telemetry data. They are chosen for their robustness with categorical features and low inference latency, making them suitable for frequent interactions.
2. Transformer-Based Models: These models combine scalar/tabular features with raw code context. They utilize a hybrid design where a 100M-parameter code generation model serves as the backbone for understanding the code scope, with tabular features encoded via a Multi-Layer Perceptron (MLP) and concatenated before the classification head.

Data and Features

The training and evaluation utilize anonymized feature usage logs from JetBrains' cloud-inferenced completion service (Mellum).

• Tabular Telemetry: Includes categorical and scalar features such as typing speed, caret scope, previous actions, and static analysis outputs.
• Code Context: Raw code snippets surrounding the cursor (collected primarily for Kotlin in offline settings due to privacy constraints).
• Labels: Completions are labeled as Positive (accepted by the user via Tab key) or Negative (ignored or explicitly rejected via Escape key, mouse click, or caret movement).

Evaluation Strategy

The study employs a three-pronged evaluation approach:

1. Offline Analysis: Comparing Boosting vs. Transformer models across different False Negative Rates (FNR) to determine the trade-off between filtering out bad suggestions and missing good ones.
2. Cross-Language Offline Evaluation: Testing Boosting models on syntactically diverse languages (Kotlin, Python, PHP, C#) to assess generalizability.
3. Online A/B Testing: Deploying Boosting-based models in a production environment to measure real-world impact on user metrics.

Key Contributions

The paper outlines three primary contributions:

1. Empirical Validation of Control Models: An offline analysis and online A/B study demonstrating that lightweight, boosting-based trigger and filter classifiers can reduce inference requests by approximately 20% while improving completion quality metrics in JetBrains IDEs.
2. Analysis of Model Architectures: Large-scale empirical evidence highlighting the potential of transformer models to combine contextual code understanding with in-IDE features, while simultaneously documenting the practical obstacles regarding latency, privacy, and backend integration that favor the adoption of smaller boosting models in production.
3. Discussion on Future Directions: A critical examination of evaluation metrics, advocating for a shift toward measuring long-term productivity gains rather than immediate acceptance rates. The authors also propose exploring user personalization and alternative architectures like end-to-end reinforcement learning.

Results

Offline Performance

• Boosting vs. Transformers: Transformer-based models demonstrated the ability to filter more generations with a smaller impact on the absolute number of completed symbols compared to boosting models. However, boosting models outperformed transformers in improving Accept Rate (AR) and Cancel Rate (CR) at higher filter rates. This is attributed to the fact that the denominator for rate-based metrics (shown completions) is less affected by the boosting approach's filtering strategy.
• Language Variance: Performance varied significantly across languages. For instance, in Kotlin, the filter model was more effective, whereas for PHP, the trigger model offered better control at specific false-negative rates. The authors note that filter models generally have access to more information (generated completion features) but are sometimes limited by smaller training sets due to the risks of disabling them in production.

Online A/B Study

The study deployed boosting-based models in a production environment for Java, Python, and Kotlin.

• Filter Models: Resulted in significant improvements in Accept Rate (+32.9% to +47.6%) and reductions in Cancel Rate (-15.5% to -36.7%). However, this came at the cost of a reduction in the Ratio of Completed Code (RoCC) (-9.6% to -14.0%), confirming offline findings that aggressive filtering reduces the total volume of code inserted.
• Trigger Models: In the trigger model ablation (where filter models were present in both groups), the model successfully prevented approximately 20% of completions from being generated. However, the observed reduction in total generated completions per user was only 13.8%. The authors attribute this to the dependence of completion events: by altering the user's interaction flow, the trigger model inadvertently created new opportunities for completions, mitigating the expected reduction in volume.

Significance and Claims

The paper positions Control Models as a necessary evolution in the integration of LLMs into developer workflows. The authors claim that:

• Efficiency and Quality: Smarter triggers and filters can decouple the cost of inference from the value delivered to the user, saving computational resources while enhancing the perceived quality of the tool.
• User-Centric Design: The focus must shift from purely optimizing the LLM's output to optimizing the interaction with the LLM. This includes aligning suggestions with the developer's flow and mental state.
• Metric Evolution: Current metrics (like Accept Rate) primarily measure perceived productivity. The authors argue for the development of metrics that capture long-term productivity gains and maintainability, noting that short-term "acceleration" does not always equate to effective software delivery.
• Personalization: The "lowest-hanging fruit" for future improvement lies in designing models that adapt to individual user preferences, as acceptance and rejection of suggestions are often matters of personal style rather than objective code quality.

The authors conclude that while transformer models offer deep contextual understanding, the practical constraints of latency and privacy in IDEs currently favor the adoption of efficient boosting-based control models, with a clear path toward more personalized and integrated interaction policies.