Technical Summary: Rethinking Technology Stack Selection with AI Coding Proficiency

1. Problem Statement

The advent of Large Language Models (LLMs) has transformed software development workflows, yet existing technology selection frameworks remain rooted in human-centric evaluation criteria. Traditional methods assess technologies (e.g., third-party libraries) based on intrinsic attributes like maintainability, performance, and reliability for human developers. However, these frameworks overlook a critical variable in the LLM era: AI coding proficiency.

The core problem is that a technology popular and effective for human developers may not be effectively leveraged by LLMs to generate high-quality code. Selecting a technology with low AI proficiency can lead to:

• High Debugging Effort: LLMs may generate code with syntax errors, deprecated API calls, or functional omissions when using certain libraries.
• Technical Debt: Low-quality generated code increases integration costs and offsets productivity gains.
• Ecosystem Risks: Proficiency gaps may trigger a "Matthew effect," where developers disproportionately adopt high-proficiency libraries, threatening technological diversity and increasing security risks associated with centralization.

The paper poses the practical question: Is a technology ready for AI-assisted development?

2. Methodology

2.1 Concept Definition

The authors introduce AI Coding Proficiency, defined as the degree to which a given LLM can accurately understand and correctly invoke a technology's APIs to produce high-quality code snippets. This is measured by the quality of code generated when the LLM is tasked with using specific libraries.

2.2 Dataset Construction

To study this phenomenon, the authors constructed a large-scale dataset involving:

• Scope: 170 third-party Python libraries across 61 distinct software development scenarios (e.g., geographic data processing, database interaction, machine learning).
• Competing Libraries: Libraries were paired based on functional overlap (e.g., TensorFlow vs. PyTorch, Shapely vs. GDAL) to allow for direct comparison.
• Prompt Generation: An automated pipeline generated 850 prompts, each asking an LLM to complete a specific task using a target library.

2.3 Experimental Setup

• Models: Six representative LLMs were evaluated: GPT-4o, GPT-4o-mini, Claude Sonnet 4, Gemini 2.5 Flash, Qwen3-Coder-Plus, and DeepSeek-R1.
• Evaluation Metrics: A multi-dimensional code quality assessment was applied to 25,500 generated code snippets, covering five dimensions:
  1. Functional Suitability: Correctness relative to task description.
  2. Performance Efficiency: Estimated time and space complexity.
  3. Maintainability: Measured via Maintainability Index (Halstead volume, cyclomatic complexity).
  4. Usability: Readability scores.
  5. Reliability: Handling of edge cases and exceptions.
• Scoring: An overall AI proficiency score was calculated as the arithmetic mean of these five dimensions. Statistical significance was verified using bootstrap resampling and Cohen's d effect sizes.

2.4 Failure Analysis & Mitigation

• Failure Patterns: Low-quality snippets (identified via interquartile range) were manually annotated by software engineering experts to categorize failure modes.
• Prompting Techniques: The study evaluated three prompting strategies (Chain-of-Thought, Few-shot prompting, and Regeneration) to determine if they could mitigate proficiency gaps.

3. Key Results

3.1 Proficiency Gaps (RQ1)

• Library Variance: Competing libraries with similar functionality and popularity exhibit significant differences in AI proficiency. In repeated runs, the quality score difference between libraries like Shapely and GDAL reached up to 84.89%.
• Model Variance: The "best" library varies significantly depending on the LLM used. For instance, GPT-4o performed better with PostgreSQL adapters, while Qwen3-Coder-Plus performed better with ArangoDB adapters for the same tasks.
• Correlation: There is no significant correlation (Spearman coefficient = 0.09) between a model's general coding capability ranking (based on standard benchmarks) and its proficiency with specific libraries.
• Prevalence: Approximately 11.17% of competing library pairs showed significant quality differences across the models tested.

3.2 Failure Patterns (RQ2)

Analysis of 814 low-quality code snippets revealed eight failure patterns. The two most critical and common were:

1. Incorrect Functionality (53.07%): The code executes but fails to meet core task requirements (e.g., missing convolutional layers in a CNN).
2. Missing Edge Handling (40.66%): Failure to handle edge cases or exceptions, posing reliability and security risks.
   Other patterns included Syntax Errors, Incorrect Library Usage, and Import Issues.

3.3 Enhancement Strategies (RQ3)

• Prompting Impact: Few-shot prompting and Regeneration significantly improved overall code quality scores and reduced proficiency gaps between competing libraries (e.g., reducing the gap by up to 44.93% on Qwen3-Coder-Plus).
• Limitations: Chain-of-Thought (CoT) showed minimal improvement, potentially because it increases implementation complexity without addressing core correctness or maintainability issues.
• Unintended Consequences: While prompting can narrow gaps, it can also cause "lead reversals" where a previously inferior library becomes superior, or widen gaps if examples align poorly with a library's API conventions.

3.4 Popularity vs. Proficiency

The study found only a weak positive correlation (0.574) between library popularity (GitHub stars) and AI proficiency. In many cases, more popular libraries had lower AI proficiency scores, indicating that popularity is an unreliable predictor for AI-assisted development suitability.

4. Key Contributions

1. Conceptualization: Proposed and defined AI Coding Proficiency as a new, essential dimension for technology selection in the LLM era.
2. Methodology: Developed a comprehensive evaluation pipeline and dataset covering 170 libraries, 61 scenarios, and six LLMs, utilizing a multi-dimensional quality assessment framework.
3. Empirical Evidence: Conducted the first large-scale study quantifying proficiency gaps, revealing that library choice significantly impacts code quality and that model selection must be technology-specific.
4. Actionable Insights: Identified specific failure patterns (Incorrect Functionality, Missing Edge Handling) and demonstrated that prompt engineering can serve as a mitigation strategy, though it is not a universal fix.

5. Significance and Claims

The paper argues that the software engineering community must fundamentally rethink technology selection frameworks. The authors claim that:

• Current frameworks are insufficient: Relying solely on intrinsic attributes or general model leaderboards leads to suboptimal choices in AI-assisted development.
• Proficiency is a critical metric: Ignoring AI proficiency risks eroding productivity gains, increasing technical debt, and introducing security vulnerabilities.
• Ecosystem health is at stake: Unchecked proficiency gaps could lead to a "winner-takes-all" market dynamic, reducing the diversity and robustness of the software ecosystem.
• Call to Action: The authors call for the integration of AI proficiency into technology selection frameworks and the development of strategies (both from the model and library sides) to preserve competitive balance and ensure high-quality, efficient development.

The study is modest in its scope, focusing primarily on the Python ecosystem and third-party libraries, acknowledging that future work is needed to generalize findings to other languages and technology types.