Technical Summary: The Influence of Large Language Models on Code

Problem Statement

The rapid integration of Large Language Models (LLMs) and LLM-powered tools (e.g., Copilot, Cursor) into software development has raised questions regarding their tangible impact on real-world programming practices. While previous research has focused on distinguishing LLM-generated code from human-written code or evaluating functional correctness, there is a lack of empirical evidence regarding whether LLMs have fundamentally transformed code style in the wild. This paper investigates the central question: Have LLMs transformed code style, and how can such transformation be characterized? Specifically, the authors examine whether the stylistic preferences of LLMs (e.g., naming conventions, complexity metrics) are permeating human-authored codebases over time.

Methodology

The study employs a dual-pronged approach combining large-scale empirical analysis of real-world repositories with controlled simulations.

1. Data Collection

• GitHub Repositories: The authors collected 22,074 repositories and over 1 million source code files linked to arXiv papers published between Q1 2020 and Q3 2025. Repositories were categorized by programming language (Python, C/C++) and scientific domain (Computer Science vs. Non-CS).
• Human-Written Baseline: To establish a pre-LLM baseline, the authors utilized Code4Bench, a dataset of accepted user submissions from Codeforces prior to 2020.
• Problem Subset: A subset of 200 algorithmic problems was selected from Code4Bench, stratified by difficulty (800–2000+ rating) and algorithm type (e.g., DP, Greedy, Math).

2. Simulation and Generation

The authors simulated LLM-assisted coding using 12 diverse LLMs (including Qwen3 series, DeepSeek-R1, GPT-4.1, Llama-4, and Gemini-2.0). Two generation strategies were tested:

• Direct Generation (ANS): Models solved problems from scratch based solely on problem descriptions.
• Reference-Guided Generation (REF): Models were provided with a correct human-written solution (AC code) and instructed to revise or rewrite it.

3. Evaluation Metrics

The study analyzed code across four dimensions:

• Naming Patterns: Categorized into formats (snake_case, camelCase, PascalCase, etc.) and measured name lengths.
• Complexity & Maintainability: Measured using Cyclomatic Complexity, Halstead metrics (Volume, Difficulty, Effort, Bugs), and Maintainability Index (Standard and Custom).
• Code Similarity: Quantified using Cosine Similarity and Jaccard Similarity between human code, direct generation, and reference-guided generation.
• Reasoning Alignment: Analyzed the match between algorithmic labels in the model's reasoning chain and the ground-truth problem tags.

Key Contributions and Results

1. Evidence of Style Transformation (Naming Patterns)

The study provides the first large-scale empirical evidence that LLMs are influencing human coding styles.

• Adoption of LLM Preferences: In Python, the proportion of snake_case function names in GitHub repositories increased from 40.7% in Q1 2023 to 49.8% in Q3 2025. This trend aligns with the preference observed in LLM-generated code.
• Variable Naming: LLMs consistently prefer longer, more descriptive variable names (e.g., max_length over ml) and avoid single-letter or digit-suffixed names. GitHub repositories show a parallel decline in short/digit-suffixed names and an increase in average name length, particularly in CS domains.
• Language Differences: While Python shows clear temporal trends in naming, C/C++ trends are less pronounced, though snake_case usage in C/C++ variables and functions still shows an upward trajectory.

2. Complexity and Maintainability

• LLM vs. Human: In Python, LLM-generated code exhibited higher cyclomatic complexity than human-written code. Conversely, in C/C++, LLMs produced code with lower cyclomatic complexity.
• Maintainability: LLM-generated code generally demonstrated higher maintainability indices and lower estimated bug counts compared to human-written solutions in the simulation.
• Reference-Guided vs. Direct: In Python, reference-guided generation (rewriting existing code) resulted in more concise code (lower Halstead volume/effort) than direct generation. In C/C++, the opposite was observed.

3. Code Similarity

• Imitation Capability: When provided with reference code, LLMs produced solutions with significantly higher similarity (both Cosine and Jaccard) to the original human code compared to direct generation.
• Divergence: Without reference guidance, LLM-generated code in IO algorithm tasks diverged significantly in style from human-written code, suggesting that LLMs have distinct "native" styles that differ from human norms unless explicitly conditioned.

4. Reasoning and Algorithmic Understanding

• Label Matching: The analysis of reasoning chains revealed that LLMs often fail to explicitly mention the correct algorithmic labels (e.g., "DP," "Greedy") associated with the problem.
• Error Rates: The error rate (identifying incorrect algorithms) generally exceeded the match rate (identifying correct algorithms).
• Language Bias: Models tended to rely more heavily on algorithmic reasoning for C/C++ tasks compared to Python, where they leaned toward practical implementation details.
• Difficulty Correlation: Both match and error rates increased with problem difficulty, suggesting that harder problems better activate the models' algorithmic reasoning capabilities, albeit with higher noise.

Significance and Claims

The paper claims to provide the first large-scale empirical evidence that LLMs are not just tools for generating code but are actively reshaping the stylistic landscape of real-world programming.

• Shift in Norms: The authors argue that human coding styles are shifting toward LLM-preferred conventions (e.g., snake_case, longer names), creating a feedback loop where human developers may be subtly aligning with machine-generated norms.
• Beyond Detection: Unlike prior work focused on detecting AI-generated code, this study focuses on the influence of LLMs on the evolution of code style, highlighting that the impact of LLMs extends beyond functional correctness to aesthetic and structural norms.
• Implications: The findings suggest that as LLM adoption grows, the distinction between "human" and "machine" style may blur, necessitating a re-evaluation of how code quality and style are assessed in the future.

The authors remain modest regarding their claims, acknowledging limitations such as the inability to quantify the exact percentage of code generated by LLMs in the wild and the potential incompleteness of the pre-LLM baseline (Code4Bench) in representing the full spectrum of human coding styles. They emphasize that their work identifies traces of influence rather than definitive causation for all observed changes.