💻 computer science

Do AI Agents Really Improve Code Readability?

This study reveals that while AI agents primarily target logic complexity and documentation when refactoring code for readability, their actions frequently degrade traditional quality metrics such as the Maintainability Index and Cyclomatic Complexity, challenging the assumption that AI-driven refactoring inherently improves code readability.

Original authors: Kyogo Horikawa, Kosei Horikawa, Yutaro Kashiwa, Hidetake Uwano, Hajimu Iida

Published 2026-03-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kyogo Horikawa, Kosei Horikawa, Yutaro Kashiwa, Hidetake Uwano, Hajimu Iida

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very talented, hyper-fast assistant named "AI Agent." You ask this assistant to tidy up your messy workshop (your computer code) so it's easier for you and your friends to find tools and understand how things work. You expect the workshop to look cleaner, brighter, and more organized.

But what if, after the assistant finishes, the workshop is actually more cluttered, even though the assistant claims it's "cleaner"?

That is exactly what this research paper discovered. The authors, a team of researchers from Japan, investigated whether AI agents are actually making code easier to read, or if they are just making it look different.

Here is the story of their findings, broken down into simple concepts:

1. The Setup: The "Tidy Up" Request

Code readability is like the legibility of a handwritten note. If the handwriting is messy, the note is useless. If the code is messy, it's hard to fix bugs or add new features.

Recently, developers started using AI Agents. Unlike a simple chatbot that gives you one answer, an AI Agent is like a project manager. It can plan a task, break it into steps, do the work, and check its own homework.

The researchers wanted to know: When these AI Agents try to "clean up" code, do they actually make it easier to read?

2. The Investigation: Looking for the "Tidy" Signs

The researchers went through a massive database of over 1.3 million code changes made by AI agents. They looked for "receipts"—specifically, commit messages (the notes developers write when saving changes) that said things like "improving readability," "making it clearer," or "easier to understand."

The First Surprise (RQ1): The Agents Don't Care Much About Reading
They found that out of every 1,000 changes the AI made, only 3 were explicitly trying to make the code easier to read.

Analogy: Imagine a chef who is hired to cook a meal. You'd expect them to focus on the food. But this chef is mostly focused on sharpening knives, washing dishes, and rearranging the spice rack. Only 0.3% of their time is actually spent on making the food taste better. The AI agents are mostly busy with other things, not readability.

3. The "What" They Changed: Deep Fixes vs. Surface Polish

When the AI agents did try to improve readability, what did they actually do? The researchers compared the AI's work to how human developers usually clean up code.

Humans usually fix the "surface stuff": renaming variables (like changing x to customer_name), fixing indentation, or adding comments. It's like organizing the tools on the wall so they are easy to grab.
AI Agents focused on the "deep stuff": simplifying complex logic and fixing documentation.
- Analogy: If the code was a tangled ball of yarn, humans would untie the knots one by one. The AI Agent, however, decided to cut the whole ball in half and rewrite the instructions on how to use it. They focused on the logic (how the machine thinks) rather than the style (how it looks).

4. The Big Twist: The "Cleaner" Code Was Actually Messier

This is the most shocking part of the study. The researchers measured the code before and after the AI touched it using standard "health metrics" (like a doctor checking blood pressure).

They expected the code to get better. Instead, they found:

56% of the time, the code became harder to maintain (the "Maintainability Index" dropped).
42% of the time, the code became more complex (the "Cyclomatic Complexity" went up).
The code actually got longer.

The Analogy:
Imagine you ask a contractor to fix a leaky faucet.

You expect: They tighten the valve, and the leak stops.
What happened: The contractor replaced the faucet with a high-tech, custom-built machine that has 50 extra pipes, a digital display, and a warning light. It works, and the leak is gone, but now the whole thing is huge, expensive to fix, and confusing to understand.

The AI agents were so focused on making the logic "smart" or adding explanations that they accidentally made the code bloated and complicated. They traded a simple, clean solution for a complex, over-engineered one.

5. The Conclusion: Don't Trust the Agent Blindly

The paper concludes that while AI agents are powerful, they currently lack the "common sense" to know when a code change is actually helpful for a human reader.

They are good at: Fixing broken logic and writing long explanations.
They are bad at: Knowing when to stop. They tend to over-engineer, making code longer and more complex in the name of "readability."

The Takeaway for Developers:
Don't just let the AI agent hit "Apply" on all its suggestions. You need to be the Quality Control Inspector. You have to look at the changes and ask: "Did this actually make it simpler, or did the AI just build a bigger, more confusing machine?"

In short: AI agents are great at thinking, but they are currently terrible at knowing when to keep things simple.

1. Problem Statement

Code readability is a cornerstone of software maintainability and sustainability. Poor readability leads to technical debt, increased bug rates, and higher communication costs. While Large Language Models (LLMs) and AI agents have emerged as powerful tools for automated refactoring, existing research has largely focused on general refactoring activities (e.g., performance optimization, structural changes) rather than specifically targeting readability.

The core research gap is that it remains unclear whether AI agents, when explicitly instructed or observed to improve readability, actually enhance code quality. There is a concern that agents might prioritize superficial changes or inadvertently degrade structural metrics while attempting to improve readability.

2. Methodology

The study employed a quantitative and qualitative analysis of commits from the AIDev dataset, which contains pull requests generated by five different coding agents across thousands of repositories.

Data Collection & Filtering

Source: 1.38 million agentic commits were mined from the AIDev dataset.
Filtering: The authors filtered commits based on:
1. Keywords: Commit messages containing readability-related terms (e.g., "readability," "clarity," "understandable").
2. Language: Only commits modifying Python source files were retained to ensure consistent metric analysis.
3. Exclusion: Merge commits were excluded.
Final Sample: 403 valid commits were selected for deep analysis (after excluding those with parsing errors).

Research Questions (RQs)

RQ1 (Prevalence): How common are readability-related commits among all AI agent-generated commits?
RQ2 (Focus): What specific types of readability issues (smells) do AI agents address compared to human developers?
RQ3 (Impact): How do quantitative code metrics change before and after these readability-focused commits?

Analysis Techniques

Qualitative (RQ2): A random sample of 231 commits was manually inspected by three experienced researchers using closed card sorting. They mapped changes to eight predefined readability smells (e.g., complex logic, bad identifiers, documentation issues) based on Oliveira et al.'s taxonomy. Inter-rater reliability was high (94.7% agreement).
Quantitative (RQ3): The authors used the Radon library to calculate metrics for code before and after each commit.
- Metrics: Maintainability Index (MI), Cyclomatic Complexity (CC), Lines of Code (LOC), Logical Lines of Code (LLOC), Source Lines of Code (SLOC), and Halstead metrics (Volume, Difficulty, Effort).
- Statistics: The Wilcoxon signed-rank test ( $p < 0.01$ ) was used to determine statistical significance, and Cliff's delta was used to measure effect size.

3. Key Results

RQ1: Prevalence

Readability-related commits are extremely rare, accounting for only 0.3% of all AI agent-generated commits.
The most common keyword used was "readability" (50.1% of readability commits), indicating agents use standardized vocabulary when addressing this topic.

RQ2: Focus of Improvements

AI Agents vs. Humans: There is a distinct bias in what agents choose to fix.
- Agents heavily prioritize Complex/Long Logic (42.4%) and Documentation (24.2%).
- Agents largely ignore surface-level issues like Bad Identifiers (14.3%) and Formatting (14.3%), which are more frequently addressed by humans.
Example: Agents tend to simplify complex string operations (e.g., replacing manual escaping with json.dumps) rather than renaming variables or fixing indentation.

RQ3: Impact on Code Metrics

Contrary to the goal of improving readability, the quantitative analysis revealed a degradation in traditional quality metrics:

Maintainability Index (MI): Decreased in 56.1% of commits (Mean $\Delta = -3.25$ ).
Cyclomatic Complexity (CC): Increased in 42.7% of commits (Mean $\Delta = +3.13$ ).
Code Size: Lines of Code (LOC) increased significantly in 71.5% of cases.
Statistical Significance: The increases in complexity and code volume were statistically significant ( $p < 0.001$ ) with large effect sizes.
Conclusion: While agents attempt to clarify logic, they often do so by adding code volume or complexity, resulting in lower maintainability scores according to standard metrics.

4. Key Contributions

Empirical Evidence on AI Readability: The study provides the first large-scale empirical analysis specifically isolating AI agent refactoring for readability, distinguishing it from general refactoring.
Identification of Agent Bias: It reveals that AI agents have a strong bias toward fixing "deep" logic and documentation issues while neglecting "shallow" but common readability smells like naming conventions.
Paradoxical Findings: It uncovers the counter-intuitive finding that commits explicitly labeled as "readability improvements" often result in worse structural quality metrics (higher complexity, lower maintainability).
Replication Package: The authors provide a public replication package containing the data and scripts used for the analysis.

5. Significance and Implications

For Practitioners: Developers should not blindly accept AI-generated refactoring suggestions labeled as "readability improvements." Manual review is critical, as these changes may increase technical debt and cognitive load.
For Tool Developers: AI agents need to be equipped with quality-checking mechanisms. Future agents should include a verification phase that rejects refactoring if it increases Cyclomatic Complexity or decreases the Maintainability Index, even if the code looks "cleaner" superficially.
For Researchers: The study highlights a gap between "perceived" readability (what agents think they are fixing) and "measured" readability (structural metrics). Future research should focus on training agents to balance functional logic simplification with structural efficiency.

Final Conclusion: AI agents currently lack a robust understanding of effective readability improvements. While they successfully target complex logic and documentation, their interventions frequently degrade structural code quality, suggesting that current agentic coding workflows require human oversight and stricter quality gates.