💻 computer science

Can LLMs Produce Better Object-Oriented Designs than Human-Involved Development?

This study compares object-oriented design quality across pre-LLM, post-LLM, and pure-LLM Java projects, finding that while LLM-generated code exhibits lower complexity and fewer code smells, it often suffers from oversimplification and weak responsibility separation, underscoring the continued necessity of human guidance in design decomposition.

Original authors: Zushuai Zhang, Elliott Wen, Ewan Tempero

Published 2026-05-20

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zushuai Zhang, Elliott Wen, Ewan Tempero

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a master architect tasked with building a complex house. You have three different teams to choose from to do the actual construction:

The Old Guard (PreAI): Experienced builders who have never used a computer assistant. They draw their own blueprints and build everything by hand.
The Hybrid Crew (PostAI): Experienced builders who now have a powerful computer assistant. They still draw the blueprints and make the big decisions, but they let the computer help them lay bricks and cut wood.
The Robot Team (PureAI): A team of advanced robots that take a simple verbal description of the house and build the entire thing from scratch, end-to-end, without a human touching a tool.

This paper asks a simple question: Which team builds the best house? Specifically, does the "Robot Team" create a design that is just as good, or even better, than the human teams?

The Experiment

The researchers set up a "construction challenge" using a classic board game called Kalah. They asked three groups to build a Java software program to play this game:

Group 1 (PreAI): Students from 2021 (before AI coding tools were common).
Group 2 (PostAI): Students from 2024 (after AI tools became popular).
Group 3 (PureAI): The same assignment given to three different top-tier AI models, which generated the code entirely on their own.

They didn't just check if the game worked; they looked at the quality of the design. Think of this as checking if the house has a logical layout, if the rooms are the right size, and if the plumbing isn't tangled.

What They Found

1. The Robot Team Built "Smaller" Houses, But They Were Too Simple

The PureAI (Robot) teams built code that looked very clean on the surface.

The Good News: Their code had fewer "smells" (bad habits like messy code) and seemed less complex overall. It was like a house with very few walls and a very open floor plan.
The Bad News: This simplicity was actually a problem. The robots oversimplified the design. Instead of building separate rooms for the "Player," the "Board," and the "Pits" (the holes in the game), the robots often smashed everything into one or two giant rooms.
The Metaphor: Imagine a house where the kitchen, bedroom, and bathroom are all just one big room with no doors. It's very simple and easy to clean (low complexity), but it's terrible for living in because you can't separate your activities (poor "responsibility separation"). The robots missed the important "abstractions" (the distinct concepts) that make a system understandable.

2. The Hybrid Crew (PostAI) Started Acting Like Robots

The students from 2024 (who used AI helpers) didn't build quite as "robotic" as the PureAI team, but they were closer to the robots than the 2021 students were.

Their designs were also getting a bit too simple, missing some of the distinct "rooms" (concepts) that the 2021 students included.
However, unlike the robots, the Hybrid Crew's code actually had more messy habits (code smells) than the old-school students. It seems that when humans try to fix or edit robot-generated code, they sometimes introduce new messiness locally, even if the overall structure looks simple.

3. The "Prompt" Matters, But It's Not a Magic Wand

The researchers tried giving the robots more specific instructions, like "Make sure you have a separate class for the Player and a separate one for the Board."

Result: When the instructions were more specific, the robots did a better job of creating separate "rooms" (concepts).
Limitation: Even with the best instructions, the robots still didn't quite match the quality of the human-designed houses. They still tended to oversimplify.

The Big Takeaway

Robots are great at laying bricks, but humans are still needed to draw the blueprints.

The paper concludes that while AI can write code that works and looks clean, it struggles with the big picture of Object-Oriented Design. It tends to lump everything together to make things "simple," which actually makes the software harder to maintain later because the distinct parts of the system are lost.

PureAI (Robots): Great at avoiding small mistakes, but bad at understanding the big structure.
PostAI (Humans with AI): Starting to lose some of the structural depth, and sometimes making the code messier in the process.
The Lesson: If you use AI to write software, you must act as the architect. You need to tell the AI exactly how to break the problem down into distinct parts (decomposition) and what each part is responsible for. If you just say "build a game," the AI will build a messy, oversimplified version that looks clean but is hard to fix later.

Technical Summary: Can LLMs Produce Better Object-Oriented Designs than Human-Involved Development?

Problem Statement

While Large Language Models (LLMs) are increasingly capable of generating code, their ability to produce high-quality Object-Oriented Design (OOD) for multi-class projects remains unclear. Prior research has largely focused on:

Functional correctness at the problem level (single methods or classes) rather than project-level architecture.
Pre-LLM baselines, failing to account for the impact of LLM-assisted human development (PostAI).
High-level artifacts (e.g., UML diagrams) rather than the maintainability and structural quality of implemented code.

This study addresses the gap in understanding how OOD quality in projects generated end-to-end by LLMs (PureAI) compares to human-involved projects produced before (PreAI) and after (PostAI) the widespread adoption of LLMs.

Methodology

The authors conducted a comparative case study using a postgraduate Java assignment: implementing the Kalah board game. The study analyzed 11 datasets across three authorship conditions:

PreAI (Human-Involved, Pre-LLM): 93 submissions from March 2021, predating the public release of major LLM coding assistants.
PostAI (Human-Involved, Post-LLM): 57 submissions from August 2024, after LLM tools became mainstream.
PureAI (LLM-Generated): 810+ generation runs using three contemporary models (gpt-5.4, gemini-2.5-pro, gemini-3.1-pro-preview) across three prompt variants varying in OOD guidance specificity (None, Broad, Specific). Only runs that passed all 19 functional test cases (after up to five repair iterations) were retained.

Evaluation Metrics

The study employed a multi-dimensional evaluation framework:

OOD Metrics: 13 metrics capturing structural characteristics, including Weighted Methods per Class (WMC), Coupling Between Object Classes (CBO), Lack of Cohesion in Methods (LCOM), Depth of Inheritance Tree (DIT), Number of Classes (#Cl), and Lines of Code (LOC).
Code Smell Density: Project-level density of 30 detected smell types (11 method-level, 19 class-level), normalized by size (KLOC).
Domain Modeling:
- Concept Representation: Whether six key domain concepts (Board, Game, Player, Pit, House, Store) were explicitly modeled as classes.
- Runtime Conformance: Whether the number of instantiated objects matched the requirements implied by the domain concepts.

Statistical Analysis

Comparisons utilized the Mann–Whitney U test with Cliff's delta ( $\delta$ ) for effect size on non-binary metrics, and Chi-square/Fisher's exact tests with Odds Ratios (OR) for binary metrics. Results were corrected for multiple comparisons using the Benjamini–Hochberg procedure.

Key Contributions

First Comparative Analysis: This is the first study to compare OOD quality across PreAI, PostAI, and PureAI conditions in a project-level setting.
Multi-Dimensional Evaluation: The study combines quantitative OOD metrics, code smell analysis, and requirement-based domain modeling to provide a holistic view of design quality.
Empirical Evidence on Oversimplification: The paper provides evidence that while LLM-generated code often appears "cleaner" (lower smell density), this is frequently associated with oversimplification (missing abstractions, weaker responsibility separation).
Insights into PostAI Trends: The study demonstrates that human-involved projects in the LLM era (PostAI) exhibit design tendencies closer to PureAI (simplification) than PreAI, yet without the same reduction in code smells.

Results

RQ1: Differences Among PreAI, PostAI, and PureAI

Structural Simplicity vs. Oversimplification: PureAI projects generally exhibit fewer classes, lower total complexity, size, and coupling compared to human projects. However, this "simplicity" correlates with oversimplification:
- Higher average and max-percentage complexity/coupling per class (burden concentrated in fewer classes).
- Lower cohesion (higher LCOM), as single classes often represent multiple domain concepts.
- Fewer explicitly represented domain concepts (missing abstractions).
Code Smells: PureAI projects show significantly lower code smell density (both method and class levels) compared to human-involved projects.
PostAI Convergence: PostAI projects are statistically closer to PureAI than to PreAI on many OOD metrics, showing similar tendencies toward simplification and missing domain concepts. However, unlike PureAI, PostAI projects have higher code smell density than PreAI, particularly at the method level.
Runtime Conformance: Once a domain concept is represented, PureAI projects are generally more likely to conform at runtime than human projects.

RQ2: Effect of Prompt Specificity in PureAI

Impact of Guidance: Increasing prompt specificity (from None $\to$ $\to$ Broad $\to$ $\to$ Specific) generally leads to:
- More classes and higher total complexity/size/coupling (moving away from oversimplification).
- Lower average and max-percentage complexity/coupling per class.
- Reduced overall and method-level code smell density.
- Improved representation of domain concepts (though effects vary by model).
Limitations: More specific prompts do not fully close the gap with human-involved projects. For some models (e.g., O54), increasing specificity from Broad to Specific yields limited additional improvement.

Significance and Implications

The paper concludes that while LLMs can effectively handle implementation details (producing code with fewer smells and high runtime conformance), they struggle with object-oriented decomposition and responsibility assignment without explicit human guidance.

Oversimplification Risk: PureAI projects often appear simpler and less "smelly," but this masks a lack of proper abstraction and responsibility separation, which can harm long-term maintainability.
PostAI Warning: The availability of LLM tools may lead human developers to produce designs that mimic LLM oversimplification (missing concepts) while introducing new local issues (method-level smells), potentially negating productivity gains.
Human-in-the-Loop Necessity: Appropriate human guidance on high-level decomposition and domain modeling remains critical. Relying solely on generic or even specific OOD prompts is insufficient to achieve human-comparable design quality.
Review Strategy: Code reviews in LLM-assisted environments should explicitly check for missing domain concepts and oversimplified structures, rather than just functional correctness or local code smells.

The study suggests that the most effective use of LLMs in OOD is to leverage them for implementation within a structure defined by human experts who understand the domain and desired architecture.