Detecting UX smells in Visual Studio Code using LLMs
This paper presents an LLM-assisted approach to detecting UX smells in Visual Studio Code by mining and classifying GitHub issues, revealing that the most prevalent usability problems concern informativeness, clarity, intuitiveness, and efficiency.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are a chef in a high-end kitchen. Your kitchen isn't just a room with a stove; it's a super-smart, automated workspace that helps you chop, mix, and cook faster than ever before. This is Visual Studio Code (VSCode), the most popular "kitchen" (or Integrated Development Environment, IDE) that software developers use to build apps and websites.
But even the best kitchens get messy. Sometimes the spice jars aren't labeled, the oven timer is confusing, or the knife block is placed in a spot where you constantly bump into it. These aren't broken appliances (bugs); they are just annoying, confusing, or inefficient design choices. In the world of software, we call these "UX Smells."
Here is a simple breakdown of what this paper is about, using some everyday analogies.
1. The Problem: The Kitchen Gets Too Crowded
Over the last decade, VSCode has grown from a simple text editor into a massive, complex kitchen with thousands of gadgets, plugins, and AI helpers. While this is powerful, it has made the kitchen harder to navigate.
Developers are constantly complaining about these annoyances in the kitchen's "suggestion box" (which is actually the GitHub issue tracker). But there are thousands of complaints. It's too much for a human to read every single note to figure out what the real problems are.
2. The Solution: Hiring a Super-Intelligent Sous-Chef (The AI)
The authors of this paper decided to use a Large Language Model (LLM)—think of it as a super-smart, tireless AI assistant—to help them read these thousands of complaints.
- The Task: They asked the AI to read every complaint, figure out if it was a "smell" (a design annoyance), and categorize it.
- The Human Check: Just like a head chef wouldn't trust a robot completely, the researchers had human experts double-check the AI's work to make sure it wasn't hallucinating or misreading the notes.
3. The Discovery: What's Actually Broken?
After the AI and humans worked together, they found something very interesting. They grouped the complaints into four main "zones" of the kitchen experience:
Zone A: The "What's Happening?" Zone (Cognitive Transparency)
- The Issue: "Why is this button grey?" "Did my save actually work?" "I can't tell which file I'm editing."
- The Result: This is where the most problems are. About 54% of all the complaints were about the system not being clear, informative, or intuitive.
- The Analogy: It's like walking into a kitchen where the lights are flickering, the labels on the jars are faded, and the oven doesn't beep when the food is done. You can still cook, but you have to guess, which is exhausting.
Zone B: The "Flow" Zone (Efficiency)
- The Issue: "I have to click five times to do one thing." "This menu is too far away."
- The Result: About 29% of complaints.
- The Analogy: The kitchen layout is okay, but you have to walk across the room to get the salt every time you cook. It slows you down.
Zone C: The "Sturdiness" Zone (Reliability)
- The Issue: "The app crashed." "The theme colors are wrong."
- The Result: Only 12% of complaints.
- The Analogy: Surprisingly, the stove and the fridge (the core mechanics) rarely break. The hardware works fine; it's just the experience of using it that's annoying.
Zone D: The "Learning" Zone (Flexibility & Empowerment)
- The Issue: "I can't customize this." "I don't know how to find this feature."
- The Result: Only 5% of complaints.
- The Analogy: Once the kitchen is stable, people stop worrying about how to learn new tricks or change the layout. They just want to cook dinner without confusion.
4. The Big Insight: "UX Debt"
The paper introduces a concept called UX Debt. Imagine you are building a house. If you rush and put the light switch in a weird spot because you're in a hurry, you create "Debt." You have to fix it later, or you just live with the annoyance.
The researchers found that VSCode has a lot of "Debt" in the "Clarity" and "Efficiency" areas.
- The "Salience-Neglect" Theory: The things developers care about most (being fast and knowing what's going on) are the exact things that are getting the most annoying. Because the software is growing so fast, the designers are adding new features but forgetting to keep the interface clear.
- The "Saturation" Theory: The "boring" stuff (like the app not crashing) is actually working really well. The problems have shifted from "Does it work?" to "Does it feel good to use?"
5. Why This Matters
This study is important because it proves that for modern software tools, the biggest problem isn't that the code is broken; it's that the user experience is getting "cognitive drag."
It's like driving a car with a perfect engine (Reliability) but a confusing dashboard and sticky steering wheel (UX Smells). You can still drive, but you get tired and frustrated much faster.
In summary: The authors used AI to read thousands of user complaints about VSCode. They discovered that while the software is technically strong, it is suffering from a "clarity crisis." Developers are struggling to understand what the software is doing and how to do things quickly. The solution isn't necessarily to fix bugs, but to clean up the design so the "kitchen" feels intuitive again.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.