Rule Taxonomy and Evolution in AI IDEs: A Mining and Survey Study
This paper presents a mixed-methods study mining 7,310 rules from 83 open-source projects and surveying 99 practitioners to establish a taxonomy for AI IDE rules, revealing a gap between developer priorities and actual configurations while demonstrating that rule evolution significantly improves software artifact compliance.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you've hired a super-smart, incredibly fast, but slightly chaotic personal assistant to help you build a house. This assistant is an AI, and it's great at laying bricks and painting walls. But sometimes, it gets confused. It might decide to build a door where a window should go, or it might use a type of wood you specifically told it not to use.
To fix this, you give the assistant a Rulebook. This isn't just a one-time note; it's a living document that stays with the project forever. Every time the assistant starts working, it reads this Rulebook to remember exactly how your house should look.
This paper is a deep dive into how real-world developers use these "Rulebooks" (called Rules in AI IDEs) and how they change them over time. The researchers looked at 83 real projects and talked to 99 developers to see what's actually happening.
Here is what they found, broken down simply:
1. The "Rulebook" Taxonomy (What's in the book?)
The researchers sorted thousands of rules into a giant filing cabinet with 5 main drawers and 25 smaller folders:
- Architecture & Design: The big picture (e.g., "Use this specific blueprint style").
- Code Implementation: The nitty-gritty (e.g., "Write code like this," "Don't use classes").
- Workflow & Management: How the team works (e.g., "Always run tests before saving").
- Quality Assurance: Safety checks (e.g., "Test everything," "No security holes").
- AI Collaboration: How the AI should behave (e.g., "Be concise," "Don't guess").
The Big Surprise:
There was a huge disconnect between what developers said was important and what they actually wrote down.
- What they value: Developers told the survey, "The most important thing is the Architecture (the big picture) and Context (how the AI understands the project)."
- What they actually wrote: The Rulebooks in the real projects were mostly filled with low-level details like "Use this font," "Put files in this folder," and "Run this test first."
- The Analogy: It's like hiring an architect to design a skyscraper, but spending 90% of your time writing them a memo about which brand of coffee to drink and how to organize the stapler, while barely mentioning the structural steel.
2. How the Rules Evolve (How the book changes)
Rules aren't static; they get updated constantly. The researchers watched 1,540 changes happen.
- The Main Action: Most of the time, developers are adding new rules. They aren't usually rewriting old ones; they are just tacking on new instructions.
- The "Why" Gap:
- What the data says: When looking at the actual code changes, developers mostly added rules to expand the project (adding new features) or enrich the context (giving the AI more background info).
- What developers say: When asked in the survey, developers said they mostly change rules to fix mistakes the AI made.
- The Analogy: It's like a parent saying, "I mostly add new chores to the list to help the kids learn new skills," but the kids say, "We only add chores when we mess up the dishes." The reality is constructive growth, but the feeling is just damage control.
- The "Correction" Habit: When developers do try to fix an AI mistake, they rarely edit the old rule. Instead, they add a new rule that says, "Do NOT do X." It's like putting a "Do Not Enter" sign next to a door instead of repainting the door.
3. Does it actually work? (The Compliance Check)
The researchers wanted to know: If you update the Rulebook, does the AI actually follow the new instructions better?
- The Result: Yes, significantly.
- The Numbers: Before a rule update, the AI followed the instructions about 49% of the time. Immediately after the update, that jumped to 72%. That's a 23% improvement.
- The Catch: This works best for concrete, easy-to-check things (like "File names must end in .ts"). It works much less well for vague, high-level ideas (like "Follow good architectural principles").
- The Analogy: If you tell the assistant, "Always wear a red hat," they will do it perfectly once you remind them. But if you say, "Be a good leader," they might still get confused. The Rulebook is great for specific commands, but less effective for abstract philosophy.
Summary of the Study's Takeaways
- Developers care about the big picture, but write about the small stuff. They value architecture but spend their time fixing formatting and workflow details.
- The "Add, Don't Edit" Strategy. Developers prefer to pile on new rules to fix problems rather than cleaning up the old ones. This makes the Rulebooks get long and messy over time.
- Updates work, but only for specific things. Changing the rules makes the AI follow instructions much better, but only if those instructions are clear and concrete.
- The "Negative Constraint" Problem. Developers often fix AI errors by adding "Don't do this" rules. This is a quick fix, but it can make the Rulebook cluttered and confusing in the long run.
In short, AI IDEs are powerful, but the "Rulebooks" we use to control them are currently a bit messy. We are using them to solve immediate, small problems rather than guiding the big, complex design, and we are building them up piece-by-piece rather than keeping them clean and organized.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.