Configuring Agentic AI Coding Tools: An Exploratory Study
This paper presents a systematic analysis and empirical study of configuration mechanisms for agentic AI coding tools across 2,923 GitHub repositories, revealing that while static context files like AGENTS.md dominate adoption, advanced features such as Skills and Subagents remain underutilized, leading to distinct tool-specific configuration cultures.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you've just hired a team of incredibly smart, super-fast robots to help you build a house. These robots are your Agentic AI coding tools (like Claude Code, GitHub Copilot, or Cursor). They can lay bricks, paint walls, and even design blueprints on their own.
But here's the catch: These robots are very literal. If you don't tell them exactly how your house should look, they might build a castle when you wanted a cottage, or use red bricks when you wanted blue.
This paper is a massive study of how humans are currently teaching these robots the rules of the house. The researchers looked at nearly 3,000 software projects (like digital construction sites) to see how developers are giving instructions to their AI helpers.
Here is the breakdown of their findings, using simple analogies:
1. The "Rulebook" vs. The "Toolbox"
The researchers found that developers have two main ways to talk to these robots:
- The Rulebook (Context Files): This is like a sticky note or a README file left on the desk. It says things like, "We always paint the kitchen yellow," or "Always use this specific type of screw."
- The Finding: This is by far the most popular method. Almost every project has one. It's the "Hello, World" of AI configuration.
- The Toolbox (Advanced Mechanisms): This includes things like Skills (pre-written scripts the robot can run) or Subagents (hiring a specialized robot to do a specific job, like just the plumbing).
- The Finding: These are powerful, but people are barely using them. It's like having a high-tech power drill in your toolbox but only using it to hammer in nails. Most people stick to the simple sticky notes.
2. The "Universal Translator" (AGENTS.md)
In the past, every robot brand (Claude, Copilot, Cursor) spoke a slightly different language. If you wanted to hire a Claude robot, you had to write a note called CLAUDE.md. If you wanted a Copilot, you wrote copilot-instructions.md.
- The Trend: The study found a new standard emerging called
AGENTS.md. - The Analogy: Think of this as a "Universal Translator" or a "Master Key." Developers are starting to write their instructions in this one file because it works with almost every robot brand. It's becoming the "lingua franca" of the AI world. Even if you don't use a specific tool yet, you might start with
AGENTS.mdjust to be ready.
3. The "Shallow Dive" Problem
The researchers noticed that while developers are writing these rulebooks, they aren't going very deep.
- Static vs. Dynamic: Most "Skills" (which could be complex, interactive scripts) are just static text documents.
- The Analogy: Imagine you hire a robot chef. You give it a recipe book (the Skill). But instead of giving it a list of ingredients and a step-by-step cooking algorithm it can execute, you just paste a Wikipedia article about "How to make a cake" into the book. The robot reads it, but it can't actually do the cooking automatically.
- The Result: People are using these advanced tools mostly for documentation (reading) rather than automation (doing).
4. Different Cultures for Different Robots
Just like different companies have different office cultures, different AI tools are developing their own "configuration cultures."
- Claude Code users are the most adventurous, using a wide variety of tools and methods.
- Cursor users love "Rules" (strict constraints on how code is written).
- Copilot users mostly stick to the basics (just the Rulebook).
5. Why Does This Matter?
The study concludes with a few key takeaways for anyone building software:
- Start Simple: If you are new to AI coding tools, just write an
AGENTS.mdfile. It's the easiest way to get started and works everywhere. - Don't Overcomplicate (Yet): You don't need to build complex sub-robots or executable scripts right away. The simple "Rulebook" is doing most of the heavy lifting.
- The Future: As these tools get smarter, we might see more people using the "Toolbox" (Skills and Subagents) to automate complex tasks, but right now, we are mostly just reading the instructions.
In a nutshell: We are in the early days of the AI revolution in coding. Everyone is currently writing simple sticky notes (AGENTS.md) to tell the robots what to do. We have fancy, powerful tools available to automate complex tasks, but most people haven't figured out how to use them yet. The industry is slowly agreeing on a common language, but we are still learning how to truly harness the power of these digital assistants.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.