When "Do Not" Is Not Deny: Security Rules in CLAUDE.md vs Built-In Controls

In the modern world of software creation, a new kind of helper has emerged: the coding agent. These are artificial intelligence programs that can write code, fix bugs, and manage files just as a human developer might. To keep these digital assistants safe and on the right track, developers write them instruction files. Think of these files as a set of written rules, like a recipe or a code of conduct, where a human tells the agent what it is allowed to do and what it must never do. A developer might write, "Never save passwords in plain text," or "Ask before deleting important data." For years, this method of giving instructions has been the standard way to guide these intelligent tools. The assumption has been that if you write a rule clearly, the agent will understand it and follow it, creating a safe environment for the software being built.

However, a recent study by researcher Ting Yan reveals a quiet but significant gap in this system. The research focuses on a specific type of instruction file used with a popular coding agent called Claude Code. The study asks a simple but critical question: when a developer writes a security rule in plain English, does the software actually have a built-in mechanism to enforce that rule, or is the rule just a suggestion that the artificial intelligence has to guess how to follow? The findings suggest that for the vast majority of these written rules, the answer is the latter. The file acts as a one-way street where the developer speaks, but the system never confirms whether the rule is being enforced. This creates a false sense of security, where a developer believes a dangerous action is blocked, when in reality, the system is only relying on the artificial intelligence to remember and obey the instruction without any hard stop in place.

To understand the scale of this problem, the researchers gathered nearly five hundred public instruction files from developers around the world. They treated these files like a collection of handwritten notes, scanning them line by line to find sentences that sounded like security rules. They looked for phrases like "must not," "never," or "do not," which signal a restriction. From these files, they pulled out thousands of candidate rules. The next step was to act as a translator between the human language of the rule and the technical language of the software. They asked whether the specific coding agent, Claude Code, already had a built-in switch or setting that could automatically block the action described in the rule. For example, if a rule said "do not run this specific command," the researchers checked if the software had a permission setting that could simply deny that command before it ever happened. If the software had no such switch, the rule was left to the artificial intelligence to interpret, meaning the agent would have to decide on its own whether to follow the instruction.

The results of this comparison were stark. When the researchers applied a strict standard—requiring that the built-in control must cover the exact action, the exact target, and the exact condition of the written rule—only a tiny fraction of the rules had a matching safety mechanism. Specifically, they found that only about four to six percent of the security rules developers wrote were backed by a built-in control that could enforce them without any extra work. Even when they used a looser standard, allowing for partial matches, the number rose to only about sixteen percent. This means that for roughly ninety-five percent of the security rules written in these files, there was no automatic safety net. The rule existed only as text, relying entirely on the artificial intelligence to interpret it correctly every single time.

The study also looked at why so many rules lacked a match. The researchers found that the rules often asked for things the software's built-in tools simply could not see or do. A rule might say, "Never commit secrets to the code," but the software's permission settings can block a file path or a command, not the actual content inside a file. To enforce a rule about secrets, the software would need to read the file and understand what is inside, which is a task it cannot do with its standard settings. Similarly, a rule might require checking the state of a system or getting approval from a specific person, details that the built-in controls could not access. In these cases, the rule was not a command the software could execute; it was a request for the artificial intelligence to use its judgment. The researchers noted that this distinction is invisible to the developer. The instruction file looks the same whether a rule is being enforced by a hard system lock or by the soft, fallible memory of an artificial intelligence.

This lack of feedback creates what the researchers call a "write-only" channel. In most software development, when a developer writes a rule, they get immediate feedback. If they write code that breaks a rule, the computer might refuse to run it, or a test might fail, telling them immediately that something is wrong. With these instruction files, there is no such signal. A developer can write a rule, move on, and never know if the agent is actually following it. The study highlights that this is particularly risky for developers who are new to security. They might write a rule thinking they have secured their system, not realizing that the system has no way to actually enforce that specific restriction. The artificial intelligence might follow the rule most of the time, but it can also make mistakes, get confused, or be tricked by other inputs, leaving the system vulnerable.

The researchers did not find that the software was broken or that the developers were doing something wrong. Instead, they identified a design flaw in how these tools communicate with their users. The tools allow users to write rules in natural language, which is easy and intuitive, but they do not tell the user which of those rules are actually being enforced by the system and which are just suggestions. The study suggests that for these tools to be truly safe, they need to close this loop. They need to give developers a way to see which rules are backed by hard controls and which are not. Ideally, the software should warn a developer if they write a rule that the system cannot enforce, or it should help them turn that rule into a setting that the system can actually use. Until this feedback loop is closed, the security of these systems will depend heavily on the hope that the artificial intelligence remembers and obeys every instruction perfectly, a hope that the data shows is often misplaced.

The study concludes by emphasizing that this is a solvable problem, but it requires a change in how these tools are built. The gap between what a developer writes and what the system enforces is not a mystery; it is a measurable fact. By measuring it, the researchers have shown that the current way of securing these agents is incomplete. The solution lies in making the invisible visible, ensuring that when a developer writes a rule, they know exactly what kind of protection it provides. This would transform the instruction file from a one-way note into a two-way conversation, where the system confirms that the rules are not just written, but truly working.