Context-Augmented Code Generation: How Product Context Improves AI Coding Agent Decision Compliance by 49%
This paper introduces a controlled benchmark demonstrating that augmenting AI coding agents with a product-context retrieval system (Brief) improves their compliance with team-specific product decisions by 49 percentage points compared to codebase-only access, achieving 95% compliance on realistic software engineering tasks.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you hire a brilliant, super-fast apprentice programmer to build a new feature for your company's website. This apprentice (the AI) can read every line of code you've ever written, understand how your computer works, and write new code that actually runs without crashing.
But here's the problem: The apprentice doesn't know your company's unwritten rules.
In a real office, teams have "secret handshakes" for coding. Maybe there's a rule that says, "We never use that specific calendar widget anymore," or "Every time we export data, we must log it for security audits," or "We always put our tests right next to the code we're testing." These rules aren't written in the code itself; they live in meeting notes, design documents, or the collective memory of the team.
Without this context, the brilliant apprentice might build a working feature that technically compiles but violates your team's most important rules. It's like building a beautiful house with the wrong type of bricks because the architect didn't know the city had a specific zoning law.
The Experiment: Two Approaches
The authors of this paper set up a test to see if giving the AI access to these "unwritten rules" helps. They created a simulated office environment (a code repository) with 8 different tasks, like "add a dark mode toggle" or "create a search bar." Hidden inside these tasks were 41 specific "traps" or rules that the AI needed to follow to get a perfect score.
They tested two versions of the AI:
- The "Code-Only" Apprentice (Baseline): This AI was given the code files and told, "Build this." It had to guess the rules by looking at existing code patterns.
- The "Context-Aware" Apprentice (Augmented): This AI had the same code files, but it also had a "Company Handbook" (called Brief). Before writing a single line of code, this AI could ask questions like, "What are our design rules?" or "What do our customers complain about?" It also had a step-by-step plan (a spec) written out before it started building.
The Results: A Massive Leap
The difference was dramatic.
- The Code-Only Apprentice followed the rules only 46% of the time. It often missed the invisible rules. For example, in one task, it looked at the code, saw a helper function for pagination, and decided, "Oh, this is already done!" and wrote zero code. It missed the fact that the current pagination was broken and needed a specific fix.
- The Context-Aware Apprentice followed the rules 95% of the time. By checking the "handbook," it knew exactly which widgets to use, which security logs to add, and how to structure the code.
The Big Win: The context-aware version improved compliance by 49 percentage points.
Why It Happened: The "Invisible" vs. "Visible" Rules
The paper found a clear pattern:
- Visible Rules: If a rule was written in a comment inside the code (e.g., "Use this function for security"), both apprentices found it and followed it 100% of the time.
- Invisible Rules: If a rule was only in a product document (e.g., "We use PostHog for feature flags, but it's not in the code"), the Code-Only apprentice got it 0% of the time. The Context-Aware apprentice got it 100% of the time because it looked up the rule before starting.
It's Not Just About "Knowing" the Rules
The paper also notes that the Context-Aware AI didn't just "look up" the rules; it also worked differently. It created a detailed plan (a spec) with a checklist of requirements before coding. It also had a "mid-build consultation," meaning it could pause and ask, "Am I doing this right?" while working.
The authors admit they can't perfectly separate how much of the success came from having the information versus having a better workflow. However, the data strongly suggests that having the right information (product context) is the key to solving the problems that the other AI couldn't see.
The Cost of Being "Right"
Interestingly, the Context-Aware AI cost a little more to run (about 28% more in API fees) because it spent time reading the handbook and writing plans. However, because it produced code that was ready to be merged into the main project immediately, the cost per successful task actually dropped by 68%.
The Code-Only AI was cheaper per attempt, but it produced code that humans had to fix and rewrite, making it more expensive in the long run. The Context-Aware AI wrote more code, wrote tests, and avoided deprecated patterns, meaning it was "merge-ready" 100% of the time.
The Bottom Line
This paper is a proof-of-concept. It shows that for AI coding agents to be truly useful in a professional team, they can't just read the code. They need access to the organizational knowledge—the decisions, the customer complaints, the design preferences, and the security rules that live outside the code files.
When you give the AI the "whole picture," it stops guessing and starts building exactly what the team actually needs.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.