ContextCov: Deriving and Enforcing Executable Constraints from Agent Instruction Files

1. Problem Statement: The Reality Gap and Context Drift

As Large Language Model (LLM) agents evolve from supervised assistants to autonomous software engineers, they rely on Agent Instruction files (e.g., AGENTS.md, copilot-instructions.md) to understand project-specific conventions, tools, and architectural boundaries. However, these instructions are currently passive text.

The paper identifies a critical phenomenon called Context Drift, where agents deviate from these instructions due to:

• Context Limitations: Instructions get lost in large context windows.
• Conflicting Inputs: Agents prioritize immediate context (e.g., legacy code patterns) over system directives.
• Capability Limitations: Agents fail to execute complex constraints without explicit guidance.

This creates a "Reality Gap" between what agents are told to do and what they actually do. Without real-time human supervision, these silent violations accumulate as technical debt, leading to style inconsistencies, architectural erosion, and workflow deviations.

2. Methodology: The ContextCov Framework

ContextCov is a framework designed to transform passive Agent Instructions into active, executable guardrails. It operates in two main phases: Check Generation and Runtime Enforcement.

A. Check Generation (Extraction & Synthesis)

1. Hierarchical Constraint Extraction:

   • The system parses Agent Instruction files into a Markdown Abstract Syntax Tree (AST).
   • It employs a Path-Aware Slicing algorithm to preserve the hierarchical context of constraints (e.g., distinguishing between "Backend Testing" and "Database" rules).
   • An LLM refines these slices into standalone, unambiguous natural language constraints.
   • Stable Identifiers are assigned to constraints to support incremental updates; only modified sections are re-processed.

2. Domain-Routed Policy Synthesis:

   • An Intent Router (LLM-based) classifies each constraint into one of four enforcement domains:
     • Process: Operational constraints (e.g., "Use pnpm, not npm").
     • Source: Coding style and patterns (e.g., "Use arrow functions").
     • Architectural (Deterministic): Structural rules verifiable via graph analysis (e.g., "No circular dependencies").
     • Architectural (Semantic): Intent-based rules requiring judgment (e.g., "Do not access another component's storage keys").
   • Specialized Code Generators synthesize executable Python checks for each domain:
     • Process: Generates shell wrapper logic.
     • Source: Generates Tree-sitter AST queries.
     • Architectural (Det): Generates NetworkX graph algorithms.
     • Architectural (Sem): Preserves natural language prompts for LLM-based evaluation.

B. Runtime Enforcement

ContextCov executes these checks through four specialized modules:

1. Process Interceptor: Uses $PATH manipulation to inject shell shims. Before a command (e.g., npm run compile) executes, the shim intercepts it, runs the generated Python check, and blocks the command if a violation is detected.
2. Universal Static Linter: Uses Tree-sitter to parse source code across multiple languages (Python, TS/JS, Go, Rust) and execute AST-based queries to detect style violations.
3. Architectural Validator: Constructs dependency graphs from import statements and runs graph algorithms to detect layer violations, cycles, or incorrect module placement.
4. Semantic Architectural Validator: Uses an LLM-as-a-Judge approach to evaluate code changes against semantic design principles. These checks produce warnings (not hard blocks) to flag potential violations for human review.

Design Philosophy: The system adopts a "fail-closed" approach. Ambiguous constraints are interpreted strictly to prevent violations, accepting that false positives can be manually overridden by developers, whereas false negatives allow technical debt to accumulate.

3. Key Contributions

1. Taxonomy of Context Drift: The paper characterizes the "Reality Gap" and categorizes violations into Process, Source, and Architectural (Deterministic/Semantic) types based on an analysis of real-world Agent Instruction files.
2. Hierarchical Extraction Algorithm: A novel path-aware parsing method that extracts constraints from Markdown while preserving their semantic scope, solving the issue of context loss in instruction files.
3. ContextCov Architecture: A modular system that synthesizes domain-specific executable checks (AST, Graph, Shell, LLM) from natural language instructions.
4. Large-Scale Evaluation: Comprehensive testing on 723 open-source repositories demonstrating the system's ability to extract and enforce constraints at scale.

4. Evaluation Results

The framework was evaluated on 723 repositories (median 2,010 stars) containing Agent Instruction files.

• Extraction Quality:
  • Successfully extracted and synthesized 46,316 executable checks.
  • 99.997% syntax validity (34,373 out of 34,374 checks passed syntax validation).
  • Distribution of checks: Source (28%), Process (26%), Architectural Semantic (26%), Architectural Deterministic (20%).
• Violation Detection:
  • Detected 10,927 violations (24% of all checks) across the repositories.
  • Total violations found: >500,000.
  • 81% of repositories contained at least one violation.
  • Long-tail distribution: A small number of specific constraints (4% of checks) accounted for 80% of the violations, indicating systematic drift in specific areas.

5. Significance and Implications

• Documentation as Code: ContextCov shifts the paradigm of project documentation from descriptive text to executable specifications. It treats Agent Instructions as a "living contract" that must co-evolve with the codebase.
• Agentic Feedback Loop: By blocking actions and providing immediate feedback, the system forces agents to reason about failures. This encourages agents to propose fixes to the instructions themselves, turning them from passive consumers into active maintainers of project specs.
• Executable Interpretability: The framework materializes the agent's understanding of natural language prompts into deterministic Python code (AST queries, graph algorithms). This allows developers to audit exactly how a prompt was interpreted, acting as a "prompt debugger."
• Guardrails vs. Sandboxes: Unlike security sandboxes that prevent malicious actors, ContextCov is designed to prevent well-intentioned but hallucinating agents from degrading code quality. It bounds the agent's action space to approved architectural and operational limits.

In conclusion, ContextCov provides a necessary automated compliance layer for the era of autonomous software engineering, bridging the gap between human intent and agent execution to prevent the silent accumulation of technical debt.