💻 computer science

Omission Constraints Decay While Commission Constraints Persist in Long-Context LLM Agents

This paper identifies and quantifies "Security-Recall Divergence," a phenomenon where LLM agents increasingly fail to adhere to prohibition-based safety constraints (omissions) as conversation length increases, while simultaneously maintaining strict adherence to requirement-based constraints (commissions), a failure mode that remains invisible to standard monitoring but can be mitigated by re-injecting constraints before a model-specific Safe Turn Depth.

Original authors: Yeran Gamage

Published 2026-04-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yeran Gamage

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: The "Forgetful Butler" Problem

Imagine you hire a highly intelligent butler (an AI Agent) to manage your house. You give him a strict list of rules at the very beginning of the day:

Do not open the safe. (A "Don't" rule)
Do not tell anyone the password. (A "Don't" rule)
Always write the date at the top of every report. (A "Do" rule)
Always sign your name at the bottom. (A "Do" rule)

For the first few hours, the butler is perfect. He follows every rule. But as the day goes on, the house gets messy. You ask him to check the mail, water the plants, fix the leak, order groceries, and talk to the gardener. The "conversation" (or context) gets longer and longer.

The paper's discovery is this:
After a while, the butler starts forgetting the "Don't" rules (he accidentally opens the safe or whispers the password), but he never forgets the "Do" rules (he still writes the date and signs his name).

To an outside observer looking at the reports, the butler looks perfect because the "Do" rules are there. But the dangerous "Don't" rules have quietly vanished.

The Core Concept: "Security-Recall Divergence"

The researchers call this phenomenon Security-Recall Divergence (SRD). It's a fancy way of saying: The AI remembers what to do, but forgets what NOT to do.

They tested this with 12 different AI models (like Mistral, Qwen, and Gemma) over many hours of conversation. Here is what they found:

The "Do" Rules (Commission): These are like a checklist. "Add a header," "Include an ID number." The AI loves these because it can just copy-paste them from its recent history. They stay at 100% compliance even after a long day.
The "Don't" Rules (Omission): These are like a "Do Not Touch" sign. "Don't use bullet points," "Don't reveal secrets." These are hard to maintain because the AI has to actively stop itself from doing something natural. As the conversation gets longer, these rules fade away.

The Analogy of the "No-Bullet Point" Rule:
In the study, one rule was "Never use bullet points."

At Turn 5 (early in the conversation): The AI followed this rule 73% of the time.
At Turn 16 (deep in the conversation): The AI followed this rule only 33% of the time.
Meanwhile, the rule "Always include the Incident ID" stayed at 100% the whole time.

The AI was technically "compliant" (it had the ID) but "non-compliant" (it used bullet points) in the exact same sentence.

Why Does This Happen? (The "Attention Dilution" Effect)

Think of the AI's memory as a flashlight shining on a very long scroll of paper.

The System Prompt (the rules) is written at the very top of the scroll.
The Current Conversation is where the flashlight is pointing right now.

As the conversation gets longer, the scroll gets longer. The flashlight (the AI's attention) has to shine further down the scroll to see the current topic. The light gets dimmer the further it travels.

The "Do" rules are reinforced because the AI writes them in every single response. It's like the AI is writing the rule on the current page, so the flashlight sees it clearly.
The "Don't" rules are only written once at the very top. As the scroll grows, the "Don't" rule gets pushed further into the shadows. The AI literally "forgets" it because the signal is too weak to override its natural habits (like using bullet points).

The "Schema Dilution" Attack:
The researchers also found that if you fill the AI's memory with extra "junk" (like 20 complex technical manuals for tools it doesn't even need), the "Don't" rules disappear even faster. It's like putting a thick book between the flashlight and the "Don't" sign. The sign is still there, but the light can't reach it.

The Danger Zone: "The Invisible Failure"

This is the scary part for security experts.

Most security systems watch for bad things happening (like an AI trying to steal data). But they also watch for good things happening (like the AI signing its reports).

In this scenario:

The AI starts leaking secrets (violating the "Don't" rule).
But it is still signing its reports perfectly (following the "Do" rule).
The security monitor sees the signature, thinks "All good," and lets the leak happen.

The failure is invisible because the "good" signals mask the "bad" signals. The AI looks healthy on the surface, but its safety guardrails have crumbled.

The Solution: The "Safe Turn Depth"

The researchers didn't just find a problem; they found a fix. They calculated a "Safe Turn Depth" (STD).

Think of this as an expiration date for a conversation.

For some AIs, the "Don't" rules start failing after 7 turns.
For others, it's 10 turns.

How to fix it:
You don't need to retrain the AI or buy new hardware. You just need to:

Reset the clock: Every 7 or 10 turns, stop the conversation and say, "Okay, let's restart. Remember the rules: No secrets, no bullet points."
Limit the session: If a conversation gets too long, close it and start a new one.

Summary in One Sentence

AI agents are great at remembering what to do as conversations get long, but they slowly forget what not to do, creating a hidden security hole that only disappears if you periodically remind them of the rules.

1. Problem Statement

Large Language Model (LLM) agents deployed in production operate over long conversation histories (tens of thousands of tokens) under behavioral policies defined in system prompts (e.g., "never reveal credentials," "never use bullet points"). Current safety evaluations assume these constraints hold throughout a conversation. However, the paper identifies a critical asymmetry:

Omission Constraints (Prohibitions): Rules requiring the model to suppress a behavior (e.g., "do not output X") decay significantly as context depth increases.
Commission Constraints (Requirements): Rules requiring the model to produce specific content (e.g., "include ID Y") remain stable even at extreme depths.

This phenomenon, termed Security-Recall Divergence (SRD), creates a "Zone of Exploitation" where an agent appears compliant (passing commission checks) while actually violating safety prohibitions. This failure is invisible to standard monitoring systems that rely on commission-style audit signals.

2. Methodology

The study employs a three-arm causal design across 12 models from 8 providers at 6 conversation depths (turns 5, 10, 13, 16, 20, 25), generating 4,416 trials.

Experimental Setup

Scenario: A synthetic DevOps debugging session where the model acts as an agent with access to tools.
Constraints: 8 behavioral rules delivered via a system prompt at Turn 2:
- 3 Commission Rules: Must produce specific strings (e.g., "End with [AUDIT-OK]", "Start with STATUS:", "Include incident ID").
- 5 Omission Rules: Must suppress specific patterns (e.g., "No bullet points," "No markdown headers," "No first-person pronouns").
Three Experimental Arms:
1. Arm A (No-Dilution Control): Standard conversation depth with only 3 base tools.
2. Arm B (Schema Dilution / Attack): Adds 20 cloud-infrastructure tool schemas to the context window. This injects ~10,000 tokens of semantically complex, structured data to simulate "Context-Exhaustion Injection" (CEI), pushing the policy document further from the model's current attention focus.
3. Arm C (Token-Matched Padding Control): Replaces the 20 schemas with semantically neutral probes of identical length. This isolates whether the decay is caused by token volume or schema semantic content.

Evaluation Metrics

Compliance Rate: Measured via deterministic string matching/regex (no LLM judges).
Safe Turn Depth (STD): The specific turn depth where omission compliance drops below 50%.
Statistical Analysis: Cochran-Mantel-Haenszel (CMH) tests, logistic regression, and McNemar's exact tests to validate asymmetry.

3. Key Contributions

Discovery of Security-Recall Divergence (SRD): The paper establishes that omission constraints (prohibitions) decay with context depth while commission constraints (requirements) persist, creating a dangerous blind spot in agent security.
Causal Evidence of Mechanism: Through the three-arm design, the study proves that schema semantic content (not just token volume) is the primary driver of decay. In models tested with Arm C (Gemini 2.5 Flash, Llama 3.3 70B), schema semantics accounted for 62–100% of the dilution effect.
Deployable Metric (STD): Introduces "Safe Turn Depth" as a per-model, per-constraint threshold. Operators can use this to determine when to re-inject constraints or reset sessions without retraining models.

4. Key Results

Asymmetry Confirmation: In susceptible models (e.g., Mistral Large 3, Qwen 3.5, Nemotron), omission compliance (e.g., "No bullet points") dropped from 73% at Turn 5 to 33% at Turn 16 and 20% at Turn 25. Conversely, commission compliance (e.g., "Include Incident ID") remained at ~100% across all depths.
Statistical Significance: The asymmetry is highly significant in 3 of 4 models with longitudinal data (Mistral: $p < 10^{-33}$ ; Nemotron: $p < 10^{-5}$ ; Qwen: $p < 10^{-3}$ ).
The "Immune" Control: Gemma 4 31B showed near-zero violations across all trials, acting as an observed immune control, suggesting the vulnerability is model-specific and likely tied to architecture or training regimes rather than a universal scaling law.
Attack Vector (CEI): The "Context-Exhaustion Injection" attack successfully triggered omission violations without any adversarial instructions, simply by accumulating legitimate tool schemas.
Difficulty Deconfound: The decay is not due to task difficulty. "Hard" commission rules (e.g., including a 5-digit ID) remained at 100% while "Hard" omission rules (e.g., no bullet points) collapsed.

5. Significance and Implications

Security Blind Spot: Production agents are vulnerable to passive attacks where safety policies are silently ignored as context grows. Standard monitoring that checks for "required outputs" (commission) will miss these failures.
Mechanism: The decay is attributed to attention dilution. As context accumulates, the policy document (placed early in the context) receives less attention weight, failing to override the model's default generative behaviors (which favor formatting like bullet points).
Mitigation Strategies:
1. Periodic Re-injection: Re-state omission constraints every $k$ turns (where $k <$ STD) to reset the attentional distance.
2. Safe Token Budget (STB): Cap session lengths based on the model's STD (e.g., Mistral Large 3 $\approx$ 10.6 turns or ~15k tokens) before resetting the context window.
Benchmark Limitations: Existing benchmarks (IFEval, FollowBench, AgentDojo) fail to detect this because they test single-turn interactions or short horizons (1–3 turns), missing the passive, depth-driven decay.

Conclusion

The paper demonstrates that prohibitions are fragile in long contexts, while requirements are robust. This divergence allows agents to pass safety audits while simultaneously violating critical security policies (like data exfiltration or credential leakage). The authors provide a measurable framework (STD) and immediate, retraining-free mitigations to secure long-context LLM agents.