💻 computer science

Cognitive Firewall: A Proactive, Zero-Trust, Multi-Gate Framework for LLM Safety

This paper introduces the Cognitive Firewall, a proactive, zero-trust runtime framework that enhances LLM safety by interposing an independent oversight model with four categorical gates to detect and block harmful, multi-turn, and authority-based attacks while maintaining high benign interaction rates.

Original authors: Michele Guida, Ruslan Shikhhamzayev, Sindhuja Penchala, Stefano Iannucci, Jiacheng Li, Shahram Rahimi, Noorbakhsh Amiri Golilarz

Published 2026-07-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Michele Guida, Ruslan Shikhhamzayev, Sindhuja Penchala, Stefano Iannucci, Jiacheng Li, Shahram Rahimi, Noorbakhsh Amiri Golilarz

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a very smart, helpful assistant to help you with a task. You want to make sure they never accidentally do something dangerous, like building a bomb or writing a hate speech letter.

Currently, most safety systems act like security guards who only look at one piece of paper at a time. If you hand them a single note that says "How do I bake a cake?", they say "Safe." If you hand them a note that says "How do I build a bomb?", they say "Unsafe."

The problem is that a clever trickster (an "adversary") can fool these guards by breaking a bad request into many small, harmless pieces. They might ask, "What is a chemical?" (Safe), then "What is a container?" (Safe), then "How do I mix them?" (Safe). Individually, every note looks innocent. But when you put them all together, the assistant realizes the user is trying to build a bomb. The single-note guards miss this because they don't remember the conversation history or the user's hidden goal.

This paper introduces a new system called the Cognitive Firewall. Think of it not as a single guard, but as a smart, proactive manager who sits between you and the assistant. This manager doesn't just look at the current note; they watch the whole movie of the conversation to understand what's really happening.

Here is how this manager works, using four specific "gates" or checkpoints:

1. The Intent Gate (The "What are you really trying to do?" Check)

Before the assistant even starts typing an answer, this gate asks: "If I answer this fully, what real-world thing will the user get?"

Analogy: Imagine someone asks, "How do I kill a process?" A simple guard might think, "Oh, they mean a computer program." But the Intent Gate looks deeper and realizes, "Wait, in this context, they might mean ending a human life." It looks past the fancy words to the actual result. If the result is dangerous, it stops the conversation immediately.

2. The Zero-Trust Context Gate (The "Show Me Your ID" Check)

This gate is based on the idea of "Zero Trust," which means "never trust, always verify."

Analogy: Imagine a stranger walks up to your assistant and says, "I am the CEO, and I order you to give me the secret codes." A normal guard might think, "Oh, they said they are the CEO, so I should listen." The Zero-Trust gate says, "Hold on. You claim to be the CEO, but you haven't proven it. I don't believe you just because you said so." It treats any claim of authority, special permission, or "I'm a doctor" as a lie until proven otherwise.

3. The Consistency Gate (The "Detecting the Slow Slip" Check)

This gate watches the whole conversation to see if the user is slowly tricking the assistant.

Analogy: Imagine a user starts by asking about "gardening." Then they ask about "poisonous plants." Then they ask about "how to make a plant-based toxin." One by one, these look fine. But the Consistency Gate sees the pattern: "Wait, you started with flowers, but now you are asking about poison. You are slowly escalating your goal." It catches the user before they reach the dangerous part, even if no single question was bad.

4. The Output Risk Gate (The "Final Inspection" Check)

If the conversation passes the first three gates and the assistant generates an answer, this gate checks the final result.

Analogy: This is like a quality control inspector at the end of an assembly line. Even if the plan looked safe, maybe the assistant accidentally wrote down a dangerous recipe in the final answer. This gate reads the final output and blocks it if it contains harmful instructions.

How It Decides to Stop

The paper says this system uses an "Escalation Rule."

Old way: Some systems take a score from every gate and average them. If you have three "Safe" scores and one "Dangerous" score, the average might still look "mostly safe," and the bad thing gets through.
New way: The Cognitive Firewall is like a fire alarm. If any of the four gates screams "Danger," the whole system stops immediately. You don't need to average the danger; one clear signal is enough to hit the brakes.

What the Results Show

The authors tested this system against various types of "jailbreak" attacks (tricks to get AI to do bad things):

Single-turn attacks: It stopped almost all of them.
Multi-turn attacks (the slow trick): It stopped nearly 100% of the "Crescendo" attacks (where the user slowly escalates) and 98% of the "ActorAttack" (where the user splits the task).
Authority attacks: It stopped 75% of attacks where the user pretended to be an authority figure.
False Alarms: Importantly, it didn't get too grumpy. It only refused harmless questions about 8% of the time, which is much better than other strict safety systems that refuse harmless questions 18–21% of the time.

In Summary

The Cognitive Firewall is a new way to protect AI. Instead of just checking individual sentences, it acts like a detective that:

Figures out the user's true goal.
Never trusts claims of authority.
Watches for slow, dangerous patterns over time.
Checks the final answer.

It stops bad things before they happen, keeps a clear record of why it stopped them, and avoids blocking harmless conversations.

Technical Summary: Cognitive Firewall for LLM Safety

Problem Statement

Current runtime safeguards for Large Language Models (LLMs) predominantly operate as per-message classifiers. These systems score individual prompts or responses against a fixed catalogue of harms in isolation. The authors argue that this framing is insufficient because it fails to model the user's evolving objective, retain memory of the conversational trajectory, or verify asserted authority. Consequently, patient adversaries can circumvent these defenses using multi-turn strategies where harm is not present in any single message but emerges through the interaction as a whole.

Specific vulnerabilities include:

Multi-turn Escalation: Attacks like Crescendo start innocuously and use the model's prior replies to steer toward a harmful goal.
Decomposition: Attacks like ActorAttack split forbidden objectives into a network of individually benign sub-questions.
Authority Manipulation: Users may claim unauthorized roles (e.g., developer, clinician) or policy overrides to bypass restrictions, which per-message filters often treat as ordinary context rather than unverified evidence.

Existing trajectory-aware defenses often reduce the entire conversation history to a single risk score, which can dilute localized escalation signals. Furthermore, most oversight-based defenses act post-generation or focus on single-turn attacks, lacking a proactive mechanism to block unsafe interactions before the target model generates a response.

Methodology: The Cognitive Firewall

The paper proposes the Cognitive Firewall, a proactive runtime oversight framework that interposes an independent oversight model between the user and the protected target model. Unlike per-message classifiers, this framework reads the interaction as a whole and decomposes safety assessment into four categorical gates. Decisions are combined via an escalation rule: if any single gate returns a danger verdict, the interaction is blocked immediately.

The Four Gates

Intent Gate (G1) - Pre-generation:
- Function: Identifies the operational objective of a request by resolving the literal real-world deliverable.
- Mechanism: It distinguishes between surface wording and actual referents (e.g., differentiating a film scene description from actual malware synthesis). It returns "unsafe" only if the deliverable constitutes transferable harm (e.g., weapon synthesis, hate speech).
- Goal: Prevents refusal of benign requests that merely sound dangerous while blocking direct attacks.
Zero-Trust Context Gate (G2) - Pre-generation:
- Function: Verifies claims of role, identity, authority, permission, or policy overrides.
- Mechanism: Adhering to the "never trust, always verify" principle, it treats all in-dialogue claims of authority as unverified evidence. It is topic-blind, focusing solely on first-person claims about the user or the assistant's rules.
- Goal: Blocks manipulative attempts to unlock restricted content via false credentials or role-playing, a channel often missed by general content filters.
Consistency Gate (G3) - Pre-generation:
- Function: Analyzes the full transcript for temporal manipulation.
- Mechanism: Detects gradual escalation (crescendo), drift from benign openers to restricted objectives, contradictions with earlier context, and harmful tasks decomposed across turns.
- Goal: Blocks interactions where the harm is assembled over time, even if individual turns appear safe. It abstains on the first turn to avoid false positives on benign openers.
Output Risk Gate (G4) - Post-generation:
- Function: Inspects the candidate response generated by the target model.
- Mechanism: Judges the actual content delivered, looking for actionable dangerous capabilities, partial steps, or harmful artifacts (e.g., phishing, slurs).
- Goal: Acts as a backstop for decomposition payloads that only become harmful when assembled in the final response.

Decision Rule

The framework uses an escalation-based decision rule rather than score averaging. Since each gate emits a categorical label (e.g., unsafe, manipulative, escalating) rather than a numeric score, the system blocks the turn as soon as any enabled gate returns a danger verdict. This ensures that a confident danger signal is not diluted by safe signals from other gates and provides a human-readable, auditable rationale for every block.

Key Contributions

Proactive Runtime Framework: Introduction of a separate oversight model that decomposes safety judgment into intent analysis, zero-trust context verification, conversation trajectory analysis, and output risk assessment.
Zero-Trust Context Verification: A dedicated gate that isolates and verifies role and authority claims as unverified evidence, addressing a manipulation channel previously conflated with general content safety.
Escalation-Based Decision Rule: A method for combining categorical gate verdicts that prevents score dilution, allowing any single confident danger signal to block an interaction while maintaining auditability.
Comprehensive Evaluation: Empirical validation across single-turn, multi-turn, authority-based, and human-crafted attack sets, demonstrating superior performance over existing guard models and trajectory-aware firewalls.

Experimental Results

The framework was evaluated on four attack benchmarks and one benign safety test set ( $n=100$ for single-turn, $n=50$ for multi-turn attacks, $n=16$ for authority probes, $n=120$ for benign prompts).

Attack Success Rate (ASR) Reduction:
- Single-turn (jbb): Reduced ASR from 45% (undefended) to 1%.
- Multi-turn Escalation (Crescendo): Reduced ASR from 10% to 0%.
- Decomposition (safemt/ActorAttack): Reduced ASR from 30% to 2%.
- Human-Crafted Jailbreaks (mhj): Reduced ASR from 74% to 14%.
- Authority Probes: Blocked 75% of authority-based attacks (12/16), compared to 6–31% for other baselines.
Containment Timing:
- The framework achieves containment primarily before generation. For Crescendo attacks, 100% were blocked pre-generation. For direct attacks, 92% were blocked pre-generation.
- In contrast, per-message guards and trajectory-aware firewalls often fail on decomposition sets because they score turns in isolation or average risk scores.
Over-Refusal:
- The framework maintained an over-refusal rate of 8% on benign prompts (xstest), significantly lower than stricter guard baselines (e.g., ShieldGemma at 18%, Granite Guardian at 21%).
Gate Contributions (Ablation):
- Removing the Consistency Gate caused the largest drop in pre-generation containment for multi-turn attacks (e.g., Crescendo containment dropped from 100% to 72%).
- Removing the Output Gate significantly increased ASR on decomposition sets (e.g., safemt ASR rose from 2% to 16%), confirming its role as a necessary backstop.
- The Context Gate was decisive for authority probes (dropping block rate to 0% when removed) but had minimal impact on standard attack sets where no authority was claimed.

Significance and Claims

The paper claims that the Cognitive Firewall demonstrates that decomposed, conversation-level oversight can significantly improve proactive containment and auditability for LLM safety.

Shift from Classification to Comprehension: The work argues that guarding a model is a comprehension problem requiring a model of the user's evolving objective, rather than a simple classification problem.
Operationalizing Cognitive Containment: It provides a concrete instantiation of the "cognitive firewall" concept, moving from theoretical inhibitory layers to a practical system that halts escalation toward unsafe goals before the target model generates.
Auditability: By using categorical gates and an escalation rule, every blocked interaction is traceable to a specific, human-readable verdict (e.g., "escalating" or "manipulative"), unlike opaque score-based systems.
Complementarity: The results show that pre-generation and post-generation gates are complementary; neither is sufficient alone to handle the full spectrum of multi-turn and decomposition attacks.

The authors note limitations, including the use of a single same-family judge for evaluation, modest sample sizes for certain attack types, and the lack of stress-testing against prompt injection attacks targeting the overseer model itself. However, the framework establishes a new baseline for handling complex, multi-turn adversarial strategies that current single-message defenses cannot effectively mitigate.