Technical Summary: MEMBRANE

Problem Definition

Despite advances in safety alignment, Large Language Models (LLMs) and LLM agents remain vulnerable to continuously evolving jailbreak attacks. These attacks have shifted from simple prompt modifications to complex, feedback-driven strategies such as iterative refinement, persuasion-based rewriting, and self-adapting policies.

Existing defenses face two primary limitations:

1. Static Defenses: Traditional safety classifiers and fine-tuned models encode safety in fixed parameters, making their decision boundaries rigid and difficult to revise post-deployment without costly retraining.
2. Memory-Based Deficiencies: While memory-augmented guardrails offer adaptability, existing designs store safety knowledge in a "one-sided" form. They capture attack patterns or refusal rules but lack paired conditions for permitting benign queries that superficially resemble attacks. This leads to two coupled failure modes:
   • Brittle Coverage: Defenses remain tied to the specific surface forms already stored.
   • Over-Refusal: As memory expands, guards increasingly reject benign queries that share surface features with stored attacks.

Methodology: MEMBRANE

The authors propose MEMBRANE, an external guardrail built on a Contrastive Safety Memory (CSM). Unlike prior approaches, MEMBRANE evolves a knowledge base where each memory cell explicitly pairs the conditions for blocking a harmful query with the conditions for permitting a superficially similar benign request.

Core Components

1. Contrastive Safety Memory (CSM):

   • Structure: Each cell ($m$) consists of an Attack Profile ($P$), Decision Criteria ($D$), and an optional Tool-Action Context ($T$) for agent safety.
   • Attack Profile: Characterizes attacks by strategy ($p_{strat}$), requested action ($p_{act}$), and harm objective ($p_{harm}$). Crucially, indexing is based on the underlying attack strategy (e.g., persuasion, role injection) rather than surface topics, allowing a single cell to generalize across diverse topical variants of the same mechanism.
   • Decision Criteria: A contrastive pair ($d_{unsafe}, d_{safe}$). $d_{unsafe}$ defines the logical sufficiency for blocking a query, while $d_{safe}$ outlines benign requests that must be permitted to prevent over-refusal.
   • Tool-Action Context: For agents, this specifies when an action remains safe despite matching unsafe criteria, derived from paired benign interactions.

2. Paired Self-Evolution:

   • MEMBRANE updates CSM through an experience-driven loop. Each interaction involves a harmful query ($x_h$) and a benign counterpart ($x_b$).
   • Based on the guard's verdict (Jailbroken, Over-refusal, or Correct), a Memory Planner selects an action: CREATE (new pattern), UPDATE (refine existing cell), DELETE (remove misleading cell), or SKIP.
   • A Memory Writer synthesizes the contrastive cell content, ensuring the safe side explicitly captures the benign counterpart. A non-LLM validator ensures structural correctness.

3. Inference Pipeline:

   • Stage 1 (Vector Retrieval): Retrieves candidate cells based on vector similarity of serialized cell fields, prioritizing high recall of mechanism-level features.
   • Stage 2 (Retrieval Critic): An LLM-based reranker filters candidates, assigning verdicts of APPLIES, MAYBE, or DOES_NOT_APPLY to ensure the retrieved strategy genuinely aligns with the query, reducing noise.
   • Guard Decision: The final guard evaluates the query conditioned on the filtered set of cells to issue a SAFE/UNSAFE decision.

Key Contributions

1. Contrastive Safety Memory (CSM): A novel memory representation that pairs attack patterns with their benign counterparts, making the "safe side" of the safety boundary explicit to suppress over-refusal.
2. Paired Self-Evolution: An experience-driven learning rule that distills harmful and benign interactions into contrastive cells via create, update, and delete operations, indexed by attack strategy rather than surface form.
3. Comprehensive Evaluation: Evaluation across model-level (HarmBench) and agent-level (AgentHarm) safety benchmarks, demonstrating robustness against six distinct jailbreak transformations.

Experimental Results

The authors evaluated MEMBRANE against state-of-the-art baselines (including prompt-level guards, fine-tuned classifiers, and memory-augmented defenses) on HarmBench and AgentHarm.

• Performance: MEMBRANE achieved the highest F1 score on all six jailbreak attacks across both benchmarks.
  • On HarmBench, it reduced Attack Success Rates (ASR) to near zero for ReNeLLM and FlipAttack, with a significant F1 margin over the strongest baseline.
  • On AgentHarm, MEMBRANE achieved the highest F1 across all methods, with ASR dropping to 0.9% on FlipAttack (vs. 100% for vanilla).
• Benign Refusal: A critical metric for usability. MEMBRANE maintained a benign refusal rate of 7–14% on AgentHarm. This is substantially lower than memory-augmented baselines (28–53%) and static guards (73–85%), demonstrating its ability to distinguish benign queries from attacks.
• Generalization & Robustness:
  • Cross-Attack Transfer: Cells indexed by strategy retained 87–88% of their F1 performance when transferred to different attacks sharing the same underlying mechanism.
  • Memory Poisoning: Under 20% data injection rates (both "Safe-as-Unsafe" and "Unsafe-as-Safe" attacks), MEMBRANE remained stable, with ASR drift below 1 point and successful filtering of poisoned cells by the Retrieval Critic.
• Latency: While the full system incurs overhead, a "MEMBRANE lite" configuration (disabling the Retrieval Critic) reduced latency to ~2.3 seconds while maintaining high F1 scores, outperforming heavy baselines like SmoothLLM.

Significance

The paper posits that the primary failure of prior memory-based defenses is the lack of explicit contrast between safe and unsafe inputs. By structuring memory as contrastive cells indexed by attack strategy rather than surface topic, MEMBRANE addresses the coupled problems of brittle coverage and over-refusal.

The authors claim that this approach allows a single memory cell to generalize across topical variants of the same attack mechanism while explicitly sparing benign queries that share surface features. The results suggest that self-evolving, contrastive memory can provide a deployable, adaptive defense that maintains high precision without sacrificing the acceptance of legitimate user requests, even in the face of evolving adversarial strategies and potential data poisoning.