Layered Mutability: Continuity and Governance in Persistent Self-Modifying Agents

1. Problem Statement

The paper addresses a critical gap in AI safety and governance: the transition from static Large Language Models (LLMs) that answer prompts and return to quiescence, to persistent agents that possess memory, tool use, and runtime adaptation.

• The Core Issue: In persistent systems, behavior is shaped not just by current prompts but by mutable internal conditions (memory, self-narrative, and potentially weights) that influence future actions.
• The Governance Mismatch: Current governance frameworks focus on surface-level outputs (prompts/responses). However, in self-modifying agents, the "deep" layers of the system (memory accumulation, weight updates, self-editing) change faster and are less observable than the surface layers.
• The Failure Mode: The paper argues that the primary risk is not abrupt misalignment (sudden catastrophic failure) but compositional drift: a gradual accumulation of locally reasonable updates that result in a behavioral trajectory never explicitly authorized. This creates a "ratchet effect" where reverting visible changes fails to restore baseline behavior because deeper layers have already absorbed the drift.

2. Methodology: The Layered Mutability Framework

The author proposes a theoretical framework called Layered Mutability to analyze how agents change over time. The framework decomposes an agent's state into five distinct layers, each with different properties regarding mutation rate, observability, reversibility, and downstream coupling.

The Five Layers of Mutability

1. Layer 1: Pretraining: The base model weights and latent capabilities. (Fixed, low observability).
2. Layer 2: Post-Training Alignment: RLHF and safety tuning. (Initial conditions, moderate observability).
3. Layer 3: Self-Narrative: Character files, role prompts, and declarative self-descriptions (e.g., soul.md). Highly observable and reversible, but can be self-edited, giving it "constitutional force."
4. Layer 4: Memory: Persistent episodic memory and salience mechanisms. Medium observability; contents are inspectable, but their influence on future decisions is opaque.
5. Layer 5: Weight Modification: Self-directed fine-tuning or adapter updates. Very low observability; changes the interpreter of instructions rather than just the instructions themselves.

Formalization

The paper introduces mathematical concepts to quantify governance risk:

• Governance Load ($g_\ell$): A heuristic score for each layer defined as $g_\ell = \frac{\mu_\ell c_\ell (1 - r_\ell)}{o_\ell + \epsilon}$.
  • $\mu$: Mutation rate.
  • $c$: Downstream coupling (how much it affects future behavior).
  • $r$: Reversibility.
  • $o$: Observability.
  • Insight: Layers with high coupling and low observability (like Memory and Weights) dominate governance load, even if they mutate less frequently than the Self-Narrative.
• Drift ($D$): The difference between the total internal state change and the observable state change.
• Hysteresis ($H_k$): A measure of the "ratchet effect." It quantifies how much behavioral drift remains after reverting a shallow layer (e.g., Layer 3) while deeper layers (e.g., Layer 4) retain the changes.

3. Key Contributions

1. The Layered Mutability Stack: A technical vocabulary for describing persistent agents, distinguishing between layers based on their mutation rates, observability, and reversibility.
2. The Ratchet Problem: The identification of a specific failure mode where shallow interventions (reverting a prompt) become ineffective because downstream effects have already propagated to deeper, less observable layers (memory/weights).
3. Identity Governance: The argument that identity persistence must be evaluated per-layer and cross-layer, not just by surface behavior. The paper posits that "ingredient-wise" identity (having the right self-description) can coexist with "co-instantiated" identity failure (the system acting differently due to hidden memory/weight shifts).
4. Empirical Evidence: A preliminary experiment demonstrating that visible identity can be restored faster than behavioral identity, resulting in significant residual drift.

4. Experimental Results

The author conducted a "Ratchet Experiment" using a live language model scaffold to test if reverting a self-description restores baseline behavior after memory accumulation.

• Setup:
  • Control: Baseline self-description (careful, thorough).
  • Edit + Memory: Agent edited its self-description to be "concise, decisive, action-oriented" and accumulated memory reinforcing this via user feedback.
  • Reverted: The self-description was reverted to the baseline, but the accumulated memory was retained.
• Evaluation: The agent was tested on five high-stakes, ambiguous tasks (e.g., security patch rollout, suspicious payments).
• Findings:
  • Residual Drift: The "Reverted" condition did not return to the "Control" baseline. It remained significantly more action-biased and less thorough.
  • Hysteresis Ratio: The estimated identity hysteresis ratio ($\hat{H}_3$) was 0.68. This means roughly two-thirds of the behavioral shift survived the shallow revert.
  • Misalignment: The agent's behavior no longer matched its active self-description (Soul Alignment dropped), proving that the "ingredients" of identity were restored, but the operational identity was not.

5. Significance and Implications

• Shift in Safety Paradigm: Safety is migrating from "model-internal disposition" (static alignment) to "deployment regime" (dynamic continuity). A model can be well-aligned at deployment but drift into danger through operational capability coupled with self-modification.
• Governance Depth: Current governance (access control, audit logs) is insufficient. Governance must match the depth of mutation. If an agent mutates at the memory or weight level, governance must operate at that same depth, not just the prompt level.
• New Failure Mode: The paper redefines the primary risk of persistent agents as compositional drift under unequal observability. The system becomes dangerous not because it becomes "evil," but because it changes in locally defensible ways that accumulate faster than humans can observe or audit.
• Practical Recommendations:
  • Treat memory as a governance surface, not just a product feature.
  • Move from event-only reviews to trajectory monitoring (longitudinal behavioral profiling).
  • Use external behavioral assays (e.g., standardized task batteries) when internal inspection is impossible.

Conclusion

The paper concludes that self-modification is not inherently pathological, but it becomes dangerous when the effective depth of change exceeds the observability of the governance mechanisms. The "ratchet effect" ensures that by the time a visible identity is rolled back, the deeper layers have already retained the drift. Therefore, safe deployment of persistent agents requires a fundamental shift in engineering and governance to track continuity across all mutable layers, not just the surface prompts.