Predictive Memory Localization: Forecasting Selective Intervention Paths from Internal Signals

Technical Summary: Predictive Memory Localization (PML)

Problem Statement
Activation steering allows researchers to control language model behavior by injecting localized internal representations into specific layers. However, the literature suggests a critical gap: while localization identifies where information resides, it does not guarantee that a specific direction admits a "selective operating regime." A direction may successfully shift a target behavior at one intervention strength while simultaneously damaging semantic neighbors or unrelated capabilities at another, or it may fail to produce a clean effect at any strength. Current methods often rely on static localization evidence or fixed-strength interventions, which fail to predict whether a usable, selective coefficient exists for a given record, direction, and layer. The core problem is distinguishing directions that offer leverage from those that cause collateral damage across a measured intervention path.

Methodology
The authors introduce Predictive Memory Localization (PML), a framework that treats the measured-grid intervention path as the predictive object of memory localization.

1. Measured-Grid Intervention Paths:
   PML defines a record-conditioned path across a pre-specified grid of intervention strengths ($\alpha$). For each record $i$, direction $v$, and layer $\ell$, the system measures:

   • Target Movement ($\Delta T$): Changes in the answer margin for desired behaviors.
   • Collateral Damage: Changes in semantic-neighbor ($\Delta N$) and general capability ($\Delta C$) margins.
   • Random Calibration: Separate 95th-percentile thresholds ($\tau_T, \tau_N, \tau_C$) are established using random Gaussian directions to define what constitutes a significant target effect or damage.
   • Outcomes: The system records four binary events at held-out stronger coefficients: Target-any, Neighbor Damage, Capability Damage, and Clean-any (target effect without damage).

2. Predictive Signals:
   PML forecasts these outcomes using a hierarchy of evidence:

   • Baseline & Metadata ($B+M$): Record properties and model configuration.
   • Static Localization ($L$): Features derived from class separation, projection, and saliency without intervention.
   • Supervised Geometry ($G$): Features from Recursive Feature Machines (RFM) and Average Gradient Outer Product (AGOP), including spectral concentration and alignment.
   • Low-Dose Response ($R$): A "strength-disjoint" causal measurement taken at a weak intervention strength ($|\alpha| = 0.1$). This serves as a cheap dynamic diagnostic.

3. Decision Policy:
   A risk-aware selector uses these forecasts to choose a specific coefficient or abstain. The utility function rewards target movement and penalizes collateral margin decreases. The policy aims to maximize utility while avoiding dense scans of the coefficient grid.

Key Contributions

• Definition of a Random-Calibrated Path: The paper defines a measured-grid path that jointly records target leverage, semantic-neighbor damage, capability damage, and clean coefficients for every record-direction-layer combination.
• Separation of Static vs. Causal Signals: The study demonstrates that static localization and supervised geometry are weak predictive priors. In contrast, a strength-disjoint low-dose response ($R$) provides the dominant signal for forecasting outcomes at stronger, disjoint strengths.
• Actionable Forecasting: PML connects forecasting to action via a held-out collateral-aware policy. This policy selects a coefficient or abstains, providing a proof of concept for reducing the computational cost of dense intervention evaluations.

Experimental Results
The study evaluates 3,000 frozen records across nine datasets and fourteen domains, generating 30,000 distinct paths. Results are confirmed across three residual-norm-matched base models (Qwen3-1.7B, Qwen3.5-2B, Ministral-3-3B).

• Direction Performance: Learned directions (specifically RFM/AGOP at layer 7) significantly outperform random controls. In the primary study, RFM/AGOP achieved 13.1% Target-any and 12.3% Clean-any incidence, exceeding random baselines by 3.6 and 3.4 percentage points, respectively.
• Predictive Hierarchy: Static localization features add little predictive value beyond baseline metadata. The low-dose response ($R$) at $|\alpha|=0.1$ is the strongest predictor for outcomes at $|\alpha| \in \{0.25, 0.5\}$.
• Model Robustness: The predictive hierarchy (Low-dose > Static) replicates across all three base models, with record-held-out macro AUROC scores ranging from 0.801 to 0.828.
• Decision Utility: A predictor-driven selector improves utility and significantly reduces semantic-neighbor damage compared to a fixed-strength policy. It reduces the need for evaluation from a dense scan (26 coefficients) to approximately 2.55 evaluations (two weak probes plus a final action).

Significance and Claims
The paper claims that PML transforms memory localization from a static claim into a falsifiable forecast of margin-level selective outcomes. The central finding is that a small, causal response is a more useful forecast of later selective behavior than static localization geometry alone.

The authors emphasize that localization does not automatically imply controllability; a direction can be well-localized yet lack a selective operating regime. By pairing localization with a cheap causal measurement, PML identifies which directions admit usable coefficients and which require abstention. The work does not claim to solve general model editing or guarantee safety across all downstream side effects, but rather provides a rigorous, risk-aware framework for selecting intervention strengths based on measured path topology.