Technical Summary: PRISM – Recovering Instruction Sets from Language Model Activations

1. Problem Statement: Instruction Set Retrieval (ISR)

As Large Language Models (LLMs) are increasingly deployed as autonomous agents, reliable monitoring requires more than observing output tokens; it necessitates understanding the internal objectives steering behavior. Current monitoring faces challenges when models infer unintended subgoals, follow contextual cues, or are influenced by prompt injections and hidden objectives.

While recent "activation-to-language" methods (e.g., Patchscopes, LatentQA, Activation Oracles) demonstrate that hidden states can reveal natural-language information, they are ill-suited for Instruction Set Retrieval (ISR). Existing approaches typically answer arbitrary questions or produce single high-level summaries. They fail to recover the full set of simultaneous instructions, constraints, prohibitions, and subgoals active in agentic settings. Consequently, they often recover only a fraction of active instructions, conflate distinct constraints, or hallucinate objectives not present in the context.

The paper formalizes ISR as the problem of learning an interpreter $\phi$ that, given an activation snapshot $H_\ell$ from a frozen target model $M$, produces a predicted list $\hat{I}$ that maximizes coverage (recovery of ground-truth instructions) while minimizing hallucination (prediction of unsupported instructions).

2. Methodology: PRISM

The authors introduce PRISM, an activation-conditioned interpreter designed specifically for faithful instruction set retrieval.

Architecture

PRISM is designed for lightweight deployment. Instead of loading a separate interpreter LLM, it reuses the frozen base weights of the target model $M$ (specifically Qwen3.5-9B in the experiments).

• Input: A snapshot of residual-stream activations $H_\ell$ from a specific layer $\ell$ (e.g., layer 16) over a fixed window of generated tokens (e.g., the final 128 tokens).
• Projection: A learned linear projection maps these activation vectors into the model's input embedding space.
• Decoding: The projected activations serve as a "soft prefix" fed into the target model. The model then autoregressively decodes a natural-language bullet list of the active instructions.
• Parameters: Only the projection layer and LoRA adapters are trained; the base model weights remain frozen.

Training Strategy

PRISM is trained in two stages to address the limitations of standard cross-entropy loss, which penalizes surface-form mismatches and fails to distinguish between missed instructions and hallucinations.

1. Supervised Pretraining (SFT): The model is trained to map activation snapshots to ground-truth instruction lists using next-token cross-entropy. This establishes a baseline activation-to-list mapping.
2. Judge-Guided Reinforcement Learning (GRPO): To optimize for set-level faithfulness, the authors employ Group Relative Policy Optimization (GRPO).
   • Judge Mechanism: A separate LLM judge evaluates candidate instruction lists against the ground-truth list. It determines which reference instructions are covered and which predicted bullets are unsupported (hallucinated). The judge's system prompt is calibrated against human annotations using Cohen's $\kappa$.
   • Reward Function: The reward $r$ is a scalar derived from coverage ($Cvg$) and hallucination ($Hall$) metrics:
     $$r = w_{inst} \cdot Cvg(\hat{I}, I) - w_{halluc} \cdot Hall(\hat{I}, I) - P_{len}$$
     Where $P_{len}$ is a two-sided length penalty to prevent reward hacking via overly long or collapsed reports.
   • Optimization: The policy is updated to maximize the group-relative advantage, directly optimizing for high coverage and low hallucination rather than token-level matching.

3. Key Contributions

1. Formalization of ISR: The paper defines Instruction Set Retrieval as a distinct problem from general activation interpretation, focusing on recovering the complete set of operative constraints and subgoals.
2. PRISM Framework: An activation-conditioned interpreter that decodes hidden states into a faithful bullet list of active instructions without relying on surface-level token inspection or model self-reporting.
3. Judge-Guided GRPO: A novel training objective that uses an LLM judge to provide semantic feedback, directly optimizing for instruction-level recall and penalizing hallucinations.
4. Calibrated Evaluation: The authors calibrate their LLM judge against human annotations and release the prompts to support reproducible evaluation of instruction set retrieval systems.
5. Resource Release: The paper commits to releasing source code, training data, and security-relevant evaluation datasets (covering prompt injection and hidden objectives) to facilitate future research.

4. Experimental Results

The evaluation uses a held-out test corpus of 1,000 examples across four settings: Benign (BN), Behavioral Constraints (BC), Hidden Objectives (HO), and Adversarial Prompts (AP). The target model is Qwen3.5-9B.

• Overall Performance: PRISM (with RL) achieves the highest average reward (0.736) and coverage (0.745) across all settings, significantly outperforming the strongest non-PRISM baseline (Activation Oracles, reward 0.532).
• Security-Relevant Recovery: In adversarial settings (HO and AP), PRISM demonstrates superior ability to recover injected or covert instructions. Specifically, on the Adversarial Prompt subset, coverage improved from 0.520 (PRISM w/o RL) to 0.652.
• Hallucination Control: PRISM maintains a low hallucination rate (0.014). While some baselines show lower hallucination rates, the paper notes this is often because they collapse prompts into generic summaries, missing many active instructions entirely.
• Ablation: The addition of the judge-guided RL stage consistently improves performance across all domains, particularly in the Adversarial Prompt setting, confirming that the RL objective successfully optimizes set-level properties rather than just fluency.

5. Significance and Claims

The paper claims that PRISM demonstrates that activations encode recoverable instruction information, suggesting that activation-based monitoring is a viable path toward transparent oversight of LLM agents.

• Monitoring Capability: The work provides a method to inspect the internal "operative instruction set" of an agent, revealing not just what the model is doing, but why (i.e., which specific constraints or goals are driving the behavior).
• Security Implications: By successfully recovering hidden objectives and prompt injections that other methods miss, PRISM offers a tool for auditing agents in high-stakes scenarios where input prompts alone may not reveal adversarial steering.
• Limitations and Scope: The authors are modest about their claims, noting that:
  • Results are specific to the Qwen3.5-9B model family, layer 16, and a 128-token window; transferability to other models or longer contexts is unproven.
  • The evaluation relies on oracle-generated labels and judge-based scoring, which, while calibrated, are not absolute ground truth.
  • PRISM is a monitoring method, not a safety intervention; recovering an instruction does not automatically dictate whether to block or modify behavior.
  • The system is not intended for high-stakes punitive decisions without human review due to the risk of missed instructions or hallucinations.

The paper concludes that while PRISM is not a complete safety solution, it represents a significant step toward understanding and auditing the internal objectives of deployed LLM agents.