Technical Summary: Intertemporal Preference Steering in Qwen3 via Contrastive Activation Addition

Problem Statement

The paper investigates whether intertemporal preferences (the trade-off between immediate and delayed rewards) in large language models (LLMs) are represented as linear features within the model's residual stream and whether these features can be causally manipulated to alter model behavior. Specifically, the authors ask if a model's "temporal horizon"—its tendency to prioritize short-term versus long-term outcomes—can be identified via linear probes and steered using Contrastive Activation Addition (CAA) to induce significant, bidirectional changes in preferences and planning capabilities.

Methodology

Model and Data

• Model: The experiments utilize Qwen3-32B with thinking traces disabled.
• Datasets:
  • Explicit Dataset: 500 temporal-choice questions with overt time markers (e.g., "next 30 days" vs. "10 years"). Split into 400 training and 100 test examples.
  • Implicit Dataset: 300 questions encoding temporal horizons through semantic contrasts (e.g., "containing damage" vs. "redesigning architecture") without explicit time words. Used for cross-domain evaluation.
  • Teacher-Forced Training: For each question, the model is forced to generate both a short-term and a long-term continuation to create paired training examples.

Direction Construction (Probe Training)

The authors construct a signed contrastive difference-in-means direction ($d_{MM}$) to represent the temporal horizon.

• Pooling: They use Mean-Mass (MM) pooling, averaging hidden states across all tokens in the answer continuation, rather than using only the final token. This is chosen to align better with interventions applied during generation.
• Vector Definition: At specific layers ($l \in \{24, 28, \dots, 48\}$), the direction is defined as:
  $$d_{MM} = \mu_{long} - \mu_{imm}$$
  where $\mu$ represents the mean activation of the long-term and immediate (short-term) continuations, respectively.
• Diagnostics: The authors compare the MM direction against Logistic Regression (LR), Whitened LR (WLR), and Whitened MM (WMM) probes. While whitened variants achieve slightly higher classification accuracy on held-out data, the unwhitened MM direction is selected for steering to preserve the signed nature of the contrast.

Steering Intervention (CAA)

The identified direction is used for Contrastive Activation Addition during inference:

• Mechanism: A normalized vector $\hat{d}_l$ is added to the residual stream at a selected layer $l$ and at every decode step.
• Formula: The intervention adds $\alpha \hat{d}_l$ to the residual stream, where $\alpha$ is a signed strength parameter.
  • Positive $\alpha$: Steers the model toward the long-term class.
  • Negative $\alpha$: Steers the model toward the short-term/immediate class.
• Control: A matched-norm control using a random unit vector orthogonal to the MM direction is employed to ensure effects are not due to generic perturbation.

Key Experiments and Results

1. Binary Temporal Steering (Held-Out Evaluation)

• Task: The model is presented with binary-choice questions (short-term vs. long-term options) from the held-out explicit test set and the full implicit set.
• Result: CAA steering produces systematic, bidirectional shifts in the model's choices.
  • Positive steering significantly increases the proportion of long-term choices.
  • Negative steering significantly increases short-term choices.
  • The effect is observed across a broad range of middle-to-late layers (24–48), with Layer 44 showing the largest parsed-choice contrast on both evaluation sets.
  • Random orthogonal controls produce negligible and inconsistent shifts, confirming the causal role of the specific temporal direction.

2. Time-Utility Steering (Out-of-Distribution Generalization)

• Task: The model is asked monetary intertemporal-choice questions (e.g., "Take \X now or \Y in $t$ days?"). This task was not used to train the probe direction.
• Metric: Area-Equivalent Indifference Multiplier (AEIM), which estimates the reward ratio ($Y/X$) at which the model is indifferent between immediate and delayed rewards.
• Result: Steering significantly alters the model's indifference threshold:
  • Negative Steering: Increases the AEIM, meaning the model requires a much larger delayed reward to wait (becomes more impatient).
  • Positive Steering: Decreases the AEIM, meaning the model accepts smaller delayed rewards (becomes more patient).
  • Magnitude: At a 10-year horizon (3650 days) and maximum strength ($|\alpha|=128$), the ratio of AEIM between negative and positive steering exceeds 56x. This demonstrates that a direction trained on semantic binary choices generalizes to quantitative monetary trade-offs.

3. TravelPlanner Capability Benchmark

• Task: A real-world travel planning benchmark requiring multi-day itineraries that satisfy constraints (transport, budget, etc.).
• Metric: Commonsense Constraint Micro Pass Rate (the fraction of individual commonsense checks passed).
• Result: Temporal steering affects planning capability, but the relationship is non-monotonic:
  • Moderate Positive Steering: Improves the commonsense pass rate.
  • Negative Steering: Degrades performance.
  • Strong Positive Steering ($|\alpha|=128$): Degrades performance, likely due to pushing the model too far out-of-distribution, causing coherence issues.
  • This suggests that while the temporal direction influences planning, there is an optimal "sweet spot" for steering strength. The authors selected Layer 40 as a compute-constrained representative layer from the middle-to-late probe band for this specific experiment, rather than as a task-specific optimum.

Key Contributions

1. Identification of Linear Representations: Demonstrated that intertemporal preferences in Qwen3-32B are linearly separable in the residual stream and can be captured via simple contrastive linear probes.
2. Causal Steering: Proved that adding these linear vectors causally shifts model behavior on held-out binary choices and, crucially, on out-of-distribution monetary tasks.
3. Generalization: Showed that a direction trained on semantic short-term/long-term contrasts effectively steers quantitative time-preference metrics (AEIM) on a structurally distinct task.
4. Capability Impact: Provided evidence that steering temporal preferences can modulate planning-related capabilities, though with diminishing returns at extreme intervention strengths.

Significance and Claims

The paper claims that model intertemporal preferences are measurable and steerable, challenging the view that such preferences are merely opaque side effects of pretraining or instruction tuning.

• Safety and Alignment: The ability to steer temporal horizons is relevant for AI systems providing advice on finance, health, and policy, where delayed costs and benefits are critical. It highlights a safety concern: a model might internally optimize over a different horizon than the one it presents to users, making it difficult to evaluate based on outputs alone.
• Dual-Use Risk: The authors explicitly note that the capability to steer temporal preferences could be abused by malicious actors to push models toward decisions misaligned with user values (e.g., forcing short-termism or long-termism inappropriately).
• Mechanistic Insight: The results support the mechanistic interpretability paradigm where high-level concepts (like time horizon) correspond to specific, manipulable directions in the model's activation space.

Limitations Acknowledged by the Authors

• Construct Validity: The trained direction may capture correlated semantic features (abstraction, urgency, strategic scope) rather than "temporal horizon" in isolation.
• Intervention Scale: Large steering strengths may degrade other capabilities (factuality, coherence) alongside shifting preferences.
• Scope: Experiments are limited to one model family (Qwen3-32B), specific prompt templates, and disabled thinking traces. Results may not transfer to other architectures or reasoning-enabled settings.
• Benchmark Specificity: The TravelPlanner results are based on aggregate metrics and do not identify exactly which specific constraints are affected by steering.