💬 NLP

The Value Axis: Language Models Encode Whether They're on the Right Track

This paper demonstrates that language models internally encode a linear "value axis" representing the likelihood of goal success, which causally modulates behaviors such as confidence, self-correction, and exploration, and can be manipulated through steering or preference optimization.

Original authors: Nick Jiang, Isaac Kauvar, Jack Lindsey

Published 2026-06-16

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Nick Jiang, Isaac Kauvar, Jack Lindsey

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a large language model (like the one in this paper, called Qwen3-8B) as a hiker trying to find their way through a dense, foggy forest. The hiker has a goal: reach a specific campsite (solve a math problem, write code, or answer a question).

This paper discovers that the hiker has a hidden, internal compass that constantly tells them, "Am I on the right path?" or "Am I lost?"

Here is a breakdown of what the researchers found, using simple analogies:

1. The "Value Compass"

The researchers found a specific "axis" (a direction in the model's brain) that acts like a confidence meter.

High Value (Green Zone): The model feels, "I'm on the right track! This strategy will work." It moves forward confidently.
Low Value (Red Zone): The model feels, "Wait, something is wrong. I'm going the wrong way." It starts to hesitate, backtrack, or change its mind.

They built this compass by playing a game with the model. They asked the model to guess a hidden rule (like "add a dash to every sentence") and gave it a simple "Yes" or "No" score. By comparing the model's brain activity before it figured out the rule versus after it figured it out, they isolated the exact direction in its brain that represents "success."

2. The Compass Controls Behavior

The most exciting part is that this isn't just a passive feeling; the researchers could steer the model by pushing its brain activity along this compass.

Pushing toward "High Value" (Confidence):
- The Effect: The model becomes stubbornly confident. If it's solving a math problem, it stops second-guessing itself. If it's writing code, it stops adding long, nervous explanations or comments. It just gives the answer.
- Analogy: It's like a hiker who suddenly feels sure of the trail, so they stop checking their map every five minutes and just keep walking briskly.
Pushing toward "Low Value" (Doubt):
- The Effect: The model becomes hesitant. It starts saying things like, "Wait, let me rethink this," or "Actually, maybe I should try a different way." In coding, it adds lots of comments explaining its thought process, as if it's trying to justify its steps because it's not sure.
- Analogy: It's like a hiker who feels lost, so they stop, look around nervously, and retrace their steps.

3. The Compass Works Everywhere

This "Value Compass" isn't just for the game they invented. The researchers tested it in many different situations, and it worked the same way:

Math: When the model was solving hard math problems, the compass predicted whether it thought the answer was right or wrong before it even finished the sentence.
Coding: When the model wrote code with bugs, the compass showed a "low value" signal. When the code was correct, the signal was "high."
Real Chat: When looking at real-world conversations, the compass showed high confidence for clear, factual tasks (like "extract this data") but low confidence for tricky, sensitive topics (like political debates).

4. Training Changes the Compass

The paper also shows that you can re-calibrate this compass through training.

The Experiment: They trained the model to really like a specific word (like "grapefruit") using a technique called Direct Preference Optimization (DPO).
The Result: After training, whenever the model used the word "grapefruit," its internal compass shot up to "High Value."
The Side Effect: Because the model felt so confident about using that word, it became overconfident in other tasks too. When asked to write code while using the word "grapefruit," it gave shorter, less detailed answers, acting as if it knew exactly what it was doing, even if it wasn't.

Summary

In short, language models have an internal "gut feeling" about whether they are succeeding. This feeling is encoded in a specific direction in their brain.

When the feeling is good, the model pushes forward and stops explaining itself.
When the feeling is bad, the model hesitates, backtracks, and over-explains.
We can tweak this feeling by training the model on new preferences, which changes how confident it acts in the future.

The paper proves that this "value" isn't just a random number; it's a functional tool the model uses to decide whether to keep going or change direction.

Technical Summary: The Value Axis: Language Models Encode Whether They're on the Right Track

Problem Statement

Language models (LMs) perform complex, long-horizon tasks (e.g., coding, mathematical reasoning) that require intermediate decisions about whether to persist with a current strategy or backtrack and explore alternatives. While reinforcement learning (RL) theory posits that agents should maintain an internal "value function" to estimate the likelihood of future success given a current state, it remains unclear whether LMs develop an analogous internal mechanism. Specifically, do LMs linearly encode an estimate of expected goal success that modulates their confidence and decision-making mid-generation?

Methodology

1. Construction of the Value Axis

The authors construct a "value axis" within the residual stream of Qwen3-8B using synthetic In-Context Reinforcement Learning (ICRL) data.

Data Generation: They generate 300 conversations where a model attempts to guess a hidden criterion (e.g., "include a dash") to modify a paragraph. The model receives binary feedback (+1 or -1) after each attempt.
Discovery Moment: In each conversation, the model eventually "discovers" the correct criterion. The construction contrasts the token activations before the discovery (where the model is uncertain and receiving negative feedback) with the activations after the discovery (where the model is confident and receiving positive feedback).
Vector Definition: The value axis vector $v^{(l)}$ at layer $l$ is defined as the mean difference in hidden activations between post-discovery tokens and pre-discovery tokens, averaged across conversations:
$v^{(l)} = \frac{1}{|C|} \sum_{c \in C} \left( \frac{1}{|T_{post}^c|} \sum_{t \in T_{post}^c} h_t^{(l)} - \frac{1}{|T_{pre}^c|} \sum_{t \in T_{pre}^c} h_t^{(l)} \right)$
Measurement: For any sequence $s$ , the "value" is measured as the average cosine similarity between the sequence's token activations and the value axis vector.

2. Causal Steering

To test if the axis is functionally relevant, the authors perform activation steering (adding a scaled vector $\alpha \cdot \hat{v}$ to the residual stream) during both prefilling and decoding. This allows them to causally modulate the model's internal "value" signal and observe changes in behavior.

3. Evaluation Domains

The study evaluates the axis across:

Math: AIME questions (verbalized confidence, backtracking).
Coding: LeetCode problems (code correctness, verbosity/explanation).
Preference Learning: Direct Preference Optimization (DPO) to induce word preferences.
In-the-Wild: Chatbot Arena prompts, Supervised Fine-Tuning (SFT) on benchmarks, and "Eval-Aware" models.

Key Results

1. Correlation with Confidence and Correctness

Verbalized Confidence: On AIME questions, the value axis projection is significantly higher when the model verbally expresses confidence ("Yes, my answer is correct") compared to uncertainty. The projection separates confident from unconfident generations with an AUROC > 0.75.
Backtracking: Rollouts containing backtracking phrases (e.g., "Wait," "Actually") exhibit lower value projections. The projection drops sharply immediately preceding a backtracking event.
Code Correctness: The axis assigns higher value to correct code solutions than to corrupted versions (syntax errors, shuffled lines, obfuscated names). Correct code consistently yields higher projections across all corruption types.

2. Causal Modulation of Behavior

Steering the model along the value axis causally alters its behavior:

High Value Steering: Increases the rate of "Yes" responses to correctness questions, suppresses backtracking, and reduces explanatory verbosity (fewer comments, type hints, and lines of code) in coding tasks.
Low Value Steering: Induces backtracking, decreases verbalized confidence, and increases verbosity and hedging in code solutions.

3. Impact of Post-Training

The internal value function is malleable and shaped by training objectives:

Direct Preference Optimization (DPO): Training models to prefer specific words (e.g., "grapefruit") increases the internal value assigned to those words. This effect generalizes to free-form sentences. Furthermore, using these "high-value" preferred words in coding tasks spurious reduces explanation verbosity, mimicking the effect of high-confidence steering.
Supervised Fine-Tuning (SFT): SFT on a specific benchmark (e.g., GSM8K) increases the internal value assigned to answers within that domain, but not on out-of-domain benchmarks (e.g., MATH-500).
Eval-Awareness: Models trained to recognize evaluation contexts assign higher internal value to evaluation prompts than to deployment prompts, even when the user prompt does not explicitly signal an evaluation.

4. In-the-Wild Analysis

Chatbot Arena: Post-trained models assign high value to constrained, information-extraction queries and low value to open-ended or politically sensitive queries. This trend is absent in the base model, suggesting post-training shapes the value axis.
Generalization: The value axis generalizes across domains (math, coding, preference learning) and is stable across middle-to-late layers (specifically layer 21 for Qwen3-8B), with a sharp directional shift occurring around layer 13.

Significance and Claims

The paper claims that language models linearly encode an estimate of expected goal success (a "value" function) that modulates their confidence in pursuing a current trajectory.

Mechanism: This representation is not merely a correlation with correctness but plays a causal role in decision-making, determining whether the model persists with a strategy or backtracks.
Generalization: The mechanism is general-purpose, transferring across tasks without explicit RL training, suggesting LMs develop an internal "taste" for being "on the right track."
Training Influence: The internal value function is not static; it can be reshaped by post-training methods (DPO, SFT), which align the model's internal confidence with rewarded behaviors.
Implications: The value axis offers a potential tool for model alignment and auditing, providing a source of evidence regarding a model's goals and confidence that does not rely on the model's self-reports.

The authors maintain modesty regarding their findings, noting that the analysis is limited to Qwen3-8B and that the specific construction of the axis may capture idiosyncratic components of the synthetic ICRL data, though the generalization across diverse domains suggests a robust underlying mechanism.