💬 NLP

VALUEFLOW: Toward Pluralistic and Steerable Value-based Alignment in Large Language Models

This paper introduces VALUEFLOW, a unified framework that addresses key gaps in value-based alignment by integrating hierarchical value extraction, a large-scale intensity-labeled database, and a calibrated evaluation system to enable the pluralistic and steerable control of value intensities in Large Language Models.

Original authors: Woojin Kim, Sieun Hyeon, Jusang Oh, Jaeyoung Do

Published 2026-06-08

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Woojin Kim, Sieun Hyeon, Jusang Oh, Jaeyoung Do

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a very smart, but somewhat rigid, robot how to act like a human. The problem isn't just that the robot needs to be "nice"; it's that humans have complex, sometimes conflicting, internal compasses called values.

Some people care deeply about kindness, others about justice, and some about power. The tricky part is that these values aren't just "on" or "off" switches. You can be slightly kind, or extremely kind. You can be mostly just, but a little bit selfish.

Current AI alignment methods are like trying to teach the robot with a simple "Good/Bad" button. It's too blunt. The paper VALUEFLOW introduces a new, much more sophisticated toolkit to help AI understand and express these human values with the right intensity.

Here is how VALUEFLOW works, broken down into three simple parts:

1. The Map: HIVES (The Hierarchical Value Embedding Space)

Imagine you have a giant library of human thoughts, but the books are thrown everywhere. Some books are about "Kindness," others about "Helping Neighbors," and others about "Donating Money." A simple search might confuse "Kindness" with "Justice" because they are related but different.

HIVES is like a super-organized library map. It doesn't just list values; it understands the hierarchy. It knows that "Kindness" is a big umbrella, and under it, you have "Caring" and "Helping." It organizes these values into a 3D space where similar values are close together, and very different ones are far apart. This helps the AI understand the structure of human values, not just the words.

2. The Reference Library: VIDB (The Value Intensity Database)

Now, imagine you want to tell the AI, "Be moderately kind, but very strict about justice." How does the AI know what "moderately" looks like?

Previously, AI judges would just guess a score (like "7 out of 10"). But different AIs guess differently, and their scores are unstable.

VIDB is a massive, pre-calibrated library of text examples. It contains thousands of sentences, each labeled with a precise "intensity score" (from -10 to +10) for specific values.

The Analogy: Think of VIDB as a set of standard weights on a scale. If you want to know how "heavy" a new sentence is in terms of "Justice," you don't just guess. You compare it against these standard weights.
How it works: Instead of asking the AI "Is this text just?" (which leads to bad guesses), the system asks the AI to rank the new text against a few standard examples from the library. "Is this text more just than Example A, but less just than Example B?" This ranking method is much more stable and reliable than guessing a number.

3. The Steering Wheel: Intensity-Aware Steering

This is the part where you actually control the AI.

In the past, if you told an AI "Be kind," it might be too kind or not kind enough. With VALUEFLOW, you can give the AI a dial.

The Analogy: Imagine driving a car. Old methods were like having a gas pedal that only had "Off" and "Full Throttle." VALUEFLOW gives you a steering wheel with a speedometer. You can say, "Drive with a 'Kindness' intensity of +8" or "Drive with a 'Justice' intensity of -2" (meaning, reject injustice).

The paper tested this on many different AI models and found:

Some models are easier to steer than others. Some respond well to your instructions, while others are stubborn.
Values fight each other. If you tell the AI to be "Very Kind" but also "Very Strict," it has to find a balance. The paper found that sometimes one value wins out, and sometimes they mix in predictable ways.
It works for groups. They used this to figure out what values different groups of people (like different political parties or cultures) hold, and then steered the AI to sound more like those groups. This made the AI's predictions about what those groups would say much more accurate.

The Big Picture

The authors built a complete system that:

Maps values into a structured space (HIVES).
Measures how strong a value is in a text by comparing it to a standard library (VIDB).
Steers the AI to express those values at specific strengths.

They aren't saying this will solve all AI safety problems or that we should use this to manipulate people. Instead, they are providing a scientific toolkit to study how AI behaves when we ask it to be "a little bit" of something versus "a lot" of something. It's a step toward making AI that can understand the nuance of human values, rather than just following a simple "good vs. bad" rulebook.

Technical Summary: VALUEFLOW

Problem Statement

Aligning Large Language Models (LLMs) with the diverse spectrum of human values remains a central challenge. Current preference-based methods often fail to capture deeper motivational principles, tending to capture surface-level or context-dependent choices rather than the enduring priorities that underpin consistent human behavior. Furthermore, existing value-based approaches suffer from three critical gaps:

Extraction: Methods often ignore the hierarchical structure of values, relying on static questionnaires or simple judgments that lack nuance across levels of abstraction.
Evaluation: Existing metrics typically measure the presence of a value rather than its calibrated intensity, often using dictionaries or coarse ratings that produce unstable comparisons across models.
Steerability: The extent to which LLMs can be reliably steered to express targeted values at specified intensities is not well characterized, particularly regarding multi-value control and dose-response behaviors.

Methodology: The VALUEFLOW Framework

To address these limitations, the authors introduce VALUEFLOW, a unified framework spanning extraction, evaluation, and steering with calibrated intensity control. The framework consists of three core components:

1. HIVES (Hierarchical Value Embedding Space)

HIVES is a unified representation mapper designed to capture the intra- and cross-theory structure of values.

Structure: It integrates heterogeneous value theories—specifically Schwartz's Theory of Basic Values (SVT), Moral Foundations Theory (MFT), Ross's Duties, and Human Rights—into a single high-dimensional space.
Training: The model undergoes a two-stage training process:
- Stage 1 (Intra-Theory): Uses hierarchical contrastive learning to align representations within a specific theory, pulling semantically related values closer based on their shared position in the hierarchy.
- Stage 2 (Inter-Theory): Aligns different theories using cross-theory anchors derived from concept pooling and curated user-friendly instances, creating shared semantic coordinates.
Output: A hierarchy-aware embedding space that disentangles value vectors while preserving local affinities.

2. VIDB (Value Intensity DataBase)

VIDB is a large-scale resource of value-labeled texts with intensity estimates, constructed to serve as a reference for evaluation and steering.

Construction: The database aggregates rankings from multiple LLMs using a Plackett–Luce (PL) model. Instead of asking models for absolute scalar ratings (which are unstable), the system presents windows of texts and asks models to rank them relative to a value definition.
Calibration: Latent intensity scores are estimated from these rankings and normalized to a consistent $[-10, 10]$ scale. A human-LLM verification loop flags and corrects potential outliers.
Scale: It covers 32 values across four theoretical frameworks, derived from 10K unique texts per value.

3. Ranking-Based Evaluator and Steering Protocol

The framework introduces a novel evaluation protocol and a steering mechanism:

Evaluation: To score a model's output, the system constructs a ranking window containing the output and $k-1$ anchor texts from VIDB. A judge LLM ranks these texts, and the output's intensity is estimated by optimizing the PL likelihood against the fixed anchor utilities. This approach yields stable, calibrated intensity scores.
Steering: The authors propose an end-to-end steering protocol that conditions generation on $(value, intensity)$ pairs. This is achieved via:
- Intensity Anchor Prompts: Explicit natural language cues (e.g., "strongly values," "slightly rejects").
- User-Text Prompts: Providing representative texts from VIDB that correspond to specific intensity bins.

Key Contributions

Unified Hierarchical Embedding (HIVES): The construction of a hierarchical value embedding space that unifies heterogeneous theories, enabling systematic study of value representation and disentanglement.
Ranking-Based Evaluation & VIDB: The proposal of a ranking-based evaluation method for value intensity, accompanied by the release of the large-scale Value Intensity Database (VIDB). This framework provides a stable, interpretable alternative to rating-based baselines, demonstrating lower deviation from human ratings (1.4 vs. 2.1–4.2) and higher win rates (up to 79%).
Steerable Pluralism: The extension of steerability to encompass both directional alignment and graded value intensity. This enables the analysis of "steerable value pluralism," allowing models to be conditioned on multiple values with specific intensities simultaneously.

Experimental Results

The authors conducted comprehensive large-scale studies across ten models (including Qwen3, Mistral, Phi-4, GLM-4, GPT-4.1, Claude-4, Grok-4, etc.) and four value theories.

Steerability Regimes: Models exhibit three qualitative regimes:
- Weakly Steerable: Models like Phi-4 and Claude-4 show limited controllability, especially for negative steering.
- Moderately Steerable: Models like Qwen3 and GPT-4.1 respond consistently but with value-dependent magnitudes.
- Strongly Steerable: Models like Grok-4, Gemma-3, and GLM-4 exhibit large shifts in both positive and negative directions.
Asymmetry and Composition:
- Dose-Response Asymmetry: Many values exhibit polarity-asymmetric behavior (e.g., responding well to positive targets but resisting negative ones).
- Composition Laws: In multi-value steering, similar values compose approximately additively, while opposed values exhibit trade-offs. Strong positive anchors often dominate, attenuating negative targets rather than reversing them.
Demographic Alignment: Using value profiles constructed from demographic data, the framework improves behavior-prediction accuracy on OpinionQA by over 10% for certain attributes (e.g., Phi-4 Religion accuracy increased from 44.5% to 57.4%) compared to attribute-only baselines.
Stability: The ranking-based evaluator significantly reduces model instability (variance, sign-flip rates) and prompt sensitivity compared to direct scalar rating.

Significance and Claims

The paper positions VALUEFLOW as the first end-to-end research stack for value-aware alignment. Its significance lies in:

Infrastructure: Establishing a scalable infrastructure for evaluating and controlling value intensity, moving beyond simple binary alignment to graded, pluralistic control.
Methodological Rigor: Providing a principled path to pluralistic alignment by grounding behavior in enduring motivational principles rather than transient preferences.
Practical Utility: Offering tools for personalized alignment (via value profiles) and cross-cultural deployment (via shared anchors).
Safety Insights: Linking steerability to safety by profiling refusal behaviors, revealing that models vary significantly in their ability to be steered away from harmful values without triggering refusals.

The authors conclude that while the framework makes trade-offs for interpretability and stability (e.g., modeling intensity as a 1D scalar), it provides common infrastructure for pluralistic audits and reproducible value-based alignment. They explicitly note that the framework is not designed to enforce a single value system but to analyze and compare pluralistic expressions.