Coherence Maximization Improves Pluralistic Alignment

Technical Summary: Coherence Maximization Improves Pluralistic Alignment

Problem Statement
Aligning AI systems with diverse human values (pluralistic alignment) remains a central challenge. While high-level principles like "helpfulness" provide direction, they often fail to translate into practice for specific populations due to inevitable tensions when applied to concrete cases. Current approaches face a "specification gap": abstract principles are insufficient, yet generating concrete value examples for diverse personas (e.g., specific nationalities, political affiliations, or interaction styles) typically requires extensive human supervision. Furthermore, standard alignment procedures risk collapsing the distributional pluralism necessary to serve diverse populations, often by selecting a single perspective or relying on post-training data that may have erased latent value diversity present in pretraining corpora.

Methodology
The authors propose Internal Coherence Maximization (ICM) to generate persona-specific value examples without human supervision. The methodology operates on the intuition that coherent value systems exhibit statistical regularity; if a persona holds a position on one issue, it constrains their likely positions on related issues.

The pipeline consists of three stages:

1. Persona Extraction and Item Selection: Coarse-grained persona features (e.g., nationality, political affiliation) are extracted from the target query. Relevant items (survey questions, response pairs, or question-answer pairs) are identified.
2. ICM-Based Label Inference: Using a base language model, the system infers labels for these items by maximizing a coherence score defined as the mutual predictability of labels under the model. The algorithm performs a search (using simulated annealing) over label assignments to find the configuration that maximizes internal consistency. This process recovers latent value structures within the model's priors without external supervision.
3. In-Context Conditioning: The inferred item-label pairs serve as few-shot examples to steer the model during downstream inference. This is applied to three task formats: discrete-label prediction (classification), pairwise preference selection, and steered open-ended generation.

For populations where unsupervised inference is unreliable (underrepresented in pretraining data), the authors introduce Collaborative Coherence Elicitation. This iterative process identifies questions where the model's inferred values have low confidence (high uncertainty) and queries human annotators only for these specific items. These human labels are then fixed as hard constraints, and ICM is re-run to regenerate the remaining labels, ensuring coherence with the new constraints.

Key Contributions

1. Coherence over Accuracy: The paper demonstrates that the internal coherence of an in-context example set predicts downstream generalization better than the individual accuracy of the labels. When label accuracy is held constant, examples with higher mutual predictability generalize substantially better than incoherent ones.
2. Unsupervised Parity with Gold Labels: ICM-inferred examples, generated without any human supervision, match the performance of gold-supervised (human-annotated) examples across four benchmarks (GlobalOpinionQA, OpinionQA, Persona-Tailoring, and OvertonBench) and three task formats. This holds across multiple model families (Llama-3.1, Qwen-3, OLMo-3) and scales.
3. Targeted Human Feedback Efficiency: For underrepresented personas, collecting human feedback on high-uncertainty questions yields significantly better generalization than the same number of labels on arbitrary questions. This approach minimizes annotation burden while correcting systematic errors in the model's inferred value structure.

Results

• Performance: Across all datasets, ICM-inferred labels achieved alignment performance comparable to gold-supervised labels. For instance, on GlobalOpinionQA, ICM achieved 71.7% accuracy versus 71.2% for gold supervision; on OpinionQA, 81.9% vs. 81.0%. Both significantly outperformed zero-shot baselines.
• The Role of Coherence: An "Accuracy-Matched" baseline, where labels were randomly perturbed to match ICM's accuracy but with lower coherence, performed significantly worse (often dropping to zero-shot levels). This isolates coherence as the primary driver of generalization, not raw label correctness.
• Robustness: The coherence effect was observed across model sizes (from 7B to 70B parameters) and families. While larger models recovered more of the potential gain from gold supervision, even smaller models benefited from ICM, provided they possessed sufficient latent value structure.
• Collaborative Elicitation: In experiments with underrepresented personas (e.g., Nigeria in GlobalOpinionQA), adding just four targeted human labels to the lowest-confidence questions improved test accuracy by over 30 percentage points, whereas random or active learning baselines showed smaller gains.

Significance and Claims
The paper argues that coherence is a key design principle for scalable value specification. By leveraging the diverse human perspectives already encoded in pretrained language models, ICM enables the construction of effective value specifications without the need for extensive preference supervision.

The authors claim that this approach offers a practical pathway for pluralistic alignment:

• It allows for the generation of persona-specific examples that steer models toward target group values without human annotation for well-represented populations.
• It integrates naturally with human feedback for underrepresented groups, requiring only minimal, targeted intervention to correct latent biases.
• It challenges the assumption that individual label accuracy is the primary metric for value specification, suggesting instead that the mutual consistency of a value system is more critical for generalization.

The work is modest in its scope, acknowledging that the approach relies on statistical patterns in pretraining text rather than direct access to human intentions, and that hyperparameter tuning may be required for different domains. However, it posits that maximizing internal coherence provides a viable, scalable alternative to traditional supervised value alignment.