💬 NLP

PALMs: Using Multi Construct-Grounded Rationales for Modeling Population Preferences in LLMs

This paper introduces Population Aligned Language Models (PALMs), a suite of models aligned to specific national populations by leveraging multi-construct-grounded rationales for preference tuning, which significantly outperforms existing baselines in simulating diverse cultural values, beliefs, and social reasoning without requiring task-specific supervision.

Original authors: Priyanka Dey, Brihi Joshi, Preyashi Poddar, Jieyu Zhao, Emilio Ferrara

Published 2026-08-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Priyanka Dey, Brihi Joshi, Preyashi Poddar, Jieyu Zhao, Emilio Ferrara

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to understand people. Right now, most robots are like generalist chefs who have tasted a little bit of everything but don't really know how to cook a specific family's favorite meal. They might know what "food" is, but they don't quite get why a grandmother in Italy might prefer a slow-cooked ragù while a teenager in Brazil might crave a spicy feijoada. This is the world of Large Language Models (LLMs)—super-smart computer programs that can write, chat, and solve problems. But here's the tricky part: these robots often act like they are one single, average person. They miss the deep, invisible rules that make different groups of people think and feel differently.

To fix this, scientists look at "preferences"—what people like, believe, and value. They know that our choices aren't just random; they are shaped by our personalities (are we adventurous or cautious?), our culture (do we value the group or the individual?), and our moral compass (what feels fair or right?). The big question researchers are asking is: How do we teach a robot to stop acting like a generic "average human" and start acting like a specific person from a specific place, with all their unique quirks and cultural background? If we can do that, these robots could be much better at helping us, understanding us, and even simulating how real people might react to new ideas.

This is where a new team of researchers steps in with a clever solution called PALMs (Population Aligned Language Models). Think of a standard robot as a student who tries to memorize the answers to a test by just looking at the final grade. They might get the right answer, but they don't really understand why it's right. The researchers behind PALMs realized that to truly understand a population, the robot needs to learn the "secret sauce" behind the answers. They didn't just feed the robot data about what people said; they taught it to write a "reasoning diary" before giving an answer.

Here is how they did it: They created five special robot versions, one for each of five countries: the USA, India, Brazil, France, and Italy. But instead of just telling the robot, "You are American," they gave it a much deeper toolkit. They taught the robot to think using five specific lenses, like a pair of special glasses that change the view:

Personality: Is the person open to new ideas or do they like routine?
Culture: Do they value independence or sticking together as a group?
Values: What drives them? Is it freedom, safety, or helping others?
World Beliefs: Do they see the world as a safe, exciting place or a dangerous, chaotic one?
Morality: What do they think is fair or sacred?

The robot was trained to write a short story using these five lenses to explain why a person from that country would choose one answer over another. For example, if a Brazilian person prefers a certain type of music, the robot doesn't just say "They like it." It writes a little note saying, "Because this person values community and sensory enjoyment (Culture and Values), and sees the world as a place full of fun opportunities (World Beliefs), this music feels right." The robot practiced this over and over, learning to hide these "reasoning notes" inside its brain so it could use them automatically later.

The results were surprisingly good. When the researchers tested these new robots, they found that PALMs were much better at understanding people than the old, generic robots. In fact, across all five countries, the new models improved their accuracy by an average of 8.59% compared to the best previous methods. This suggests that grounding a robot's learning in deep psychological and cultural theories works much better than just showing it surface-level data or simple demographic labels like "age" or "gender."

The paper also checked if these robots could do other things, like acting as a "personalized reward model" (deciding what a specific person would like) or simulating how a group might react to a new policy. In these tests, PALMs again outperformed the competition, showing improvements of 5.19% in personalized rewards and 6.34% in population simulations. Even when asked to solve social puzzles about human emotions and intentions (a task called Social IQA), the robots got better, reaching 80.23% accuracy on the test.

However, the researchers are careful to note that this isn't a magic fix-all. They found that simply feeding the robot more survey data without this "reasoning diary" approach actually made things worse in some cases, causing the robot to lose its ability to see different perspectives. They also admit that treating a whole country as one single group is a simplification; inside every country, there are many different regions and cultures that their model doesn't fully capture yet. But the main takeaway is clear: if you want a robot to truly understand a group of people, you have to teach it to think like them, not just talk like them. By giving the robot a structured way to reason through personality and culture, we get a much clearer picture of the diverse human world.

Technical Summary: Population Aligned Language Models (PALMs)

Problem Statement

Large Language Models (LLMs) are increasingly used to simulate individual user behavior, yet they struggle to faithfully represent the systematic variation in values, beliefs, and cultural norms that distinguish one population from another. Existing approaches typically rely on shallow demographic prompting (e.g., "act as a 30-year-old from Brazil") or direct fine-tuning on behavioral survey data. The paper argues that these methods fail because they optimize for surface-level response distributions rather than capturing the underlying psychological and cultural constructs that jointly shape human judgment. Consequently, models often over-represent majority viewpoints or collapse into generic representations when fine-tuned directly on observed responses, failing to preserve the rich compositional structure of human preferences.

Methodology

The authors introduce Population Aligned Language Models (PALMs), a framework that aligns LLMs to specific populations by grounding preference learning in multi-construct rationales. The methodology operates on the premise that human preferences emerge from the interaction of individual-level psychology and group-level cultural variation.

1. Construct Space

The approach utilizes five established families of social science theory to form a latent space of reasoning:

Personality (OCEAN): Individual traits (Openness, Conscientiousness, Extraversion, Agreeableness, Neuroticism).
Cultural Dimensions (Hofstede): Group-level tendencies (e.g., Individualism vs. Collectivism, Power Distance).
Human Values (Schwartz): Motivational priorities (e.g., Self-Direction, Security, Benevolence).
Primal World Beliefs (Clifton): Individual assumptions about the world (e.g., Safe vs. Dangerous, Enticing vs. Dull).
Moral Foundations (Haidt): Ethical intuitions (e.g., Care/Harm, Fairness/Unfairness, Loyalty/Betrayal).

2. Training Pipeline

PALMs are trained using a two-stage process on the Community Alignment dataset, covering five countries: USA, India, Brazil, France, and Italy.

Stage 1: Supervised Fine-Tuning (SFT): A base model (Llama-3.1-8B-Instruct) is fine-tuned on persona-conditioned preference instances $(p, q, a^+)$ to initialize preference prediction.
Stage 2: Preference Optimization with Latent Supervision: The core innovation involves Direct Preference Optimization (DPO) augmented with construct-grounded rationales.
- For each preference pair, synthetic rationales are generated by an external LLM (Qwen2.5-7B-Instruct). These rationales decompose the preference decision across the five construct families, explaining why a specific demographic profile would prefer $a^+$ over $a^-$ .
- During DPO, these rationales are inserted into the prompt context but masked from the loss function. The model is not trained to generate the rationale tokens explicitly; instead, the rationales serve as structured, latent inductive bias to shape the model's internal representations of preference.
- This contrasts with "generic" rationales (without construct grounding) or direct behavioral fine-tuning, which the authors argue lack the necessary structural signal.

3. Inference

At inference time, the model is prompted with a user persona and a query. It generates a construct-grounded rationale internally (before producing the final output) to guide its preference decision. The rationale tokens are not required as input during inference, allowing the model to generalize without external annotation.

Key Contributions

PALMs Framework: The introduction of a suite of models aligned to specific populations using multi-construct rationales grounded in psychological and cultural theory as latent supervision.
Superiority of Construct-Grounded Rationales: Empirical evidence demonstrating that grounding preference learning in psychological and cultural constructs outperforms both demographic prompting and behavioral fine-tuning (including culture-specialized models like CultureLLM).
Generalization to Downstream Tasks: Demonstration that PALMs generalize to downstream applications—personalized reward modeling, population simulation, and social reasoning—without task-specific supervision, suggesting that construct-grounded representations capture transferable latent structures of human diversity.

Results

The authors evaluated PALMs across four dimensions: personality, values/beliefs, cultural norms, and morality, using ground-truth human distributions from the target populations.

In-Domain Alignment: PALMs consistently outperformed all baselines (including zero-shot Llama-3.1-8B-Instruct, Gemini-2.5-Flash, and CultureLLM) across all five populations. The full PALM model achieved an average 8.59% relative improvement over the best baseline. Notably, the "SFT + DPO + Generic" configuration (using rationales without construct grounding) performed better than standard DPO but worse than PALMs, isolating the specific contribution of the psychological constructs.
Downstream Performance:
- Personalized Reward Modeling: PALMs improved accuracy by 5.19% on PersonalizedRewardBench and PRISM compared to baselines, with significant gains in society-related topics (religion, law, politics).
- Population Simulation: On SIMBENCH, PALMs achieved the best alignment on 15 out of 17 datasets, with particularly large gains (up to ~20%) on tasks aligned with the construct families (e.g., OpinionQA, OSPsychMACH).
- Social Reasoning: In a case study on SOCIALIQA, PALMs achieved 80.23% accuracy on the test set, outperforming the strong Gemini-2.5-Flash baseline (79.72%) despite being a smaller 8B model, with the largest gains in reasoning about human motivations.

Significance and Claims

The paper claims that rationalizing through established psychological and cultural constructs provides a richer inductive signal for preference learning than demographic conditioning or survey-based training data alone.

The authors argue that directly fitting models to observed behavioral outputs often leads to a collapse of representational diversity, whereas grounding the learning process in latent constructs allows the model to internalize the "why" behind preferences. This approach enables the creation of population-specific models that are not merely statistical mimics of survey data but are structurally aligned with the psychological mechanisms driving human judgment. The work advocates for the broader adoption of social science theory as a structured inductive bias in alignment research, offering a principled foundation for modeling populations in a multilingual and fine-grained manner.

The authors acknowledge limitations, including the use of country-level abstraction as a proxy for populations (which flattens within-country variation), the non-exhaustive nature of the selected construct families, and the current reliance on English-only training data. They posit that future work should explore finer-grained population definitions and native multilingual modeling.