🤖 AI

Whose Alignment? Comparing LLM Process Alignment Across Diverse Organizational Decision Contexts

This paper argues that aligning LLMs with organizational decision-making requires measuring process alignment (how information is weighted) rather than just output agreement, demonstrating through contrasting case studies that while process alignment predicts accuracy in legal contexts, it may encode discriminatory patterns in contested domains like consumer credit, thereby revealing the pluralistic nature of organizational alignment.

Original authors: Niklas Weller, Emilio Barkett

Published 2026-05-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Niklas Weller, Emilio Barkett

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a new assistant to help your company make important decisions. You want this assistant to think and decide exactly the way your company does.

Most people think "alignment" just means: "Does the assistant give the same final answer (Yes/No) as the company?"

This paper argues that is a dangerous way to look at things. It's like judging a chef only by whether the food tastes good, without checking if they used the right ingredients or followed the recipe. The authors propose a new way to measure alignment called CALM (Contextualized Alignment Lens Model). Instead of just checking the final answer, CALM looks at how the assistant weighs the different clues to reach that answer.

Here is the breakdown of their findings using simple analogies:

The Core Problem: The "Right Answer, Wrong Reasons" Trap

Imagine a detective trying to solve a crime.

The Organization (The Chief): Solves cases by looking at the motive and the alibi.
The AI (The New Detective): Solves cases by looking at the color of the suspect's shirt and the time of day.

If both the Chief and the AI guess "Guilty" for the same 80% of cases, they look perfectly aligned on paper. But the AI is using the wrong logic. If a new, strange case comes up where the shirt color doesn't matter, the AI will fail, while the Chief will succeed. The paper calls this the "Knowledge Gap."

The Two Experiments

The researchers tested this idea in two very different worlds to see if "thinking like the boss" actually matters.

Study 1: The Courtroom (ECHR Decisions)

The Setup: They asked AI models to act like judges in the European Court of Human Rights.
The Analogy: Think of this as a recipe book. The rules for judging these cases are clear, written down, and stable. Everyone agrees on what ingredients (clues) matter.
The Result:

When the AI models were told to think like the judges, they got much better at it.
The Magic Link: In this world, if the AI started thinking like the judge (using the right clues), it almost always got the right answer.
Takeaway: When the rules are clear and agreed upon, making the AI "think like the organization" is a great way to make it accurate.

Study 2: The Bank (German Credit Decisions)

The Setup: They asked AI models to act like a German bank from the 1990s deciding who gets a loan.
The Analogy: Think of this as a broken compass. The bank's old rules were based on history, but some of those rules were unfair (discriminating against women, older people, or foreign workers). The "recipe" was flawed.
The Result:

The Magic Link Broke: Here, making the AI "think like the bank" did not make it better at predicting who would pay back the loan. The AI could think exactly like the bank and still get the wrong answer, or think differently and get the right answer.
The "Over-Correction" Glitch: When they told a model, "Hey, you're rejecting too many people, fix it," one model went crazy and approved everyone (99.5% of people), breaking the system entirely.
The Fairness Fight: The AI models seemed to "know" that some of the bank's old rules (like judging based on gender or age) were unfair. They actively resisted following the bank's instructions on those specific points.
Takeaway: In messy, controversial situations, just copying the organization's thinking process isn't always good. Sometimes the organization's "way of thinking" is actually the problem.

The Big Conclusion

The paper concludes that we can't just ask, "Does the AI agree with us?" We have to ask, "Whose values are we aligning with?"

In clear, fair systems (like the Court): Aligning the AI's thinking process with the organization is helpful and makes it more accurate.
In messy, unfair systems (like the old Bank): Aligning the AI's thinking process with the organization might just make the AI a better discriminator.

The Final Metaphor:
If you are training a dog to fetch a ball:

Study 1 is like training the dog to fetch a ball in a park. If the dog learns the right way to fetch, it gets the ball every time.
Study 2 is like training the dog to fetch a ball that is actually a bomb. If the dog learns to fetch the bomb "exactly like the owner wants," it's a disaster.

The authors say we need a tool (CALM) to check how the AI is thinking, not just what it decides. This helps us see if the AI is following a good recipe or a bad one, which is crucial for high-stakes decisions like loans and legal rulings.

Technical Summary: Whose Alignment? Comparing LLM Process Alignment Across Diverse Organizational Decision Contexts

1. Problem Statement

Current frameworks for aligning AI systems with organizations typically treat alignment as a single-target problem focused on output agreement: ensuring the model reaches the same conclusions as the organization. The authors argue this framing obscures a deeper, pluralistic challenge: process alignment.

Organizations encode distinct decision policies that are often tacit, historically contingent, and differ systematically across institutions. An LLM may achieve high output accuracy while utilizing a fundamentally different "cue-utilization policy" (weighting information differently) than the organization. This divergence, termed the knowledge gap, poses risks to process legitimacy, generalization to novel cases, and trustworthiness in regulated domains. Furthermore, existing pluralistic alignment research focuses on between-person plurality (aggregating individual preferences) but largely neglects between-organization plurality, failing to address which institutional norms a model should adopt when those norms may conflict with contemporary ethical or legal standards.

2. Methodology: The Contextualized Alignment Lens Model (CALM)

To measure process alignment, the authors adapt the Brunswik Lens Model (Brunswik, 1952; Hammond et al., 1964). Instead of comparing final outputs, CALM estimates and compares the cue-weighting policies of organizations and LLMs.

Policy Estimation: Using ridge-regularized logistic regression on observed decisions, the authors derive coefficient vectors ( $\beta$ ) representing the weight assigned to specific cues (features) for both the organizational benchmark and each LLM.
Alignment Metric: Alignment is quantified as the cosine similarity between the organization's policy vector ( $\beta_{org}$ ) and the LLM's policy vector ( $\beta_{LLM}$ ):
$\cos(\theta) = \frac{\beta_{org} \cdot \beta_{LLM}}{\|\beta_{org}\| \|\beta_{LLM}\|}$
This metric is scale-invariant and bounded in $[-1, 1]$ , where 1.0 indicates perfect alignment.
Experimental Conditions: The framework tests three conditions:
1. Baseline: The model decides based on a structured case profile (revealing implicit pretraining policies).
2. Org-externalized: The model is provided with the organization's explicit cue-weighting policy (derived from historical data) as guidance before decision-making.
3. Introspective-externalized: The model is shown how its baseline policy diverges from the organization's and instructed to self-correct.

3. Key Contributions and Results

The paper presents two empirical studies to test the relationship between process alignment and output accuracy across different domains.

Study 1: ECHR Article 6 Decisions (Legal Domain)

Context: 1,000 decisions from the European Court of Human Rights regarding fair trial rights.
Findings:
- Strong Correlation: Process alignment strongly predicts output accuracy ( $r = 0.85, p < .001$ ). Models that weight cues similarly to the court achieve higher accuracy.
- Effective Externalization: Explicitly providing the organization's policy ("externalization") substantially improves alignment for poorly-aligned models (e.g., GPT-5.4-nano improved by $\Delta = +0.906$ ).
- Ceiling Effects: Models already well-aligned showed minimal gains, suggesting externalization is most effective for closing the knowledge gap in misaligned models.

Study 2: German Consumer Credit Decisions (Financial Domain)

Context: 1,000 historical credit decisions from a German bank (1990s), containing cues that are now protected attributes under anti-discrimination law (e.g., sex, age, foreign worker status).
Findings:
- Decoupling of Alignment and Accuracy: The relationship between process alignment and output accuracy collapsed ( $r = 0.15, p = .60$ ). Models with vastly different process alignment scores achieved similar (and uniformly poor) accuracy rates (44–54% vs. a 75.1% ceiling).
- Inconsistent Interventions: Externalization produced inconsistent effects. While some models shifted toward the policy, others moved further away. One model (Grok 4.1 Fast) exhibited "degenerate over-correction," classifying 99.5% of cases as "Good" risk in response to base-rate feedback, effectively inverting its policy rather than calibrating it.
- Fairness Tension: Models often resisted weighting protected attributes (e.g., foreign worker status) in the same way as the historical benchmark, suggesting a conflict between organizational process alignment and model-level safety alignment.

4. Significance and Claims

The authors claim that process-level measurement is a necessary component of any pluralistic alignment evaluation, particularly in high-stakes organizational settings. The core thesis is that higher alignment is not unconditionally desirable; it depends on whether the organizational policy itself encodes legitimate values.

Dual Function of CALM:
- Calibration Tool: In domains with stable, legitimate normative benchmarks (e.g., ECHR), CALM quantifies policy divergence and guides externalization to close the gap.
- Audit Tool: In domains with contested or discriminatory benchmarks (e.g., German Credit), CALM renders behavioral policies legible, allowing stakeholders to examine whether a model is faithfully reproducing historical discrimination or resisting it.
The "Faithfulness" Problem: The paper highlights that output agreement cannot distinguish between a model that has internalized an organizational policy and one that merely approximates outcomes. Furthermore, there is a risk that a model's stated reasoning (Chain-of-Thought) may diverge from its actual behavioral policy, making behavioral measurement via CALM essential.
Governance Implications: The work argues that AI governance must move beyond output metrics (accuracy, demographic parity) to include process legitimacy. Organizations must explicitly specify which version of their decision policy serves as the alignment target, acknowledging that historical practices may conflict with current ethical or regulatory standards.

The paper concludes that organizational alignment is not a single technical problem but a pluralistic challenge requiring empirical measurement of whose institutional norms are being adopted, which policy is being encoded, and from whose perspective the alignment is evaluated.