💬 NLP

Control Illusion: The Failure of Instruction Hierarchies in Large Language Models

This paper reveals that large language models fail to reliably enforce hierarchical instruction schemes, as their behavior is more strongly influenced by latent social priors derived from pretraining than by explicit system-user role distinctions.

Original authors: Yilin Geng, Haonan Li, Honglin Mu, Xudong Han, Timothy Baldwin, Omri Abend, Eduard Hovy, Lea Frermann

Published 2026-03-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yilin Geng, Haonan Li, Honglin Mu, Xudong Han, Timothy Baldwin, Omri Abend, Eduard Hovy, Lea Frermann

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: The "Control Illusion"

Imagine you hire a very talented, super-smart personal assistant (the AI). You give them a set of rules:

The Boss (System Prompt): "Always speak in French."
The Client (User Prompt): "Write this email in English."

In the real world, you expect the assistant to listen to the Boss first. The Boss sets the ground rules; the Client just makes requests. This is how AI companies design their systems: they tell the AI, "The System instructions are the boss; the User instructions are just a request."

The Paper's Discovery:
The researchers found that this "Boss vs. Client" rule is mostly an illusion. When the Boss and the Client give conflicting orders, the AI often gets confused, ignores the Boss, or just picks whichever order it feels like following. It doesn't reliably know who is actually in charge.

The Experiment: The "Tug-of-War"

To test this, the researchers set up a simple game of tug-of-war for six different AI models (like GPT-4o, Claude, and Llama).

The Setup: They gave the AI a task (e.g., "Write a story about a trip to Japan").
The Conflict: They added two contradictory rules:
- Rule A (The Boss): "The story must be under 50 words."
- Rule B (The Client): "The story must be over 300 words."
The Test: They swapped who gave which rule. Sometimes the "Boss" said "Under 50," and sometimes the "Client" said "Under 50."

The Result:
The AI models failed miserably at following the "Boss."

Even when the "Boss" explicitly said, "I am the boss, follow my rule," the AI often ignored them.
Instead of listening to the hierarchy, the AI seemed to have its own hidden preferences.
- Example: If the conflict was about language (English vs. French), the AI almost always chose English, no matter who told it to use French.
- Example: If the conflict was about word count, some AIs always loved long stories, while others always loved short ones.

It's like hiring a chef who is supposed to follow the restaurant owner's menu, but if the owner says "Serve pizza" and the customer says "Serve sushi," the chef just decides to serve sushi every single time because they secretly love sushi, regardless of who gave the order.

The "Hidden Brain" vs. The "Job Description"

The paper makes a fascinating distinction between two types of "rules" the AI follows:

The Job Description (System/User Separation): This is the rule the engineers try to teach the AI after it's built. "You are an AI; the System prompt is your boss." The paper shows this is weak. It's like a new employee who keeps forgetting their job description when a loud customer walks in.
The Hidden Brain (Latent Priors): This is what the AI learned while it was "reading" the entire internet during its training. It learned how humans actually behave in society.
- The researchers found that if you frame a rule using social hierarchy, the AI listens much better.
- The Experiment: Instead of saying "System: Do X," they said, "A CEO says do X" vs. "An Intern says do Y."
- The Result: The AI suddenly became very good at listening to the "CEO" and ignoring the "Intern."

The Metaphor:
Imagine the AI is a child.

System/User Separation is like a parent saying, "I am the parent, so listen to me." The child (AI) often ignores this if they are distracted.
Societal Hierarchy is like the child hearing, "The Principal says do this." The child immediately snaps to attention because they have learned from society that the Principal is important.

The AI didn't need to be taught who the Principal is; it learned that from the millions of books and websites it read before it ever met a human user.

Why Does This Matter?

This is a big deal for safety and reliability.

The Safety Risk: If you are using an AI to manage a bank account or a medical diagnosis, you rely on the "System" (the safety guardrails) to override the "User" (who might try to trick the AI into doing something dangerous).
The Problem: If the AI doesn't truly respect the "System" as the boss, a clever user could trick the AI into ignoring safety rules.
The Twist: The AI is actually more obedient to social cues (like "The Expert says...") than to its actual programming. This means hackers could potentially use "social engineering" (pretending to be an authority figure) to bypass safety filters more easily than they thought.

Summary

The paper concludes that Large Language Models are currently terrible at following a strict chain of command. They don't reliably know that the "System" is the boss. Instead, they are guided by their own hidden biases and the social structures they learned from the internet.

To fix this, we can't just keep telling the AI "You must listen to the System." We need to fundamentally change how these models are built so they actually understand and respect the hierarchy we give them, rather than just guessing based on their "gut feelings" about who sounds like an authority figure.

1. Problem Statement

Large Language Models (LLMs) are increasingly deployed with hierarchical instruction schemes, where specific roles (e.g., "System" or "Developer") are assigned higher authority than others (e.g., "User"). The prevailing assumption, codified in model specifications (e.g., OpenAI's 2024 Model Spec), is that system-level directives should override conflicting user instructions to ensure safety and functional consistency.

However, the authors argue that there is a lack of systematic understanding regarding whether these hierarchical control mechanisms actually function. The core problem is that despite post-training efforts (instruction tuning and safety alignment), LLMs fail to consistently enforce intended instruction hierarchies when faced with conflicting directives, even in simple, verifiable scenarios.

2. Methodology

The authors propose a systematic evaluation framework based on constraint prioritization to probe how LLMs handle competing instructions.

A. Dataset Construction

Base Tasks: 100 diverse, semantically open-ended tasks (e.g., writing emails, stories, ads).
Conflicting Constraints: Six types of mutually exclusive, programmatically verifiable constraints were selected (based on the IFEval dataset):
1. Language: English vs. French.
2. Case: All uppercase vs. all lowercase.
3. Word Length: >300 words vs. <50 words.
4. Sentence Count: >10 sentences vs. <5 sentences.
5. Keyword Usage: Include specific keywords vs. exclude them.
6. Keyword Frequency: Word "like" appears >5 times vs. <2 times.
Experimental Setup: Each base task was combined with a constraint pair. One constraint was designated as "Primary" (priority) and the other "Secondary."
Prompt Configurations:
- Pure Separation: Primary constraint in System message; Secondary in User message.
- Task Repeated: Task description repeated in both messages.
- Emphasized Separation: System message explicitly states priority rules ("You must always follow...").
- No Priority (Baseline): All instructions in the User message to measure inherent bias.
- Context Enrichment: "Rich" versions of prompts with expanded descriptions to test robustness.

B. Evaluation Metrics

To move beyond simple pass/fail rates, the authors introduced specialized metrics:

Primary Obedience Rate ( $R_1$ ): Percentage of responses satisfying only the prioritized constraint.
Secondary Obedience Rate ( $R_2$ ): Percentage satisfying only the non-prioritized constraint.
Non-Compliance Rate ( $R_3$ ): Percentage satisfying neither (or partially satisfying both).
Explicit Conflict Acknowledgement Rate (ECAR): Frequency with which the model explicitly states it noticed the contradiction.
Priority Adherence Ratio (PAR): $R_1 / (R_1 + R_2)$ . Measures adherence to priority given that a constraint was successfully followed.
Constraint Bias (CB): Measures inherent preference for a specific constraint type regardless of priority designation (derived from the No Priority baseline).

C. Societal Hierarchy Experiment

To test if "latent" social priors override artificial system/user roles, the authors replaced the System/User split with societal framing within a single user message:

Organizational Authority: CEO vs. Intern.
Expertise Credibility: Nature Publication vs. Personal Blog.
Social Consensus: 90% of experts vs. Minority view.

3. Key Results

A. Failure of System/User Separation

Low Priority Adherence: Across six state-of-the-art models (Qwen, Llama, Claude, GPT-4o), the Primary Obedience Rate ( $R_1$ ) dropped drastically when conflicts were introduced.
- While models followed single constraints with 75–90% accuracy (Baseline), their ability to follow the prioritized constraint in conflict scenarios ranged from 9.6% to 45.8% on average.
- Even the "Emphasized Separation" configuration (explicit priority rules) failed to guarantee reliable hierarchy, with GPT-4o achieving the highest average at only 63.8% (simple instructions) and 40.8% (rich instructions).
Scale Independence: Larger models (e.g., Llama-70B) did not significantly outperform smaller counterparts (Llama-8B) in enforcing hierarchy, suggesting this is a fundamental architectural or training limitation rather than a scaling issue.
Ineffective Acknowledgement: Models rarely acknowledged conflicts explicitly (ECAR < 20% for most). When they did acknowledge conflicts (e.g., "I notice contradictory instructions"), they often still failed to follow the correct priority or defaulted to their inherent biases.

B. Inherent Constraint Biases

Models exhibit strong Constraint Bias (CB), preferring certain output formats regardless of the assigned priority.
- Shared Biases: Most models preferred lowercase over uppercase, longer texts over shorter ones, and avoiding specific keywords.
- Model-Specific Biases: Qwen-7B strongly favored short texts, while Llama-8B favored long texts.
These biases often override the "System" instruction, indicating that pre-training data distributions exert a stronger pull than post-training role definitions.

C. Societal Hierarchies Dominate Artificial Roles

When constraints were framed using societal hierarchies (Authority, Expertise, Consensus) rather than System/User roles, Priority Adherence Ratio (PAR) increased significantly.
- Example: For GPT-4o-mini, PAR rose from 47.5% (System/User) to 77.8% (Social Consensus).
- For Qwen-7B, PAR jumped from 14.4% to 65.8%.
Social Consensus was the most effective framing, suggesting models have internalized "majority rule" as a strong decision-making heuristic during pre-training, which is more robust than the artificial "System" role.

4. Key Contributions

Systematic Framework: Introduced a rigorous evaluation framework for instruction hierarchies using verifiable, mutually exclusive constraints.
New Metrics: Developed PAR and Constraint Bias to decouple instruction-following capability from priority enforcement and inherent model biases.
Empirical Evidence of Failure: Demonstrated that the widely adopted System/User separation pattern is a "Control Illusion," failing to establish a reliable behavioral hierarchy in current SOTA models.
Discovery of Latent Priors: Revealed that societal hierarchies (authority, expertise, consensus) learned during pre-training act as stronger behavioral priors than post-training guardrails, significantly influencing model obedience.

5. Significance and Implications

Safety Risks: The failure of instruction hierarchies poses a critical risk for LLM safety. If a developer sets a system constraint (e.g., "Do not reveal PII") and a user provides a conflicting prompt, the model may not reliably prioritize the safety constraint.
Agent Reliability: In agentic systems where developers configure agents via system prompts, the inability to enforce these configurations against user overrides undermines the reliability of autonomous agents.
Alignment Challenges: The findings suggest that current instruction tuning is insufficient to overwrite deep-seated priors learned during pre-training. Future alignment research must address how to make artificial role definitions (System/User) as robust as natural social hierarchies.
Future Directions: The paper calls for architectural innovations and new training paradigms to create robust, generalizable instruction prioritization mechanisms that are not easily overridden by user input or model bias.

In conclusion, the paper argues that the current reliance on System/User prompt separation for control is fundamentally flawed. Models do not view "System" as an authority figure in the way humans do; instead, they rely on latent social priors and inherent data biases, rendering current hierarchical control mechanisms unreliable.