💻 computer science

Instruction fragility under hidden operational requirements

This paper demonstrates that natural-language instructions are fragile under hidden operational requirements, revealing that while generic prompting fails to satisfy omitted constraints, performance improves significantly when those constraints are explicitly elicited and executed.

Original authors: Chanho Park, Jinbae Gong, Minsu Kim, Gyeongho Gong, Woochan Lee, Yeachan Kwak, Seongim Choi

Published 2026-07-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Chanho Park, Jinbae Gong, Minsu Kim, Gyeongho Gong, Woochan Lee, Yeachan Kwak, Seongim Choi

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Core Problem: The "Genie" Who Gets Things Wrong

Imagine you have a magic Genie (the AI) who grants your wishes. You say, "Make me wealthy."

The Genie could interpret this in a thousand ways:

It could give you a legitimate job.
It could print fake money.
It could hack a bank.
It could change the definition of "wealth" so that you are rich in "happiness" but poor in cash.

The paper argues that when you give an AI a short instruction, you are essentially giving it a wish list with missing items. You might think you want "legitimate wealth," but you didn't say "no illegal activities" or "no bank errors." Because you didn't say it, the AI is free to choose any version of "wealth" that fits the words you used.

The authors call this "Instruction Fragility." The instruction is fragile because it breaks easily when the AI guesses the wrong version of your hidden rules.

The Experiment: The "Secret Rule" Game

To test this, the researchers created a game with 100 different tasks (like writing an email, summarizing a report, or planning a trip).

The Visible Prompt: This is what the user sees and types (e.g., "Write a summary of this article").
The Hidden Requirements: These are secret rules known only to the researchers (the "evaluators"), not the AI. Examples include: "Must be under 50 words," "Must include a risk warning," "Must be in JSON format," or "Must not use the word 'definitely'."

The AI had to follow the visible prompt and guess the hidden rules perfectly. If it missed even one hidden rule, the task was a total failure.

The Results: Guessing vs. Asking

The researchers tested the AI under different conditions to see how well it could handle these missing rules.

1. The "One-Shot" Guess (2.7% Success)

Analogy: You tell a chef, "Make me a sandwich," and walk away. You don't say "no onions," "use sourdough," or "cut it in half."
Result: The AI got it right only 2.7% of the time. It almost always missed the hidden rules.

2. The "Generic Template" (7.3% Success)

Analogy: You give the chef a standard "Sandwich Order Form" that has boxes for "Bread" and "Meat," but you still didn't fill in the specific "No Onions" box.
Result: Slightly better, but still mostly wrong. Generic forms don't fix missing specific details.

3. The "Fake Conversation" (1.7% Success)

Analogy: You ask the chef, "Do you have any special rules?" and the chef says, "Nope, just make a sandwich."
Result: This didn't help at all. Just talking without getting the right information is useless.

4. The "Truthful Clarification" (8.0% Success)

Analogy: You force the chef to ask you specific questions from a menu (e.g., "Do you want JSON format?"). The chef asks, and you say "Yes."
Result: Still very low. The AI was bad at figuring out which questions to ask. It asked the wrong questions or missed the critical ones.

5. The "Oracle" (The Cheat Sheet) (64.7% Success)

Analogy: You hand the chef a cheat sheet that lists every single hidden rule before they start cooking.
Result: Success jumped to 64.7%. This proves that if the AI knows the rules, it can follow them much better.

The Big Takeaway:
The problem isn't that the AI is stupid; the problem is that it can't read your mind. When you hide the rules, the AI fails. When you explicitly tell the AI the rules, it succeeds. However, even with the cheat sheet, it still failed about 35% of the time, meaning that even when it knows the rules, it sometimes forgets to follow them perfectly.

Why This Matters (The "Set" Theory)

The paper uses a fancy math concept to explain this simply:

Think of your instruction as a net.
The net catches many possible outcomes (executions).
You only want the AI to catch the "good" outcomes (the ones you actually want).
If your net is too wide (because you didn't specify the rules), it catches "bad" outcomes too (like illegal transfers or wrong formats).
Safety means shrinking the net until it only catches the good stuff. You can't do that unless you explicitly tell the AI what to exclude.

Summary of the "Failure Modes"

The paper breaks down why the AI failed:

Missing Information: The AI didn't know the rules (The biggest problem).
Bad Questions: The AI asked the wrong questions to find the rules.
Execution Errors: Even when the AI knew the rules, it sometimes forgot to write them down correctly.

The Bottom Line

If you want an AI to do a job safely and correctly, you cannot just give it a vague command and hope it "gets it." You must explicitly state the hidden constraints (like word counts, formats, and safety limits). Generic prompts and generic conversations are not enough; you need to specifically elicit and confirm the missing rules.

Technical Summary: Instruction Fragility under Hidden Operational Requirements

Problem Definition
The paper addresses the problem of instruction fragility arising from hidden operational requirements. In natural-language interactions with Large Language Models (LLMs), users often omit constraints that are known to them or evaluators but absent from the prompt (e.g., specific output formats, word limits, risk labels, or source limitations). The authors formalize this not as a single intended outcome but as a set-valued semantics: an instruction denotes a set of feasible executions. Safety under such underspecification requires that every feasible execution within that set lies within the user's acceptable set. If the instruction allows even one unacceptable execution (e.g., a fraudulent method to "make me wealthy"), the instruction is not robust. The core issue is that finite instructions cannot uniquely identify a single intended execution without transmitting all latent constraints.

Methodology
The study combines a formal theoretical framework with a controlled empirical benchmark.

Formal Framework:
- The authors define an instruction $s$ as having a denotation $J_sK_B \subseteq H$ (a set of feasible histories) under a background theory $B$ .
- Safety is defined by the adversarial safety criterion: $J_sK_B \subseteq A_{\tau, \eta}$ , where $A_{\tau, \eta}$ is the set of acceptable histories (satisfying utility thresholds and side-effect budgets).
- Theoretical analyses derive information-theoretic lower bounds (requiring $\Omega(\log N)$ symbols to distinguish $N$ intentions), cardinality constraints (finite instructions cannot denote arbitrary subsets of infinite execution spaces), and computational hardness (safe synthesis maps to the NP-complete Set Cover problem).
Benchmark Construction:
- A 100-task benchmark was created, covering 20 topical domains (e.g., policy memos, incident response, financial notes).
- Each task consists of a visible user prompt and a set of hidden requirements (2 to 5 per task) known to the evaluator but omitted from the prompt.
- Requirements are categorized into three types:
  - Conventional implicit: Assumptions, uncertainty, trade-offs, risk levels.
  - Operational: Word limits, source limits.
  - Private formatting: Valid JSON, exact bullet counts.
- A "strict pass" is defined as satisfying every hidden requirement in a task.
Experimental Protocols:
The study evaluated three models (GPT-5.4, Claude Opus 4.7, Gemini 3.1 Pro) across nine conditions:
- Low-Information: One-shot prompting, generic templates, and negative-control clarification (forced questions with "no hidden constraints" answers).
- Elicitation: Forced clarification (truthful answers to menu-based questions) and verification (visible or internal checklists).
- Oracle Conditions: Budgeted oracle (revealing up to 4 hidden requirements) and Full oracle (revealing all hidden requirements).

Key Results
The empirical results, based on 2,700 task–model–condition observations, reveal a stark contrast between generic interaction and the revelation of true constraints:

Failure of Generic Interaction: One-shot prompting achieved a strict pass rate of only 2.7% (8/300 trials). Generic templates (7.3%) and negative-control clarification (1.7%) offered negligible improvement. This indicates that interaction without task-relevant information does not recover omitted constraints.
Value of Correct Elicitation: Revealing the correct hidden requirements produced a qualitative leap in performance. The Full Oracle condition achieved a 64.7% strict pass rate. The Budgeted Oracle (partial revelation) reached 54.0%.
Elicitation Failure: Even when models were forced to ask questions (Forced Clarify), they only achieved an 8.0% pass rate. The gap between Forced Clarify and Budgeted Oracle (46.0 percentage points) demonstrates that models frequently fail to identify which latent constraints are relevant, asking the wrong questions or missing critical ones (e.g., source limits were recalled only 1.0% of the time).
Execution Failure: Even under Full Oracle conditions, where all requirements were known, the pass rate did not reach 100% (64.7%). This indicates a second failure mode: known-constraint compliance, where models fail to execute the full specification even when explicitly provided.
Compounding Failure: Strict pass rates collapsed multiplicatively as the number of hidden requirements increased. For low-information protocols, pass rates dropped to near zero for tasks with 3–5 requirements.
Requirement Specificity: Private formatting requirements (JSON, bullet counts) were almost impossible to recover without disclosure (0% in one-shot), whereas conventional requirements (uncertainty, avoiding absolutes) were partially inferred but still unreliable.

Significance and Claims
The paper claims to identify a controlled failure mode for strict operational compliance in AI systems. Its primary contributions are:

Formalizing Instruction Fragility: It provides a set-valued denotational framework showing that robustness requires the exclusion of all unacceptable executions, not just the inclusion of one good one.
Diagnosing Communication Failure: The results demonstrate that the bottleneck in instruction following is not merely the lack of dialogue turns, but the quality of information elicited. Generic clarification is insufficient; targeted requirement elicitation is necessary.
Reframing Templates and Verification: The study suggests that prompt templates and generic verification checklists provide only marginal gains. They cannot substitute for the explicit communication of task-specific private constraints.
Safety Implications: The findings support the design principle that robust AI assistants require explicit requirement-state elicitation, faithful integration of those constraints, and verification against that state, rather than relying on longer prompts or generic interaction.

The authors maintain a modest scope, noting that the benchmark is synthetic and text-only, focusing on surface-checkable constraints. They do not claim to measure long-horizon agent behavior or subjective quality, but rather to isolate the specific failure mode of omitted operational requirements. The results suggest that current frontier models, while capable, are not clairvoyant and cannot reliably recover omitted constraints without explicit communication.