💻 computer science

LLMs Know the Constraint But Do Not Use It: Activation Bottlenecks in Pragmatic Constraint Reasoning

This paper argues that LLMs' failure to adhere to implicit feasibility constraints stems not from a lack of knowledge, but from a routing bottleneck where the encoded constraint is not consistently activated during decision-making, a limitation that prompt-based interventions cannot overcome but targeted activation patching can partially repair.

Original authors: Yubo Li, Ramayya Krishnan, Rema Padman

Published 2026-08-14

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Yubo Li, Ramayya Krishnan, Rema Padman

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a super-smart robot how to make decisions. You give it a massive library of books so it knows almost everything about the world. But then, you ask it a tricky question: "I want to wash my car, and the car wash is 50 meters away. Should I walk or drive?" A human would instantly realize, "Wait, if I drive, I need my car to be there to drive it!" But the robot, despite having read every book about cars and washing, might confidently say, "Drive!" because it sees the word "car" and the word "drive" and thinks, "Oh, that's a match!"

This paper dives into a specific corner of artificial intelligence called Large Language Models (LLMs). These are the brainy AI systems that chat, write stories, and solve problems. The big mystery researchers are trying to solve is the difference between knowing something and using it. Think of it like a student who has memorized the entire rulebook for a game but, when playing, keeps forgetting the rules because they get distracted by the shiny pieces on the board. Scientists have long suspected that these AIs might "know" the hidden rules of logic but fail to apply them when the situation gets tricky. This matters because if we want AI to be safe and reliable in the real world—like helping doctors or driving cars—we need to know if it's actually thinking or just guessing based on surface-level clues.

The researchers behind this study decided to play detective with 14 different AI models to figure out exactly what's going on in their "brains" when they make these mistakes. They set up a clever experiment using a "quartet" of scenarios. Imagine four versions of the same story: one where the hidden rule is there (the car is missing), one where the rule is removed (the car is magically there), one where the rule is shouted out loud ("Remember, the car is missing!"), and one where they just add a random sentence of the same length to see if the AI just likes long stories.

Here is the twist they discovered: The AI isn't stupid; it's just distracted. But more importantly, the researchers found that the AI's "distraction" happens in two very different ways.

When the researchers looked inside the AI's "brain" (its hidden layers of data), they found that the AI does know the rule. They could build a simple detector that read the AI's internal thoughts and said, "Yes, the car is missing!" with over 88% to 94% accuracy. The knowledge was right there, sitting in the middle of the model's processing, just like a fact sitting on a desk.

However, the problem is that the AI doesn't use that knowledge when it's time to make the final decision. It's like having a GPS that knows the road is closed, but the driver (the decision-making part of the AI) ignores the GPS and keeps driving anyway. The researchers call this a routing problem. The information exists, but the path to the answer is blocked or ignored.

Crucially, they discovered two distinct types of robots failing in different ways:

The "Over-Active" Robots: Some models (like Llama-4 and Claude Opus) are so eager to be safe that they apply the "constraint-heavy" answer (e.g., "Walk") even when the constraint is actually absent. If you remove the car from the story, they still say "Walk" because they are biased toward the "safe" answer. They know the rule, but they can't turn it off when they should, leading them to fail the logic test by applying a rule that doesn't exist.
The "Under-Active" Robots: Other models (like GPT-OSS) fail to apply the rule even when it is there. They see the missing car but still say "Drive." They know the rule is there, but their internal wiring is broken so badly that the knowledge never reaches the decision button at all.

They tested this by doing something called "activation patching." Imagine you could take the "thought" from a moment when the AI did remember the rule (like when you shouted it out loud) and paste it directly into the moment when it forgot the rule. For the balanced models (like Qwen3-14B), this magic paste worked perfectly! The AI suddenly got the answer right, proving the knowledge was there and just needed a nudge to get to the decision. But for the under-active models, the paste did nothing. Even though they knew the rule, their internal wiring was broken in a way that prevented that knowledge from ever reaching the decision button. (Note: The researchers did not test this patching on the "over-active" group, so we don't know if the paste would fix them).

The paper also tested the most popular ways people try to fix AI mistakes, like asking the AI to "think step-by-step" or "list the prerequisites." The results were surprising: None of these tricks actually fixed the problem. Instead, they made the AI more likely to guess the "safe" answer (like "walk") even when it shouldn't. It turns out that when you tell an AI to "think harder," it just gets louder about mentioning the rule, but it still doesn't know when to apply it. It's like a student who, when told to "show their work," writes down the rule on every single line of their homework, even the ones where the rule doesn't matter.

In short, the paper suggests that the failure isn't because the AI lacks knowledge. It's because the AI has a "routing bottleneck." It knows the truth, but it can't figure out when to let that truth guide its answer. The current methods of just asking the AI to "try harder" or "think more" aren't fixing the broken wiring; they are just making the AI more confident in its wrong guesses. To truly fix this, we might need to rebuild the internal pathways that connect what the AI knows to what it decides to say, and we need different fixes for the robots that are too eager versus the ones that are too passive.

Technical Summary: Activation Bottlenecks in Pragmatic Constraint Reasoning

Problem Statement

Large Language Models (LLMs) frequently fail to apply implicit feasibility constraints when a salient surface cue suggests a different course of action. A canonical example involves a model answering "walk" to a car wash scenario, ignoring the implicit constraint that a car wash requires a car to be present. While aggregate accuracy metrics often reward correct answers, they conflate two distinct behaviors: genuine inference of the hidden constraint and conservative defaulting (choosing a safe option for unrelated reasons).

Previous work established that this failure is widespread but left ambiguous whether correct answers reflect true constraint reasoning or a bias toward safe options. This paper argues that hidden-constraint failures are primarily routing problems rather than knowledge problems. The models often possess the constraint information internally but fail to route it into the final decision-making process.

Methodology

The authors formalize conditional constraint activation through four falsifiable conditions (K, S, R, P) and instantiate them via a "quartet diagnostic" and mechanistic interventions:

The Quartet Diagnostic: For each scenario, four conditions are constructed:
- ACTIVE: The original scenario with an implicit constraint.
- REMOVED: A counterfactual where the constraint is removed but the surface cue remains.
- EXPLICIT: The constraint is stated as a declarative sentence.
- SALIENCE CONTROL: A length- and frame-matched neutral filler replaces the constraint cue.
- Metric: The Conservative Bias Index (CBI) ( $Acc_{Active} - Acc_{Removed}$ ) distinguishes between over-activation (high CBI) and under-activation (low/negative CBI).
Mechanistic Probes (K, S, R):
- Knowledge (K): Linear probes on hidden states decode the presence of the constraint.
- Symmetry (S): Probe accuracy is tested for symmetry between ACTIVE and REMOVED prompts.
- Routing (R): The correlation between the probe-projected magnitude and the decision logit gap is measured.
Activation Patching (P): Hidden states from an EXPLICIT prompt (donor) are patched into the hidden states of a failing ACTIVE or REMOVED prompt (recipient) at the probe's best layer to test if the constraint direction causally shifts the decision.
Mitigation Frontier: Four prompting strategies (Chain-of-Thought, precondition listing, goal decomposition, counterfactual checking) are evaluated on a two-dimensional frontier: Active Gain (improvement on ACTIVE items) vs. Pair Harm (degradation on REMOVED items).
Models: The study evaluates 14 models (including frontier API models and open weights) and performs deep mechanistic analysis on two open-weight models: Qwen3-14B (balanced behavior) and GPT-OSS-20B (under-activation).

Key Results

1. Two Distinct Failure Modes

Analysis of 14 models reveals two populations separated by CBI:

Over-activation (7 models): Models (e.g., Llama-4, Claude Opus 4.6) apply the constraint even when it is absent, leading to high accuracy on ACTIVE items but significantly lower accuracy on REMOVED items.
Under-activation (2 models): Models (e.g., GPT-OSS-20B) fail to apply the constraint when required, performing better on REMOVED items than ACTIVE items.
Balanced (5 models): Models show minimal difference between ACTIVE and REMOVED performance.

2. Mechanistic Dissociation (K/S/R/P)

Knowledge is Present: Linear probes decode the constraint applicability with high accuracy (>88%) in both Qwen3-14B and GPT-OSS-20B, satisfying condition K.
Symmetry Holds: Probe accuracy is symmetric across ACTIVE and REMOVED prompts, satisfying condition S.
Routing Failure: The probe-projected magnitude correlates weakly with the decision logit gap (failing condition R).
Patching Results:
- Qwen3-14B: Patching constraint information from an EXPLICIT prompt into an ACTIVE prompt increases the gold–shortcut logit gap by +6.4 nats, effectively repairing the decision. This satisfies condition P.
- GPT-OSS-20B: Despite the constraint being decodable (88% probe accuracy), patching the same information yields no effect ( $\Delta \approx -0.07$ nats). The constraint direction is present but unread by the decision head.

3. Failure of Prompted Mitigations

All four tested prompting strategies (CoT, precondition listing, goal decomposition, counterfactual checking) fail to reach the "repair corner" (positive gain, non-positive harm).

Harm Without Repair: Strategies consistently inflate the Conservative Bias Index (PairHarm +0.28 to +0.65) while providing negligible Active Gain (+0.01 to +0.04).
Mediation Analysis: The strategies work by inducing a single surface behavior: mentioning the prerequisite. This mention occurs at high rates (0.84–0.96) regardless of whether the constraint applies. The "correctness" is mediated by this mention, but the mention triggers a conservative bias that harms performance on REMOVED items.
Thinking Budget: Increasing the reasoning budget in thinking-mode models exacerbates over-activation (increasing CBI) without repairing the routing issue.

Significance and Claims

The paper claims that hidden-constraint failure in LLMs is a circuit-level conditional activation problem, not a knowledge deficit.

Routing vs. Knowledge: The failure is not that models "don't know" the constraint; they encode it symmetrically. The failure is that the constraint signal is not reliably routed into the decision head.
Prompting Limitations: Prompted interventions (CoT, etc.) cannot repair this routing failure. They merely increase the global probability of mentioning the constraint, which inadvertently inflates conservative bias on items where the constraint does not apply.
Remedy Direction: The authors conclude that effective remedies must operate on the routing direction itself (e.g., learned activation gates or circuit-level interventions) rather than relying on surface-level prompting or increased compute.

The study provides a formal diagnostic framework (K/S/R/P) and empirical evidence that aggregate accuracy metrics obscure the distinction between genuine reasoning and conservative defaulting, necessitating a shift toward mechanistic evaluation of constraint activation.