🤖 AI

Unpredictability dissociates from structured control in language agents

This paper demonstrates that in language agents, stochastic unpredictability cannot substitute for structured control mechanisms coupling reasons, memory, and self-state to action selection, as targeted lesions to these structured components consistently degrade performance while high-stochasticity variants fail to replicate structured action-field coupling across extensive benchmarks.

Original authors: Jia Xiao

Published 2026-05-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jia Xiao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Question: Is "Chaos" the Same as "Control"?

Imagine you are watching a puppet show.

Scenario A: The puppeteer is pulling strings with a clear plan. The puppet moves because of specific reasons (memory of the plot, a decision to stop, or a rule to follow).
Scenario B: The puppet is just being tossed around by a gust of wind. It moves wildly and unpredictably, but it has no plan, no memory, and no reason for where it goes next.

For a long time, people assumed that if a computer program (an "AI agent") acted in a wild, unpredictable way, it must be "thinking" or "controlling" itself. This paper asks: Is that true?

The researchers say no. They found that just because an AI is unpredictable (like the wind-blown puppet), it doesn't mean it has structured control (like the puppeteer). In fact, true control requires specific internal "organs" like memory, reasoning, and the ability to say "no" (inhibition).

The Experiment: The "Lesion" Lab

To prove this, the researchers built a special AI agent that acts like a human with a "control panel." They could turn specific parts of the agent's brain on or off, like a surgeon performing a "lesion" study (removing parts of the brain to see what happens).

They had two main versions of the agent:

The "Structured" Agent: This one had all its tools turned on. It had a Reason module (to explain why it chose something), a Memory module (to remember past events), a Self-State module (to keep track of its goals), and a Veto module (to stop itself from making a bad choice).
The "Chaos" Agent: This one had the Reason, Memory, Self, and Veto modules turned off. Instead, they turned up the "temperature" (randomness) to make it guess wildly and unpredictably.

The Analogy:
Think of the Structured Agent as a chef following a recipe, tasting the food, and deciding to add salt or stop cooking.
Think of the Chaos Agent as a blender spinning at maximum speed. It throws ingredients everywhere (unpredictable), but it isn't "cooking" anything; it's just making a mess.

What They Found

The researchers ran these agents through 7 different types of tasks (like telling a story, making a choice, or remembering a fact) and checked 74,000+ interactions. Here is what happened:

1. Chaos is not Control
The "Chaos" agent was indeed much harder to predict. It jumped around more. However, it failed to act like a controlled agent. It didn't remember its goals, it didn't change its mind based on new reasons, and it couldn't stop itself from making mistakes.

The Result: Being unpredictable did not make the agent smarter or more in control. It just made it random.

2. The "Post-Hoc" Trick
Some people might argue, "Maybe the Chaos agent is controlling itself, it just writes its reasons after it makes a choice."
To test this, the researchers forced the Chaos agent to write down reasons, memories, and veto decisions after it acted, just to see if it looked like it was in control.

The Result: Even when forced to write a story about why it did something, the Chaos agent's actions didn't actually match the story. The "reasons" were just decoration, not the real cause of the action.

3. The "Finite Action" Test
To be absolutely sure, they stripped away all the fancy text and looked only at the final decision (e.g., "Choose Option A" or "Stop"). They gave the agents a secret test: "If I change the reason, you should change your choice. If I give you a fake reason, you should ignore it."

The Result: The Structured Agent passed. It changed its choice when the reason changed and ignored fake reasons. The Chaos Agent failed miserably; it couldn't follow the rules of logic or memory.

The "Blindfolded" Check

To make sure they weren't biased, three human experts looked at 1,200 examples without knowing which agent was which. They agreed almost perfectly (96% agreement) that the Structured Agent was the only one actually "in control."

The Limits (What the Paper Didn't Say)

The paper is very careful about what it claims. It does not say:

That AI can never be controlled.
That randomness is useless.
That this applies to every AI in the world.

Instead, it says: In the specific type of AI agent they built, simply making the AI more random (stochastic) did not create true, structured control. You cannot substitute a "wild guess" for a "reasoned decision."

The Bottom Line

If you want an AI to be truly in control—able to remember, reason, and stop itself—you need to build those specific parts into it. You cannot just turn up the "randomness knob" and hope it figures out how to be smart. Unpredictability is not the same as intelligence.

Technical Summary: Unpredictability Dissociates from Structured Control in Language Agents

Problem Statement
The paper addresses a fundamental conflation in the evaluation of language agents: the assumption that behavioral unpredictability (stochastic dispersion) is synonymous with structured control. While a system may be difficult to predict due to broad sampling, noise, or a large output space, this does not necessarily imply it possesses goal-relevant representations, inhibitory mechanisms, or context-sensitive action selection. The study investigates whether stochastic sampling can substitute for explicit, structured mechanisms—specifically those coupling reasons, memory, self-state, and inhibition (veto) to action selection—in a language-agent implementation where these components can be selectively disabled (lesioned).

Methodology
The research employs a "lesionable" language-agent family designed to isolate specific control components. The experimental design is structured around three evidential axes:

Stochastic Dispersion: Measuring whether increasing stochasticity (via high-temperature sampling and disabling control modules) increases action unpredictability.
Trace–Action Coupling: Assessing whether reported reasons, memory, self-state, and veto information remain coupled to the final action, distinct from merely being present in the output format.
Behavioral Action Responsiveness: Testing if finite action codes change in expected directions under counterfactual probes and targeted lesions, independent of free-form trace wording.

Experimental Setup:

Variants: The study compares a Structured-Control Variant (A4), which enables branching, self-state, memory, reason-graph, and veto, against a High-Stochasticity Comparator (A5), which disables the control modules and increases sampling temperature. Targeted lesions (A4_no_reason, A4_no_veto) selectively remove specific components.
Controls: The study includes matched-interface controls (post-hoc, scrambled, random, and verbosity controls) that use the same reporting schema to rule out format availability as a confounding factor. It also tests strong stochastic alternatives (lower-temperature calibration, compute-matched samplers, self-consistency) and extends to open-weight models (Qwen2.5 7B/14B/32B, Mistral-7B) and external APIs.
Metrics:
- Action Entropy: Shannon entropy over final action strings to measure unpredictability.
- Action-Field Coupling Index (AFCI): A composite score measuring the alignment of reason, memory, veto, and self-state with the final action.
- Finite-Action-Code Benchmark: A format-independent test where final actions are mapped to a six-code ontology (e.g., ACTION_A, VETO, DEFER). Scores (BRSI, BMCI, BVEI, BSCI) measure responsiveness to specific probes while ignoring free-form trace wording.
Scale: The baseline lesion matrix comprises 74,352 calls across seven datasets. Matched-interface controls span 26,946 generations. The finite-action-code benchmark includes 57,816 scored records. Open-weight runs cover 126,000 generations per model across 20 task families.

Key Results

Dissociation of Unpredictability and Control: In the baseline lesion matrix (7/7 datasets), the high-stochasticity comparator was significantly more unpredictable than the structured-control variant. However, this increased dispersion did not reproduce the structured-control profile. Removing reason or veto components (lesions) specifically reduced the expected structured-control metrics in all 7 datasets, confirming that the structured profile relies on these specific mechanisms.
Coupling vs. Format: The structured agent maintained a significantly higher Action-Field Coupling Index (AFCI) than all stochastic, post-hoc, scrambled, and verbosity controls across all datasets. The lower 95% bootstrap bound for the AFCI difference was 0.4519, indicating that the coupling is not merely an artifact of the reporting format.
Behavioral Responsiveness: In the finite-action-code benchmark, the structured-control variant exceeded the high-stochasticity comparator and the targeted lesions in 7/7 datasets for all predefined behavioral components (reason, memory, veto, self-continuity). The dataset-cluster bootstrap lower 95% bound was 0.8557.
Robustness and Generalization:
- Stochastic Substitution: Stronger stochastic controls (including entropy-matched attempts) failed to recover the structured-control profile. No calibration candidate satisfied the predefined four-level entropy closeness criterion (threshold 0.15).
- Model Generalization: The dissociation held across Qwen2.5 (7B, 14B, 32B) and Mistral-7B families. Structured accuracy exceeded distribution-matched and scrambled-context controls in 236/240, 240/240, and 238/240 cells respectively.
- Blinded Audit: A three-annotator blinded audit over 1,200 items showed high agreement (mean pairwise Cohen's $\kappa$ = 0.961), validating the scoring of control dimensions.
Limitations: The study explicitly notes that strict entropy matching and strict token/compute matching did not meet their predefined gates. Consequently, the conclusion is bounded to the tested matched controls and does not claim that no perfectly entropy-calibrated stochastic comparator could ever succeed. Similarly, the formal counterfactual-flip stress test did not meet its 90% flip-following gate.

Significance and Claims
The paper claims to provide a controlled dissociation between unpredictability and structured control within a specific language-agent family. Its primary contribution is demonstrating that stochastic unpredictability does not reproduce structured, action-coupled control when the latter is defined by the coupling of reasons, memory, self-state, and veto to action selection.

The study argues that variability, reflective text, and action control are easily conflated in agent evaluation. A model can produce diverse answers or plausible post-hoc rationales without actually coupling them to action selection or respecting inhibition. By using a lesionable architecture and format-independent behavioral benchmarks, the paper establishes that the "structured-control profile" is a distinct operational property that stochastic sampling alone cannot substitute.

The findings are modestly bounded to the tested agent family, prompts, and measured trace/action protocols. The authors do not claim to have proven a universal law of agency but rather to have experimentally separated two distinct operational modes in the evaluated systems. The work serves as a caution against equating spontaneity or high entropy with instrumental control or goal-directed agency in language agents.