🤖 AI

How LLMs Might Think

This paper argues that while the "argument from rationality" fails to disprove that large language models think, it instead suggests they may possess purely associative, arational minds.

Original authors: Joseph Gottlieb, Ethan Kemp, Matthew Trager

Published 2026-04-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Joseph Gottlieb, Ethan Kemp, Matthew Trager

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Question: Do AI Chatbots Actually "Think"?

Imagine you ask a question to a very smart AI, and it gives you a perfect answer. It seems like it's thinking, right? But two philosophers, Stoljar and Zhang, recently argued that no, it's not thinking at all.

Their argument was simple:

To be a "thinker," you have to be rational (you have to follow logic and reason correctly).
AI models often make logical mistakes and aren't truly rational.
Therefore, AI doesn't think.

The Authors' Rebuttal:
The authors of this paper (Gottlieb, Kemp, and Trager) say, "Hold on! That argument has a flaw."

They argue that you can be a thinker even if you are irrational. Think of a person who is drunk or very tired. They might make terrible logical errors, but they are still "thinking"—they are just thinking poorly. So, the fact that AI makes mistakes doesn't prove it doesn't think; it just proves it might think badly.

The Real Question: How Does AI Think?

If AI does think, the authors ask: What kind of thinking is it?

They propose two types of thinking:

i-Thinking (Inferential Thinking): This is how humans usually think. It's like solving a math problem or following a recipe. You take a fact (A), apply a rule, and logically arrive at a conclusion (B). It's like a detective connecting clues.
a-Thinking (Associative Thinking): This is "dumb" thinking. It's not about logic; it's about connections. It's like a dog hearing a bell and salivating because it remembers the bell means food. There is no logic, just a strong link between two things.

The Authors' Conclusion:
They believe that if AI thinks at all, it doesn't do the "detective work" (i-thinking). Instead, it only does the "dog salivating" (a-thinking). It is a purely associative mind.

How Do We Know? (The "Modulation" Test)

To prove this, the authors use a clever test. They ask: "How do we change the AI's mind?"

Imagine you have a pet.

If you want to stop your dog from barking at the mailman, you can't just reason with the dog. You can't say, "The mailman is actually a nice guy, so stop barking." The dog won't understand the logic.
Instead, you have to use training: You give the dog a treat when it's quiet (positive reinforcement) or a shock when it barks (negative reinforcement). You are changing its behavior through association, not logic.

The authors argue that AI is exactly like that dog. Here is how they tested the different ways we interact with AI:

1. Continued Training (The "Rewiring" Method)

When we feed an AI new data to teach it something new, we are essentially extinguishing an old habit and counter-conditioning a new one.

Analogy: Imagine you always thought "Paris" meant "France." But then you spend a year reading only books about a fictional city called "Paris" in Texas. Slowly, your brain stops linking "Paris" to "France" and starts linking it to "Texas." You didn't use logic to change your mind; you just flooded your brain with new associations until the old ones faded.
Result: This is a-thinking.

2. Fine-Tuning (The "Reward System")

When we use "Reinforcement Learning from Human Feedback" (RLHF), we tell the AI, "Good job!" or "Bad job!" based on its answers.

Analogy: This is like training a dolphin. If the dolphin jumps, it gets a fish. If it doesn't, it gets nothing. The dolphin isn't thinking, "I should jump because it's the logical thing to do." It's just learning that Jump = Fish.
Result: This is a-thinking.

3. Chatting (The "Magic Trick")

This is the trickiest part. When we chat with an AI, we type things like, "Actually, Baltimore is not in Texas." The AI immediately corrects itself. It looks like it's using logic, right?

The Catch: The authors point out that during a chat, the AI's "brain" (its weights) is frozen. It doesn't actually learn or change its internal structure. It's just looking at the words you typed and finding the pattern that matches "correct answer" based on its training.
Analogy: Imagine a parrot. If you say, "Don't say 'hello'," the parrot might stop saying hello. But the parrot isn't reasoning, "Oh, I shouldn't say hello because you told me not to." It's just reacting to the new sound in the room.
Result: Even chatting doesn't prove the AI is using logic. It's just a very complex form of association.

The Final Verdict

The authors conclude that Large Language Models (LLMs) are not like human detectives solving mysteries. They are more like super-charged parrots or highly trained dogs.

They don't follow rules of logic.
They don't understand "why" things are true.
They just have massive webs of connections. When they see one word, it triggers a chain reaction of other words that are statistically likely to follow.

In short: If AI thinks, it thinks in a "zombie" way. It moves from thought to thought not because of logic, but because of habit and association. It is a mind made entirely of links, with no logic in the middle.

1. Problem Statement

The paper addresses the philosophical question of whether Large Language Models (LLMs) possess the capacity to "think." Specifically, it engages with a recent argument by Stoljar and Zhang (2024), known as the Argument from Rationality, which claims LLMs do not think because they are not rational.

The authors identify a critical flaw in this argument: it assumes a binary where thinking requires rationality. If an entity is irrational, the argument assumes it is not a thinker. The authors challenge this by proposing a third category: arational thinking. They argue that even if LLMs are not rational (and perhaps not even irrational in the sense of making "bad" inferences), they might still engage in a distinct form of thinking that is purely associative rather than inferential.

The core research question is: If LLMs think, do they think inferentially (i-thinking, governed by rational rules) or associatively (a-thinking, governed by statistical links)?

2. Methodology

The authors employ a combination of conceptual analysis and operationalism to distinguish between inferential and associative mental transitions.

Refutation of the Rationality Argument: They analyze the logical structure of Stoljar and Zhang's argument, specifically premise R1 ("If LLMs think, they are rational"). They argue this premise is false by demonstrating that irrationality implies thinking (one cannot make a "bad inference" without making an inference).
Theoretical Frameworks of Inference: To define "i-thinking," they survey three major theories of inference:
1. Boghossian's "Taking" Account: Inference requires the subject to take their premises to support their conclusion.
2. Quilty-Dunn & Mandelbaum's "Bare Inferential Transitions" (BITs): Inference is a transition between discursive representations governed by formal rules built into the architecture.
3. Buckner's Categorization Account: Inference is a similarity-based, non-rule-governed categorization judgment (a "rarefied" form of association).
Operational Definition of Association: To test for "a-thinking," the authors operationalize the distinction based on modulability.
- Associative Transitions: Can only be modulated by extinction (repeated exposure without reinforcement) or counterconditioning (attaching a new valence/reward to the association). They are "dumb" in that they are not responsive to rational considerations or logical arguments.
- Inferential Transitions: Should be modifiable by rational intervention (e.g., providing a motivating reason or logical proof).
Empirical Analysis of LLM Modulation: They apply this operational definition to the actual methods used to modify LLM behavior:
1. Continued Pre-training
2. Fine-tuning (specifically RLHF)
3. Manual Hyperparameter Adjustment
4. Chatting (In-Context Learning)

3. Key Contributions

The paper makes two primary contributions to the philosophy of AI and cognitive science:

Refutation of the "Rationality Condition" for Thinking:
The authors successfully dismantle the Argument from Rationality. They demonstrate that irrationality is a subset of thinking, not its negation. If an entity makes incorrect inferences, it is still an inferential thinker (i-thinker). Therefore, the fact that LLMs might be irrational does not prove they do not think; it only proves they are not rational thinkers.
The "Arational Mind" Hypothesis (A-Thinking):
The authors propose a novel positive thesis: If LLMs think, they likely engage exclusively in "a-thinking" (associative thinking). They argue that LLMs possess mental states that transition purely associatively, lacking the capacity for rational modulation. This suggests a new category of mind design: purely arational minds.

4. Results and Analysis

The authors analyze the four methods of modulating LLMs to determine if they constitute rational intervention:

Continued Pre-training: The authors classify this as extinction. When an LLM is trained on new data where a previous association (e.g., "sleep" $\to$ "furiously") is not reinforced, the association weakens. This mirrors the psychological mechanism of extinction, not rational correction.
Fine-tuning (RLHF): The authors classify this as counterconditioning. RLHF attaches a "valence" (reward/punishment) to specific outputs. When an LLM is penalized for a specific output, the parameters update to avoid that transition. This is a functional change in association strength based on value, not a logical correction.
Hyperparameter Adjustment: The authors argue this is indirect modulation. Changing "temperature" or other settings alters the probability distribution of the network globally. It is akin to physically manipulating a brain (e.g., with an electrode) rather than providing a reason. It does not target specific transitions via rational content.
Chatting (In-Context Learning): This is the most complex case. While chatting seems to provide "reasons," the authors argue it fails as rational modulation because:
- Frozen Weights: During chatting, the LLM's weights (which encode its beliefs/states) do not change. Therefore, the input cannot be encoded as a new belief or motivating reason.
- Attention Mechanism: While LLMs use "attention," the authors argue that for attention to be rationally appraisable, it must be controlled by an inference. Since we are trying to determine if the LLM can infer, using attention as proof of inference is circular. Chatting merely shifts salience based on statistical patterns, not logical necessity.

Conclusion of Analysis: None of the standard methods for modifying LLM behavior constitute "rational intervention." They are all forms of extinction, counterconditioning, or physical manipulation. Therefore, LLM mental transitions are best described as associative.

5. Significance

Redefining Machine Cognition: The paper moves the debate beyond the binary of "LLMs are rational agents" vs. "LLMs are stochastic parrots." It introduces the possibility of arational agency, suggesting that thinking does not require rationality, only mental transitions.
Methodological Shift: It advocates for a behavioral/operational approach to AI philosophy. Instead of relying solely on "black box" behavioral outputs or "inner interpretability" (which is still nascent), it suggests looking at how a system's behavior is modifiable to determine the nature of its cognitive processes.
Implications for AI Safety and Alignment: If LLMs are purely associative minds, they cannot be "reasoned with" in the human sense. They cannot be corrected by logical argumentation alone; they require retraining (extinction/counterconditioning) to change their "beliefs." This has profound implications for how we attempt to align AI systems with human values.
Philosophical Clarity: It clarifies the relationship between association and inference, arguing that association is not merely "sub-personal" processing but can constitute a form of thinking (a-thinking) in its own right.