Why Sampling Is Not Choosing: Intentionality, Agency, and Moral Responsibility in Large Language Models

Technical Summary: Why Sampling Is Not Choosing

Problem Statement
Recent advances in Large Language Models (LLMs), particularly those based on transformer architectures, have sparked claims that these systems exhibit agency or qualify as moral agents. While LLM outputs can be normatively evaluated (assessed for correctness, appropriateness, or conformity to moral standards), it remains unclear whether the models themselves are appropriate subjects of moral responsibility. The central problem addressed is the conflation of normatively evaluable outputs with genuine agency. The paper challenges the attribution of moral responsibility to transformer-based systems, arguing that their operational mechanisms—specifically probabilistic input-output mappings and stochastic sampling—do not constitute the form of free will or commitment-bearing agency required for moral accountability.

Methodology and Conceptual Framework
The paper employs a philosophical analysis grounded in the philosophy of mind and action, utilizing a specific conceptual framework to define the necessary conditions for moral responsibility:

1. Intentionality: The paper distinguishes between intrinsic intentionality (meaning inherent to a system, directed toward objects for the system itself) and derived intentionality (meaning attributed to a system by external agents). It draws on Searle (1980), Harnad (1990), and Brentano (1995) to argue that computational systems possess only derived intentionality.
2. Moral Responsibility: Defined not merely as causal participation but as "commitment-bearing agency." This requires an agent to be the author of its actions, capable of self-attribution, and able to endorse or repudiate actions based on reasons.
3. Free Will: Characterized not as indeterminism or randomness, but as the capacity for reasons-responsive control and ownership of action.
4. Technical Analysis of Transformers: The paper analyzes the architecture of transformer-based models, focusing on:
   • Training: Optimization via next-token prediction using teacher forcing on human-generated corpora, learning statistical regularities rather than semantic understanding.
   • Inference: Generation via sampling from a probability distribution (using methods like greedy decoding, temperature scaling, top-k, or nucleus sampling).
   • Reasoning Models: An examination of systems using chain-of-thought or tree-based search, arguing these remain statistical selections rather than genuine deliberation.

Key Arguments and Results
The paper constructs a deductive argument to demonstrate that transformer-based systems fail to satisfy the conditions for moral responsibility:

• Premise 1: Moral responsibility requires commitment-bearing agency.
• Premise 2: Commitment-bearing agency requires intrinsic intentionality and the capacity for self-attributed action.
• Premise 3: Such agency constitutes the form of free will relevant to responsibility.
• Premise 4: Transformer-based systems lack intrinsic intentionality and self-attributed action.
  • Evidence: Their operation is a probabilistic mapping learned from data. The "aboutness" of their outputs is derived from human interpretation, not internal to the model.
  • Evidence: The variability introduced by stochastic sampling is not "choice" or "authorship"; it is randomness, which lacks the ownership and reasons-responsiveness required for free will.
  • Evidence: Even "reasoning" models (e.g., chain-of-thought) simulate deliberation but ultimately select tokens based on statistical likelihood or heuristic scoring, not the recognition of reasons as reasons.
• Conclusion: Therefore, transformer-based systems are not appropriate subjects of moral responsibility.

The paper addresses and refutes potential counter-arguments, including the Intentional Stance (attributing beliefs/desires for predictive utility), Functionalism (defining mental states by function), and Compatibilism (reconciling determinism with responsibility). The authors argue that none of these frameworks suffice to establish genuine agency in systems lacking intrinsic intentionality and self-attribution.

Key Contributions

1. Novel Commitment-Based Account: The paper develops a specific account of agency that links intrinsic intentionality to moral responsibility through the concepts of authorship and free will. It clarifies that responsibility is not about the output being evaluable, but about the source being capable of commitment.
2. Systematic Technical-Philosophical Analysis: It applies this philosophical framework to contemporary transformer architectures, explicitly demonstrating how the mechanics of next-token prediction and stochastic sampling preclude the necessary form of agency.
3. Clarification of "Reasoning" Models: It extends the critique to advanced "reasoning" models, arguing that intermediate steps (like chain-of-thought) do not transform statistical selection into genuine deliberation or authorship.

Significance and Claims
The paper claims to provide a unified philosophical and technical explanation for why attributing moral agency to LLMs is misguided. Its significance lies in:

• Distinguishing Evaluation from Agency: It rigorously separates the ability to normatively evaluate a system's output from the system's capacity to be held responsible for that output.
• Revisiting Classical Critiques: It re-engages classical arguments regarding syntax-semantics distinctions and symbol grounding (Searle, Harnad) in the context of modern LLMs, showing their continued relevance.
• Practical Implications: The paper concludes that responsibility for the behavior and deployment of these systems must remain with human designers, deployers, and institutional frameworks. Efforts to align or regulate LLMs should be viewed as constraining human tools within normative practices, not as transforming the systems into moral agents.

The paper does not propose new experimental protocols or future applications; rather, it serves as a critical boundary-setting work, arguing against the anthropomorphization of probabilistic input-output mechanisms.