The End of Human Judgment in the Kill Chain? Relocating Initiative and Interpretation with Agentic AI

1. Problem Statement

The paper addresses a critical gap in the governance of Artificial Intelligence (AI) in military contexts. While international frameworks (such as the UN GGE-CCW and REAIM) mandate "context-appropriate human judgment and control" in the use of lethal force (the "kill chain"), the emergence of Large Language Model (LLM)-based Agentic AI threatens to render this human oversight substantively ineffectual.

The core problem is that the specific features making LLM-based agents operationally superior for data fusion, intelligence analysis, and battlefield management (e.g., initiative, dynamic interpretation, goal re-prioritization, and memory) simultaneously displace human normative and epistemic authority. Consequently, human involvement becomes "window dressing," unable to provide the meaningful explanation or justification required for ethical and legal accountability in lethal outcomes.

2. Methodology

The paper employs a conceptual and normative analysis grounded in specific technical use cases and emerging military strategies.

• Use Case Analysis: The author examines the architecture of LLM-based agents, using hypothetical and emerging examples like Anduril's "Lattice" Command and Control (C2) system. The analysis focuses on how these agents process heterogeneous data (sensor feeds, SIGINT, human reports) to make decisions.
• Feature Deconstruction: The methodology involves isolating four specific technical capabilities of LLM agents: Initiative, Interpretation, Goal-Orientedness, and Dynamic Memory.
• Normative Mapping: The author maps these technical features against the requirements of existing international governance frameworks (GGE-CCW, REAIM) to demonstrate a fundamental incompatibility. The argument relies on the premise that for human judgment to be valid, it must be a primary driver in the causal explanation of an outcome.
• Objection Handling: The paper systematically addresses counter-arguments regarding pre-deployment testing (TEVV), human-on-the-loop oversight, and the sufficiency of final human sign-off on target selection.

3. Key Contributions

The paper makes several distinct technical and ethical contributions:

• Identification of the "Displacement Mechanism": The author argues that LLM agents do not merely automate tasks; they relocate initiative and interpretation. Unlike traditional AI which follows static, pre-programmed rules, LLM agents dynamically decide when to seek human input, how to interpret ambiguous data, and which sub-goals to prioritize based on real-time context.
• The Four Critical Features: The paper defines four features that create a "black box" of authority:
  1. Initiative: Agents decide autonomously when to task sensors or seek human help, effectively deciding the threshold for human intervention.
  2. Interpretation: Agents interpret unstructured data (e.g., voice transcripts, raw sensor noise) and reconcile conflicting inputs without rigid schemas, introducing stochastic opacity.
  3. Goal-Orientedness: Agents decompose high-level commander intent into sub-goals and dynamically reprioritize them based on shifting contexts, potentially re-conceptualizing the original intent.
  4. Dynamic Memory: Agents integrate past observations and evolving world states into current reasoning, meaning their decisions are based on a contextual history invisible to the human operator.
• The "Ineffectual Human" Thesis: The paper concludes that when these agents operate in the data fusion stage (upstream of the final strike), they displace the human from the causal chain of the decision. If a human cannot explain why a specific target was selected because the data selection and fusion were performed by an agent with its own dynamic memory and initiative, the human's role is merely nominal.
• Policy Critique: The paper challenges the sufficiency of current governance proposals that rely on "human-in-the-loop" or "human-on-the-loop" models, arguing these are insufficient for LLM-based agentic systems where the human is too far removed from the actual determinants of the outcome.

4. Results and Findings

• Incompatibility with Governance: The study finds that LLM-based agents used for data fusion and battlefield management are fundamentally incompatible with the requirement for "context-appropriate human judgment" as defined by current international policy. The features that make them effective (speed, adaptability, autonomy) are the same features that make human oversight impossible.
• Failure of Oversight Mechanisms:
  • Pre-deployment (TEVV): Testing and validation cannot account for the stochastic and dynamic nature of agents that learn and adapt in real-time.
  • Real-time Oversight: Human monitoring is impossible at the speed and complexity of LLM data fusion; any human oversight would itself need to be mediated by another AI, creating a recursive loop of opacity.
  • Final Sign-off: A human making the final decision based on data fused by an agent is not exercising judgment on the determinants of the outcome (the data selection and prioritization), rendering the final decision ethically vacuous.
• Risk of Responsibility Attraction: The displacement of epistemic authority (the right to form beliefs) and normative authority (the right to make moral choices) to the machine makes it impossible to attribute responsibility for wrongful outcomes to a specific human actor.

5. Significance and Recommendations

Significance:
The paper provides a crucial technical warning that the next generation of military AI (Agentic AI) represents a qualitative shift from previous "autonomous weapons." It argues that the current international discourse is dangerously outdated, as it assumes human control is possible via traditional oversight models. If data fusion is automated by LLMs, the "human in the kill chain" becomes a legal fiction, undermining the foundations of International Humanitarian Law (IHL) regarding distinction, proportionality, and accountability.

Recommendations for the Governance Community:
Since a total ban on LLM agents is deemed unrealistic given their current deployment and strategic necessity, the author proposes a shift in strategy:

1. Stop Relying on "Human Judgment" as a De-risking Strategy: Policymakers must stop using "human-in-the-loop" as a primary safety mechanism for LLM-based data fusion, as it provides a false sense of security.
2. Develop New TEVV Standards: Create testing, evaluation, validation, and verification methods specifically for agentic systems, including red-teaming and real-world testing of human-machine teams.
3. Implement Technical Guardrails: Mandate oversight mechanisms that include non-LLM interpretable layers, counterfactual processing, and hard-coded constraints that cannot be dynamically re-interpreted by the agent.
4. Redefine Human Engagement: If human judgment is retained, it must be redefined to acknowledge that it may not be "close" to the outcome. This requires new frameworks for training and responsibility attribution that recognize the human is overseeing the process of the agent, not the specific content of the battlefield decision.

In conclusion, the paper asserts that without a fundamental restructuring of how human authority is conceptualized and technically enforced, the deployment of LLM-based agents in lethal kill chains will inevitably lead to a loss of meaningful human control and accountability.