Why Avoid Generative Legal AI Systems? Hallucination, Overreliance, and their Impact on Explainability

1. Problem Statement

The paper addresses the critical epistemic and legal risks associated with the rapid deployment of Generative Legal AI (GLAI) systems within the legal profession. While these systems promise efficiency, the author argues they pose fundamental threats to judicial independence and fundamental rights due to two interconnected phenomena:

• Hallucination (Confabulation): GLAI systems, built on Large Language Models (LLMs), prioritize statistical token prediction and linguistic fluency over factual accuracy. This leads to the systematic generation of plausible but fabricated legal authorities, case law, and facts.
• Overreliance (Automation Bias): The human-like, persuasive, and confident tone of GLAI outputs encourages legal professionals to defer uncritically to algorithmic recommendations. This erodes professional skepticism and critical thinking.

The core problem is that these two factors synergistically undermine the principle of explainability, a cornerstone of European AI governance (GDPR and the AI Act). If an AI generates decisions based on non-existent facts (hallucinations) and users accept them without scrutiny (overreliance), the "meaningful human oversight" required by law becomes impossible, rendering regulatory safeguards merely formal.

2. Methodology

The paper employs a qualitative, doctrinal, and interdisciplinary analysis combining legal theory, technical architecture, and cognitive psychology:

• Conceptual Definition: The author defines GLAI as an umbrella term for systems adapted for the legal domain (e.g., document drafting, decision support, criminal justice prediction), distinguishing them from traditional rule-based legal systems.
• Historical & Technical Review: The paper traces the evolution of AI from symbolic rule-based systems (1950s–1970s) to statistical machine learning (1980s–2000s) and finally to deep learning/foundation models (2010s–present). It analyzes the technical architecture of LLMs (next-token prediction on massive, unstructured corpora) to explain why hallucinations occur (statistical probability vs. ground truth).
• Comparative Legal Analysis: The study examines current deployment trends using reports from the International Bar Association (IBA) and the European Commission for the Efficiency of Justice (CEPEJ). It maps GLAI applications across five functional domains: legal information discovery, drafting, decision support, public legal assistance, and Online Dispute Resolution (ODR).
• Regulatory Framework Analysis: The paper situates the technical risks within the EU AI Act and GDPR, specifically analyzing Articles 13, 14, and 22 regarding high-risk AI systems, human oversight, and the right to explanation.
• Case Study Integration: The author integrates real-world examples (e.g., the 2025 US District Court incident where a judge withdrew an opinion due to fabricated citations, and a Turkish enforcement office relying on AI rent calculations) to illustrate the practical consequences of these risks.

3. Key Contributions

• Definition of GLAI: The paper formally categorizes "Generative Legal AI" not just as a tool, but as a distinct class of systems with specific architectural limitations (statistical prediction vs. legal reasoning) that make them inherently unsuited for high-stakes legal reasoning without strict constraints.
• The "Hallucination-Overreliance" Synergy: A novel theoretical contribution is the argument that hallucinations and overreliance are not isolated issues but a reinforcing cycle. The "anthropomorphic" nature of GenAI (the "uncanny valley" and "Computers as Social Actors" paradigm) masks the stochastic nature of the output, leading professionals to trust fabricated data, which in turn makes the lack of explainability irreversible.
• Critique of "Explainability by Design": While acknowledging "explainable AI" (XAI) as a potential solution, the paper argues that standard XAI techniques (e.g., feature importance scores) are insufficient for law. Legal explainability requires domain-aware explanations that capture argumentative structures, defeasibility, and normative justifications, which current GenAI cannot provide.
• Regulatory Gap Analysis: The paper highlights that while the EU AI Act mandates human oversight, the current technical reality of GLAI makes "meaningful" oversight practically impossible if the underlying data is fabricated and the user is cognitively biased toward the machine.

4. Results and Findings

• Technical Limitation: GLAI systems are fundamentally designed for linguistic fluidity, not factual accuracy. They operate on probabilistic patterns found in training data (which includes fiction and bias), making them incapable of distinguishing "truth" from "plausible fiction" in a legal context.
• Erosion of Explainability: Hallucinations introduce unverifiable data into decision pipelines. Consequently, a legal professional cannot provide a credible justification for a decision derived from an AI, violating the GDPR's requirement for "meaningful information about the logic involved."
• Psychological Vulnerability: Legal professionals are susceptible to automation bias due to the efficiency and authoritative tone of GenAI. Empirical evidence (e.g., the COMPAS recidivism tool case) shows that even when AI is opaque or biased, humans tend to outsource moral responsibility to the algorithm.
• Current State of Adoption: Despite the risks, GLAI is already integrated into five key legal domains (research, drafting, analytics, public assistance, ODR). However, the paper notes that full integration is still in a "slow phase," suggesting a window for intervention.
• Ineffectiveness of Current Mitigations: Standard verification methods are insufficient because the effort required to verify AI hallucinations often exceeds the time saved by using the tool, leading to a net decrease in work quality.

5. Significance and Conclusion

The paper concludes with a strong normative argument: The routine adoption of GLAI in legally consequential decision-making is currently incompatible with legal and ethical standards.

• Judicial Independence: The reliance on "black box" systems that hallucinate facts threatens the independence of the judiciary, as judges may become passive recipients of algorithmic outputs rather than active reasoners.
• Fundamental Rights: The inability to explain decisions based on fabricated data violates the right to a fair trial and the protection of fundamental rights.
• Recommendation: The author advocates for strong restraint. Until the technical problems of hallucination and the psychological problem of overreliance are solved (which the author deems improbable at the current stage), GLAI should not be used for core legal reasoning or decision-making.
• Future Direction: If GLAI is to be used, it requires "domain-aware" explainability mechanisms that go beyond technical transparency to address the specific argumentative structure of law, alongside robust human oversight that is not merely formal but substantive.

In essence, the paper serves as a cautionary manifesto against the uncritical integration of generative AI into the legal sphere, emphasizing that efficiency cannot come at the cost of truth, accountability, and the rule of law.