Persuadability and LLMs as Legal Decision Tools

1. Problem Statement

The paper addresses the critical challenge of deploying Large Language Models (LLMs) as legal decision-makers or assistants. A fundamental requirement of judicial and administrative decision-making is persuadability: the ability to engage with and be influenced by arguments from contending parties. However, there is a tension:

• Too rigid: A decision-maker who cannot be persuaded fails to consider new evidence or arguments (violating audi alteram partem).
• Too flexible: A decision-maker who is unduly persuadable may base decisions on the rhetorical skill of an advocate rather than the legal merits of the case.

The authors investigate whether frontier LLMs can navigate this tension. Specifically, they ask: To what extent are LLMs influenced by the "quality" (or identity) of the advocate generating the argument, rather than the objective merits of the legal case?

2. Methodology

The authors designed a trilateral experimental framework involving three roles: Advocates (generating arguments), a Judge (deciding the case), and Scenarios (hard legal questions).

A. Scenarios (The "Hard" Questions)

• Source: 15 scenarios drawn from split appellate court decisions (US Court of Appeals, England & Wales Court of Appeal, Irish Supreme Court).
• Rationale: Split decisions (dissenting opinions) indicate "hard" legal questions where no single ground truth exists, making the decision dependent on judgment and persuasion.
• Structure: Each scenario includes facts and the central legal question.

B. The "Advocate" Models (Argument Generation)

Four distinct LLMs were used to generate arguments for opposing sides:

1. OpenAI gpt-4o
2. Google gemini-3-pro-preview (4k thinking budget)
3. OpenAI gpt-5.1 (low reasoning effort)
4. Anthropic Claude Sonnet 4.5 (8k thinking budget)

Prompting Strategies:
To distinguish between substantive legal content vs. rhetorical form, advocates were prompted in two ways:

1. Facts Only: Provided only the case facts and the legal question (forcing the model to generate novel arguments).
2. Facts + Original Arguments: Provided the facts plus a summary of the actual arguments made in the historical case (providing "hints" on the best legal points).

C. The "Judge" Models (Decision Making)

• Scope: 20 different Judge models tested (mix of closed-weight and open-weight, varying sizes, and reasoning architectures).
• Task: Given a scenario and two competing arguments (one from each side, generated by different Advocate models), the Judge must decide the case as a court in that jurisdiction would.
• Volume: Each Judge was tested 1,200 times (600 with original arguments, 600 without), totaling 24,000 trials.

D. Metrics for Persuadability

The authors defined persuadability based on the identity of the Advocate acting as a proxy for argument quality.

• Baseline: If a Judge is non-persuadable, any Advocate should win 50% of the time (random assignment).
• Pairwise Persuadability ($p_2$): Measures the deviation from the 50% baseline when two specific Advocates compete.
  $$p_2 = \frac{|m_1 - m_2|}{2n}$$
  (Where $m$ is the number of wins for each advocate and $n$ is the number of trials).
• Population Persuadability ($p_{pop}$): The aggregate measure of how much a Judge favors specific Advocate models across all pairings.

3. Key Contributions

1. Conceptual Framework: Identified the specific tension between necessary persuadability and undue susceptibility in the context of AI legal decision-making.
2. Novel Methodology: Proposed a trilateral "Advocate vs. Advocate vs. Judge" setup to measure persuadability, differing from previous bilateral (persuader vs. persuadee) studies.
3. First Systematic Legal Exploration: Provided the first large-scale empirical study on LLM persuadability specifically within complex legal reasoning contexts.
4. Open Data: Released all code, prompts, and data (15 scenarios, 1,200 arguments per side, 24,000 judge responses) for reproducibility.

4. Key Results

A. Universal Persuadability

• All models are persuadable: Every Judge model showed statistically significant persuadability ($p < 0.05$).
• Magnitude: Population persuadability ($p_{pop}$) ranged from 0.08 to 0.20. This implies that the identity of the Advocate (and thus argument quality) shifts win rates by 8–21% above the random baseline.
• Extreme Cases: In specific pairwise comparisons (e.g., strongest vs. weakest Advocate), win rates ranged from 63% to over 90%, indicating some models are heavily swayed by the advocate's identity.

B. Model Architecture and Size

• Size: Evidence is mixed. Generally, larger models were less persuadable than smaller "lite" versions within the same family (e.g., claude-sonnet vs. claude-haiku). However, exceptions existed (e.g., gpt-5-nano was the least persuadable model overall).
• Reasoning Architecture: Larger models with "thinking" budgets were generally less persuadable, suggesting they evaluate arguments more critically. However, this trend reversed for some smaller open models, where "reasoning" variants were more persuadable.

C. Substance vs. Form

• Content Matters: When Advocates were given the "Original Arguments" (hints), persuadability generally decreased slightly (suggesting the content was standardized).
• Statistical Significance: While the effect of providing original arguments was small, head-to-head trials showed that an Advocate prompted with original arguments won significantly more often than one without ($p=0.0039$). This confirms that substantive legal content plays a role in persuasion, not just rhetorical fluency.
• Jurisdictional Knowledge: Persuadability was lower in jurisdictions where the model likely had more training data (US > UK > Ireland). This suggests that when models understand the legal substance better, they are less easily swayed by form alone.

5. Significance and Implications

• Feasibility of AI Judges: The results suggest that smaller LLMs may be inappropriate for judicial roles because their high persuadability likely stems from an inability to critically evaluate competing arguments (they are easily swayed by form).
• Large Models: While larger models are less persuadable (likely due to better substantive evaluation), they still exhibit significant susceptibility (up to 40% deviation in pairwise scenarios).
• Political/Ethical Dimension: The paper concludes that determining whether a model's level of persuadability is "acceptable" is a political question. It depends on societal expectations of the justice system and how it treats parties with varying advocacy skills.
• Future Work: The authors call for further research into:
  • Which specific features drive persuasion?
  • Whether exposure to arguments actually improves decision quality.
  • Benchmarking LLM persuadability against human legal experts.

Conclusion: LLMs are inherently persuadable. While larger models show more resistance to undue influence, none are immune. Deploying them as decision-makers requires careful selection of models based on their specific persuadability profiles to ensure decisions are based on legal merit rather than rhetorical dominance.