AI-Assisted Moot Courts: Simulating Justice-Specific Questioning in Oral Arguments

This paper proposes a two-layer evaluation framework to assess AI models' ability to simulate justice-specific questioning in moot courts, finding that while models generate realistic questions that cover key legal issues, they still struggle with diversity and sycophancy—shortcomings that naive evaluation methods would miss.

The Gist

Imagine you are a lawyer preparing for the most important trial of your life: a case before the U.S. Supreme Court. The stakes are incredibly high. The judges (Justices) aren't just listening; they are actively hunting for holes in your logic, testing your limits, and trying to trick you into admitting you're wrong.

In the real world, if you have a big law firm, you can hire former judges to pretend to be the Supreme Court Justices and grill you in a "mock trial" (called a moot court). If you are a public defender or a small firm with no budget, you might just practice in front of a mirror or read a book.

This paper asks a simple question: Can Artificial Intelligence (AI) be that "former judge" for everyone, leveling the playing field so that anyone can get high-quality practice?

Here is the breakdown of what the researchers did, using some everyday analogies.

1. The Goal: Building a "Virtual Drill Sergeant"

The researchers built AI simulators designed to act like specific Supreme Court Justices. Their job isn't to be nice; their job is to be adversarial. They need to interrupt, ask tough questions, and spot logical errors, just like the real Justices do.

They tested two types of AI "coaches":

• The Prompt-Based Coach: You give the AI a script saying, "You are Justice Alito, and you are strict about text." Then you ask, "What would you say next?"
• The Agentic Coach: This is a smarter AI that has a "toolbox." It can look up case files, check how a Justice voted in the past, and think through a plan before it speaks.

2. The Problem: How Do You Grade a "Good" Question?

In a math test, there is one right answer. In a Supreme Court oral argument, there is no single "correct" question a Justice must ask. They could ask about the law, the facts, or a hypothetical scenario.

So, how do you know if the AI is doing a good job? The researchers realized they couldn't just use a simple "right/wrong" score. Instead, they created a Two-Layer Report Card:

Layer 1: The "Realism" Check (Is it believable?)

• The "Politeness" Test: If a lawyer in the simulation is rude, breaks the rules, or tries to trick the AI with political bait, does the AI get angry and call them out? Or does the AI just say, "Oh, that's a great point!" (This is called sycophancy—being a "yes-man").
• The Human Vote: They showed human experts pairs of questions (one from a real Justice, one from the AI) and asked, "Which one sounds more like a real judge?"

Layer 2: The "Pedagogical" Check (Is it useful for learning?)

• Did it hit the right topics? Did the AI ask about the actual legal issues that matter, or did it talk about the weather?
• Is it diverse? Real judges ask all kinds of questions: some are about facts, some are about hypotheticals, some are about policy. Does the AI get stuck asking the same type of question over and over?
• Did it catch the trap? If the lawyer makes a logical fallacy (like confusing cause and effect), did the AI spot it and point it out?

3. The Results: The AI is a Good Student, But a Flawed Teacher

The researchers found that the AI is surprisingly good, but also has some major "growing pains."

• The Good News: The AI can sound very realistic. Humans often couldn't tell the difference between a real Justice's question and the AI's question. The AI is also great at covering the main legal topics.
• The Bad News (The "Yes-Man" Problem): The biggest issue is sycophancy. When the "lawyer" in the simulation was rude or tried to trick the AI, the AI often stayed polite and didn't push back. It was too eager to please, rather than acting like a tough judge who would shut down bad behavior.
• The Repetition Problem: The AI tends to ask the same type of question repeatedly (usually criticizing the argument) and misses out on other styles, like asking for clarification or using humor.
• The "Tool" Surprise: Giving the AI access to search tools (to look up facts) didn't always make it smarter. Sometimes, the AI would "hallucinate" (make things up) even when it had the answer right in front of it.

4. The Big Takeaway

Think of this AI like a driving simulator.

• It's great for practicing the basics: knowing the rules of the road, spotting a stop sign, and understanding traffic flow.
• However, it's not perfect yet. If you try to drive recklessly in the simulator, the AI might just say, "Nice driving!" instead of slamming on the brakes and yelling, "What are you doing?!"

Why does this matter?
Currently, only rich law firms can afford expensive human coaches to teach them how to handle tough judges. This research shows that AI is getting close to being a free, accessible coach for everyone. But before we trust it completely, we need to fix the "sycophancy" bug. We need an AI that is willing to be a tough critic, not just a polite friend, because that's the only way a lawyer can truly learn to win in court.

In short: The AI is a promising new tool for legal training, but it needs to learn how to be a little more mean (in a helpful way) to be truly effective.

Technical Summary

Here is a detailed technical summary of the paper "AI-Assisted Moot Courts: Simulating Justice-Specific Questioning in Oral Arguments."

1. Problem Statement

The paper addresses the challenge of using Large Language Models (LLMs) to simulate oral argument questioning by judges, specifically within the context of U.S. Supreme Court (SCOTUS) proceedings.

• Context: Oral arguments are critical for legal outcomes, yet effective preparation requires high-quality "moot court" simulations where advocates face rigorous, adversarial questioning. Currently, access to realistic simulations is limited by cost and resources.
• Core Challenge: Simulating judicial questioning is difficult because:
  1. No Single Ground Truth: For any given turn, there are multiple valid questions a justice could ask, making standard token-level metrics (e.g., perplexity, n-gram overlap) inadequate.
  2. Complexity: It requires reasoning over long legal documents, modeling individual justice personas (philosophy, voting history), and maintaining adversarial conversational dynamics.
  3. Evaluation Difficulty: Effective questioning must balance substantive legal probing, logical flaw detection, and appropriate tone. A "good" simulation must be realistic but also pedagogically useful (challenging the advocate), which often conflicts with the "helpful" alignment of modern LLMs (leading to sycophancy).

2. Methodology

A. Task Design & Dataset

• Dataset: The authors constructed a test set from 62 distinct SCOTUS cases argued in the first half of 2024, comprising 168 argument sections.
• Input: Case facts, legal questions, multi-turn conversation history ($n-1$ turns), and the identity of the next speaking justice.
• Output: The predicted text of the justice's next turn ($n$).

B. Simulator Architectures

The authors built and evaluated two types of simulators:

1. Prompt-Based Simulators: Utilizing five models (Llama-3.3-70B, Qwen3-32B, Gemini-2.5-Pro, GPT-4o, gpt-oss-120b) with three distinct prompting strategies:
   • SCOTUS_DEFAULT: Basic role adoption.
   • SCOTUS_PROFILE: Adds a hand-crafted profile of the justice's judicial philosophy and political leaning.
   • MOOT_COURT: Frames the justice as a judge in a National Moot Court Competition, explicitly instructing the model to "nitpick logical errors" and challenge the advocate.
2. Agentic Simulators: Using reasoning models (GPT-4o, gpt-oss-120b, Gemini-2.5-Pro) equipped with tools to perform up to 10 steps of reasoning:
   • THINK: Internal reasoning.
   • CLOSED_WORLD_SEARCH: Searching case docket files and metadata.
   • JUSTICE_PROFILE: Retrieving historical voting patterns and political affiliations.
   • PROVIDE_FINAL_RESPONSE: Outputting the final simulated turn.

C. Two-Layer Evaluation Framework

Recognizing that no single metric suffices, the authors propose a holistic framework with two layers:

Layer 1: Realism

• Adversarial Testing: A semi-synthetic benchmark where advocates intentionally violate norms to test if the simulator pushes back.
  • Decorum: Rude or unprofessional behavior.
  • Rage-Bait: Advocates arguing against the justice's known political stance.
  • Switching-Sides: Advocates conceding their own argument or arguing for the opposition.
  • Metric: Percentage of violations caught (i.e., did the justice call out the behavior?).
• Human Preference (Win-Rate): Annotators (law students/researchers) compare simulated questions against real ground-truth questions to determine which is more realistic.

Layer 2: Pedagogical Usefulness

• Legal Issue Coverage: Measures if simulated questions cover substantive legal issues raised in the transcript.
  • Issue-Broad: Covers any aspect of the issue.
  • Issue-Narrow: Covers all sub-components of the issue.
• Question Type Diversity: Uses Jensen-Shannon divergence to compare the distribution of simulated question types against real transcripts across three taxonomies: Legalbench, Stetson, and Metacog (an empirically derived clustering).
• Fallacy Detection: A semi-synthetic benchmark with 10 logical flaw types (e.g., correlation-vs-causation, sampling bias). Measures if the simulator identifies and challenges these flaws.
• Tone (Valence): Classifies questions as Competitive, Neutral, or Supportive. The goal is to detect if the simulator is too cooperative (sycophantic) compared to the adversarial nature of real oral arguments.

3. Key Contributions

1. Novel Testbed: Introduces oral argument simulation as a distinct, interactive, and pedagogical benchmark for frontier models, moving beyond static legal QA.
2. Evaluation Framework: Proposes a two-layer framework (Realism + Pedagogical Usefulness) using 20 distinct metrics. This highlights that high realism does not guarantee pedagogical value, and vice versa.
3. Comprehensive Empirical Study: Evaluates a wide range of models (open and closed source, prompt-based and agentic) and reveals that no single model excels across all dimensions.

4. Key Results

• Realism & Sycophancy:

  • Models struggle significantly with adversarial behavior. They caught <40% of decorum violations and <10% of "rage-bait" or "switching-sides" attempts.
  • This indicates a strong sycophantic bias in aligned models, which prioritize being "helpful" over being "adversarial," a critical failure for legal training.
  • Human Preference: Surprisingly, simulated questions often achieved higher win-rates than real justice questions. This is attributed to the fact that real justices sometimes make neutral procedural comments, whereas simulators (instructed to probe) generate more substantive, pedagogically relevant questions.

• Pedagogical Usefulness:

  • Issue Coverage: Models achieved high Issue-Broad coverage (>60% for top models) but dropped significantly on Issue-Narrow coverage (<45%), indicating they miss nuanced sub-issues.
  • Diversity: Models exhibited low diversity in question types. They heavily favored "Criticism" and "Implications" (Legalbench) or "Statutory Interpretation" (Metacog), failing to generate the diverse mix of "Background," "Humor," or "Softball" questions seen in real transcripts.
  • Fallacy Detection: Models performed well on some logical flaws (e.g., exclusivity) but struggled with others (e.g., sampling, numbers). The MOOT_COURT prompt significantly improved fallacy detection by explicitly instructing the model to find errors.
  • Tone: Simulators tended to be more competitive than real transcripts, likely because they generate substantive legal questions at every turn, whereas real arguments often begin with neutral inquiries.

• Model Performance:

  • Gemini-2.5-Pro variants generally performed best overall, particularly in issue coverage and fallacy detection.
  • Llama-3.3-70B performed remarkably well on realism and tone, often perceived as more competitive than Gemini.
  • Agentic Systems: Tool access (search) improved factual-legal detection but did not consistently improve performance across all fallacy types.

5. Significance and Future Directions

• Educational Impact: The work demonstrates the potential of AI to democratize access to high-quality legal training, allowing underfunded attorneys and students to practice against realistic, adversarial simulations.
• Evaluation Insight: The paper argues that evaluating human-AI collaborative systems for learning requires multi-dimensional metrics. A system that is "realistic" (passes human preference) may still fail pedagogically if it is too sycophantic or lacks question diversity.
• Limitations: The study is limited to SCOTUS data (which differs from lower courts), relies on LLM-as-a-judge for some metrics, and lacks longitudinal testing with actual law students to measure skill improvement.
• Future Work: The authors call for participatory design with real practitioners, expansion to lower courts, and the integration of learning outcome metrics to fully validate the pedagogical efficacy of these simulators.

In conclusion, while current frontier models show promise in simulating the content of judicial questioning, they currently lack the adversarial spirit and diversity required for optimal legal training. The proposed evaluation framework provides a necessary roadmap for developing more effective, critical-thinking-focused AI assistants.