Can LLMs Synthesize Court-Ready Statistical Evidence? Evaluating AI-Assisted Sentencing Bias Analysis for California Racial Justice Act Claims

Imagine the legal system as a giant, complex machine. For decades, this machine has sometimes treated people differently based on the color of their skin, even when they committed the same crimes. In California, a new law called the Racial Justice Act (RJA) was passed to fix this. It says: "If we can prove with numbers that the system is biased, we should give people a second chance."

But here's the problem: Proving bias with numbers is like trying to solve a massive, 10,000-piece puzzle while wearing blindfolds. It requires expensive statisticians, months of digging through dusty records, and complex math that most lawyers (and certainly most defendants) can't afford. This has created a "Second-Chance Gap"—hundreds of people who could be freed or have their sentences reduced, but can't because they can't prove the bias.

Enter Redo.io, a new tool built by the author, Aparna Komarla. Think of Redo.io as a "Digital Detective" powered by Artificial Intelligence (AI). Its job is to help lawyers find that missing puzzle pieces quickly and turn raw data into a story a judge can understand.

Here is how the paper breaks down, using simple analogies:

1. The Problem: The "Second-Chance Gap"

Imagine you are in a race where the starting line is moved back for some runners just because of their skin color. The new law (RJA) says, "If you can show the starting line was moved, we'll let you run again."

The Issue: To prove the starting line was moved, you need a surveyor with expensive equipment to measure the track. Most runners (defendants) don't have a surveyor. They have to do the math themselves, which is nearly impossible.
The Result: Many people stay in prison even though they might be innocent of the unfairness of their sentence.

2. The Solution: The "Digital Detective" (Redo.io)

Redo.io is a free, open-source platform that acts as a bridge between raw data and legal stories.

The Data: The system has already gathered 95,000 prison records (like a giant library of case files) using public records laws.
The Math: It uses traditional, reliable math (like a calculator) to crunch the numbers. It compares how different groups were sentenced for similar crimes.
The AI Role: This is the magic part. The AI doesn't do the math itself (because AI can be bad at math). Instead, the AI acts as a translator. It takes the cold, hard numbers and writes a "court-ready" story. It explains what the numbers mean, why they matter, and where the data might be missing pieces.

3. How It Works: The "Recipe" Analogy

Imagine you are baking a cake (the legal argument).

The Ingredients: The lawyer puts in the details: "I want to compare Black defendants to White defendants in Contra Costa County who were charged with robbery."
The Mixing Bowl: The system runs three different types of "math tests" (Odds Ratio, Relative Risk, Chi-Square). Think of these as three different chefs tasting the batter to see if it's sweet enough.
The AI Baker: Once the math is done, the AI writes the recipe card. It says: "The batter is too sweet (biased). Here is the evidence. But be careful, we only have a small sample of this specific cake, so the taste might be slightly off."
The Safety Check: The lawyer (the human) must read the card and say, "Yes, this looks right," before handing it to the judge. The AI is the assistant, not the boss.

4. The Test: Did the AI Pass the Exam?

The researchers tested this "Digital Detective" by having it write 30 reports and then comparing them to reports written by human statisticians. They used a "Judge AI" to grade the work.

The Good News (The AI's Superpowers):

Ethics: The AI was perfect at making sure the story didn't blame the people for the bias. It correctly stated that the system was the problem, not the defendants' behavior.
Honesty: It was very good at admitting when the data was messy or incomplete (e.g., "We couldn't see all the records because of privacy laws").

The Bad News (Where the AI Stumbled):

The "Small Sample" Trap: The AI sometimes struggled to explain why a small number of cases makes the math less reliable. It's like a weather forecaster saying, "It might rain," based on looking out the window for only 10 seconds. A human expert would say, "That's not enough time to know for sure."
Conflicting Math: Sometimes the three "chefs" (math tests) gave slightly different answers. The AI had a hard time explaining why they differed, which is a crucial part of a legal argument.

5. The Big Picture: A Tool, Not a Replacement

The paper concludes that AI is not a replacement for human lawyers or statisticians. You wouldn't let a robot drive a car in a storm without a human holding the wheel.

Instead, AI is a powerful co-pilot.

Before: A lawyer might spend 3 months and thousands of dollars trying to find one piece of evidence.
Now: With Redo.io, they can do it in an hour for free.

The Takeaway

This paper is about democratizing justice. It's about giving the "little guy" (the public defender or the person representing themselves) the same high-tech tools that big, wealthy law firms use.

The AI isn't perfect yet—it needs human supervision to catch its math blind spots—but it has the potential to open the "Second-Chance Gap" wide open, allowing hundreds of people to finally get a fair hearing on whether their sentences were influenced by racial bias. It turns a mountain of confusing data into a clear, understandable story that a judge can actually read and act upon.

Here is a detailed technical summary of the paper "Can LLMs Synthesize Court-Ready Statistical Evidence? Evaluating AI-Assisted Sentencing Bias Analysis for California Racial Justice Act Claims."

1. Problem Statement

The paper addresses the "second-chance gap" in California's criminal legal system. Despite the Racial Justice Act (RJA) of 2020, which allows defendants to challenge convictions based on statistical evidence of racial disparities in sentencing and charging, implementation has lagged.

The Barrier: Proving RJA claims requires sophisticated statistical analysis (e.g., Odds Ratios, Chi-Square tests) and the ability to synthesize these findings into legal narratives.
The Inequity: Public defenders and self-represented litigants often lack the resources to hire statisticians or build complex databases, while District Attorneys have significant advantages.
The Limitation of Existing Tools: Current tools (e.g., the Paper Prisons Project) provide raw disparity metrics but fail to control for confounding variables (like prior criminal history) or offer interpretive guidance for non-statisticians.
The Core Challenge: Can Large Language Models (LLMs) reliably synthesize complex statistical outputs into "court-ready" evidence reports that are accurate, ethically sound, and legally contextualized, without perpetuating systemic biases?

2. Methodology

The authors developed Redo.io, an open-source platform designed to bridge the gap between raw data and legal argumentation.

A. System Architecture

Data Source: The system processes 95,000 prison records acquired via the California Public Records Act (CPRA) from the California Department of Corrections and Rehabilitation (CDCR).
Hybrid Approach: The system deliberately separates statistical computation from narrative generation to mitigate LLM hallucination risks:
1. Statistical Engine: Uses traditional programming (Python/R) to perform precise calculations for Odds Ratios (OR), Relative Risk (RR), and Chi-Square ( $\chi^2$ ) tests.
2. LLM Interpretive Layer: An LLM (specifically GPT-5-mini) is used only for downstream tasks: synthesizing the computed results into cohesive narratives, contextualizing confidence intervals, and flagging data limitations.
User Workflow: Attorneys define "similarly situated" cohorts (e.g., filtering by offense type, county, and prior commitments) and select reference groups. The system computes disparities and prompts the LLM to generate an evidence report.

B. Evaluation Framework

The authors evaluated the LLM's performance against human statisticians using a LLM-as-a-Judge framework alongside human evaluators.

Dataset: 30 generated reports covering various counties, outcomes, and ethnic groups.
Evaluation Rubric (6 Dimensions):
1. Confidence Interval (CI) Contextualization: Does the report correctly interpret CI width and the inclusion/exclusion of 1.0?
2. P-Value Contextualization: Does it correctly identify statistical significance ( $p < 0.05$ )?
3. Dataset Limitations: Does it acknowledge missing data (e.g., juvenile records, non-convictions) due to CPRA privacy restrictions?
4. Sample Size & Balance: Does it contextualize findings based on group representativeness and sample adequacy?
5. Appropriate Attribution: Does it attribute disparities to systemic factors (prosecutorial discretion, bias) rather than inherent criminal behavior?
6. Method Comparison: Does it reconcile divergent results between OR, RR, and $\chi^2$ (e.g., explaining why OR overestimates risk when outcomes are common)?

3. Key Contributions

Redo.io Platform: A free, open-source tool that democratizes access to complex statistical analysis for RJA claims, reducing the time to generate a bias analysis from weeks/months to ~1 hour and the cost from thousands of dollars to free.
Hybrid AI Architecture: Demonstrates a viable design pattern where LLMs act as interpretive bridges rather than statistical reasoners, preserving numerical accuracy while leveraging LLM fluency for legal narrative construction.
Combinatorial Synthesis: Shows how LLMs can handle the exponential complexity of synthesizing multiple statistical methods (16,384+ potential template combinations) better than rigid rule-based systems.
Ethical Guardrails: The system is designed to force the attribution of disparities to systemic causes, preventing the AI from reinforcing harmful narratives about racial criminal propensity.

4. Results

The evaluation compared GPT-5-mini against human statisticians on a 0–1 scale.

Overall Performance: The LLM achieved an average score of 0.71, compared to 0.76 for human evaluators.
Strengths:
- Attribution (Score: 1.00): The model perfectly attributed disparities to systemic factors, avoiding inherent racial bias in its language.
- P-Value Contextualization (Score: 0.89): The model reliably identified statistical significance thresholds.
- Dataset Limitations: The model acknowledged data constraints, though human evaluators scored this dimension higher (0.94 vs. 0.75), suggesting humans are more generous or nuanced in recognizing specific privacy exclusions.
Weaknesses:
- Sample Size Contextualization (Score: ~0.49–0.61): The LLM frequently failed to contextualize findings with respect to group balance ratios or the reliability implications of small sample sizes.
- Cross-Method Comparison (Score: 0.69): The model struggled to explain why OR and RR diverge in specific scenarios, showing high variability ( $\sigma = 0.11$ ) in its reasoning.
Implication: The AI is effective for prima facie showings and initial case screening but requires mandatory attorney review before court submission, particularly regarding sample adequacy and reconciling conflicting statistical signals.

5. Significance

Access to Justice: The tool significantly lowers the barrier to entry for RJA claims, potentially unlocking resentencing opportunities for hundreds of incarcerated individuals who are currently unable to prove systemic bias due to resource constraints.
AI in High-Stakes Domains: The paper provides a critical case study on the "Human-in-the-Loop" necessity. It argues that LLMs are best suited as drafting assistants that augment professional judgment rather than autonomous agents for statistical reasoning in legal settings.
Legal-Statistical Interface: It highlights the tension between statistical significance (p-values) and legal standards of "meaningful difference," suggesting that future AI tools must evolve alongside judicial precedents to remain effective.
Future Directions: The authors call for further research into deployment safeguards, user studies on attorney interaction, and improved evaluation of LLMs on edge cases (e.g., highly imbalanced cohorts).

In conclusion, the paper demonstrates that while LLMs cannot yet replace statisticians, they can serve as powerful, ethically constrained accelerators for generating court-ready evidence, provided they are integrated into a workflow that retains human oversight for critical statistical interpretation.