JUDICATA: An Agentic Neuro-Symbolic Architecture for Legal Reasoning with Adversarial Defense and Procedurally Contestable Argumentation

Imagine a world where computers can read and write like humans, but they have a very tricky habit: they sometimes make things up with total confidence. This is the world of Large Language Models (LLMs), the super-smart AI brains behind many of the chatbots we use today. While these models are amazing at writing stories or answering general questions, they struggle in places where being wrong is dangerous, like a courtroom. In law, you can't just guess; you need to connect specific facts to specific rules to reach a conclusion. If an AI invents a fake law or a fake court case, it could ruin someone's life. This paper tackles that exact problem: how do we build an AI lawyer that doesn't just "hallucinate" (make things up) but actually follows the strict, logical rules of the law, while also being tough enough to stop hackers from tricking it?

The researchers behind this study, called JUDICATA, decided to stop trying to make the AI smarter on its own and instead built a "safety net" around it. Think of a standard AI like a talented but impulsive student who loves to guess. JUDICATA is like putting that student in a strict courtroom with a team of guardians. First, there's a Retrieval Team that acts like a super-fast librarian, digging up the exact laws and past cases needed for the answer, rather than letting the AI guess from memory. Second, there's a Security Team that acts like a bouncer at a club, checking every question to make sure no one is trying to sneak in a trick or a lie. Finally, there's a Logic Team that forces the AI to write its answer in a specific format, like a legal argument with a clear claim, evidence, and a rebuttal, so humans can check the work.

The paper shows that when you combine these teams, the AI stops making up fake laws. In tests, the system managed to block about 94.4% of malicious attempts to trick it, and it significantly reduced the number of fake facts the AI tried to pass off as truth. The researchers found that the most important part of this whole system was the "Logic Team" using a method called the Toulmin Model. When they removed this part, the system's performance crashed, and the AI started making up facts again. This proves that the secret isn't just having a bigger, smarter brain, but having a strict, step-by-step process that keeps the AI honest.

The Story of JUDICATA: An AI Lawyer with a Safety Net

Imagine you are a lawyer preparing for a big trial. You need to find the perfect law to support your case, but you can't afford to make a mistake. If you cite a law that doesn't exist, you lose the case. Now, imagine you have a robot assistant to help you. The problem is, this robot is a bit of a daydreamer. It's so good at sounding confident that it often invents fake laws and fake court cases just to fill in the gaps. This is called "hallucination," and in the legal world, it's a disaster.

The authors of this paper, JUDICATA, asked a simple question: How do we stop the robot from daydreaming and make it follow the rules? They didn't try to retrain the robot's brain to be perfect. Instead, they built a neuro-symbolic architecture. That's a fancy way of saying they combined the robot's "intuition" (neural) with a strict "rulebook" (symbolic). They built a system that acts like a three-layered fortress, designed to catch errors before they ever reach the judge.

The Three Guardians of the System

The JUDICATA system is organized into three main parts, working together like a well-oiled machine.

1. The Super-Librarian (HyPA-RAG)
First, the system needs to find the right information. Standard AI systems often just search for words that look similar, which can lead to mistakes. JUDICATA uses a "Tri-Modal" search, which means it looks for answers in three different ways at once:

• The Word Match: It looks for exact legal terms (like "habeas corpus") to make sure it gets the right vocabulary.
• The Meaning Match: It understands the idea behind the question, even if the words are different.
• The Structure Match: It looks at how laws are connected in a hierarchy (like how a specific article fits inside a paragraph).

The system mixes these three search results together. If the question is simple, it trusts the word match more. If the question is complex, it leans on the structure and meaning. This ensures the AI is always holding a real law in its hand, not a fake one it made up.

2. The Bouncer (Adversarial Defense)
Before the AI even gets to read the laws, it has to pass a security check. Hackers and tricksters often try to "jailbreak" AI by asking weird questions or hiding bad instructions inside legal-sounding text. JUDICATA has a Dual-Pass Defense system that acts like a strict bouncer.

• Pass 1 (The Syntax Check): It scans the question for suspicious patterns, like code injections or known trick phrases.
• Pass 2 (The Meaning Check): It uses a smart brain to see if the question feels like a trick, even if the words look normal.

In their tests, this system stopped 94.4% of attacks (755 out of 800) without blocking any legitimate questions. It's like a bouncer who knows exactly who to let in and who to turn away, even if the bad guys are wearing a disguise.

3. The Judge's Clerk (LSIM and Toulmin)
Once the AI has the right laws and has passed security, it has to write the answer. This is where the magic happens. Instead of letting the AI write a free-form essay, JUDICATA forces it to use a specific structure called the Toulmin Model. This model breaks an argument down into six parts:

• Claim: What you are saying.
• Data: The facts you have.
• Warrant: Why the facts support the claim.
• Backing: Extra support for the warrant.
• Rebuttal: What could go wrong or what the other side might say.
• Qualifier: How sure you are (e.g., "probably" or "definitely").

By forcing the AI to fill in these boxes, the system prevents it from making things up. If the AI tries to invent a fact, it won't fit into the "Data" box, and the system catches it. This makes the AI's reasoning procedurally contestable, meaning a human lawyer can look at the answer and say, "Okay, here is your claim, here is your evidence, and here is your rebuttal. I can check this."

What the Experiments Showed

The researchers tested this system with three different AI brains: a small one running on a local computer (Mistral 7B), a huge one in the cloud (Kimi K2), and a commercial one (Gemini 2.5 Flash). They wanted to see if the system worked no matter which brain they used.

The results were clear: The structure matters more than the size of the brain.

• Reducing Fake Facts: Without the JUDICATA system, the AI was prone to making up facts. With the system, the risk of hallucination dropped significantly. For the local Mistral model, the risk of hallucination went down to 0.898 (on a scale where 0 is perfect and 1 is total chaos), and for the cloud Kimi model, it dropped to 0.467.
• The "Safety Tax": The system did make the AI slightly more cautious. In some tests, the AI cited fewer laws because it refused to cite ones it wasn't 100% sure of. The authors call this a "safety tax"—it's better to cite fewer laws correctly than to cite many laws incorrectly.
• The Critical Component: The most important finding came from an "ablation study," where they turned off parts of the system one by one. When they turned off the Toulmin Model (the structured argument), the system's performance crashed. The "Composite Score" (a measure of how good the answer was) dropped by about 27%, and the risk of hallucination skyrocketed to over 0.94. This proved that the strict argument structure is the most important part of keeping the AI honest.

Why This Matters

The paper argues that we don't need to wait for AI to become perfect on its own. Instead, we can build systems that force AI to be honest. By combining a smart search engine, a tough security guard, and a strict argument format, JUDICATA creates an AI that can be trusted in a courtroom.

The system also respects privacy. Because it can run entirely on a local computer (without sending data to the cloud), it follows strict privacy laws like Brazil's LGPD and the EU's AI Act. This means a lawyer can use the AI to help with a sensitive case without worrying that the client's secrets are being sent to a big tech company.

In the end, JUDICATA shows that the future of legal AI isn't about having a bigger, smarter robot. It's about building a better team around the robot, making sure it checks its work, follows the rules, and never, ever makes things up. As the authors suggest, this approach turns a chaotic, daydreaming AI into a reliable, contestable, and safe legal assistant.