Hyperparameter Analysis in Retrieval-Augmented Generation Systems Applied to the Public Prosecutor’s Office of the State of Espírito Santo Corpus

Imagine you have a super-smart robot friend who can write essays, tell jokes, and answer almost any question you ask. This robot is like a brilliant student who has read the entire internet. But here's the catch: this student has a terrible memory for facts. If you ask it about a specific law, it might confidently make up a rule that sounds perfect but is completely fake. This is called "hallucination," and it's a big problem when you need accurate information, like in a courtroom or a hospital.

To fix this, scientists invented a clever trick called Retrieval-Augmented Generation (RAG). Think of it like giving your robot friend a stack of textbooks right before the exam. Instead of guessing from its own memory, the robot first looks up the answer in the books, then writes its response based only on what it found. It's like a detective who checks the evidence file before writing the report. But there's a twist: how you organize those books matters. Do you tear the pages into tiny scraps? Do you keep whole chapters together? And how many pages do you let the robot read before it starts writing? If you get these settings wrong, the robot might miss the crucial clue or get confused by too much noise. This is exactly the puzzle a team of researchers in Brazil decided to solve.

---

The Detective's Dilemma: Tuning the Machine

In the state of Espírito Santo, Brazil, the Public Prosecutor's Office (MPES) has a massive library of legal documents—thousands of laws, rules, and ordinances. For years, finding a specific rule was a nightmare. Civil servants had to know exactly where to look, or they'd get stuck. To help, they built a virtual assistant named Fabi (named after a real, super-knowledgeable employee). Fabi uses the RAG trick: she reads the legal library and answers questions for the staff.

But Fabi wasn't perfect. Sometimes she missed important details; sometimes she gave vague answers. The researchers, led by Heitor Quartezani and his team, asked a simple question: What settings make Fabi the best detective possible? They didn't just guess; they treated the problem like a giant science experiment. They tested every combination of four different "knobs" on the system to see which one produced the most accurate, honest, and helpful answers.

Here is what they turned and what they found:

1. The "Cutting" Strategy (Chunking)

Imagine you have a long, complex legal story. How do you cut it up to give it to the robot?

• The Old Way (Recursive): Cut the story into fixed-size chunks, like slicing a loaf of bread into 500-character pieces. This is easy, but you might cut a sentence in half, losing the meaning.
• The Smart Way (Semantic): Cut the story only when the topic changes. If the law switches from "vacation rules" to "overtime pay," that's where you cut. This keeps the ideas whole.

The Finding: The researchers discovered that the "Smart Way" (Semantic cutting) was the clear winner. Specifically, cutting based on the 95th percentile of topic changes (meaning, only cutting when the topic shifts significantly) kept the most information intact. The "Old Way" of slicing into tiny 500-character pieces was terrible; it broke the legal logic apart, making it impossible for the robot to find the right answer. In legal work, missing a single detail can be disastrous, so keeping the story whole was the most important factor.

2. The Search Method (Retrieval)

How does Fabi find the right pages in the library?

• Vector Search: This is like asking the robot, "Find me things that feel like this question." It's great for understanding meaning but might miss exact words.
• Lexical Search: This is like a classic library card catalog. It looks for exact word matches. It's great for specific terms but bad at understanding context.
• Hybrid Search: This is the best of both worlds. It uses a special math trick (called Reciprocal Rank Fusion) to combine the "feeling" search and the "exact word" search.

The Finding: The Hybrid Search was the champion. By combining both methods, Fabi could find the right documents even if the user asked a question in a weird way or used a very specific legal term. The "feeling" search alone wasn't enough, and the "exact word" search alone was too rigid. Together, they made the system much more reliable.

3. How Much to Read (Top-K)

When Fabi finds the best documents, how many should she read before answering? Should she read 5 pages? 20?

• The Finding: The more she read, the better she got—at least up to a point. Reading 20 documents gave the best results. It turned out that giving the robot a bigger "net" to catch information helped it find the right clues, even if some of the extra pages were a bit noisy. However, the researchers noticed that after 15 documents, the improvement in the final answer quality stopped getting statistically significant. So, while reading 20 was slightly better, it cost more computer power. They suggested that in the future, the system could use a smarter filter to read fewer pages but still get the same great results.

4. The Brain Power (LLM Choice)

Finally, they tested two different "brains" for Fabi: the massive, super-smart gpt-4o and the smaller, faster gpt-4o-mini.

• The Surprise: You might think the bigger, more expensive brain would win. But the smaller gpt-4o-mini actually performed just as well, and sometimes even better!
• Why? The giant brain was so smart that it sometimes tried to be too creative. When the robot read 20 documents, the big brain would start mixing ideas or over-interpreting the text, leading to small errors. The smaller brain, however, was more obedient. It stuck strictly to the text provided, extracting the facts without adding its own "flavor." For a legal assistant who needs to be precise, being a bit more literal was actually a superpower.

The Final Verdict

The researchers didn't just guess; they ran 3,840 different experiments (that's 96 different settings tested against 40 different questions) and used advanced math to prove their results.

They concluded that to build the best legal assistant:

1. Cut the documents by topic, not by fixed size.
2. Use a Hybrid Search that combines meaning and exact words.
3. Read about 20 documents to ensure nothing is missed.
4. Use the smaller, cheaper AI model (gpt-4o-mini) because it follows instructions more strictly and makes fewer mistakes.

This study is a big deal because it moves beyond "let's try this and see what happens." It uses rigorous science to tell us exactly how to tune these powerful tools so they don't just sound smart, but actually are smart and reliable. For the Public Prosecutor's Office, this means Fabi can now help them find the right laws faster and with fewer mistakes, making the legal system work better for everyone.