RAGAL: A Frugal, Fully Local Retrieval-Augmented Assistant for Technical Support at a Government Agency

Imagine you have a super-smart robot librarian who can read millions of books and answer any question you ask. Usually, to make this robot work, you have to send your questions and the books to a giant, cloud-based brain in a distant data center. But what if your library contains top-secret government files that are illegal to leave the building? What if you can't use the cloud at all? This is the puzzle that sits at the heart of Retrieval-Augmented Generation (RAG). Think of RAG as a two-step dance: first, the robot quickly scans a specific pile of documents to find the most relevant pages (retrieval); second, it uses those pages to write a smart, accurate answer (generation). The big question for many organizations is: Can we build this robot entirely inside our own walls, using only a single, modest laptop, without ever sending a single byte of data to the outside world?

This paper tells the story of RAGAL, a project by a Romanian government agency that built exactly such a robot. They faced three impossible-sounding rules: no data could ever leave their building, the robot could only draft answers (humans had to hit the "send" button), and they had to do it all on a consumer laptop with only 8 GB of memory. Instead of giving up, they turned this "frugal" setup into a lab for clever engineering. They discovered that the secret to success wasn't buying a bigger, more expensive brain, but rather teaching the robot how to look for information better. By fine-tuning their search engine on real support tickets and using a clever trick called "anchor distillation" to prevent the robot from making up fake database commands, they created a system that works safely and effectively. They even built a "slow-motion judge"—a massive AI model that runs so slowly on their CPU it can't chat in real-time, but is perfect for grading the robot's homework overnight. The result is a blueprint for how any organization with strict privacy rules can build a helpful AI assistant without ever needing the cloud.

The Story of RAGAL: A Robot That Never Leaves Home

Meet RAGAL, the new assistant for a Romanian government agency that manages rural investment funds. This agency deals with thousands of sensitive documents and support tickets every day. Because these files contain private citizen data, they are strictly forbidden from leaving the agency's building. This means they couldn't use the usual, powerful AI tools that live in the cloud. They had to build their own "fully local" robot, and they had to do it on a single laptop with a graphics card that only had 8 GB of memory—a tiny amount by AI standards.

The team had to follow three strict rules, which they call constraints:

1. Zero Egress: Not a single piece of data could ever leave the building. No sending questions to the cloud, no downloading training data from the internet. Everything had to happen on their own hardware.
2. Read-Only: The robot was a writer, not an actor. It could draft SQL code (instructions for fixing database errors) or write explanations, but it could never actually run those commands. A human had to review and execute them.
3. Frugality: The entire system was developed, trained, and tested on one consumer laptop.

The Big Discovery: Better Search Beats Bigger Brains

The team started with a common assumption: to get better answers, you need a bigger, smarter AI model. But they quickly found out that was wrong. Their first major win came not from upgrading the "brain" (the generator), but from upgrading the "search engine" (the retriever).

They realized that simply asking the robot to find documents wasn't enough. They needed to teach it how to look. They built a hybrid search system that combined two methods: a "dense" search that understands the meaning of words, and a "sparse" search that looks for exact keyword matches. They also added a "traffic cop" (intent routing) that decides which type of document to look at based on the question. If you ask about a procedure, it looks at manuals; if you ask about a database fix, it looks at old support tickets.

This simple change was a massive upgrade. Before any fancy training, this hybrid search boosted their success rate from 62% to 81%. It was the cheapest and most effective improvement they made.

The 8 GB Miracle: Training on a Laptop

The next challenge was training the search engine on their own data. They had about 15,000 resolved support tickets and internal documents. They wanted to teach the search engine to understand the specific language of their agency.

Usually, training a model like this requires a massive supercomputer. But the team had only 8 GB of memory. They discovered a hidden trap: on their Windows laptop, the graphics card driver would silently "spill" data into the regular computer memory when it ran out of space, making the training incredibly slow without ever crashing. It was like trying to run a marathon with a backpack full of water that you didn't know you were carrying.

To fix this, they used a special recipe:

• 8-bit Optimizer: A way to shrink the math the computer has to do.
• Gradient Checkpointing: A technique that saves memory by recalculating some steps instead of storing them.

With these tricks, they managed to fully train their search engine in just 72 minutes on that single laptop. The results were stunning: the ability to find the right document in the top 10 results (Recall@10) jumped from 0.663 to 0.850.

The Silent Trap: One Domain, Two Worlds

Here is where the story gets tricky. When they trained the search engine only on the support tickets, it got really good at finding tickets. But, in a silent and invisible way, it got worse at finding the official documents. The training had "deformed" the search engine's understanding, making it forget how to handle the formal documents.

They only discovered this because they built a separate test set just for documents. If they hadn't, they would have thought their system was perfect, while it was actually failing on half their data. To fix this without sending data to the cloud, they used a clever trick called GenQ. They used a local AI to generate fake questions for their documents, creating a new training set that balanced both worlds. This repaired the damage, and the search engine became great at finding both tickets and documents again.

Safety First: Stopping the Robot from Lying

Since the robot was drafting database commands, the team had to make sure it never made things up (hallucinated). They used a technique called Anchor Distillation. Instead of letting the AI write the SQL code from scratch, they took real, working code from past tickets, copied it exactly, and only let the AI write the text around it (explaining what it does and when to use it). This made it impossible for the robot to invent fake database commands because the code itself was a real, verified "anchor."

They also found a surprising benefit to masking PII (hiding names and ID numbers). When they replaced real names with placeholders like <user> or <project_id> before training, the robot actually wrote better answers. It stopped copying specific people's names from old tickets and started writing flexible templates that could be used for anyone.

The Slow Judge

Finally, how do you grade a robot if you can't use a cloud-based AI to check its work? The team built a "slow judge." They used a massive AI model (744 billion parameters) that was too big to run fast. It ran on their computer's main processor (CPU) instead of the graphics card, moving data from the hard drive as it went. It was incredibly slow, taking about 10 to 13 minutes to grade a single answer. But, because it was so smart, it was perfect for grading the robot's homework overnight.

They found that this slow judge was a great "second opinion," but not perfect. It sometimes disagreed with human experts, especially on the very best or very worst answers. The lesson was to use the judge to find trends and patterns, but always have a human double-check the specific verdicts.

The Takeaway

The RAGAL project proves that you don't need a supercomputer or the cloud to build a powerful, safe AI assistant. By focusing on clever engineering—better search strategies, careful data handling, and strict safety rules—a small team with a single laptop can create a system that respects privacy and delivers real value. They showed that sometimes, the best way to move forward is to move slower, keep everything local, and pay close attention to the details.