Business Truth, not SQL Accuracy: A Rule-Gated 7B Analytics Agent Outperforms a Direct-Prompted 32B Baseline

The Detective vs. The Calculator: Why Being Right Matters More Than Being Fast

Imagine you are in a giant, chaotic library where millions of books are constantly being rewritten, moved, or thrown away. You ask a librarian, "How many red books were sold last Tuesday?" A traditional computer program acts like a super-fast calculator: it immediately starts searching, pulls out a number, and shouts it back at you. If the math is perfect, the computer feels proud. But what if "red" meant "crimson" to you but "scarlet" to the librarian? Or what if the red books were actually moved to a different building last week, making the question impossible to answer? The calculator would still give you a number, but it would be the wrong one, and nobody would know until it was too late.

This is the world of LLM analytics agents. These are AI systems designed to talk to databases (the libraries) and answer business questions. For a long time, scientists measured their success by checking if the AI wrote the correct "code" (SQL) to ask the library a question. It's like grading a student only on whether they used the right grammar, ignoring whether they actually understood the story. But in the real world, a "perfect" sentence that leads to a wrong number is a disaster. It's like a GPS that gives you perfect directions to a house that burned down last year. The real challenge isn't just writing the code; it's knowing when not to answer, when to ask for clarification, and when to say, "I can't help with that." This paper dives into that messy, real-world problem, asking: Is it better to have a giant, expensive brain that guesses confidently, or a smaller, rule-following detective that double-checks everything?

The Story of QueryProof: The Rule-Bound Detective

In this study, a researcher named Morris Lee built a new kind of AI agent called QueryProof. The goal was to test a bold idea: maybe we don't need a massive, 32-billion-parameter "brain" to be smart. Maybe we just need a smaller, 7-billion-parameter brain that is forced to follow strict rules and check its own work.

To test this, the researcher created a special playground called WarehouseReliabilityBench. Imagine two fake libraries (one for an online store, one for a software company) filled with tricky traps. Out of 400 questions, about half were designed to be impossible to answer correctly with a simple number. Some questions were ambiguous (like asking for "revenue" without saying if you meant "gross" or "net"), some asked for data that didn't exist, and some tried to trick the AI into breaking the rules. In these cases, the correct answer wasn't a number; it was a polite refusal, a request for more details, or a "I can't do that" message.

The experiment compared three main characters:

1. The Giant Brain (32B): A huge model prompted directly to answer everything. It had no safety net, no rules, and no one checking its work.
2. The Rule-Following Detective (QueryProof): A smaller 7B model, but this one was wrapped in a "deterministic state machine." Think of this as a strict supervisor who forces the AI to check a rulebook before speaking. If the question is ambiguous, the supervisor says, "Stop! Ask for clarification." If the data is missing, the supervisor says, "Stop! Refuse to answer." The AI only gets to speak if it passes these checks.
3. The Cost-Matched Baseline: A smaller model with some extra examples (few-shot) to see if just giving it more hints helps.

The Big Reveal: Rules Beat Raw Power

The results were surprising and very clear. The Giant Brain was confident but often wrong. It tried to answer every single question, even the impossible ones, and ended up giving wrong business numbers on most of them. Its "False Success Rate" (the ratio of wrong answers to all answers it returned) was a staggering 0.754. It was like a student who guesses on every test question and gets a high score on "trying," but fails the actual test.

QueryProof, the smaller detective with the rulebook, completely outperformed the giant.

• Accuracy: QueryProof achieved a Business Truth Rate of 0.537 (meaning it got the right answer or the right behavior 53.7% of the time), while the Giant Brain only managed 0.300.
• Safety: When QueryProof did return an answer, it was wrong less often than the Giant Brain. Specifically, 35.1% of the answers QueryProof returned were incorrect, compared to 75.4% of the answers returned by the Giant Brain. Crucially, on the specific questions that could be answered, the Giant Brain returned zero wrong numbers that went undetected, whereas QueryProof returned 13 answers to questions that actually required clarification or refusal. The key win was that QueryProof stopped the "silent failures" where a wrong number is returned without any warning, ensuring that no wrong business number was ever returned on an answerable task.
• Cost: Here is the kicker. QueryProof was 71.0% cheaper per correct answer than the Giant Brain. It didn't need a massive model to be reliable; it needed a good system of checks.

The paper found that the "magic" wasn't in the AI's brain size. It was in the deterministic layer—the rigid rules that checked the question against the database's actual structure before the AI ever spoke. This layer acted like a bouncer at a club, stopping bad questions before they could cause trouble.

What Didn't Work (And Why That's Important)

The researchers also tried to add some "smart" features to QueryProof, hoping to make it even better, but these failed.

• The Confidence Model: They tried to teach the AI to "feel" how confident it was and to skip questions it wasn't sure about. This didn't work. On the test, the AI's learned confidence was actually worse than a simple, hand-written rule. It was like a student who thinks they know the answer but is actually guessing, while the simple rule was just being honest.
• The Routing System: They tried a system that would send hard questions to a bigger model. This backfired on the test set, causing the AI to refuse too many questions it could have answered.

The paper is very careful to say that these failures are real. The "smart" parts didn't generalize from practice to the real test. The only thing that worked was the boring, rigid, rule-based checking.

The Catch: How Sure Are We?

The researchers are honest about the limits of their story.

• The Playground: The libraries were synthetic (fake), built from a single seed. While this makes the test fair and reproducible, we don't know for sure if this works in a messy, real-world company with decades of bad data.
• The "Leak": The researchers admitted that during the setup, they accidentally found some specific phrases that matched the test questions. They removed them before the final test, but they acknowledge that the test isn't perfectly "clean" because they knew about the questions while building the rules. They treat the results as a "frozen evaluation" with a known flaw, not a perfect proof.
• The Stats: When they re-ran the math using "families" of questions instead of individual questions, the confidence intervals got wider. This means the direction of the result (Rules > Raw Power) is likely true, but the exact size of the win might be a bit fuzzy.

The Takeaway

The main lesson here is that for business analytics, reliability comes from structure, not just size. A smaller AI, when wrapped in a strict system of rules that check the data and the question before answering, can be more accurate, safer, and cheaper than a massive AI that just guesses.

The paper suggests that if you want an AI that doesn't lie to your boss, you shouldn't just buy a bigger model. You should build a better "rulebook" and a "bouncer" to check the work. The "smart" learning parts (like confidence scores) might be nice to have, but in this specific case, they didn't help. The real hero was the boring, deterministic check that said, "Wait, let's make sure this question makes sense before we answer."