When JSON Is Not Enough: Semantic Reliability of Schema-Constrained LLM Ordering Agents
This paper introduces OrderBench, a benchmark demonstrating that while JSON Schema constraints ensure syntactic validity for LLM ordering agents, they fail to guarantee semantic reliability or safety, necessitating additional domain verification and fail-closed execution mechanisms.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are hiring a very talented, super-fast robot chef to take your order at a busy restaurant. You speak to it in plain English, saying things like, "I want a burger, no pickles, and make sure it's gluten-free." The robot's job is to translate your words into a strict, digital ticket that the kitchen's computer can read. For a long time, the big worry was that the robot might write the ticket in a messy format the computer couldn't understand—like scribbling on a napkin instead of filling out a form. To fix this, engineers gave the robot a rigid template (called a "schema") to force it to write the ticket perfectly, ensuring every box was checked and every number was in the right place.
But here is the twist: just because the ticket looks perfect on the form doesn't mean the order is actually right. The robot might fill out the form flawlessly but still write "add pickles" when you said "no pickles," or it might order a burger when the kitchen is out of meat. This paper dives into that specific danger. It asks: If the robot follows the rules of the form perfectly, can we trust it to actually get the food order right? The researchers built a test kitchen to see if these smart robots are truly reliable or if they are just really good at faking the paperwork.
The Paper: When the Form is Perfect, but the Order is Wrong
This paper, titled "When JSON Is Not Enough," investigates a critical gap in how we use AI agents to handle real-world tasks like ordering food. The authors, led by Yin Li from the University of Birmingham, created a test called OrderBench. Think of OrderBench as a rigorous, no-nonsense exam for AI robots trying to take restaurant orders. They didn't just ask, "Did the robot write a valid ticket?" They asked, "Did the robot actually understand what you wanted?"
The researchers tested four different AI models using 300 specific scenarios. These scenarios covered tricky situations like:
- Negation: "I want a pizza with no cheese."
- Scope: "I want two burgers, but only the first one needs extra sauce."
- Safety: "I'm allergic to peanuts, but the menu says this sauce has them."
- Availability: "I want the special of the day," when the special is actually sold out.
They ran these tests in two ways:
- Prompt-only: Telling the robot, "Please write your answer in a JSON format" (a standard data format).
- JSON-schema mode: Forcing the robot to use a strict, pre-defined template that guarantees the output is technically valid.
The Big Surprise
The results were a wake-up call for engineers. The study found that making the output look perfect (schema validity) does not mean the content is correct (semantic reliability).
Here is what the data showed:
- The "Perfect" Paperwork Trap: Even the strongest AI model tested, GPT-OSS 120B-fast, achieved 100% success in creating technically perfect tickets in both modes. However, when it came to getting the actual order right, its success rate dropped to 83.0% in the prompt-only mode and 81.3% in the strict schema mode.
- The Dangerous Illusion: The most shocking finding came from the smaller, weaker models. The Gemma-2-2B model produced 100% technically valid tickets when forced to use the strict schema. Yet, it only got the actual order right 2.0% of the time. Worse, it committed unsafe acceptances—accepting orders that should have been rejected (like ordering an allergen-heavy dish for someone with an allergy)—41.7% of the time.
- The Middle Ground: Another model, Qwen3-30B-A3B, also hit 100% on the technical validity test, but its real-world success was only around 30%, with unsafe acceptances hovering near 15%.
What This Means for the Future
The paper argues that relying solely on "structured output" (forcing the AI to fill out a form) is not enough. It is like having a robot that can fill out a tax form without a single typo, but the numbers it writes down are completely wrong.
The authors conclude that schema validity is a necessary first step, but it is not a substitute for checking the meaning. Just because an AI agent can produce a perfectly formatted JSON object doesn't mean it is safe to let that agent charge your credit card or reserve your food. The study suggests that we need a "fail-closed" system: if the AI isn't 100% sure the order is correct and safe, it shouldn't just send the ticket; it should stop and ask for clarification.
In short, the paper warns us not to be fooled by the neatness of the form. A robot that follows the rules of the form perfectly can still be a terrible chef. We need to verify the content of the order, not just the format.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.