Token-Oriented Object Notation vs JSON: A Benchmark of Plain and Constrained Decoding Generation

This benchmark study evaluates Token-Oriented Object Notation (TOON) against JSON for LLM data serialization, finding that while TOON offers promising token efficiency for complex structures via in-context learning, its advantage is often negated by prompt overhead in short contexts and it currently underperforms constrained decoding for simple structures, suggesting its true potential follows a non-linear scaling curve dependent on task complexity.

The Gist

Imagine you are a chef (the AI) trying to send a complex recipe to a sous-chef (the computer system) so they can cook it perfectly.

This paper is a taste test comparing three different ways to write down that recipe:

1. JSON (The Classic Cookbook): The standard, well-known format everyone uses. It's reliable, but sometimes a bit wordy.
2. JSON-SO (The Strict Sous-Chef): A version where the chef is forced to follow a rigid checklist. They can't make a mistake, but they might get confused if the checklist is too strict.
3. TOON (The New Shorthand): A brand-new, ultra-compact language designed specifically to save space. It's like writing a recipe in a secret code that uses fewer words.

The author, Ivan Matveev, wanted to see if this new "secret code" (TOON) is actually better than the old ways, especially when you have to pay for every word you write (tokens).

The Setup: The "Prompt Tax"

Here's the catch: JSON is like a language the chef already knows. You just say, "Write a recipe," and they do it.

TOON, however, is new. The chef has never seen it before. So, before they can write the recipe, you have to spend a lot of time explaining the rules: "Okay, use 2-space indentation, put arrays in brackets, don't forget the commas..."

This explanation is called the "Prompt Tax." It's like paying an entrance fee before you can even start cooking. If the recipe is short, that entrance fee might cost more than the savings you get from using the shorthand.

The Results: What Happened in the Kitchen?

1. The "Simple Salad" Test (Easy Data)

When the task was simple (like a list of users or a simple order), JSON-SO (The Strict Sous-Chef) was the winner.

• Why? Because the strict rules prevented the chef from making typos, and since the recipe was short, the "Prompt Tax" of TOON made it too expensive.
• Analogy: If you are just ordering a coffee, it's faster to say "Coffee" than to explain a new, complex ordering system.

2. The "Complex Banquet" Test (Hard Data)

When the task got complicated (deep nesting, like a company with departments and employees), TOON started to struggle.

• The Problem: The new shorthand is great for flat lists, but it gets messy when you have to go deep into layers (like a Russian nesting doll). The chef got confused by the rules and made mistakes.
• The Fix: The system had to try again and again (repair loops), which ate up all the time and money saved by the shorthand.

3. The "Sweet Spot"

TOON shined brightest in the middle ground: Standard business documents like invoices or orders.

• These are structured enough to be predictable but not so deep that the rules break.
• The Verdict: If you are sending huge amounts of data (like a massive database dump), the savings from TOON eventually pay off the "Prompt Tax." But for small tasks, it's not worth the effort.

The Big Takeaways (In Plain English)

• Don't reinvent the wheel for small jobs: If you are just sending a simple list, stick to standard JSON or use the "Strict Sous-Chef" (JSON-SO). It's faster and cheaper.
• TOON is a specialist, not a generalist: It works amazing for "tabular" data (rows and columns, like Excel sheets) but fails miserably at complex, deep hierarchies (like a family tree or a complex file system).
• The "Prompt Tax" is real: Because TOON is new, you have to teach the AI how to use it every time. If your data is small, that teaching cost is too high. You only save money if you are generating massive amounts of data.
• The "Repair Loop" Trap: If the AI messes up the new shorthand, fixing it is expensive. Because the instructions are so long, sending the error back to the AI doubles the cost.

The Final Recommendation

Think of TOON like a high-speed train.

• If you are traveling a short distance (small data), driving a car (JSON) is faster because you don't have to walk to the station and buy a ticket (the Prompt Tax).
• If you are traveling thousands of miles (massive datasets), the train is unbeatable. It's fast and efficient, but only if you stay on the tracks. If you try to take the train off-road (complex, deep data structures), it derails.

Bottom line: TOON is a promising tool for specific, high-volume jobs, but it's not ready to replace the standard JSON format for everything just yet.

Technical Summary

Here is a detailed technical summary of the paper "Token-Oriented Object Notation vs JSON: A Benchmark of Plain and Constrained Decoding Generation" by Ivan Matveev (February 2026).

1. Problem Statement

Large Language Models (LLMs) are increasingly used to generate structured data, typically in JSON format. However, JSON can be verbose, leading to high token consumption and increased costs/latency. Token-Oriented Object Notation (TOON) has been proposed as a lightweight alternative designed specifically for LLMs to reduce token usage.

While TOON has shown promise in comprehension tasks, there is a lack of empirical data regarding its generation capabilities compared to standard JSON. Key questions remain:

• Can LLMs generate valid TOON via one-shot in-context learning (ICL) without training data?
• Does the token savings in the output offset the "prompt tax" (the overhead of instructional examples required to teach the model the new syntax)?
• How does TOON compare to Constrained Decoding (Structured Output/JSON-SO), which enforces syntax via state machines (e.g., xgrammar)?

2. Methodology

The author conducted a comprehensive benchmark comparing three output generation tracks across 21 different LLMs (ranging from 2B to 480B parameters).

Experimental Setup

• Datasets: Four test cases varying in structural complexity:
  1. Users: Simple tabular structure.
  2. Order: Nested structure with arrays.
  3. Company: Deep hierarchy (departments/employees).
  4. Invoice: Items and totals (mixed complexity).
• Tracks Compared:
  1. Plain JSON (J): Standard generation without constraints.
  2. JSON-SO (JSO): JSON generation using Constrained Decoding (guided decoding via xgrammar) to enforce valid syntax at the token level.
  3. TOON (T): Generation using a universal one-shot prompt containing TOON syntax rules and examples.
• Metrics:
  • One-shot Accuracy: Success rate on the first attempt.
  • Final Accuracy: Success rate after up to 3 repair cycles (re-prompting with error logs).
  • Token Budget: Total tokens (Prompt + Completion) consumed to achieve a valid result.
• Validation: Outputs were canonicalized and validated against Pydantic models. TOON outputs were decoded to JSON via CLI before validation.

3. Key Contributions & Findings

A. The "Prompt Tax" vs. Syntax Savings

• TOON's Efficiency is Context-Dependent: While TOON syntax is shorter, the large instructional prompt required for one-shot learning often negates these savings for small or medium-sized outputs.
• The Crossover Point: TOON only becomes token-efficient when the generated payload is large enough to amortize the fixed cost of the system prompt. For simple tasks, Plain JSON or Constrained Decoding is often cheaper.

B. Performance of Constrained Decoding (JSON-SO)

• Weak Models: Constrained decoding acts as a crucial "safety net." For smaller models (e.g., Qwen2.5-Coder-7B), plain JSON generation failed completely (0% accuracy, massive token waste), while JSON-SO achieved 75% accuracy with stable token usage.
• Strong Models: For large reasoning models (e.g., Hermes-4-405B), strict grammar constraints degraded one-shot accuracy (dropping from 92.5% to 35.0%). The constraints forced the model into suboptimal token paths, interfering with its natural probability distribution, though it recovered to 100% after repair cycles.

C. TOON's "Domain Alignment" Hypothesis

The paper identifies a critical dichotomy in TOON's performance based on data topology:

1. Aligned Cases (Tabular/Uniform): For flat arrays and standard nested business entities (Users, Orders, Invoices), TOON performs well.
   • Result: In the "Users" case, TOON achieved 90.5% one-shot accuracy using 22% fewer tokens than plain JSON.
2. Non-Aligned Cases (Deep/Recursive): For deep hierarchies (Company case), TOON performance collapses.
   • Result: One-shot accuracy dropped to 0% for the Company case. While repair cycles eventually brought final accuracy to ~48%, the cost of retries made it inefficient compared to JSON.
   • Note: Interestingly, for the Company case, TOON's final token count was slightly lower than JSON due to the massive syntactic overhead of JSON in deep nesting, but the failure rate made it unreliable.

D. Scaling Hypothesis

The authors propose that TOON's true efficiency follows a non-linear curve. It is likely only viable for high-volume, large-dataset scenarios where cumulative syntax savings eventually outweigh the initial prompt overhead. However, this introduces risks regarding "indentation drift" in long-context windows.

4. Results Summary Table (Averages)

Metric	Plain JSON (J)	JSON-SO (JSO)	TOON (T)
One-Shot Accuracy	Highest (often >90% for strong models)	Moderate (High for weak models, low for strong)	Low (High for aligned, 0% for non-aligned)
Final Accuracy	High	High	Moderate (Dependent on repair loops)
Token Efficiency	Baseline	Best for simple tasks (e.g., Users case: 556 tokens)	Best for large, aligned tasks (if prompt tax is amortized)
Robustness	High	High (Guardrails for weak models)	Low (Fails on complex structures)

5. Significance and Recommendations

Significance

This benchmark challenges the assumption that "lighter syntax" automatically equals "better efficiency" for LLMs. It highlights that:

1. Prompt overhead is a dominant factor in token economics for novel formats.
2. Constrained decoding is not a universal win; it can hinder strong models while helping weak ones.
3. Data topology dictates format viability. TOON is not a general-purpose replacement for JSON but a specialized tool for specific data types.

Recommendations

1. Use TOON for "Aligned" Data: Adopt TOON for high-volume, cost-sensitive pipelines involving tabular data, SQL dumps, or standard transactional documents (Orders/Invoices) where schemas are uniform.
2. Avoid TOON for Deep Hierarchies: Do not use TOON for complex state trees, DOM structures, or deeply nested configurations. Stick to Plain JSON or JSON-SO.
3. Leverage Constrained Decoding for Weak Models: Use JSON-SO to stabilize generation for smaller, less capable models.
4. Future Work: Further research is needed to test TOON with per-case schema examples (to reduce prompt tax) and to validate its behavior on significantly larger datasets to confirm the non-linear scaling hypothesis.

Conclusion

TOON shows promise as a specialized serialization format for aligned, tabular, and transactional data, offering potential token savings in high-volume scenarios. However, it currently lacks the robustness to replace JSON for general-purpose structured generation, particularly for complex, non-uniform structures, due to high prompt overhead and sensitivity to structural complexity. Constrained decoding remains a powerful tool for ensuring validity, especially for smaller models, but may hinder the performance of top-tier reasoning models.