💬 NLP

StructEval: Benchmarking LLMs' Capabilities to Generate Structural Outputs

The paper introduces StructEval, a comprehensive benchmark comprising 18 formats and 44 task types that evaluates LLMs' ability to generate and convert both non-renderable and renderable structured outputs, revealing significant performance gaps where even state-of-the-art models struggle with structural fidelity, particularly in generation and visual content tasks.

Original authors: Jialin Yang, Dongfu Jiang, Lipeng He, Sherman Siu, Yuxuan Zhang, Disen Liao, Zhuofeng Li, Huaye Zeng, Yiming Jia, Haozhe Wang, Benjamin Schneider, Chi Ruan, Wentao Ma, Zhiheng Lyu, Yifei Wang, Yi Lu

Published 2026-04-06

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jialin Yang, Dongfu Jiang, Lipeng He, Sherman Siu, Yuxuan Zhang, Disen Liao, Zhuofeng Li, Huaye Zeng, Yiming Jia, Haozhe Wang, Benjamin Schneider, Chi Ruan, Wentao Ma, Zhiheng Lyu, Yifei Wang, Yi Lu, Quy Duc Do, Ziyan Jiang, Ping Nie, Wenhu Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a very talented, super-fast chef (a Large Language Model, or LLM) to help you run a restaurant.

In the past, we mostly tested this chef on how well they could write a menu description. Could they describe a burger so deliciously that you could almost taste it? Could they answer questions about the ingredients? We gave them tests like "Write a poem about a tomato" or "Explain the history of pizza."

But in the real world of software, the chef doesn't just need to write pretty words. They need to build the actual kitchen equipment and organize the pantry perfectly. If they write a recipe that looks great but has the wrong measurements, the cake collapses. If they organize the pantry but put the flour in the spice jar, the whole kitchen breaks.

This is exactly what the paper StructEval is about. It's a new, tough test to see if these AI chefs can actually build structured things, not just talk about them.

Here is the breakdown of their new test, explained with some kitchen metaphors:

1. The Two Types of "Cooking"

The researchers realized that "building" comes in two flavors, so they split the test into two sections:

The "Pantry Organizer" (Text-Only):
Imagine the chef needs to organize a spreadsheet of ingredients. They need to make sure the data is in a strict format like JSON (a digital filing cabinet), YAML (a list of instructions), or CSV (a grocery list).
- The Test: "Here is a messy list of ingredients. Please put them into a perfect JSON file."
- Why it matters: If the file is even slightly broken, the computer program trying to read it will crash.
The "Architect" (Visual Rendering):
Now, imagine the chef needs to build a physical display window or a website. They need to write code for HTML (the skeleton of a webpage), React (interactive buttons), or SVG (digital art).
- The Test: "Here is a description of a travel itinerary. Please build a webpage that shows it, with a big title, a table with 3 rows, and a button at the bottom."
- Why it matters: The code must not only look right on paper but actually work when you open it in a browser.

2. The Two Ways to Take the Test

The paper introduces two different ways to challenge the AI:

The "Blank Canvas" Challenge (Generation):
The AI is given a natural language prompt (e.g., "Make a schedule for a space trip") and has to create the structure from scratch. This is like asking the chef to invent a new dish from scratch. It's hard because they have to remember all the rules of the format while being creative.
The "Translation" Challenge (Conversion):
The AI is given a structure in one format (e.g., a JSON file) and asked to translate it into another (e.g., a YAML file). This is like taking a recipe written in French and translating it perfectly into Spanish without losing any ingredients.

3. How Did They Grade the Chefs?

In the past, humans just read the output and said, "Looks good!" But that's too subjective. StructEval uses a Robot Judge with three specific tools:

The Syntax Check (The Grammar Police): Does the code actually run? If you ask for JSON, is it valid JSON, or is it broken garbage? If it doesn't parse, the score is zero.
The Keyword Hunt (The Inventory Check): Did the chef include the specific items you asked for? If you asked for a "Title" and a "Date," did they actually put those exact words in the right spots?
The Visual Eye (The Art Critic): For the visual tasks (like webpages), they use a special AI that looks at the rendered image of the code. It asks questions like, "Is the title centered?" or "Is the button blue?" If the image doesn't match the instructions, points are deducted.

4. The Shocking Results

The researchers tested the top chefs (the best AI models available today, like GPT-4o, o1-mini, and open-source models like Llama). Here is what they found:

The Gap is Real: Even the "Star Chefs" (the most expensive, commercial models) only got about 75 out of 100. They are good, but they are far from perfect.
The Open-Source Struggle: The free, community-built models (like Llama) scored about 10 points lower than the paid giants. They are getting there, but they still trip over the complex rules.
Harder Than It Looks:
- Translation is easier than Creation: It's easier for the AI to turn a JSON file into a YAML file than to invent a JSON file from a paragraph of text.
- Visuals are the hardest: Making a webpage that looks right is much harder than just writing a text list. The AI often forgets to center the text or misses a specific button.
- The "Niche" Nightmare: The AI models are great at common things like HTML or JSON. But if you ask them to make a diagram in Mermaid or a typesetting file in TikZ (specialized formats), they often fail miserably, scoring below 50%.

The Big Takeaway

The paper concludes that while AI is amazing at writing essays and chatting, it is still clumsy when it comes to building strict, structured systems.

Think of it like this: The AI can write a beautiful story about a house, but if you ask it to actually draw the blueprints so a construction crew can build it without the roof falling in, it often makes mistakes.

StructEval is the tool we need to fix that. It's a rigorous training ground to teach these AIs that in the real world, structure matters just as much as style. If the format is wrong, the whole application breaks.

1. Problem Statement

While Large Language Models (LLMs) have achieved remarkable success in natural language generation, reasoning, and code generation, their ability to produce strictly structured outputs remains a critical bottleneck for real-world software engineering and data pipeline applications.

The Gap: Existing benchmarks primarily focus on semantic quality or reasoning (e.g., MMLU, GSM8K) or specific modalities like code generation. They lack comprehensive evaluation across diverse structured formats (both non-renderable like JSON/YAML and renderable like HTML/SVG).
The Challenge: Generating structured output requires not only semantic coherence but also strict adherence to syntactic rules, hierarchical schemas, and visual fidelity. Errors in formatting can lead to parsing failures, broken pipelines, or non-functional user interfaces.
Current Limitations: Prior benchmarks often rely on manual verification, lack unified metrics for text vs. visual outputs, or cover only a narrow set of formats, leaving the performance of state-of-the-art (SOTA) models on complex structural tasks largely unknown.

2. Methodology: The StructEval Framework

The authors introduce StructEval, a comprehensive benchmark designed to systematically evaluate LLMs across 18 distinct formats and 44 task types.

A. Dataset Construction

The dataset consists of 2,035 examples organized into two complementary subsets:

StructEval-T (Text-only): Evaluates generation and conversion of non-renderable formats (e.g., JSON, XML, YAML, CSV, TOML, Markdown).
StructEval-V (Visual): Evaluates generation and conversion of executable code that renders visually (e.g., HTML, React, SVG, Mermaid, Matplotlib, LaTeX, TikZ, Vue, Angular).

Task Types:

Generation: Converting natural language prompts into structured outputs.
Conversion: Translating between two structured formats (e.g., JSON $\to$ YAML, HTML $\to$ React).

Annotation Pipeline:
To ensure high quality and scalability, the authors employed a three-stage pipeline:

Task Curation: Defining 44 unique task types spanning 18 formats.
LLM-based Synthesis: Using LLMs to generate initial candidate examples (queries and metrics) to reduce manual effort.
Expert Review: A two-pass human review process using LabelStudio to validate task clarity, completeness, and metric correctness.

B. Evaluation Metrics

StructEval introduces a novel, automated evaluation framework combining three core metrics:

Syntax Score ( $s_s$ ): Verifies structural validity.
- Text: Parsed using format-specific Python parsers (e.g., json.loads).
- Visual: Rendered using headless engines to check for execution success.
- Score: 1 if valid/successful, 0 otherwise.
Keyword Matching Score ( $s_k$ ): Evaluates structural fidelity using dot-path references.
- Checks for the presence of specific keys, nested fields, list indices, and attributes (e.g., planet.moons[0].name).
- Supports complex rules including wildcards, CSV headers, and XML attributes.
Visual Question Answering (VQA) Score ( $s_v$ ): Exclusive to StructEval-V.
- Uses a Vision-Language Model (VLM, specifically GPT-4.1-mini) to answer questions about the rendered image (e.g., "What is the text in the header?").
- Ensures the visual output matches the semantic requirements.

Final Scoring:
The final score is a weighted combination of these metrics:

Text Tasks: $s = 0.2 \cdot s_s + 0.8 \cdot s_k$
Visual Tasks: $s = 0.2 \cdot s_s + 0.1 \cdot s_k + 0.7 \cdot s_v$

3. Key Contributions

Comprehensive Benchmark: The first benchmark to cover 18 diverse formats (both code and non-code) and 44 task types, bridging the gap between text-only and visual structural generation.
Unified Evaluation Framework: A novel metric system that unifies syntactic parsing, path-based keyword matching, and vision-based VQA into a single scoring pipeline.
High-Quality Dataset: A rigorously curated dataset of 2,035 examples with human-verified VQA pairs (88.92% fairness rate) and robust keyword definitions.
Systematic Analysis: A detailed empirical study revealing performance gaps between commercial and open-source models, and between generation vs. conversion tasks.

4. Key Results

The authors evaluated 12 models (6 open-source, 6 commercial) in a zero-shot setting.

Overall Performance:
- SOTA Performance: Even the best commercial model, GPT-4o, achieved an average score of 76.02%.
- Open-Source Gap: The best open-source model, Qwen3-4B, scored 67.04%, trailing commercial models by ~9 points.
- Top Performer: o1-mini (OpenAI) scored 75.58%, showing strong reasoning capabilities but still lagging behind GPT-4o.
Task Difficulty Analysis:
- Generation vs. Conversion: Models generally perform better on conversion tasks than generation tasks. However, commercial models excel at generation in StructEval-T, while open-source models sometimes perform better on conversion.
- Visual vs. Text: Generating code for visual rendering (StructEval-V) is significantly harder than generating text-only structures.
- Format Specifics:
  - Easy: Text $\to$ Markdown, Text $\to$ HTML, YAML $\to$ JSON (Scores > 90%).
  - Hard: Text $\to$ Mermaid, Text $\to$ SVG, Matplotlib $\to$ TikZ (Scores < 50% for many models).
Model Disparities:
- Phi-3-mini performed poorly (40.79% avg), particularly failing on structural conversions (e.g., TOML $\to$ YAML) due to malformed tags and missing hierarchical relationships.
- Commercial vs. Open Source: The performance gap is widest in StructEval-T generation tasks (especially TOML), suggesting commercial models have superior training on complex schema adherence.

5. Significance and Future Directions

Real-World Impact: StructEval highlights that while LLMs are fluent, they are not yet reliable for production-grade structured output generation, which is essential for APIs, data pipelines, and UI generation.
Research Direction: The results suggest that future models need to optimize for structural reasoning and schema adherence, not just semantic fluency.
Limitations: The current benchmark focuses on static, non-interactive rendering. Future work should extend to dynamic behaviors (animations, interactions) and include broader bias audits in the dataset generation.

In conclusion, StructEval provides a critical tool for the community to measure progress in LLMs' ability to act as reliable "structural engineers," revealing that significant challenges remain in generating complex, error-free structured data and visual code.