SO-Bench: A Structural Output Evaluation of Multimodal LLMs

1. Problem Definition

The paper addresses a critical gap in the deployment of Multimodal Large Language Models (MLLMs) in real-world agentic settings. While MLLMs are increasingly used for tasks like web automation, data extraction, and tool use, these applications require outputs that are not only semantically correct but also structurally valid (i.e., conforming to a predefined JSON schema).

Current limitations include:

• Lack of Benchmarks: Existing benchmarks focus on text-only structured generation or simple Key Information Extraction (KIE) with flat templates. None systematically evaluate the ability of MLLMs to extract information from diverse visual inputs (UIs, documents, charts, natural images) and align them with complex, nested, customized JSON schemas.
• Schema Adherence vs. Accuracy: Models often struggle to balance following strict schema constraints (syntax) with accurately extracting visual content (semantics), leading to outputs that are either syntactically invalid or semantically hallucinated.

2. Methodology: SO-Bench Construction

The authors introduce SO-Bench, a comprehensive benchmark designed to evaluate visual structured output capabilities.

A. Dataset Composition

• Scale: 1,800 curated image–schema pairs derived from a pool of 112K+ images and 6.5K+ JSON schemas.
• Domains: Covers four distinct visual domains:
  1. UI Screens: Mobile, web, and desktop interfaces.
  2. Natural Images: General photography.
  3. Documents: Receipts, posters, and forms.
  4. Charts: Data visualizations.
• Schema Complexity: Schemas range from simple to highly nested (up to 22 levels deep) with over 2,000 fields, sourced from real-world APIs and synthetic generation.

B. Multi-Stage Auto-Labeling Pipeline

To ensure high-quality data, the authors developed a pipeline involving:

1. Schema Generation & Association:
   • Uses CLIP embeddings to match images with relevant schemas from a repository.
   • Employs multi-image grouping to generate unified schemas for clusters of similar images (e.g., a multi-item menu).
2. User Intent Generation:
   • Synthesizes 60K diverse user profiles (age, occupation, locale) to generate varied instructions (conversational, direct, ambiguous) for each image-schema pair.
3. Response Generation & Refinement:
   • Uses frontier models (e.g., GPT-5, Gemini-2.5-Pro) as generators.
   • Implements a Critic-Refiner loop: An LLM-based critic validates outputs against schema constraints and suggests improvements.
   • Human-in-the-Loop: Domain experts inspect intermediate outputs at every stage to filter errors.

C. Evaluation Metrics

The evaluation pipeline decomposes performance into three granular metrics:

1. Schema Validation Accuracy: Percentage of outputs that are syntactically valid JSON and conform to the schema definition.
2. Field Matching Accuracy (FMA): Measures the recall of extracted fields. Uses Exact Match for clear data and Fuzzy Match (normalized edit distance/relative error) for ambiguous visual data.
3. Full Structure Matching Accuracy (FSMA): Requires the entire nested structure to match the ground truth (Exact or Fuzzy).

3. Key Contributions

1. SO-Bench Benchmark: The first systematic framework for evaluating schema-grounded visual reasoning across diverse domains and complex nested structures.
2. Comprehensive Benchmarking: Extensive evaluation of over 20 models, including open-source (Qwen, Gemma, InternVL) and proprietary frontier models (GPT-5, Gemini 2.5-Pro, Claude).
3. Training Insights: Demonstrated that targeted Supervised Fine-Tuning (SFT) and Reinforcement Learning with Verifiable Rewards (RLVR) significantly improve structured output capabilities.

4. Experimental Results

A. Benchmark Performance

• Schema Adherence: Frontier models show strong schema following. Gemini-2.5-Pro leads with 98% schema validation accuracy, followed closely by GPT-5 (96%).
• Structural Fidelity: There is a massive gap between schema adherence and full content accuracy.
  • Even the best models (Gemini-2.5-Pro) achieve only ~19% Full Match (Fuzzy) accuracy.
  • Smaller models (e.g., 3B–7B parameters) struggle significantly, often failing to maintain schema compliance as depth increases.
• Correlation Analysis: SO-Bench performance correlates strongly with Agentic Tool Use (BFCL), Visual Instruction Following (MIABench), and General Knowledge (MMMU), but weakly with simple OCR tasks, suggesting structured output requires higher-level reasoning than just text extraction.

B. Impact of Training

The authors conducted training experiments on a 3B internal model using a 114K training set:

• Supervised Fine-Tuning (SFT): Training on diverse data (real + synthetic schemas) significantly boosted performance. A 3B model trained on 50K examples achieved performance comparable to untrained 30B+ models.
  • Key Finding: Models trained only on synthetic schemas failed to generalize to real-world schemas, highlighting the need for diverse training data.
• Reinforcement Learning (RLVR): Using a reward function based on Field Match Accuracy and schema validity:
  • RLVR improved Schema Validation by ~13% over the baseline.
  • However, RLVR alone underperformed compared to SFT. The best results came from SFT followed by RLVR, though gains were marginal, suggesting the baseline model's capacity was a limiting factor.

5. Significance and Future Directions

• Bridging Perception and Reasoning: SO-Bench highlights that while MLLMs are good at "seeing," they are not yet reliable at "acting" via structured outputs. This is a bottleneck for autonomous agents.
• Training Paradigm: The work proves that structured output is a learnable skill. Targeted SFT on diverse schema-image pairs is more effective than relying solely on pre-training or simple RL.
• Limitations:
  • Reliance on frontier models for initial labeling may introduce bias or errors.
  • Current metrics focus on recall; future work needs better precision metrics for hallucinated fields.
  • Multiple valid ground truths exist for some visual tasks, which current single-label evaluation may penalize.

Conclusion: SO-Bench establishes a new standard for evaluating MLLMs in agentic workflows. It reveals that while schema compliance is improving, the ability to generate fully accurate, complex structured outputs from visual inputs remains a significant challenge, solvable largely through targeted, diverse training data.