💻 computer science

Self-Healing JSON Generation in Small Language Models: A Systematic Evaluation of Syntax Repair, Schema Coercion, and Validator Feedback

This paper presents a self-healing framework for small language models that combines syntax repair, schema-guided conversion, and validator feedback to significantly improve JSON validity from 16.7% to 78.1% without modifying model weights, while revealing that such syntactic corrections do not guarantee semantic accuracy.

Original author: Ahmad Abdelrahman

Published 2026-11-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original author: Ahmad Abdelrahman

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the author. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where computers can read a messy handwritten note, a rambling email, or a spoken story and instantly pull out the key facts to fill a neat, organized form. This is the promise of turning free-flowing language into structured data, a task that powers everything from digital calendars to inventory systems. To do this, computers often use small, efficient versions of artificial intelligence that can run on ordinary devices without needing massive servers. These small models are cheap and private, but they have a flaw: when asked to write their answers in a strict format called JSON, they frequently make mistakes. They might forget a comma, wrap the answer in extra symbols, or invent a phone number that doesn't exist. If a computer program tries to read this broken answer, it crashes. For years, the solution was to force the computer to think in a very rigid way while it was writing, but this is difficult to set up and can sometimes make the computer less smart. A new study asks a simpler question: what if we let the computer write its answer freely, and then we gently fix the mistakes afterward?

Researchers set out to test a three-step process that acts like a repair crew for these digital answers. They took three different small artificial intelligence models and asked them to turn 120 different text descriptions into structured records, such as contact details or event schedules. The first step in their repair crew was a syntax fixer. This tool looked at the raw text and removed common formatting errors, like extra quotation marks or missing brackets, without needing to ask the computer to think again. The second step was a schema guide. This tool checked the content against the rules of the form. If the computer wrote a date as "October 8, 2025" but the form required "2025-10-08," the guide converted it. If the computer wrote "yes" instead of "true," the guide changed it. Crucially, this step never invented new information; it only reshaped what was already there. The third step was a feedback loop. If the answer still didn't fit the rules, the system showed the computer the specific errors and asked it to try again.

The results showed that the first two steps did the heavy lifting. Before any repair, only about one in six answers from the models were actually usable. After the syntax fixer and the schema guide did their work, the number of usable answers jumped to nearly two out of three, all without asking the computer to generate a single new response. For the most capable model in the study, this simple cleanup made almost every answer perfect. The researchers found that the computer's biggest problem was often just a habit of wrapping its answers in code symbols, like a triple-backtick fence, which confused the strict parsers but was easily removed by the repair tool. For the smallest, least capable model, the problem was deeper. It often misunderstood the structure of the request or copied the rules of the form itself instead of the data. In these cases, simply asking the computer to try again after seeing its errors helped, but only if the repair tools had already cleaned up the basic formatting first.

However, the study uncovered a surprising and important truth: just because an answer is formatted correctly does not mean it is true. Even after the repair crew fixed the syntax and the structure, nearly half of the valid answers still contained wrong information. The models would sometimes leave a field blank, invent a plausible-looking phone number, or pick the first option from a list of choices without checking if it was right. This happened even when the computer was told to try again. The researchers observed that the computer could satisfy the rules of the form perfectly while still failing the task of telling the truth. For instance, a model might correctly format a contact card but fill the phone number with a placeholder like "123-456-7890" because it didn't know the real number.

The study suggests that relying solely on whether an answer fits a format is not enough. To truly trust these small models, systems need to check if the information inside the answer matches the original text. The researchers concluded that the best approach is to use simple, automatic tools to fix the formatting and structure first, as this solves most problems for free. Only then should the system ask the computer to try again, and even then, it must verify that the new answer contains real facts, not just a perfect-looking shell. This work shows that while small models can be made much more reliable with a little post-processing, the goal of extracting accurate information remains a challenge that requires checking the content, not just the shape.

Technical Summary: Self-Healing JSON Generation in Small Language Models

Problem Statement

Small language models (SLMs) offer a cost-effective and privacy-preserving alternative for structured information extraction, particularly for on-device applications. However, their outputs frequently fail to meet the strict requirements of JSON Schema validation. These failures manifest as unparseable syntax (e.g., missing brackets, incorrect quotes) or semantic non-compliance (e.g., wrong data types, missing required fields).

While constrained decoding (restricting token generation to a grammar) is a known solution, it requires access to model logits, which is often unavailable in hosted APIs or embedded runtimes. Furthermore, forcing strict generation can distort the model's probability distribution and degrade task performance. Consequently, there is a need for a post-generation approach that repairs JSON outputs without modifying the model's decoder or requiring additional infrastructure.

Methodology

The paper proposes a three-layer, model-agnostic "self-healing" pipeline that operates sequentially on the generated text. The process is designed to minimize additional model calls while maximizing schema validity.

Layer 1: Lenient Syntactic Repair (L1)

This layer operates purely on the text string before parsing. It employs a tolerant lexer and a bracket-stack parser to handle common deviations from strict JSON:

Removal of wrappers: Strips Markdown code fences (```json) and introductory text.
Lexical flexibility: Accepts single quotes, typographic quotes, comments (//, /* */), and non-standard literals (e.g., True, None, NaN).
Structural repair: Uses a stack-based assembler to insert missing commas, remove trailing/duplicated commas, supply null for keys followed by closing brackets, and close truncated containers.
Flagging: Each modification is recorded to categorize the type of syntax error.

Layer 2: Schema-Guided Coercion (L2)

Once the text is parsed into an object, L2 attempts to coerce values to match the target JSON Schema without inventing new content. Operations include:

Type conversion: Converting numeric strings (with currency symbols) to numbers, "yes/no" to booleans, and integral floats to integers.
Normalization: Case-insensitive enum matching and key renaming (e.g., orderId to order_id).
Structural adjustments: Wrapping scalars in arrays, splitting comma-separated strings, and removing properties forbidden by additionalProperties: false.
Conservative parsing: Rewriting strings into required patterns (e.g., dates, 24-hour times) only when the parse is unambiguous. Crucially, L2 does not create missing information, except for adding explicit null for nullable properties.

Layer 3: Validator-Guided Re-prompting (L3)

If the object fails validation after L1 and L2, the system initiates a retry. The validator's error messages (field path and error description) are appended to the prompt, asking the model to correct the specific issues. The new output is then re-processed through L1 and L2. This loop continues up to a retry budget ( $K=2$ ).

Experimental Setup

The author evaluated this pipeline on 120 text-to-JSON tasks across four schema families (contact, order, event, paper) using three instruction-tuned SLMs:

SmolLM2-360M
Qwen2.5-0.5B
Qwen2.5-1.5B

Six experimental conditions were tested, ranging from raw output to the full self-healing pipeline, including a "blind retry" baseline (resampling without feedback) to isolate the value of validator errors.

Key Results

1. Efficacy of Deterministic Repair

The most significant gains were achieved without any additional model calls.

Raw Validity: Only 16.7% of raw responses satisfied the schema.
L1 + L2 Impact: Applying syntactic repair and schema coercion raised validity to 67.8% on average.
Model Variance:
- Qwen2.5-1.5B: Achieved 99.2% validity with L1+L2 alone; retries added negligible value.
- Qwen2.5-0.5B: Validity jumped from 3.3% (raw) to 72.5% after L1 (primarily by removing Markdown code fences), and further to 83.3% after adding L2.
- SmolLM2-360M: L1 had no effect (validity remained at 9.2%), but L2 raised validity to 20.8%.

2. The Role of Validator Feedback (L3)

Feedback vs. Blind Retry: The full self-healing pipeline (L1+L2+L3) achieved 78.1% validity with an average of 1.57 calls per task. This was statistically comparable to blind resampling (74.4%, $p=0.060$ ).
Model Specificity: Feedback was highly effective for SmolLM2-360M, improving validity from 32.5% (blind retry) to 46.7% ( $p=0.003$ ). In contrast, it offered no benefit for Qwen2.5-0.5B, which continued to wrap outputs in code fences even after feedback.
Failure Modes: Feedback failed to correct "formatting habits" (like code fences) that the model persisted in repeating. It was most useful for semantic errors (e.g., missing required fields).

3. Validity Does Not Equal Correctness

A critical finding is that schema validity does not guarantee factual accuracy.

Error Rate: Among valid, self-healed responses, 45.6% contained at least one incorrect field.
Common Errors:
- Empty Placeholders: Models filled required string fields with empty strings ("") to satisfy type constraints.
- Fabricated Values: Models invented plausible but false data (e.g., phone numbers like "123-456-7890").
- Default Enums: Models selected the first enum value (e.g., "pending") regardless of context.
Conclusion: Even with a perfect pipeline, small models often hallucinate content to satisfy structural constraints.

Significance and Contributions

The paper makes four primary contributions:

A Precise Pipeline: It defines a three-layer self-healing workflow (L1, L2, L3) that is model-agnostic and requires no access to model logits.
Ablation of Deterministic Layers: It demonstrates that deterministic repair (L1 and L2) accounts for the majority of validity gains (83.3% of repaired outputs required no extra calls), making it the most cost-effective step.
Failure Taxonomy: It identifies two distinct failure regimes:
- Formatting Habits: Trivial to repair via code (e.g., code fences).
- Schema Misunderstanding: Difficult to repair (e.g., copying the schema structure, wrong types), often requiring semantic intervention.
Validity vs. Correctness Distinction: It provides evidence that validator feedback can lead to "valid but wrong" outputs. The author argues that structured-output evaluations must measure field-level accuracy alongside schema validity, as small models may prioritize passing the validator over extracting accurate information.

Practical Recommendations

Based on the findings, the author suggests:

Always apply deterministic repair and coercion first, as it is free and resolves most issues.
Use validator feedback only for models where remaining errors are semantic; for stochastic errors, blind resampling is equally effective.
Implement content validation (e.g., rejecting empty strings for required fields) alongside schema validation.
Design schemas that mirror the surface structure of the source text to minimize the need for complex semantic transformations.