💬 NLP

FMBench: Adaptive Large Language Model Output Formatting

This paper introduces FMBench, a benchmark for evaluating large language models on adaptive Markdown formatting, and proposes a lightweight alignment pipeline combining supervised fine-tuning and reinforcement learning to enhance structural compliance while balancing semantic fidelity.

Original authors: Yaoting Wang, Yun Zhou, Henghui Ding

Published 2026-02-09

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yaoting Wang, Yun Zhou, Henghui Ding

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, talkative robot assistant. When you ask it a question, it usually gives you a brilliant answer with the right facts. But sometimes, the way it presents those facts is a mess. It might write a list that looks like a jumbled pile of bricks, create a table where the columns don't line up, or forget to close a code block. To a human, it looks a bit messy; to a computer program trying to read that answer later, it's a complete disaster.

This paper, FMBench, is about teaching these robots to be not just smart, but also neat and organized.

Here is the breakdown of their work, using some simple analogies:

1. The Problem: The "Messy Room"

The authors noticed that while AI models are great at knowing what to say, they often fail at how to say it in a specific format called Markdown (the language used to make text bold, create lists, tables, and code blocks).

The Analogy: Imagine a chef who cooks a delicious meal (the content) but serves it on a dirty plate, spills the sauce on the table, and forgets to cut the steak (the formatting). The food tastes good, but you can't eat it properly.
The Issue: These formatting errors are "small" (like a missing comma or a broken list), but they break the "machines" that need to read the answer later.

2. The Solution: The "FMBench" Gym

To fix this, the team built a special training ground called FMBench.

What it is: Think of this as a gym specifically for "organizing skills." Instead of just asking the robot to answer a math problem, they ask it to answer the problem while following strict rules: "Make sure the list has three items," "Put the code in a box," and "Use three levels of headings."
The Data: They created 1,100 practice exercises. They took real documents (like academic papers or business reports), cleaned them up, and then asked the AI to rewrite them into perfect Markdown structures. Humans then checked the work to make sure it was actually good.

3. The Training Method: "Learn, Then Polish"

The paper proposes a two-step training recipe to turn a messy robot into a neat one, without needing to force it with rigid computer code during the actual conversation.

Step 1: Supervised Fine-Tuning (SFT) - "The Copycat Phase"
- The Analogy: This is like showing the robot a stack of perfect, neatly organized essays and saying, "Copy this style." The robot learns to mimic the structure.
- The Result: The robot gets much better at understanding the meaning of the instructions and keeping the facts correct.
Step 2: Reinforcement Learning (RL) - "The Coach's Whistle"
- The Analogy: Now that the robot can write, a "coach" (an automated system) watches it. If the robot writes a list that breaks, the coach gives a "thumbs down." If it writes a perfect table, the coach gives a "thumbs up." The robot tries again and again, learning to avoid the "thumbs down" and chase the "thumbs up."
- The Result: This step makes the robot much more robust. It learns to handle tricky instructions where the formatting rules are hard to follow, ensuring the final output is structurally perfect.

4. The Discovery: The "Tightrope Walk"

The researchers found something interesting: Being smart and being neat are two different skills.

The Analogy: Imagine walking a tightrope. If you focus too hard on looking at the scenery (the content/meaning), you might stumble on the rope (the structure). If you focus too hard on the rope, you might forget to look at the scenery.
The Finding: The "Copycat" step (SFT) made the robot smarter about the content. The "Coach" step (RL) made it better at the structure. But if you push too hard on one side, the other might suffer. The best results came from a balanced mix of both steps, especially for bigger, more powerful robots.

5. The Bottom Line

The paper concludes that to make AI truly useful in the real world (where computers need to read the answers), we can't just rely on the AI being "smart." We have to explicitly train it to be organized.

They showed that by using their two-step training method (Copycat + Coach), they could make different types of robots (from small to large) produce answers that are both factually correct and perfectly formatted, ready for humans to read and computers to process. They made their "gym" (FMBench) and their "training manual" available for others to use.

Technical Summary: FMBench: Adaptive Large Language Model Output Formatting

Problem Statement

While Large Language Models (LLMs) have demonstrated strong capabilities in following natural language instructions and generating semantically coherent responses, they frequently fail to adhere to strict output formatting constraints required in production environments. Specifically, Markdown—a de facto standard for chat assistants, documentation, and tool-augmented pipelines—is prone to subtle but critical structural errors. These include broken nested lists, malformed tables, inconsistent heading hierarchies, and unbalanced code blocks. Such errors, while often minor at the token level, disproportionately degrade downstream usability, parsing reliability, and human readability.

Existing approaches face significant limitations:

Prompt Engineering: Highly sensitive to phrasing and context, often lacking robustness.
Constrained Decoding: Guarantees syntactic validity but introduces inference overhead and may conflict with the model's tokenization or semantic fluency, particularly in hybrid outputs.
Supervised Fine-Tuning (SFT): Often overfits to superficial regularities in training data rather than learning robust, context-aware formatting principles.
Reinforcement Learning (RL): While promising, existing methods often rely on task-specific reward engineering or rigid rule definitions that lack flexibility across diverse formatting schemas.

Furthermore, current benchmarks rarely isolate the Markdown setting, making it difficult to diagnose formatting failures under realistic instruction-following scenarios. There is a lack of a focused evaluation framework that balances semantic correctness with structural compliance.

Methodology

1. FMBench: A Benchmark for Markdown Formatting

The authors introduce FMBench, a benchmark designed to evaluate adaptive Markdown output formatting under realistic instruction-following settings.

Data Construction Pipeline: A semi-automated, four-stage pipeline generates high-quality, strictly formatted data:
1. Document Crawling: Collecting raw documents from eight domains (academic, legal, technical, etc.).
2. Document Cleaning: Standardizing and removing noise.
3. LLM Formatting: Applying three formatting rules to generate structured data at varying difficulty levels.
4. Human Refining: Expert review to correct grammar, polish style, and ensure structural compliance.
Dataset Composition: The dataset contains 1,100 high-quality Markdown documents (800 training, 300 test). It features a balanced structural design where complexity is driven by nested list depth and item counts rather than extreme section counts.
Evaluation Metrics:
- Semantic Score: Measured via BERTScore-F1 between the generated output and the reference text to assess content preservation.
- Structure Score: A structural reward measuring compliance with target formatting and document organization (e.g., hierarchy, block types).

2. Training Pipeline: SFT + Reinforcement Learning

The paper proposes a lightweight post-training pipeline to improve Markdown compliance without relying on hard decoding constraints at inference time.

Stage 1: Supervised Fine-Tuning (SFT):
- Base models (OpenPangu and Qwen) are fine-tuned on instruction-response pairs using Low-Rank Adaptation (LoRA).
- This stage focuses on adapting the model to the formatted text generation task and improving semantic alignment.
Stage 2: Reinforcement Learning Fine-Tuning (RLFT):
- Following SFT, the model undergoes RL fine-tuning using GRPO (Group Relative Policy Optimization).
- Reward Design: A composite objective balances semantic fidelity and structural correctness:
  $r = \lambda_1 r_{sem} + \lambda_2 r_{struct}$
  - $r_{sem}$ : Computed via BERTScore-F1 between the output and reference (using a Longformer encoder for long documents).
  - $r_{struct}$ : Targets high-level organizational similarity by comparing abstractive summaries of the generated output and the reference.
- The policy is optimized via group-wise relative reward comparisons, sampling four parallel candidate generations per prompt.

Key Contributions

Benchmark (FMBench): The first benchmark exclusively focused on Markdown output formatting, spanning diverse structural requirements (hierarchical sections, mixed prose-list-table composition, code blocks) and evaluating both semantic and structural axes.
Method: A practical post-training recipe combining SFT and RL (specifically GRPO) that improves Markdown compliance while maintaining semantic fidelity, eliminating the need for hard constraints during inference.
Analysis: Empirical results across two model families (OpenPangu and Qwen) revealing the inherent trade-off between semantic and structural objectives and demonstrating the necessity of carefully designed rewards.

Experimental Results

Experiments were conducted on OpenPangu (1B, 7B) and Qwen3 (1.7B, 8B) models.

Overall Performance: The full pipeline (SFT + GRPO) consistently outperformed pretrained baselines in both semantic fidelity and structural compliance.
Model Scale Effects:
- SFT Impact: SFT consistently drove improvements in semantic alignment across all model sizes. Structural performance remained largely unchanged or slightly improved after SFT alone.
- GRPO Impact: Adding GRPO on top of SFT further improved or better preserved structural scores. For smaller models, GRPO sometimes slightly traded off semantic accuracy for stronger structural consistency. However, for larger models (e.g., Qwen3-8B), the combination achieved the best overall performance in both metrics, suggesting that higher model capacity is crucial for jointly optimizing these competing objectives.
Key Finding: SFT is the dominant factor for semantic improvement, while GRPO serves as a complementary mechanism to reinforce structural robustness, particularly when initialized from a strong SFT policy.

Significance and Claims

The paper posits that reliable LLM deployment in user-facing and system-integrated workflows depends not only on semantic correctness but also on stable, human-usable formatting.

Formatting as an Alignment Objective: The authors argue that formatting behavior should be treated as an explicit alignment objective rather than a byproduct of instruction following.
Trade-off Awareness: The results highlight an inherent tension between semantic and structural goals. Naïvely increasing structural reward weights can lead to over-regularized, template-like outputs that distort content. Therefore, reward designs must be both validator-grounded and instruction-conditioned.
Practical Utility: FMBench is presented as a diagnostic tool to motivate further research on alignment methods that treat formatting as a first-class reliability requirement for dependable structured generation.

The authors acknowledge limitations, including the inability to cover the full long tail of real-world authoring styles, the subjectivity of certain structural choices, and the reliance on reference-based evaluation which may miss multiple valid realizations of the same intent. They suggest future work should explore hybrid constraint-learning approaches, expansion to other structured formats (JSON/YAML), and human-centered evaluation.