💬 NLP

HoWToBench: Holistic Evaluation for LLM's Capability in Human-level Writing using Tree of Writing

This paper introduces HoWToBench, a large-scale Chinese writing benchmark, and proposes Tree-of-Writing (ToW), a tree-structured evaluation framework that explicitly models sub-feature aggregation to achieve high correlation with human judgments and robustness against textual disturbances, thereby addressing the limitations of existing metrics in assessing LLMs' human-level writing capabilities.

Original authors: Andrew Zhuoer Feng, Cunxiang Wang, Yu Luo, Lin Fan, Yilin Zhou, Zikang Wang, Xiaotao Gu, Jie Tang, Hongning Wang, Minlie Huang

Published 2026-04-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Andrew Zhuoer Feng, Cunxiang Wang, Yu Luo, Lin Fan, Yilin Zhou, Zikang Wang, Xiaotao Gu, Jie Tang, Hongning Wang, Minlie Huang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Robot Judge" is Getting Confused

Imagine you are a famous chef. You've just cooked a complex, 10-course meal. You want to know if it's good.

In the past, to judge AI writing, we used two main methods:

The "Copy-Paste" Judge: We compared the AI's writing to a human's writing word-for-word. If they didn't match perfectly, the AI failed. This is like judging a jazz musician by how well they can play a song exactly like a recording. It misses the creativity.
The "One-Shot" Robot Judge: We asked another AI to read the writing and give it a score (e.g., "8 out of 10"). But here's the catch: The robot judge is bad at math. When it tries to combine scores for "grammar," "creativity," and "logic," it often just averages them out or gets confused. It might give a high score to a boring essay just because it was long, or a low score to a brilliant story just because it had one typo.

The researchers call this "Negotiation Inconsistency." It's like asking a committee of robots to decide the winner of a talent show, but every time you ask them, they change their minds because they can't agree on how to weigh the different talents.

The Solution: The "Tree of Writing" (ToW)

The authors built a new way to judge writing called Tree-of-Writing (ToW).

The Metaphor: The Tree vs. The Smoothie

Old Way (The Smoothie): You throw all the ingredients (grammar, plot, emotion, formatting) into a blender. You get a smoothie (a single score). You can't taste the strawberry separately from the banana. If the blender is broken, the whole drink tastes bad.
New Way (The Tree): Imagine a tree.
- The Root is the final score.
- The Main Branches are big categories: Content (the story), Format (the structure), and Impression (the "vibe").
- The Leaves are tiny details: "Did the opening grab you?" "Is the logic sound?" "Are the paragraphs the right size?"

How it works:
Instead of one robot guessing the whole score, the system sends different "Expert Agents" to specific leaves.

One agent checks if the logic makes sense.
Another checks if the formatting (like headers and lists) is correct.
A third checks the emotional depth.

Crucially, a "Negotiator" (a smart AI) decides how much each leaf matters before the scoring starts. For a poem, "Emotion" gets a heavy weight. For a legal contract, "Logic" gets the heavy weight. This prevents the robot from getting confused about what is important.

The New Test: HoWToBench

To test this new judging system, they built a massive test called HoWToBench.

The Metaphor: The "Master Chef's Kitchen"
Most AI tests are like asking a robot to fill in a blank in a sentence. It's easy.
HoWToBench is like giving the robot a blank canvas and saying: "Write a mystery novel," or "Write a speech for a mayor," or "Write a contract for a space hotel."

12 Genres: They tested everything from poetry and fiction to official government documents and business plans.
3 Difficulty Levels:
1. Completion: "Here is a story, finish the middle." (Easy)
2. Guide: "Here is an outline, write the story." (Medium)
3. Open: "Here is a topic, write whatever you want." (Hard)

They used real human experts (writers, journalists, editors) to grade the AI's work, creating a "Gold Standard" to see if the Tree-of-Writing system was actually right.

The Surprising Discoveries

When they ran the tests, they found some things that surprised everyone:

Longer is NOT Better:
- The Myth: "If the AI writes a huge essay, it must be smart."
- The Reality: The study found that longer inputs and outputs often got lower scores. If an AI just piles on words to look smart, it actually hurts its score. Humans prefer concise, punchy writing over rambling nonsense.
- Analogy: It's like a comedian who tells a joke. If they tell the punchline in 5 seconds, it's funny. If they talk for 20 minutes before getting to the point, it's not funny.
The "Fragile" Judges:
- They tested if the judges were robust. They took a good AI story and did weird things to it: deleted a paragraph, repeated a paragraph, or changed the genre.
- Result: Old judging methods (like simple word-matching or standard AI judges) got confused and gave wildly different scores.
- The Tree: The Tree-of-Writing stayed calm. It recognized that the story was still good even if a paragraph was missing, or that the repetition was bad. It was robust.
The "Mimicry" Trap:
- Many AIs are great at following instructions when they have a lot of help (like a recipe). But when you take away the recipe (Open Writing), they struggle to be truly creative. They are good at mimicking humans, but not yet at being human-level writers.

Why This Matters

This paper is a big step forward because it stops treating writing like a math problem (counting words) and starts treating it like art.

By using a Tree structure, they made the judging process transparent. We can now see why an AI got a bad score (e.g., "It failed on Logic, but did great on Emotion"). This helps developers fix their models and helps us trust AI when it writes stories, contracts, or speeches.

In short: They built a smarter, more human-like way to grade AI writing, proving that quality isn't about how long you write, but how well you write.

1. Problem Statement

Evaluating the writing capabilities of Large Language Models (LLMs) remains a significant challenge due to the multidimensional nature of human writing (e.g., creativity, logic, style, and structure). Existing evaluation methods suffer from several critical limitations:

Inadequacy of Traditional Metrics: Reference-based metrics like BLEU and ROUGE fail to capture semantic nuance, creativity, and implicit requirements in open-ended writing.
Limitations of "LLM-as-a-Judge": While LLMs are increasingly used as evaluators, they often exhibit Negotiation Inconsistency. When asked to aggregate sub-scores (e.g., fluency, logic, instruction following) into a final score, LLMs implicitly negotiate weights between these dimensions. This leads to opaque, unstable, and biased assessments that vary across runs.
The "Mimicking Game" Bias: Current benchmarks often focus on explicit instruction following (checking if constraints are met) rather than the holistic quality of human-level writing, which requires handling implicit intents and nuanced expression.
Robustness Issues: Existing metrics and LLM judges are vulnerable to textual disturbances (e.g., repetition, paragraph dropping) and often correlate positively with verbosity (longer text = higher score), which does not align with human judgment.

2. Methodology

The paper proposes two core innovations: the Tree-of-Writing (ToW) evaluation framework and the HOWTOBENCH benchmark.

A. Tree-of-Writing (ToW) Framework

ToW simulates the human decision-making process by structuring evaluation as a tree rather than a flat list of criteria. It addresses the "Negotiation Inconsistency" problem by explicitly modeling aggregation weights.

Tree Structure:
- Root Node ( $R$ ): Represents the overall score.
- Primary Nodes: Three task-agnostic dimensions: Content ( $V_C$ ), Format ( $V_F$ ), and Impression ( $V_I$ ).
- Leaf Nodes: Granular traits (e.g., "Opening & Ending," "Logic," "Emotion," "Paragraphing") connected to their parent nodes.
Explicit Weighting (The Negotiator):
- Unlike standard LLM-as-a-judge methods that implicitly average scores, ToW employs an LLM Negotiator ( $J_W$ ) to generate an Aggregation Plan for each specific writing instruction.
- The Negotiator explicitly assigns edge weights ( $w$ ) to sub-nodes based on the genre and task type (e.g., "Logic" might be weighted higher for an argumentative essay, while "Emotion" is weighted higher for a poem).
- Weights are constrained to sum to 1 and can range from -1 to 1, allowing for negative weighting if a trait is detrimental to the specific genre.
Scoring Mechanism:
- Leaf Scoring: Specialized "Expert Agents" (LLMs) score leaf nodes.
  - Content: Uses a rubric-based approach with reference texts (1-10 scale).
  - Format: Uses a hybrid approach (Rule-based Regex for titles/structure; LLM for plot/paragraphing).
  - Impression: A holistic subjective score (1-10 scale).
- Aggregation: Scores are aggregated via Depth-First Search (DFS) using the explicit weights determined by the Negotiator.

B. HOWTOBENCH Benchmark

ToW is validated on HOWTOBENCH, a large-scale Chinese writing benchmark designed to emulate professional human writing.

Scale: 1,302 writing instructions across 12 genres (e.g., fiction, poetry, contracts, speeches, reports).
Task Categories (Progressive Difficulty):
1. Completion: Fill in missing parts of a text (Context-rich).
2. Guide: Expand text based on a provided outline (Context-medium).
3. Open: Generate text from a single topic sentence with no outline (Context-poor/Open-ended).
Data Quality: References are curated from expert-written sources (human professionals), not AI-generated. A rigorous human-in-the-loop process ensures 96.85% data quality.

3. Key Contributions

Tree-of-Writing (ToW) Framework: A novel evaluation architecture that replaces implicit aggregation with explicit, instruction-adaptive weight planning. This eliminates negotiation bias and ensures reproducible, transparent scoring.
HOWTOBENCH: The first large-scale benchmark specifically designed to evaluate human-level writing capabilities across diverse genres and varying levels of context, moving beyond simple instruction-following.
Robustness Analysis: The paper demonstrates that ToW is robust against common textual disturbances (repetition, dropping paragraphs, genre transfer), whereas traditional metrics and standard LLM judges are highly susceptible to these perturbations.
Insight on Input/Output Length: The study uncovers a negative correlation between input/output length and content quality scores in open-ended tasks. This refutes the "longer is better" bias often found in LLM evaluators, showing that piling up information does not equate to better writing.

4. Experimental Results

The authors evaluated 10 flagship LLMs (including GPT-4o, Claude-3.5, DeepSeek-R1, and Gemini-2.0) using ToW and compared it against baselines (BLEU, ROUGE, Auto-Planning, Elaborated Rubrics).

Human Alignment: ToW achieves a Pearson correlation of 0.93 with human judgments across all tasks, significantly outperforming all baselines.
- Auto-Planning (standard LLM judge): ~0.88 correlation.
- BLEU/ROUGE: ~0.75 correlation (and negative correlation in some open tasks).
Stability: ToW shows significantly lower fluctuation in sub-score aggregation compared to Auto-Planning methods, confirming the effectiveness of explicit weight assignment.
Robustness: When subjected to disturbances (e.g., repeating paragraphs), ToW scores dropped consistently (reflecting quality degradation), while Auto-Planning and BLEU often showed erratic or positive score changes, indicating they can be "hacked" by structural noise.
Model Performance:
- Models like DeepSeek-R1 and GPT-4o excelled in Completion tasks but showed significant performance drops (15-18%) in Open tasks.
- Performance varied heavily by genre; models performed well on structured formats (Letters) but struggled with creative demands (Fiction).

5. Significance

Paradigm Shift in Evaluation: This work moves the field from "instruction compliance" checking to "holistic writing quality" assessment. It acknowledges that human writing involves complex trade-offs between logic, emotion, and style that require dynamic weighting.
Reliability: By solving the "Negotiation Inconsistency" problem, ToW provides a more reliable and interpretable metric for comparing LLMs, which is crucial for the development of AI writing assistants.
Benchmarking Standard: HOWTOBENCH sets a new standard for evaluating creative and functional writing, highlighting that current state-of-the-art models still struggle with open-ended, human-level creativity, particularly when context is limited.
Future Directions: The findings suggest that improving LLM writing requires more than just scaling data or input length; it necessitates better reasoning capabilities to handle implicit requirements and genre-specific nuances.

In conclusion, HoWToBench and Tree-of-Writing provide a rigorous, human-aligned framework for evaluating LLMs, revealing that while models are improving, they still lack the nuanced adaptability of human writers in complex, open-ended scenarios.