Technical Summary: On Stable Long-Form Generation: Benchmarking and Mitigating Length Volatility

Problem Statement

While Large Language Models (LLMs) have achieved significant advancements in long-context understanding (e.g., handling inputs exceeding 100k tokens), they exhibit critical limitations in long-form generation. Existing research has primarily focused on the quality of single-generation outputs, largely overlooking output volatility. This volatility manifests as significant fluctuations in output length when the same instruction is processed multiple times, leading to unpredictable token consumption, high computational costs, and unreliable application in real-world scenarios. Current models struggle to meet precise length targets, often failing to surpass the 2k-word threshold or lacking fine-grained control over the generation process. Furthermore, existing benchmarks rely heavily on unstructured tasks with subjective evaluation criteria and fail to systematically quantify the stability of generation across multiple runs.

Methodology

The authors propose a three-stage investigation: Benchmarking, Probing, and Mitigation.

1. Benchmarking: VOLTBench

The paper introduces VOLTBench (VOlatility in Long-form Text Benchmark), a novel heterogeneous-task benchmark designed to systematically quantify length volatility.

• Scope: It covers both unstructured tasks (e.g., stories, dialogues, diaries) and structured tasks (e.g., code libraries, company profiles, mathematical formulas).
• Dimensions: The benchmark evaluates performance across languages (English/Chinese), instruction complexities (simple to fine-grained constraints), and output scales (up to 500 sections or ~100k words).
• Metrics: Unlike previous works focusing on single-run quality, VOLTBench introduces Length Volatility as a core metric. It calculates:
  • Length Standard Deviation (LSD): Absolute volatility of output lengths.
  • Length Variation Coefficient (LVC): Relative volatility with respect to the mean.
  • Mean Length Accuracy (MLA): Adherence of the mean output length to the target.
  • Generation Quality: Measured via Format Standard Deviation (FSD), Structured Content Accuracy (SCA) using execution-based verification, and Unstructured Content Accuracy (UCA) via LLM-as-a-Judge.

2. Probing: Attention Trace Analysis

To understand the internal mechanisms driving volatility, the authors analyze attention traces during generation. They hypothesize that fluctuations in attention toward constraint-encoding tokens correlate with output variability.

• Findings: The analysis reveals two primary internal failure patterns:
  1. Attention Collapse: Attention weights toward prompt constraints drop to near-zero, leading to premature termination or task abandonment.
  2. Attention Instability: Erratic attention spikes disrupt the generation flow, causing behaviors like "section skipping" (jumping from early sections directly to the end) or repetitive loops.
• Correlation: These attention dynamics are diagnostic evidence that output volatility is not random but preceded by measurable internal state failures.

3. Mitigation: GLoBo

To address these internal patterns without additional training, the authors propose Stable Generation via Logits Boosting (GLoBo). This is a lightweight, decoding-stage optimization strategy that modifies logits in real-time.

• Hybrid Structural Enforcement: GLoBo balances natural flow with strict length control. It employs a "Soft Waiting" mechanism that boosts logits for the next section title only after a natural interruption (e.g., punctuation) occurs once the target length is reached. A "Hard Enforcement" fallback prevents infinite loops by forcing a transition if the length exceeds a grace period.
• Proactive Failure Prevention: The method applies strong negative biases to suppress tokens associated with known failure modes, such as conversational fillers (e.g., "I hope these...") or premature end-of-sentence tokens before the task is complete.
• Free-form Adaptation: For tasks without predefined section titles, GLoBo uses virtual milestones and natural-interruption tokens to maintain stability.

Key Results

Extensive experiments on VOLTBench and external benchmarks (LongBench-Write, WritingBench) validate the approach:

• Volatility Reduction: GLoBo reduces the Length Variation Coefficient (LVC) by 69% compared to the base model (from 45.4% to 14.02% on a 100-section task).
• Length Accuracy: The method improves the Mean Length Accuracy (MLA) by 148%, increasing the average output length from 6,320 words (LongWriter-8B) to 15,651 words while maintaining high adherence to targets.
• Quality Preservation: Despite the increased length and stability, GLoBo maintains or improves generation quality. It achieves 100% Structured Content Accuracy (SCA) on structured tasks and significantly reduces n-gram repetition rates (from ~69% to <4%), improving lexical diversity (Type-Token Ratio).
• Generalization: The method generalizes across different model architectures (Qwen, Llama, Mamba) and scales, and performs effectively on external open-ended benchmarks, outperforming specialized long-form models like LongWriter-8B and proprietary models like Claude 3.5 Sonnet in specific length-constrained categories.
• Efficiency: GLoBo introduces minimal runtime overhead (approx. 10.8% reduction in throughput on an H20 GPU), significantly lower than other decoding strategies like Lookahead Decoding.

Significance and Claims

The paper claims to provide the first systematic confirmation of severe long-form output instability in mainstream LLMs. Its significance lies in:

1. Shifting the Paradigm: Moving beyond single-generation quality to prioritize stability and reliability, which are critical for cost-effective and dependable long-form applications.
2. Diagnostic Insight: Identifying Attention Collapse and Attention Instability as root causes of generation failure, linking internal attention dynamics directly to output volatility.
3. Training-Free Solution: Demonstrating that severe length volatility can be mitigated through a lightweight, training-free decoding intervention (GLoBo), avoiding the computational costs of retraining or complex data pipelines.
4. Benchmarking Standard: Establishing VOLTBench as a comprehensive framework that evaluates both generation quality and stability across structured and unstructured domains, addressing the gap in current evaluation methodologies.

The authors conclude that their method successfully enhances the robustness and controllability of LLMs for complex, long-context applications, enabling the generation of significantly longer, more stable, and higher-quality text without requiring additional model training.