Technical Summary: Brevity is the Soul of Inference Efficiency

Problem Statement

The conventional paradigm for improving inference efficiency in Large Language Models (LLMs) and Vision-Language Models (VLMs) focuses on reducing the cost per token through model compression techniques such as distillation, pruning, quantization, and sparse routing. This approach implicitly treats the output length (token count) as a fixed quantity. However, empirical trends indicate that output lengths are inflating rapidly—particularly in reasoning models, which now emit roughly 8× more tokens than non-reasoning peers. Consequently, per-token efficiency gains are often consumed by increased verbosity, leading to higher total inference costs (dollars per correct answer) and latency.

The paper argues that brevity is the missing lever for inference efficiency. While decoding-time constraints (e.g., length penalties, sampling controls) attempt to shorten outputs post-hoc, they often fight against a model trained to be verbose. The authors posit that data curation at the pretraining stage is a more effective mechanism: by training on concise, correct data, models internalize brevity, learning to answer in fewer tokens without sacrificing quality. This approach treats inference efficiency as a "tokens-per-correct" problem rather than a "FLOPs-per-token" problem.

Methodology

The study utilizes a controlled experimental setup comparing models trained on curated data against those trained on standard, uncurated data, alongside frontier open-weight models.

1. Data Curation Pipeline: The authors apply their VLM curation pipeline to the MAmmoTH-VL single-image training corpus. This pipeline filters for concision and correctness.
2. Model Families:
   • DatologyAI Models: Trained on the curated MAmmoTH-VL mixture.
   • Mammoth Baselines: Trained on the standard, uncurated MAmmoTH-VL data (serving as the uncurated control).
   • Frontier Comparators: Open-weight models from the Qwen (3, 3.5, 3-VL), InternVL, and Isaac families.
3. Evaluation Scope:
   • Scales: 1B, 2B, and 4B activated parameters.
   • Benchmarks: A controlled 20-evaluation suite (excluding aggregate-only evals for regression analysis) covering capabilities like math, chart interpretation, OCR, and grounding.
4. Metrics:
   • Cost-of-Pass: Defined as FLOPs per correct answer ($2N \times \bar{n}_{out} / \text{accuracy}$), where $N$ is active parameters and $\bar{n}_{out}$ is mean output length. This metric captures both model size and verbosity.
   • Length-Controlled Accuracy: To disentangle brevity from quality, the authors fit a logistic Generalized Linear Model (GLM) with fixed effects for benchmark and scale, controlling for output length. This isolates the accuracy gain attributable to curation independent of token count.
   • Per-Example Contingency: Analysis of error sets to determine if verbose reasoning models solve problems concise models miss, and vice versa.

Key Contributions and Results

1. Drastic Reduction in Cost-of-Pass

The curated models achieve a Pareto-dominant position on the inference cost frontier.

• 35× Advantage: At the 4B scale, the Datology model achieves a 35× lower Cost-of-Pass (0.41 TFLOPs per correct answer) compared to the most verbose comparator, Qwen3.5-4B (14.58 TFLOPs), while maintaining nearly identical accuracy (0.691 vs. 0.704).
• Scaling Efficiency: The curated 1B model (0.10 TFLOPs/correct) is 143× cheaper per correct answer than Qwen3.5-4B.
• Mechanism: The cost reduction is driven almost entirely by reduced token count. At fixed parameter counts, per-token FLOPs are identical across models; the variance lies in the number of tokens emitted to reach a correct answer.

2. Accuracy Gains at Matched Length

Contrary to the hypothesis that brevity might sacrifice reasoning depth, the curated models are more accurate even when output length is held constant.

• Length-Controlled Gain: Across all scales, the curated model is +17.55 percentage points (pp) more accurate than its uncurated Mammoth baseline when output length is controlled.
• Super-linear Scaling: This accuracy advantage grows with model scale: +16.7 pp at 1B, +15.6 pp at 2B, and +21.2 pp at 4B.
• Hallucination Reduction: In captioning tasks, the curated model reduces hallucinated mentions by 19% while cutting caption length by 59%, without sacrificing grounding precision.

3. The Limits of Verbosity

The study investigates whether verbose tokens contribute to correctness.

• Generic Verbosity: The uncurated Mammoth model (verbose but non-reasoning) never outperforms the concise Datology model in any capability group or scale. Generic verbosity buys no accuracy.
• Reasoning-Structured Verbosity: The Qwen3.5 family (verbose with Chain-of-Thought) shows a shrinking advantage. At 2B, it beats the concise model on 4 of 8 capability groups. At 4B, this advantage collapses to 1 of 8 groups (OCR & Document). In 4 of the 8 groups at 4B, the concise model actually outperforms the verbose reasoning model.
• Complementary Error Sets: While the concise model generally achieves higher mean accuracy, per-example analysis reveals that the verbose reasoning model solves some problems the concise model misses (and vice versa), suggesting reasoning and brevity are distinct curation targets.

4. Frontier Comparison

Against external frontier models, the curated model sits on the cost-efficient Pareto front against six of seven comparators.

• It outperforms InternVL and Perceptron variants on both accuracy and cost.
• Against the Qwen family, it trades 4–7 pp of length-controlled accuracy for a 2.4× to 3.6× cost advantage.
• Qwen3.5-4B Exception: A length-controlled comparison with Qwen3.5-4B is impossible (only 2.2% overlap in output length distributions), as Qwen3.5-4B's accuracy is inextricably bundled with its extreme verbosity (~1,284 tokens vs. ~32 tokens for Datology).

Significance and Claims

The paper claims that brevity is the soul of inference efficiency, achievable through amortized inference via data curation.

• Shift in Optimization Target: The authors argue that the industry is optimizing for the wrong metric (FLOPs per token) while ignoring the larger driver of cost (tokens per correct answer).
• Amortization of Cost: Unlike decoding-time constraints that pay a "length penalty" at every query, data curation pays the cost of concision once during training. The model then "caches" this efficiency in its weights, producing short, correct answers automatically (System 1) rather than deliberating token-by-token (System 2) at inference time.
• Scalability: The benefits of curation are not static; they compound with model scale. As models grow, the relative advantage of curated brevity over uncurated verbosity increases, both in cost reduction and matched-length accuracy.
• Practicality: The approach requires no architectural changes, no specialized hardware, and no complex decoding algorithms. It is a "data-first" solution that directly targets the inflation of inference bills driven by longer reasoning chains.

In summary, the paper demonstrates that training on curated, concise data allows VLMs to achieve state-of-the-art cost-efficiency (35× improvement) without sacrificing quality, challenging the notion that reasoning requires verbosity.