💻 computer science

How Response Parsing Manufactures Scaling Laws: A Replication and Measurement-Validity Audit of Constraint-Tracking Capacity in LLMs

This paper replicates the existence of bounded constraint-tracking capacity in LLMs but invalidates the previously reported power-law scaling exponent by demonstrating that parser fallbacks, denominator selection biases, and format non-compliance artificially inflated the original results, while also revealing that the claimed architectural bound fails to distinguish between the models used for the calculation.

Original authors: Yangyi Zhang

Published 2026-10-05

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yangyi Zhang

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a computer program that can read a list of rules and follow them perfectly, no matter how long the list gets. This is the promise of modern artificial intelligence, specifically a type of system called a large language model. These programs are trained on vast amounts of text and can answer questions, write stories, or solve problems. As these systems grow larger, with more internal components and more data, they generally become better at complex tasks. Scientists have long wondered if there is a hard limit to how many independent instructions a single model can hold in its "mind" at once. If you give a model ten rules to follow, can it do it? What about twenty? Does the ability to follow rules grow steadily as the model gets bigger, or does it suddenly break down?

A recent study claimed to have found a precise answer to this question. It suggested that as models get larger, their ability to track multiple rules grows according to a specific mathematical pattern, like a predictable curve. The researchers proposed that a small model might fail after just a few rules, while a larger model could handle many more, and that this relationship followed a strict law. This idea was exciting because it offered a way to predict exactly how capable a future machine would be based on its size. However, a new investigation has looked closely at the original numbers and the methods used to get them, and it tells a different story. The new work finds that while the basic idea of a limit is real, the specific law describing how that limit grows is not supported by the evidence.

The investigation began by re-examining the original data logs, which are the records of what the models actually wrote in response to the tests. The researchers built a new, stricter way to read these responses. In the original study, the computer program used to grade the answers had a safety net: if it couldn't find enough clear answers in a response, it would scan the entire text again, looking for the words "yes" or "no" anywhere they appeared. If it found them, it would count them as correct answers, even if the model had never actually intended them as answers to the specific questions. This safety net fired in nearly a quarter of the tests. It essentially filled in the blanks for the models, giving credit for words the models had written by accident or as part of a sentence, rather than as direct answers.

When the new researchers removed this safety net and forced the computer to only count answers that were clearly and correctly formatted, the results changed. The phenomenon of a limit still existed. A smaller model did struggle and eventually fail to follow many rules, while a larger model from the same family performed much better. This confirmed that the core observation was real: models do have a capacity limit, and size matters. However, the specific curve that described how this capacity grew with size disappeared. The original claim that the ability to follow rules doubled or tripled in a predictable way as the model got bigger was an illusion created by the way the answers were counted.

The researchers then tested this corrected method on a much wider range of models, including eight different families of systems available through online services. They found that the original mathematical law did not hold up. On one type of task, the models did not show the predicted growth at all; instead, they hit a ceiling where they could not handle more than a certain number of rules, regardless of how big they were. On another task, the models seemed to follow all the rules perfectly, but this was because they were simply repeating the instructions back in a long paragraph rather than actually tracking each rule individually. In several cases, the models failed not because they lacked the intelligence to follow the rules, but because they did not follow the strict format required to get a passing grade. They wrote long, thoughtful essays when the test demanded a simple list, and the scoring system counted this as a total failure.

The investigation also looked at the theory used to explain why the models behaved the way they did. The original study suggested that the limit was determined by a specific architectural feature of the models, a number calculated from their internal structure. The new audit found that the two models used to prove this theory had the exact same value for this feature, yet they performed very differently. This meant the theory could not actually explain the difference between them. The proposed explanation was too broad to be useful; it allowed for any result, so it could not predict the specific outcome observed.

The final conclusion is that the ability of these machines to track multiple constraints is a real and measurable thing, but the precise mathematical law that was claimed to describe it is not yet supported. The original finding was a mix of a genuine discovery and a measurement error. The error came from a common, invisible choice in how the computer read the answers, a choice that accidentally filled in missing data and created a false pattern. The researchers emphasize that this does not mean the models are perfect or that we fully understand their limits. It simply means that the specific claim about a predictable scaling law was built on shaky ground. To move forward, future studies must use stricter rules for reading answers, count every missing answer as a failure, and test many more models before declaring a universal law. The path to understanding these machines is clear, but it requires looking at the data without the safety nets that can hide the truth.

Technical Summary: "How Response Parsing Manufactures Scaling Laws"

Problem Statement
Recent work (Zhang, 2026a) proposed that Large Language Models (LLMs) possess a bounded capacity ( $k^*$ ) to track independent constraints, which follows a power-law scaling relationship $k^* \propto \theta^\beta$ with an exponent $\beta \approx 2.0$ . This hypothesis suggests that as model size ( $\theta$ ) increases, the number of constraints a model can simultaneously track increases superlinearly. The original study reported a "Safety Dimensionality Collapse" where smaller models (e.g., 1.5B parameters) collapse to near-chance performance while larger models (e.g., 7B) maintain high capacity. However, the original paper was desk-rejected without technical review, leaving the validity of its quantitative claims and the robustness of its measurement protocol unverified.

Methodology
The author conducted a rigorous replication and measurement-validity audit involving three distinct phases:

Log Re-derivation: The author re-analyzed the original response logs using both the original parser and a corrected, strict parser to isolate parsing artifacts.
Local Replication: They re-ran the measurement protocol on the same two local models (Qwen2.5-1.5B and Qwen2.5-7B) using a corrected parser that eliminated fallback mechanisms.
API-Scale Extension: The protocol was extended to eight API-served model families (including Qwen3, GLM-4.6, DeepSeek-V3.2, and Kimi-K2) totaling 800 trials to test the generalizability of the scaling law.

Key Findings and Results

Replication of the Phenomenon (The Collapse): Under a strict parsing regime, the qualitative phenomenon of "collapse" replicates. The 1.5B model saturates near chance ( $k^* \approx 1$ ), while the 7B model maintains significantly higher capacity ( $k^* \approx 13$ ). This confirms that the capacity to track constraints is a real, measurable property that varies by model size.
Failure of the Scaling Law ( $\beta \approx 2.0$ ): The specific claim that capacity scales with $\beta \approx 2.0$ $β \approx 2.0$ is unsupported.
- Parser Contamination: The original parser contained a "fallback" mechanism that scanned the entire response for "yes"/"no" tokens if the line-by-line parse failed. This fallback triggered in 23% of trials, effectively injecting answers the model never generated.
- Denominator Selection Bias: The scoring denominator only counted entries filled by the fallback, artificially inflating accuracy for models that wrote less.
- API Results: When tested on eight API models with a strict parser, the scaling exponent was not $\approx 2.0$ . On the binary judgment task, the fitted exponent was negative ($-1.04$) due to format non-compliance (models failing to output the requested format), and on the accumulative task, all models saturated at the ceiling ( $k=20$ ), suggesting the task measured output stuffing rather than constraint tracking.
Inert Architectural Bound: The original paper's theoretical bound ( $k^* \le O(dL/H)$ ) was found to be vacuous for the models used to derive the scaling law. Both the 1.5B and 7B models used for the fit share an identical architectural quantity ( $dL/H = 3584$ ), meaning the bound cannot distinguish between them or explain the observed order-of-magnitude difference in capacity.

Taxonomy of Failure Modes
The paper identifies three specific failure modes that manufactured the original scaling claim:

Parser Contamination: The fallback mechanism inserted answers (up to 2.7 per trial) that the model did not generate.
Denominator Selection: Scoring accuracy only over "filled" entries penalized models that wrote more (and thus had more chances to fail) while rewarding those that wrote less.
Format Non-Compliance: Treating empty strings or off-format responses (e.g., a model outputting a blog post instead of "yes/no") as incorrect judgments rather than formatting failures. This artificially drove $k^*$ to zero for larger models, creating a negative scaling trend.

Significance and Claims
The paper concludes that while the phenomenon of constraint-tracking collapse is real and replicable, the scaling law ( $k^* \propto \theta^{2.0}$ ) and its theoretical explanation are artifacts of flawed measurement protocols.

The author asserts that:

Published scaling laws for constraint tracking are not yet supported by the data.
The observed scaling was "manufactured" by ordinary, invisible parsing choices (regex fallbacks and denominator selection).
A credible measurement of capacity requires a protocol that forbids whole-response token scans, fixes the scoring denominator before seeing data, reports compliance rates as a primary metric, and utilizes at least three well-separated data points with intervals before calculating exponents.

The work serves as a methodological audit, demonstrating how specific implementation details in evaluation harnesses can generate spurious scaling laws, and provides a corrected protocol to prevent such artifacts in future LLM evaluation.

Technical Summary: "How Response Parsing Manufactures Scaling Laws"

More like this