🤖 machine learning

Inferring the Size of Large Language Models From Popular Text Memorization

This paper introduces a black-box method that infers conservative lower bounds on the parameter counts of large language models by analyzing their memorization accuracy of popular texts, enabling the ranking of model sizes and revealing hidden industry scaling strategies even for closed-weight systems.

Original authors: Ivica Nikolic

Published 2026-05-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ivica Nikolic

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you walk into a room where several people are reciting famous stories like Alice in Wonderland or the Bible. You can't see them, you can't ask them how big their brains are, and you can't peek at their notes. You can only listen to what they say next when you stop them in the middle of a sentence.

This is exactly the problem researchers faced with modern AI models (Large Language Models). Big tech companies build these powerful AIs but keep their "blueprints" secret. They won't tell you how many "neurons" (parameters) the AI has, which is the standard way to measure how smart or expensive a model is.

This paper introduces a clever "black-box" detective method to guess the size of these secret AI brains just by listening to how well they finish sentences.

The Core Idea: The "Memory Test"

The researchers realized that almost every big AI is trained on the same massive pile of internet text. This means they have all read the same popular books, religious texts, and famous documents.

Think of it like a memory contest. If you ask a person to finish a sentence from a book they've memorized, a person with a bigger brain (more parameters) will generally get it right more often than someone with a smaller brain.

The researchers tested this by:

Picking 37 famous, public-domain texts (like Hamlet, the Bible, and the US Constitution).
Cutting them up into thousands of little snippets.
Asking different AI models to predict the very next word in those snippets.

They found a clear pattern: The bigger the AI, the better it was at guessing the next word in these famous stories. Even though other factors (like how the AI was taught) matter, the size of the brain was the biggest driver of success.

The Two Detective Tools

To turn these "memory scores" into a size estimate, the team built two different tools:

1. The "Scaling Law" (The Map)
Imagine you have a group of known people (open-source AIs) whose brain sizes you do know. You plot their memory scores against their known sizes and draw a smooth curve connecting the dots. It looks like a ladder: as the score goes up, the size goes up exponentially.

How it works: When a secret AI takes the test, you see where it lands on the ladder. If it scores like a 100-billion-parameter model, you estimate it's roughly that size.
The Catch: Because the secret AI might be built differently (like a "Mixture of Experts" model where only part of the brain is active at once), the researchers decided to be very conservative. They don't guess the exact size; they guess a minimum floor. They say, "This AI is at least this big."

2. The "Head-to-Head" Test (The Race)
Sometimes, you don't need an exact number; you just need to know who is bigger.

How it works: The researchers pit two secret AIs against each other on the same memory test. They use a statistical "race" to see if one AI is consistently better at finishing the sentences than the other.
The Result: If AI A beats AI B in the race, they can confidently say, "AI A is definitely larger than AI B," even if they don't know the exact numbers.

What They Discovered

The team tested their method on 19 open-source models (where they knew the answers) and got it right 95% of the time. Then, they turned their eyes on the "secret" models from companies like OpenAI, Google, Anthropic, and Alibaba.

Here is what the "detective work" revealed about the industry's hidden strategies:

The "Big Brain" Strategy (Google & Anthropic): These companies seem to be building models with massive parameter counts. For example, their top-tier models showed signs of having hundreds of billions of parameters, suggesting they are getting bigger and bigger with each new generation.
The "Efficiency" Strategy (OpenAI & Alibaba): Surprisingly, the researchers found that OpenAI's newer models (like GPT-4o and GPT-5 variants) didn't seem to be getting significantly bigger in terms of raw parameter count compared to their older versions. They seem to be hitting a "size ceiling." Instead of making the brain bigger, they appear to be making the existing brain smarter through better training or smarter algorithms.
The Hierarchy Check: The method correctly identified the internal ranking of products. For example, it correctly guessed that Anthropic's "Opus" model is bigger than its "Sonnet" model, which is bigger than its "Haiku" model, without being told that order beforehand.

The Limitations (The Fine Print)

The paper is very honest about what it can't do:

It's a Lower Bound: The method is designed to be safe. It will tell you the AI is at least X size, but the real size could be much bigger. It's like saying, "This box weighs at least 10 pounds," when it might actually weigh 50.
The "MoE" Problem: Some modern AIs use a "Mixture of Experts" architecture. Imagine a library where only a few specific librarians are called in to answer a question, rather than the whole staff. Because the researchers can't see how many librarians are total versus how many are active, they can't get an exact total count for these models, only a conservative minimum.
No Cheating: The method assumes the AI isn't trying to trick them. If a company intentionally programmed their AI to forget famous books to hide its size, this method would fail.

The Bottom Line

This paper proves that you don't need a company's secret blueprints to get a good idea of how big their AI is. By simply testing how well an AI remembers famous stories, you can infer a reliable "minimum size" and uncover whether a company is growing its models by making them bigger or by making them smarter. It turns the "black box" of AI into something we can peek inside, even if we can't see everything.

Technical Summary: Inferring the Size of Large Language Models From Popular Text Memorization

1. Problem Formulation

The total parameter count ( $|f|$ ) of a Large Language Model (LLM) is a primary metric for interpreting capabilities, costs, and performance comparisons. However, for the most advanced closed-weight models developed by major tech companies (e.g., OpenAI, Google, Anthropic), this metric is almost always withheld. This opacity creates ambiguity in comparing closed and open models and obscures the economic drivers of API pricing and latency.

The paper addresses two specific inference problems under strict black-box constraints (access limited to submitting prompts and observing text outputs, with no access to logits, weights, or hidden states):

Absolute Size Inference: Estimating the parameter count $|f|$ of an unknown model $f$ .
Relative Size Inference: Determining if $|f| > (1 + \tau)|g|$ for a threshold $\tau > 0$ , establishing whether one model is significantly larger than another.

The core challenge is isolating the signal of model size from confounding variables such as training data distribution, instruction tuning, and architectural choices, which also influence text generation quality.

2. Methodology

The proposed approach relies on the observation that popular, widely-circulated texts (e.g., classical literature, religious texts, foundational documents) are present in virtually every large-scale pretraining corpus. The authors hypothesize that a model's ability to accurately predict the next token in these memorized texts is fundamentally limited by its total parameter count.

Data Collection and Signal Extraction:

Source Texts: A diverse set of 37 popular public-domain texts (e.g., Alice in Wonderland, The Bible) and 4 obscure baseline texts are used.
Accuracy Metrics: For a model $M$ , text $T$ , and prefix length $l$ , the authors compute raw accuracy ( $\hat{\alpha}_r$ ) and lifted accuracy ( $\hat{\alpha}_l = \hat{\alpha}_r - \hat{\beta}$ , where $\hat{\beta}$ is the baseline accuracy on obscure texts).
Accuracy Profile: By sampling multiple prefix lengths and texts, the method constructs a high-dimensional accuracy profile vector ( $A_{T,L}(M) \in \mathbb{R}^{2KL}$ ) for each model, aggregating performance across varying contexts.

Inference Framework:
The paper introduces two complementary methods to translate these accuracy profiles into size estimates:

Scaling-Law Estimator (Absolute Inference):
- PCA Projection: Principal Component Analysis (PCA) is applied to the accuracy profiles of known open-weight models to extract a one-dimensional latent size index ( $z_M$ ). This index captures the dominant variance in performance, assumed to be driven by parameter count.
- Exponential Scaling: Based on neural scaling laws, the relationship between the latent index $z$ and parameter count $\theta$ is modeled as exponential: $\hat{\theta}(z) = A \cdot e^{B \cdot z}$ . This curve is calibrated using open-weight models with known sizes.
- Conservative Bound: For closed models, the estimate is derived from this curve but adjusted by a factor of 0.5 (based on cross-validation error margins) to produce a conservative lower bound.
Pairwise Statistical Test (Relative Inference):
- Blocked Sign-Permutation Test: To compare two models $f$ and $g$ , the method computes the difference in their accuracy profiles. Because measurements within the same text are correlated, the test treats each source text as a "block."
- Hypothesis Testing: A one-sided blocked sign-permutation test evaluates whether the accuracy differences systematically favor one model over the other. The null hypothesis ( $H_0$ ) assumes no systematic advantage. Rejection of $H_0$ indicates that the superior model is likely larger.
- Thresholding: The test determines if $|f| > (1+\tau)|g|$ with a controlled false positive rate.

Scope and Limitations:
The method assumes general-purpose models trained on standard internet-scale corpora. It is explicitly scoped to exclude models with catastrophic forgetting due to specialized fine-tuning (e.g., code-only models) or those employing strategic evasion (e.g., intentional unlearning of public texts). For Mixture-of-Experts (MoE) architectures, the method provides a conservative lower bound on the total parameter count, as the active parameters during inference are a subset of the total.

3. Key Contributions

Memorization-based Size Signal: The paper demonstrates that aggregating next-token accuracy on popular texts yields a robust, high-dimensional signal strongly correlated with model size, even in the absence of internal model access.
Dual Lower-Bound Framework: It proposes a dual-method approach combining a PCA-driven scaling-law estimator and a rigorous pairwise statistical test. The final inference for any target model is the maximum of the bounds produced by these two methods.
Validation and Application: The framework is validated on 19 open-weight dense models (achieving $R^2 = 0.95$ for absolute estimation and ~90% precision/recall for relative testing), 16 open-weight MoE models, and 21 popular closed-weight models.

4. Results

Open-Weight Dense Models: The scaling-law estimator achieved an $R^2$ of 0.9504 on the calibration set and 0.9387 in leave-one-out cross-validation. All predictions fell within a factor-of-two error band.
Open-Weight MoE Models: The framework successfully established valid lower bounds for all 16 evaluated MoE models. While the "tightness" (ratio of bound to true size) varied significantly (e.g., 86% for WizardLM-2-8x22b vs. 11% for Mimo-v2-pro), the bounds were consistently conservative (never exceeding the true size).
Closed-Weight Models: Applied to 21 models from Anthropic, Google, OpenAI, and Alibaba, the framework:
- Correctly recovered internal product hierarchies (e.g., Opus > Sonnet > Haiku for Anthropic).
- Revealed divergent industry scaling strategies:
  - Google and Anthropic: Showed significant generational scaling, with lower bounds exceeding 400B parameters for flagship models (e.g., Claude Opus 4.6 at 433B, Gemini 3 Flash at 405B).
  - OpenAI and Alibaba: Operated under strict parameter ceilings. Flagship models like GPT-4o and Qwen3.6-Max were bounded near 110B, suggesting performance gains are driven by algorithmic optimization or data quality rather than massive parameter expansion.
- Identified specific bounds for smaller variants (e.g., GPT-5.4-nano at 2B, Claude Haiku 4.5 at 49B).

5. Significance and Claims

The paper claims to provide the first systematic, black-box methodology to probe the hidden design choices of closed LLMs. By demonstrating that model size leaves a detectable "fingerprint" in the memorization of public-domain texts, the authors show that:

Transparency: It is possible to infer conservative lower bounds on model size without access to weights or logits.
Strategic Insight: The method reveals that the industry is not monolithic; some providers are scaling parameters aggressively, while others appear to have hit a parameter ceiling, relying on other levers for capability improvements.
Robustness: The approach is robust against variations in training data and architecture for general-purpose models, provided the memorization signal is not intentionally suppressed.

The authors remain modest, acknowledging that the bounds are conservative by construction and that recovering the exact parameter count for MoE architectures or models with strategic evasion remains an open challenge. The work does not claim to replace internal metrics but offers a rigorous external probe for the "black box" of modern AI development.