🤖 AI

Towards a Densing Law for User Representation Learning at Billion-Scale Capacity

This paper proposes the "User Behavioral Densing Law," a quantitative scaling pattern linking data size to minimum sufficient tokenization capacity, and introduces ALGN, an adaptive tokenization method that overcomes billion-scale data bottlenecks and outperforms existing baselines in user representation learning.

Original authors: Bin Dou, Junru Zhang, Zhaoyi Yuan, Wuliang Huang, Letian Gong, Baokun Wang, Huan Li, Yu Cheng, Weiqiang Wang

Published 2026-08-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Bin Dou, Junru Zhang, Zhaoyi Yuan, Wuliang Huang, Letian Gong, Baokun Wang, Huan Li, Yu Cheng, Weiqiang Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the digital world, every click, purchase, and search leaves a trace. For the companies that build the platforms we use daily, these traces are not just records; they are the raw material for understanding who we are. By analyzing the long, winding history of a person's actions, artificial intelligence systems try to build a single, powerful summary of that individual's preferences and habits. This summary, often called a user representation, acts as a digital fingerprint that helps decide what news to show, what products to recommend, or which advertisements to display. For years, the prevailing belief among engineers was simple: to make these fingerprints more accurate, you simply needed more data. The logic seemed sound—feed the system more users, longer histories of their lives, and bigger computer brains to process it all, and the results would naturally get better.

However, a new study challenges this assumption of "more is better." Researchers from Ant Group and Zhejiang University investigated whether this strategy of endless expansion actually works when dealing with the massive scale of real-world data, such as the billions of transactions processed by Alipay. They found that there is a point where adding more raw information stops helping and starts hurting. Just as a room can become so cluttered with furniture that it is impossible to move, a system can become so flooded with repetitive, low-value data that it loses its ability to see what truly matters. The team discovered that the bottleneck is not the size of the computer model, but the quality and density of the information being fed into it. They propose a new approach that focuses on compressing this massive history into a compact, high-value summary, allowing the system to learn more from less.

The researchers began by testing the limits of the traditional approach on a dataset containing hundreds of millions of users. They systematically increased the number of users, the length of the time window they observed, and the size of the computer model itself. They found that performance improved quickly at first, but then hit a hard wall. When they added more users beyond a certain point, or looked at histories longer than about sixty days, the system stopped learning anything new. The extra data was mostly just the same old habits repeated over and over, like a person buying the same coffee every morning for a decade. Even when they made the computer model significantly larger, it failed to get smarter; it simply memorized these repetitive details without gaining any real insight into the user's true preferences. This phenomenon, which the authors call the "raw behavioral scaling wall," showed that simply throwing more data at the problem was no longer a viable strategy.

To break through this wall, the team introduced a method of "densing" the data. Instead of feeding the system the entire, messy history of raw events, they first translated those events into a compact set of digital tokens, similar to how a book can be summarized into a few key sentences without losing the plot. They used a technique that groups similar behaviors together and strips away the redundancy, keeping only the most informative signals. When they trained their models on these compressed, tokenized versions of the user histories, the results were striking. The system continued to improve even as the data grew larger, whereas the system trained on raw data had already stalled. The compressed data allowed the model to see the forest rather than getting lost in the trees, maintaining its ability to learn and adapt even when the input volume was massive.

The study went further to define a precise rule for how much compression is needed as the data grows. They found that the amount of "space" required to store a user's summary does not need to grow in direct proportion to the amount of raw data. Instead, it follows a predictable pattern where the necessary capacity grows slowly as the data volume increases. This means that for a system handling a billion users, you do not need a billion times more memory or computing power; you only need a carefully calculated, much smaller amount of high-quality tokens. This discovery, which they term the "Behavioral Densing Law," provides a mathematical guide for engineers to know exactly how much compression to apply for any given amount of data, ensuring they do not waste resources on unnecessary information.

Finally, the researchers developed a new tool called the Adaptive Length Gated Network to put this law into practice. Previous methods treated every user's history the same way, assigning the same amount of compressed space to everyone, regardless of how complex their life actually was. The new tool is smarter; it looks at each user individually and decides how much detail to keep. For a user with a very routine, predictable life, it assigns a very short summary. For a user with a complex, diverse set of interests and habits, it assigns a longer, more detailed summary. This dynamic approach ensures that no computing power is wasted on boring, repetitive data, while complex users still get the attention they need. In their tests, this adaptive method not only outperformed all previous systems in accuracy but also used significantly less computing capacity, proving that efficiency and performance can go hand in hand.

The implications of this work extend far beyond a single app or website. It suggests a fundamental shift in how we build intelligent systems for the future. Instead of the endless race to collect more data and build bigger models, the path forward lies in learning to extract more value from the data we already have. By focusing on the density of information rather than the sheer volume, we can build systems that are not only more accurate but also more efficient and sustainable. The study demonstrates that in the age of big data, the most powerful tool is not a bigger bucket, but a better filter.

Technical Summary: Towards a Densing Law for User Representation Learning at Billion-Scale Capacity

1. Problem Statement

In industrial user representation learning, performance has traditionally been scaled by increasing the user population, extending behavioral sequence lengths, and enlarging model capacity. However, this paper identifies a critical phenomenon termed the "Raw Behavioral Scaling Wall."

Empirical analysis on billion-scale Alipay data reveals that simply increasing raw data volume or model size yields rapidly diminishing returns once specific thresholds are reached (e.g., >0.03B users, >60 days of history, >0.2B parameters). Beyond these points:

Redundancy Dominance: Additional data primarily introduces repetitive, low-information events (e.g., habitual payments) rather than novel task-relevant signals.
Loss-Quality Dissociation: Larger models achieve lower pre-training losses by fitting redundant details but fail to improve downstream accuracy (e.g., CTR prediction, retrieval).
Bottleneck Shift: The limiting factor is no longer model capacity but the information density of the input. Raw behavioral sequences contain too much noise and redundancy for encoders to extract additional value efficiently.

The core challenge is to determine how to scale user representations effectively when raw data scaling saturates, specifically by characterizing the quantitative relationship between data scale and the minimum sufficient tokenization capacity.

2. Methodology

The paper proposes a shift from volume-oriented scaling to density-oriented scaling, operationalized through three main components:

A. Behavioral Densing via Residual Quantization

To address the scaling wall, the authors employ Residual Quantized Variational Autoencoders (RQ-VAE) as a behavioral tokenizer.

Process: Long, multi-source raw behavioral logs (PayBill, SPM, MiniProgram) are mapped to continuous embeddings, then compressed into compact, discrete semantic tokens via multi-stage residual quantization.
Mechanism: This transforms repetitive raw sequences into a fixed-length sequence of discrete codes. Early codebooks capture macro-level semantics (e.g., consumption categories), while later stages refine residual variations (e.g., merchant preferences).
Goal: To suppress redundancy and increase the information density of each input unit without altering the downstream encoder architecture.

B. The Behavioral Densing Law

The authors formulate a quantitative law governing the relationship between data scale and the optimal tokenization capacity.

Definition: The law defines the Minimal Sufficient Capacity ( $C^*$ ) required to achieve Pareto-optimal performance for a given behavioral scale ( $s$ ).
Theoretical Formulation: Derived from a Pareto optimization problem balancing downstream utility ( $U$ ) and representation cost ( $C$ ), the law posits that the optimal capacity grows as a power law of behavioral complexity.
Empirical Formula:
$\ln C^*(s) = \beta + \alpha \ln(s/s_0)$
Where $s$ represents data scale (user count or time horizon), and $\alpha$ is a coefficient dependent on data diversity and the tokenization method's representation space efficiency.
Validation: Experiments across different data sources (PayBill, SPM, MiniProgram) and tokenization methods (RQ-VAE, VQ-VAE, SARQ) confirm that the minimal sufficient capacity follows this logarithmic scaling trajectory.

C. Adaptive-Length Gated Network (ALGN)

Moving from scale-level capacity selection to instance-level allocation, the paper proposes ALGN, a novel tokenization method inspired by the Densing Law.

Motivation: Fixed-length tokenization is suboptimal because user behavior heterogeneity varies; some periods are information-rich while others are routine.
Mechanism: ALGN dynamically decides the length of the semantic ID for each user sample. At each quantization level, a gating network evaluates:
1. Residual Norm: The amount of unexplained information.
2. Cumulative Uncertainty: The ambiguity of the current semantic path.
3. Marginal Cost: The cost of adding another code level.
Decision Rule: The tokenizer continues to the next level only if the expected marginal utility (high residual/uncertainty) exceeds the marginal cost. This results in variable-length tokens, allocating more capacity to complex behaviors and fewer to redundant ones.

3. Key Contributions

Scaling Diagnosis: The paper empirically identifies the "Raw Behavioral Scaling Wall," demonstrating that industrial user representation learning suffers from severe redundancy saturation. It shows that increasing model parameters beyond a certain point under fixed data leads to loss-quality dissociation.
Behavioral Densing Law: It formulates a predictive "recipe" for tokenization configuration. The law quantifies how the minimal sufficient tokenization capacity scales with data volume, providing a principled guide for configuration selection in billion-scale settings.
Adaptive Tokenization (ALGN): It introduces a variable-length tokenization strategy that adaptively allocates capacity based on behavioral complexity. ALGN outperforms fixed-length baselines and existing adaptive methods in both performance and efficiency.

4. Experimental Results

The proposed methods were evaluated on a billion-scale Alipay dataset across three downstream tasks: Classification (Linear Probing), Text-based Retrieval, and User-to-User (U2U) Retrieval.

Breaking the Scaling Wall: Tokenized representations (RQ-VAE) consistently outperformed raw sequence modeling, particularly in the saturated regime (longer histories, larger user populations). While raw data performance flattened, tokenized data showed sustained gains.
Efficiency Gains:
- ALGN vs. Baselines: ALGN achieved state-of-the-art performance (e.g., +1.07% AUC over the best baseline SARQ) while reducing capacity usage by 13.23%.
- Scaling Slope: The application of ALGN reduced the scaling slope of the Densing Law from 0.77 (RQ-VAE) to **0.59**, indicating a more efficient utilization of representation space.
Generalizability: The Densing Law held true across different data sources (PayBill, SPM, MiniProgram) and tokenization architectures (RQ-VAE, VQ-VAE, SARQ), with the scaling coefficients correlating with intra-datasource uniqueness.

5. Significance and Claims

The paper claims to provide the first systematic, large-scale empirical test of the assumption that "more data and larger models always yield better representations." Its significance lies in:

Paradigm Shift: It argues for a transition from collecting more raw data to optimizing the information density of existing data. This offers a principled alternative to the computationally and environmentally costly practice of indiscriminate data scaling.
Practical Guidelines: The identified saturation thresholds (e.g., 0.03B users, 60 days) offer concrete guidance for industrial practitioners regarding data retention policies and training budget allocation.
Theoretical Foundation: By establishing the Behavioral Densing Law, the paper provides a theoretical framework for predicting the optimal tokenization capacity required for specific data scales, moving user representation learning from heuristic tuning to quantitative estimation.
Cost-Performance Frontier: The work demonstrates that smaller encoders operating on densified, tokenized inputs can outperform larger models trained on longer raw histories, effectively shifting the performance-cost frontier.

The authors modestly note that the current study focuses on a single data modality (textual behavioral logs) and that the generalizability of the Densing Law to other modalities (e.g., video consumption) requires further validation.