🤖 AI

EntropyMoE: Entropy-Aware Sparse Expert Routing for Tokenizer-Free LLMs

EntropyMoE introduces an entropy-aware Mixture-of-Experts architecture for tokenizer-free large language models that dynamically routes byte-level patches to specialized experts based on their entropy and length, achieving superior compression efficiency and comparable accuracy without relying on fixed tokenizers.

Original authors: Bo Liu, Muxuab Yu, Yu Zhang, Pengfei Gao, Yongping Zhang

Published 2026-08-10

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Bo Liu, Muxuab Yu, Yu Zhang, Pengfei Gao, Yongping Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to read. For a long time, the best way to do this was to chop every sentence into pre-made "chunks" called tokens, like cutting a pizza into fixed slices before serving it. But what if the robot could read the raw ingredients instead—the individual letters or bytes? This is the world of "tokenizer-free" models. They are like chefs who don't use pre-cut ingredients but instead handle the whole vegetable, deciding on the fly how big a slice to take based on how tricky the texture is. If the vegetable is tough (uncertain), they take a tiny slice; if it's soft (predictable), they take a big one. This makes the reading process much more flexible.

However, there's a catch. Even though these robots are smart enough to chop the text into variable-sized pieces, they still treat every single piece exactly the same way once it's on the cutting board. They use the same amount of brainpower to process a simple, boring word as they do for a complex, confusing one. It's like hiring a team of 100 geniuses to solve a math problem, but making all 100 of them work on every problem, whether it's "2+2" or "quantum physics." It's a huge waste of energy. The big question scientists are asking is: Can we make the robot smarter by only calling in the right experts for the right job, without wasting time on the easy stuff?

This is exactly what the paper "EntropyMoE" tackles. The researchers built a new system called EntropyMoE that acts like a super-efficient manager for a team of AI experts. Instead of looking at the whole, complicated meaning of a text chunk to decide who should work on it, this manager only looks at one simple number: entropy. In everyday terms, entropy here is just a measure of "surprise" or "uncertainty." If a piece of text is very predictable (low entropy), the manager knows it's easy. If it's full of surprises (high entropy), it's hard.

The magic happens because the manager uses this "surprise score" to pick exactly two experts out of a team of eight to handle each text chunk. The best part? The manager doesn't need to read the whole text to make this decision; it just needs that single "surprise" number. This is like a bouncer at a club who only needs to see your ID to decide if you get in, rather than interviewing you about your whole life story. By doing this, the system saves a massive amount of computer memory and processing power. In their tests, the EntropyMoE system used only 400 tiny parameters to make these routing decisions, whereas the old way required over 400,000 parameters just to do the same job.

The results were impressive. When they tested this new system against older, "dense" models (where everyone works on everything) and other smart-routing models, EntropyMoE turned out to be the most accurate at predicting the next byte of text. It made the fewest mistakes, measured in "bits per byte." While it didn't completely crush the competition in every single game (like a specific trivia quiz called HellaSwag where it did slightly better), it proved that using "surprise" as a guide is a winning strategy. The paper suggests that this approach is a solid, efficient way to build smarter AI, though the researchers admit that the current version is still a bit slower to run than the older, simpler models because the "expert switching" takes a little extra time. Ultimately, they've shown that you don't need a super-complex brain to decide who does the work; sometimes, a simple measure of how surprising the text is is all you need to organize a team of experts.

Technical Summary: EntropyMoE

Problem Statement
Recent advances in tokenizer-free Large Language Models (LLMs), specifically the Byte Latent Transformer (BLT), have improved efficiency by grouping raw bytes into dynamically sized patches based on local predictive uncertainty (entropy). In this architecture, predictable byte sequences form longer patches, while uncertain sequences form shorter ones. However, existing byte-patch architectures apply uniform, dense feed-forward computation to every patch regardless of its semantic difficulty or granularity. This "one-size-fits-all" approach fails to adapt model capacity to the variations in patch semantics. Furthermore, standard Mixture-of-Experts (MoE) models are typically token-native, routing based on high-dimensional hidden states, which ignores the explicit uncertainty signal already computed during the patching process and introduces unnecessary routing overhead.

Methodology: EntropyMoE
The paper proposes EntropyMoE, a patch-native sparse expert architecture designed to replace the dense feed-forward layers in a global patch Transformer with Top-K MoE layers. The core innovation lies in using scalar patch entropy as the sole input for the routing mechanism, rather than high-dimensional hidden states.

Architecture: Built upon a BLT-style pipeline, EntropyMoE retains the local byte encoder, global self-attention, and decoder but replaces dense feed-forward sublayers with expert pools. Each dynamic patch serves as a single routing unit.
Entropy-Only Routing: Unlike conventional routers that project hidden states to expert logits, the EntropyMoE router maps only the normalized scalar entropy ( $H_i$ ) of a patch to expert logits via a learned affine transformation ($s = aH + b$). This reduces the routing parameters from $O(d \times M)$ to $O(M)$ , where $d$ is the hidden dimension and $M$ is the number of experts. In the reported experiments (25 layers, 8 experts), this results in only 400 trainable routing parameters compared to over 400,000 for hidden-state routers.
Top-K Selection: The router selects exactly $K=2$ experts per patch based on the entropy-derived logits. The patch's hidden state is processed by the selected experts but is explicitly excluded from the selection process.
Byte-Weighted Workload Accounting: Since patches vary in length, counting patches does not accurately reflect computational load. EntropyMoE introduces a byte-weighted load balancing objective. The load for an expert is calculated as the sum of the lengths of the patches assigned to it, normalized by the total byte count. This ensures that experts handling longer (low-entropy) patches are not under-utilized relative to those handling many short (high-entropy) patches.
Initialization: Sparse models are initialized from a dense BLT checkpoint by copying the dense feed-forward weights to all experts, ensuring the model starts with identical behavior before diverging during continuation training.

Key Contributions

Patch-Native Sparse Computation: The formulation of sparse conditional computation for tokenizer-free LLMs where dynamic byte patches are the fundamental routing units, decoupling routing decisions from high-dimensional semantic representations.
Entropy-Only Router: The introduction of an entropy-only Top-2 router that utilizes the causal uncertainty signal inherent to the patching pipeline. This separates patch granularity (determined by the patcher), expert selection (determined by entropy), and workload accounting (determined by byte length).
Controlled Empirical Validation: A rigorous comparison against Dense BLT, Hidden-only MoE, and Hidden+Entropy MoE under matched data, update counts, active parameter counts, and initialization.

Experimental Results
The authors evaluated EntropyMoE on the BLT-1B backbone using the FineWeb-Edu dataset.

Modeling Quality: EntropyMoE achieved the lowest held-out Bits Per Byte (BPB) (0.8351) among all tested configurations, outperforming the Dense BLT baseline (0.8442) and both hidden-state MoE variants. The improvement was statistically significant (95% confidence intervals strictly below zero).
Downstream Performance: Downstream accuracy remained broadly comparable to baselines. Notably, EntropyMoE showed a statistically significant improvement on the HellaSwag task (+1.16% over Dense BLT) while maintaining competitive performance on other tasks like PIQA and MMLU.
Routing Efficiency: EntropyMoE reduced routing parameters by three orders of magnitude (400 vs. ~400k) compared to hidden-state routers. It achieved the best iteration time and valid-byte throughput among the sparse models tested, though it remained approximately twice as slow as the Dense BLT due to MoE dispatch and communication overhead.
Routing Patterns: Analysis revealed that EntropyMoE induces a highly selective routing structure. Unlike hidden-state routers which distribute assignments broadly, EntropyMoE concentrates assignments on a small subset of experts (the top 2 experts account for 89.1% of the mass). It also demonstrated a clear divergence in routing behavior between low- and high-entropy patches (Entropy JSD of 0.1263 vs. 0.0040 for Hidden-only), indicating that entropy effectively differentiates expert utilization.
Workload Accounting: Byte-weighted load balancing consistently improved BPB over patch-count accounting, validating the necessity of accounting for byte coverage in variable-length patch scenarios.

Significance and Claims
The paper claims that patch entropy serves as an effective, low-dimensional routing coordinate for sparse conditional computation in tokenizer-free LLMs. By leveraging the uncertainty signal already computed for patching, EntropyMoE organizes sparse computation without consuming the full semantic representation for routing decisions.

The authors modestly state that while the results establish patch entropy as a promising basis for conditional computation and demonstrate improved byte-level modeling, they do not claim to have established indispensable expert identities or a compute-matched advantage over the dense backbone (as the current sparse implementation is still slower). The observed improvements are attributed to the entropy-conditioned routing geometry and byte-aware workload balancing rather than sparse capacity alone, as hidden-state MoE variants with similar capacity did not achieve the same gains. The work extends MoE modeling beyond tokenizer-based representations, offering a pathway to more efficient and adaptive byte-level language modeling.

Technical Summary: EntropyMoE

More like this