💬 NLP

TextNCA: Neural Cellular Automata for Language Modeling via Hierarchical Local Attention

This paper introduces TextNCA, a hierarchical Neural Cellular Automaton for language modeling, to demonstrate that a staged narrow-to-wide attention schedule is the primary driver of performance, while iteration and specific architectural components provide smaller, bounded benefits.

Original authors: Avni Mittal, Avinash Anand, Ashutosh Kumar, Dikshant Kukreja, Kritarth Prasad, Sushane Dulloo, Erik Cambria, Timothy Liu, Zhengkui Wang, Rajiv Ratn Shah

Published 2026-08-04

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Avni Mittal, Avinash Anand, Ashutosh Kumar, Dikshant Kukreja, Kritarth Prasad, Sushane Dulloo, Erik Cambria, Timothy Liu, Zhengkui Wang, Rajiv Ratn Shah

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to write a story. For a long time, the best way to do this has been to build a massive, multi-story skyscraper of logic. Each floor of the skyscraper is a different "layer" that looks at the whole story at once, passing its understanding up to the next floor until the robot knows exactly what word comes next. This is how most modern AI language models work: they are tall, complex, and look at everything globally.

But what if we tried a different approach? What if, instead of a skyscraper, we used a tiny, simple robot that only looks at its immediate neighbors? This is the idea behind Neural Cellular Automata (NCA). Think of it like a game of "telephone" played on a grid, or a line of people passing a note. Each person only sees the person next to them, writes a small note, and passes it on. They do this over and over again. If they all follow the exact same simple rules, a complex picture or pattern can eventually emerge from the chaos. Scientists have used this to grow digital plants and textures, but they've never really tried it to write sentences. The big question is: Can this tiny, local, repetitive robot actually learn to write a story, and if it can, is it because it's local, because it repeats itself, or because of something else entirely?

The Experiment: A Tiny Robot with a Big Plan

In this paper, the researchers built a new kind of language model called TextNCA. Instead of a skyscraper, they built a "hierarchical" version of that tiny neighbor-robot. They gave it a very specific job description: it has to look at a small group of words, update its understanding, and then pass that updated understanding to the next group.

To make this work, they set up a three-stage relay race:

Stage 1 (The Narrow View): The robot looks at just 8 words at a time. It repeats this process 4 times, refining its understanding of that tiny neighborhood.
Stage 2 (The Medium View): It then moves to a window of 32 words. Again, it repeats the process 4 times, now seeing a bit more context.
Stage 3 (The Wide View): Finally, it looks at 128 words. It repeats the process 4 more times, finally seeing the whole sentence structure.

The robot uses the exact same "brain" (weights) for all 4 repetitions in each stage, just like a real cellular automaton. The researchers trained this on a massive dataset of text (WikiText-103) and compared it to standard AI models.

The Big Surprise: It's the Schedule, Not the Repetition

The researchers expected that the magic would come from the robot repeating its steps over and over (iteration) or from it only looking at local neighbors. But when they ran the experiments, they found something surprising.

The main driver of success was the "Narrow-to-Wide" schedule.
The model worked best because it started small and slowly got bigger. It was like reading a book: you first understand a single word, then a phrase, then a sentence, then a paragraph.

When they kept the schedule but stopped the robot from repeating itself (making it just a one-pass model), it still performed almost as well as the full repeating model. The difference was tiny (only about 4 points in a score called "Perplexity," which measures how confused the model is).
However, when they reversed the schedule (starting wide at 128 words and getting narrower), the model crashed. It became terrible at writing, scoring 131.1 compared to the good score of 60.3.
Even if they kept the same windows but shuffled them randomly (like 8, then 128, then 32), the model struggled.

This tells us that the "local" and "repeating" parts of the NCA design weren't the main heroes. The hero was the order of operations: starting narrow and expanding gradually.

What About the Repetition?

So, does repeating the steps help at all? Yes, but only a little bit, and only under very specific conditions.

The researchers found that repeating the steps exactly 4 times was the "sweet spot." If they repeated it 2 times, the model was confused. If they repeated it 6 or 8 times, the model actually got worse. It's like a student who studies a chapter once or twice and learns it well, but if they study it 10 times, they start to forget what they learned.
This repetition only worked because the model had a special "gate" (a GRU gate) and learned "step embeddings" (special markers that tell the robot which step of the process it is on). Without these, the repetition did nothing or even made things worse.

The Verdict: A Controlled Reading, Not a New Champion

The authors are very honest about their results. They admit that their TextNCA model is not better than the standard "skyscraper" models (Transformers). In fact, the standard models were much better at predicting the next word.

The standard 6-layer Transformer had a score of 52.8.
Their best TextNCA model had a score of 60.3.
The standard 12-layer Transformer was even better at 44.7.

So, this isn't a paper saying "We built a better AI." Instead, it's a paper saying, "We built a controlled experiment to figure out why these cellular automata models behave the way they do."

What We Learned (and What We Didn't)

The "Narrow-to-Wide" rule is king: The most important thing for this type of model is to start with a small view and slowly expand it. If you don't do this, the model fails.
Repetition is a dimmer switch, not a lightbulb: Iterating (repeating) the steps adds a small amount of power, but only if you stop at the right number (4). Doing too much repetition hurts performance.
Local attention has limits: When they tested the model on a reading comprehension task (SQuAD), it struggled to find answers that required looking far away in the text. This suggests that while the "start small, grow big" strategy helps, the model still can't see the whole picture as well as the standard models that look at everything at once.

The "Knob" Experiment

One cool side discovery was that they could train a version of the model that allows you to change the number of repetitions while it is running (at "inference time"). Usually, if you change how many times a model repeats itself after it's trained, it breaks. But by training the model with random numbers of repetitions, they created a "knob" that lets them adjust the depth on the fly. The catch? The model became much worse overall (a score of 101 instead of 60.3). It's like having a car that can drive at any speed you want, but it only goes half as fast as a normal car.

Conclusion

This paper doesn't give us a new, super-powerful language model. Instead, it gives us a clear map of how these "cellular automaton" models actually work. It proves that the magic isn't in the repetition or the local rules themselves, but in the hierarchical schedule—the careful, step-by-step expansion from a narrow view to a wide one. It's a reminder that sometimes, the way you organize your steps matters more than the steps themselves.

Technical Summary: TextNCA: Neural Cellular Automata for Language Modeling via Hierarchical Local Attention

Problem Statement
The paper investigates whether the three core properties of Neural Cellular Automata (NCAs)—strictly local perception, weight-shared iteration, and gated in-place updates—can support language modeling when applied to 1D causal sequences. While NCAs have been successfully used for image generation and texture synthesis, their application to language remains unexplored. Furthermore, existing language models occupy different corners of the design space: Universal Transformers use iteration and weight sharing but retain global attention; ALBERT shares weights without iteration; and sliding-window transformers use local attention without iteration or weight sharing. No existing architecture combines all three NCA properties simultaneously. The authors aim to determine which of these properties actually drives model behavior when applied to language.

Methodology: TextNCA Architecture
The authors introduce TextNCA, a 1D causal windowed-attention realization of the NCA primitive. The architecture is defined by the following components:

Core Update Rule: The model maintains a sequence of token states $h_t$ $h_{t}$ . At each step, it applies a shared local rule iteratively: $h_t = \text{Gate}(h_{t-1}, \text{FFN}(\text{LN}(\text{Perceive}(h_{t-1} + s_t))))$ $h_{t} = Gate (h_{t - 1}, FFN (LN (Perceive (h_{t - 1} + s_{t}))))$ .
- Perception: A causal windowed attention mechanism with a specific window size $w$ .
- Iteration: The same weights are applied $T_s$ times per stage.
- Gating: A GRU-style update gate combines the perception output with the previous state, allowing for in-place updates.
- Step Embeddings: A learnable step embedding $s_t$ is added at each iteration to allow shared weights to specialize across different steps.
Hierarchical Staging: The model processes tokens through $K$ $K$ sequential stages, each with a distinct parameter set $\theta_k$ $θ_{k}$ and a specific window size $w_k$ $w_{k}$ . The stages cascade with strictly increasing window sizes: $w \in \{8, 32, 128\}$ $w \in {8, 32, 128}$ .
- Schedule: The model uses a "narrow-to-wide" schedule. Stage 1 uses $w=8$ , Stage 2 uses $w=32$ , and Stage 3 uses $w=128$ .
- Iterations: Each stage iterates $T_s=4$ times, resulting in a total of 12 NCA steps.
Training Protocol: The model is trained on a 500M-token subset of SlimPajama using the Mistral 32K BPE vocabulary, with a sequence length of 512. It is evaluated on WikiText-103. The model has approximately 30M parameters and is trained for 60k steps.

Key Contributions and Ablation Studies
The paper treats TextNCA as an analytical probe rather than a performance bid, conducting a seven-axis ablation to isolate the "load-bearing" components:

The Staged Window Schedule is Dominant: The most critical finding is that the "narrow-to-wide" schedule is the primary driver of performance.
- Reversing the schedule ( $[128, 32, 8]$ ) increases Perplexity (PPL) by +70.8.
- Using a uniform window ( $[32, 32, 32]$ ) increases PPL by +59.4.
- A non-iterating sliding-window Transformer that reuses the same narrow-to-wide schedule achieves a PPL of 64.5, only +4.1 worse than the iterated TextNCA (60.3).
- This suggests the schedule accounts for the bulk of the architecture's effect, while iteration is a secondary contributor.
Iteration as an Effective Depth Knob: Iteration adds a bounded benefit (~4 PPL) but behaves non-monotonically.
- A sweep of iterations per stage ( $T_s$ ) reveals a sharp U-shaped curve with an optimum at $T_s=4$ .
- Increasing iterations beyond 4 degrades performance, ruling out an "iterative refinement" reading where more steps always help.
- Iteration requires both the GRU gate and learned per-step embeddings to be effective; removing either causes significant PPL degradation.
Component Sensitivity:
- Gating: Replacing the GRU gate with a simple residual update increases PPL by +16.3.
- Perception Kernel: While Gated Linear Attention (GLA) outperforms local softmax at a single scale, it underperforms in the hierarchical body, indicating a strong interaction between the kernel and the schedule.
- Stochastic Training: Training with random $T_s$ (stochastic iteration) enables an inference-time iteration-count knob, allowing the model to generalize to different iteration depths. However, this comes at a substantial cost, with the best PPL rising to ~101 compared to the deterministic flagship's 60.3.

Results

Performance: At the 30M parameter scale, TextNCA (PPL 60.3) does not match parameter-matched Transformers (Transformer-6L: 52.8; Transformer-12L: 44.7).
Compute Efficiency: TextNCA is not computationally Pareto-superior. When matched for FLOPs or wall-clock time, the performance gap widens significantly (TextNCA PPL rises to 70.6–80.1).
Downstream Tasks:
- Classification: TextNCA is statistically tied with Transformer baselines on sentiment and topic classification (IMDb, AG News, SST-2).
- Extractive QA: TextNCA underperforms on SQuAD (F1 26.3 vs. Transformer-6L's 28.6). The authors attribute this to the local attention design, as global-attention models consistently outperform local variants on span-level QA.

Significance and Claims
The paper positions its contribution as a "controlled reading" of NCA-style computation in language modeling. The authors claim that:

The staged narrow-to-wide window schedule is the load-bearing architectural factor, explaining the majority of the model's behavior.
Iteration and weight sharing provide a smaller, bounded benefit that functions as an effective-depth knob, contingent on the presence of specific gating mechanisms and step embeddings.
The model demonstrates that NCA-style computation can function as a language model, but at the studied scale, it does not outperform standard Transformers. The work isolates which parts of the NCA primitive carry the weight, suggesting that the "global structure emerging from local rules" in NCAs is heavily dependent on the specific scheduling of receptive fields rather than the iteration mechanism alone.
The paper explicitly avoids claiming "test-time compute scaling" in the sense of chain-of-thought or search, noting that the inference-time iteration knob achieved via stochastic training incurs a high absolute PPL penalty.