💬 NLP

A Shared Subcircuit Lets LLMs Count Down Across Tasks

This paper identifies a shared "countdown subcircuit" in Llama-3.1-70B-Instruct that enables language models to track remaining tokens across diverse tasks, revealing a generalizable mechanism for estimating time-to-goal that transfers across different models and contexts.

Original authors: Jacob Dunefsky, Wes Gurnee, Emmanuel Ameisen

Published 2026-07-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jacob Dunefsky, Wes Gurnee, Emmanuel Ameisen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you're playing a game where you have to write a sentence, but there's a strict rule: it must be exactly 10 words long, and the very last word has to be "telescope." If you write 9 words, you lose. If you write 11, you lose.

For a long time, we thought Large Language Models (LLMs)—the super-smart AI brains behind chatbots—just guessed their way through these tasks. But a new study suggests something much more organized is happening inside the machine. The researchers found that the AI has a tiny, specialized "countdown subcircuit" that acts like a built-in stopwatch, ticking down the remaining words until it hits the finish line.

The Hidden Stopwatch

The researchers started by instructing a specific model, Llama-3.1-70B-Instruct, to write sentences of a fixed length. They wanted to know how the model knew when to stop. By peering inside the model's "brain" (specifically looking at how it processes information layer by layer), they discovered a specific group of three attention heads (think of these as tiny, specialized workers) in layer 18 that are responsible for this countdown.

These three workers—labeled L18H24, L18H28, and L18H30—don't just guess. They actually measure the distance between where the sentence is right now and where it needs to end.

One worker gets active early in the sentence.
Another kicks in during the middle.
The third one fires up right at the end.

Together, they create a smooth "countdown" signal that gets stronger and stronger as the model gets closer to the target word count. It's like having three different alarm clocks set to go off at different times, ensuring the model knows exactly how much time (or in this case, how many words) it has left.

The Magic of "Implicit" Counting

Here is where it gets really cool. You might think this countdown only works if you explicitly tell the AI, "Write a 10-word sentence." But the researchers found that the model is smarter than that.

They tested the model with prompts that didn't give a number at all. Instead, they gave the model a pattern. For example, they showed it a list of DNA sequences or base64 codes, all exactly the same length. When asked to continue the list, the model didn't need a number; it just "figured out" the pattern. It inferred the target length from the previous examples and used that same countdown subcircuit to stop at the right spot.

The study suggests that the model stores this "invisible" length information right at the end of the previous strings (like a hidden note saying "Next one should be this long"). When the model sees the end of one string, it reads that note and starts its countdown for the next one.

A Universal Tool Across Models and Tasks

The researchers didn't stop there. They used a technique called "unsupervised probing" to see where else this countdown tool was hiding in the wild. They scanned through massive amounts of text, including news articles, chat logs, and code, looking for places where these specific workers were firing.

They found the countdown subcircuit popping up in all sorts of surprising places:

Embedded Tweets: The model realized it was reading a tweet and counted down to the 280-character limit, even though the text didn't say "tweet."
ASCII Tables: When formatting a table, the model counted the spaces to make sure every column was the same width.
Haikus: It tracked the syllable count for the 5-7-5 structure.
SHA Hashes: It knew exactly when a 64-character code was finished.

This suggests that the model isn't just memorizing rules for specific tasks; it has a general-purpose "countdown engine" that it reuses for almost anything that requires tracking a length.

A Shared Blueprint?

Perhaps the most fascinating finding is that this isn't just a quirk of the Llama model. The researchers compared the internal "geometry" (the shape of the data) of this countdown subcircuit to a different model, Claude 3.5 Haiku, which was studied on a different task (wrapping text to fit a line).

They found that both models use the exact same mathematical shape to store and process length information. It's as if two different car manufacturers, working in different factories, both decided to use the exact same engine design for their cars. This suggests that this "countdown motif" might be a fundamental building block that many AI models discover on their own when they need to keep track of time or space.

What This Means (and What It Doesn't)

The paper is careful to say that they have reverse-engineered this specific behavior. They proved that if you mess with the signals in these specific heads, the model stops counting correctly. They have measured the signals and observed the patterns.

However, they don't claim to have solved the mystery of how AI thinks in general. They are simply showing that for the specific job of "knowing when to stop," the AI uses a very specific, reusable, and surprisingly universal tool. It's a small but powerful piece of the puzzle, showing that even in the complex, chaotic world of AI, there are neat, organized subroutines that help the machine make sense of the world.

So, the next time you see an AI write a perfect haiku or format a table without being told exactly how many spaces to use, remember: somewhere deep inside, three little workers in layer 18 are quietly ticking down the clock, making sure everything ends just right.

Technical Summary: A Shared Subcircuit Lets LLMs Count Down Across Tasks

Problem Statement

Large Language Models (LLMs) frequently perform tasks requiring the tracking of token counts relative to a target goal, such as writing a sentence of a specific length, formatting ASCII tables, or ending DNA sequences at the correct codon. While LLMs demonstrate proficiency in these tasks, the internal mechanisms enabling them to track "time remaining" until a target length are not fully understood. Specifically, it is unclear whether models possess a generalizable mechanism for counting down that operates across different contexts (explicit vs. implicit targets) and whether this mechanism is shared across different models.

Methodology

The authors employed a bottom-up approach to mechanistic interpretability, focusing on the Llama-3.1-70B-Instruct model. The methodology proceeded in three stages:

Causal Localization via Patching:
- Task: The model was tasked with a "sentence planning" prompt: writing a sentence of a specified length ( $N$ ) ending in a specific word.
- Technique: The authors performed residual stream patching. They constructed "source" prompts (target length $n_{src}$ ) and "destination" prompts (target length $n_{dst}$ , where $n_{dst} > n_{src}$ ). Activations from the source prompt's target length token and length unit token were patched into the destination prompt at various layers.
- Goal: To identify the specific layers and tokens where length information is causally sufficient to determine the output length.
Identification of Attention Heads:
- Once the critical layer was identified, the authors analyzed the output norms of attention heads at that layer, specifically looking at the flow of information from the length unit token to downstream tokens.
- They examined the geometry of the representations (keys and queries) used by these heads to determine how they compare current position against target length.
Unsupervised Probing and Generalization:
- Implicit Length: The authors investigated settings where target length was not explicitly stated but inferred from context (e.g., repeated strings of identical length in Base64 or DNA sequences). They used patching to verify if ending delimiters in these strings causally stored length information.
- Geometric Analysis: They compared the geometric manifolds of the identified subcircuit in Llama-3.1-70B against a previously identified "line-wrapping" subcircuit in Claude 3.5 Haiku using Principal Component Analysis (PCA) and cosine similarity.
- Unsupervised Discovery: Using the output activations of the identified "countdown heads," they constructed linear probes. These probes were run on a large, unlabeled corpus (The Pile and LMSYS-1M chat dataset) to find diverse, high-activation examples where the model was implicitly counting down.

Key Contributions and Results

1. Identification of the Countdown Subcircuit

The study successfully isolated a specific subcircuit in Llama-3.1-70B-Instruct responsible for tracking remaining tokens:

Localization: Length information is moved to the "length unit token" starting around Layer 4 and is causally used until Layer 18.
Specific Heads: Three attention heads at Layer 18 (L18H24, L18H28, L18H30) were identified as the "countdown heads."
Mechanism: These heads exhibit a pattern of overlapping receptive fields. L18H28 activates early, L18H24 in the middle, and L18H30 near the end. Their combined output norm peaks immediately before the target length, indicating a continuous countdown mechanism rather than a sudden spike at the target.

2. Implicit Length Storage and Universality

The authors demonstrated that this subcircuit functions even when the target length is not explicitly provided in the prompt:

Implicit Storage: In "repeated string" settings (e.g., multiple Base64 strings of the same length), the model infers the target length from context. Patching experiments confirmed that ending delimiters of these strings causally store the implicit target length information.
Representation Similarity: Cosine similarity analysis revealed that the key activations for implicit lengths (in delimiters) and explicit lengths (in prompt tokens) lie on similar manifolds. The relationship between implicit and explicit lengths is highly linear ( $0.93 \le r^2 \le 0.97$ ), suggesting a unified representation of length information.
Cross-Model Motif: The geometric structure of the countdown subcircuit (curved manifolds for keys and queries with specific alignment properties) was found to be identical to the "line-wrapping" subcircuit previously identified in Claude 3.5 Haiku. This suggests the motif is shared across different models and tasks.

3. Unsupervised Discovery of Diverse Applications

Using unsupervised probing with the countdown heads' outputs, the authors identified a wide variety of tasks where this subcircuit is active, including:

Embedded Tweets: Predicting the end of a 280-character tweet embedded in news text.
DNA Sequences: Ending sequences at the correct codon based on previous sequences.
SHA-256 Hashes: Detecting the end of a 64-character hex string.
Haiku Generation: Tracking syllable counts and line breaks.
ASCII Tables: Aligning cells to a specific width.
Coin Tosses and Rounding: Tracking character or digit counts in generated sequences.

Significance and Claims

The paper claims that reverse-engineering a subcircuit from a single, narrow task (sentence planning) allows researchers to understand how specific behaviors generalize to a broad spectrum of tasks and even across different models.

Generalization: The work demonstrates that a single subcircuit is reused for both explicit counting (prompt-specified) and implicit counting (context-inferred).
Shared Mechanisms: The identification of the same geometric motif in Llama-3.1 and Claude 3.5 Haiku suggests that certain computational primitives for tracking time/length are fundamental and shared across frontier models.
Methodological Approach: The authors propose a "bottom-up" workflow for mechanistic interpretability: isolate a subcircuit on a controlled task, reverse-engineer its geometry, and then use unsupervised probing to map its usage across the model's entire operational space. They suggest this approach could be automated, similar to feature dictionary analysis, to systematically uncover other generalized behaviors in LLMs.

The authors remain modest, noting that while they have identified a shared motif, the work focuses on the causal interventions and geometry of length processing, rather than claiming to have solved all aspects of LLM planning or generalization.