🤖 AI

Subliminal Clocks: Latent Time Modelling in Diffusion Language Models

This paper demonstrates that Diffusion Language Models, despite lacking explicit timestep conditioning, internally encode a decodable latent representation of denoising progress within their residual streams—specifically within the activation $h_{n,t}$ , where $n$ denotes the layer index and $t$ denotes the denoising timestep—which can be manipulated to systematically control model confidence and entropy, with the KL divergence exhibiting a non-linear relationship with the distance between the estimated and actual timesteps.

Original authors: Maximo Rulli (Sapienza University of Rome), Thomas Fontanari (Sapienza University of Rome), Simone Petruzzi (Sapienza University of Rome), Federico Alvetreti (Sapienza University of Rome), Giorgio Str

Published 2026-07-03

📖 5 min read🧠 Deep dive

CC0 1.0

Original authors: Maximo Rulli (Sapienza University of Rome), Thomas Fontanari (Sapienza University of Rome), Simone Petruzzi (Sapienza University of Rome), Federico Alvetreti (Sapienza University of Rome), Giorgio Strano (Sapienza University of Rome), Donato Crisostomi (Sapienza University of Rome), Giorgos Nikolaou (EPFL), Tommaso Mencattini (EPFL), Andrea Santilli (Independent researcher), Emanuele Rodolà (Sapienza University of Rome), Simone Scardapane (Sapienza University of Rome), Alessio Devoto (Independent researcher)

Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a Diffusion Language Model (DLM) as a master sculptor working in a dark room. Unlike a traditional writer who builds a story one word at a time from left to right, this sculptor starts with a block of stone completely covered in clay (represented by [MASK] tokens). Their job is to chip away the clay, revealing the hidden words underneath, one layer at a time, until the final statue is revealed.

The big mystery this paper solves is: Does the sculptor know how much clay is left?

In traditional writing models, the computer knows exactly how many words it has written. But in these "sculpting" models, the paper asks if the model has an internal "clock" or a sense of progress, even though it wasn't explicitly taught to count.

Here is what the researchers found, explained simply:

1. The "Subliminal Clock"

The researchers discovered that even though the model isn't told "You are 50% done," it secretly keeps track of its own progress. It's like a sculptor who doesn't have a watch but can feel the weight of the stone in their hands and instinctively know, "I'm halfway through."

To understand how this works, we look at the model's internal activation state, denoted as $h$ . This state is defined by two key indices:

$n$ (Position): This represents the specific location of a token within the sequence (e.g., the 5th word in the sentence).
$t$ (Time Step): This represents the current stage of the denoising process (e.g., step 20 out of 100 total steps).

The researchers found a hidden signal inside the model's brain (specifically in its "residual streams," which are like the model's internal thought pathways) associated with these activations $h_{n,t}$ . This signal acts as a latent clock. It tells the model exactly how much of the "clay" (the masked tokens) has been removed at time step $t$ and how much is left, regardless of the token's position $n$ .

2. Reading the Mind (Probing)

To prove this clock exists, the researchers built a simple "decoder" (a probe). They looked at the model's internal thoughts at different stages of the process.

The Result: The decoder could accurately read the model's internal clock. It could tell, just by looking at the model's activation, whether the model was at the very beginning (lots of clay), the middle, or the very end (almost no clay left).
The Analogy: It's like being able to look at a person's posture and instantly know if they are just starting a marathon or if they are about to cross the finish line, even if they haven't said a word.

3. Hacking the Clock (Steering)

The most exciting part is that the researchers didn't just read the clock; they rewired it. They found a specific direction in the model's brain that corresponds to "time" or "progress."

The Experiment: They artificially pushed the model's internal clock forward or backward.
- Pushing Forward: They told the model, "You are almost done!" (even if it wasn't). The model immediately became more confident, spoke with less hesitation (lower entropy), and acted like it was ready to finish.
- Pushing Backward: They told the model, "You are just starting!" (even if it was near the end). The model became confused, less confident, and started second-guessing its choices.
The Takeaway: This proves the clock isn't just a passive observation; it's a control mechanism. If you change the model's sense of time, you change how it behaves. Notably, the impact of this manipulation is not a simple, straight-line increase in error; the divergence between the model's true state and its manipulated state does not scale linearly with the distance of the time shift ( $\hat{t}-t$ ), suggesting a more complex, non-linear relationship in how the model processes temporal information.

4. The Shape of Time

The researchers also looked at the geometry of this "clock." They found that the model doesn't represent time as a messy, chaotic cloud. Instead, it organizes time into a very neat, smooth curve (like a parabola).

The Analogy: Imagine the model's understanding of time as a rollercoaster track. As the denoising process moves forward, the model's internal state slides smoothly along this track. The researchers found that almost all the information about "how far along we are" fits onto this single, low-dimensional track.

5. The Self-Correcting Mechanism

One of the coolest findings is that the model is smart enough to fix mistakes.

If the researchers messed with the clock in the early layers of the model (the "beginning" of the thought process), the model would often "correct" itself as the information moved through the deeper layers. It would realize, "Wait, I was told I'm at the finish line, but I'm actually still in the middle," and it would adjust its internal state back to the truth.
However, if they messed with the clock in the very final layers, the model couldn't fix it, and the behavior changed permanently.

Summary

This paper reveals that Diffusion Language Models have a hidden, internal sense of "how much work is left to do."

They have a clock: They encode their progress in their internal brain states ( $h_{n,t}$ ), tracking both position ( $n$ ) and denoising step ( $t$ ).
We can read it: We can measure this progress accurately.
We can control it: By tweaking this internal clock, we can make the model more or less confident.
It's structured: This "time" is organized in a neat, predictable geometric shape within the model's brain.

Essentially, the model is like a person walking through a tunnel who knows exactly how far they are from the light, even if no one told them the distance. The researchers found the part of the brain that holds that knowledge and showed they could turn the dial to change how the person feels about the journey.

Technical Summary: Subliminal Clocks: Latent Time Modelling in Diffusion Language Models

Problem Statement

Diffusion Language Models (DLMs) have emerged as a promising alternative to autoregressive models, generating text by progressively denoising masked tokens rather than predicting them sequentially. Unlike standard diffusion models that are explicitly conditioned on a timestep variable $t$ , DLMs typically do not receive an explicit timestep input during inference. This raises a fundamental question regarding their internal mechanics: Do DLMs internally represent the progress of the denoising process, and if so, how is this information encoded and utilized?

While the fraction of unmasked tokens ( $\tau_t$ ) serves as an observable proxy for denoising progress, it remains unclear whether the model's internal representations (residual streams) encode a latent signal corresponding to this progress, and whether manipulating this signal causally affects generation.

Methodology

The authors investigate this phenomenon using two large-scale masked diffusion models: LLaDA-1.5 and Dream. Their approach is structured around three research questions (RQs) addressed through probing, causal steering, and geometric analysis.

1. Probing for Latent Time (RQ1)

To determine if denoising progress is encoded, the authors train MLP probes on the residual stream hidden states of the models.

Input: Hidden states $h_{t,l,n}^j$ from specific layers $l$ and token positions $j$ . Here, $t$ denotes the current denoising step in the generation process, and $n$ represents the specific sample index within the batch or dataset used for training the probe.
Target: The empirical denoising progress $\tau_t$ , defined as the fraction of unmasked tokens at step $t$ .
Setup: Probes are trained separately for each layer and for different token subsets ([MASK], non-[MASK], and all tokens) to test the localization of the signal.

2. Causal Steering (RQ2)

To test the causal importance of the identified signal, the authors employ mean activation vector steering.

Signal Extraction: They compute mean activation vectors $\mu_{t,l}$ for each denoising step $t$ and layer $l$ by averaging hidden states across examples.
Intervention: During inference, they replace the current layer's activation with a perturbed version: $\tilde{h}_{t,l}^j = h_{t,l}^j - \mu_{t,l} + \mu_{\hat{t},l}$ , where $\hat{t}$ is a target denoising step. This shifts the model's internal state toward the representation of a different progress level.
Control: They compare these targeted perturbations against random perturbations sampled from the empirical covariance matrix with matched norms.
Metrics: They measure downstream effects on entropy, confidence (max probability), and KL-divergence between the original and steered token distributions.

3. Geometric Characterization (RQ3)

The authors analyze the structure of the recovered signal:

Dimensionality: They perform Principal Component Analysis (PCA) on the mean vectors to determine the effective dimensionality of the $\tau$ representation.
Cross-Layer Alignment: They compute cosine similarities between mean vectors of the same denoising step across different layers to assess if a unified, model-level representation exists.
Intra-Layer Components: They examine how the signal is represented within specific layer components (Self-Attention, MLP gate, MLP up-projection, etc.).

Key Results

1. Existence of a Latent Denoising Signal

Probe Accuracy: MLP probes can reliably predict $\tau_t$ from residual stream activations across all layers, achieving $R^2 > 0.5$ in most layers. The signal is present in both [MASK] and non-[MASK] tokens, though slightly stronger in [MASK] tokens.
Conclusion: DLMs internally encode a signal related to the current denoising step, even without explicit timestep conditioning.

2. Causal Impact on Generation

Predictable Modulation: Steering the model toward a higher $\hat{t}$ (more progress) increases model confidence and decreases entropy. Conversely, steering toward a lower $\hat{t}$ decreases confidence and increases entropy.
KL-Divergence: The divergence between clean and steered distributions does not scale linearly with the distance $|\hat{t} - t|$ . Instead, the relationship is non-linear, reflecting the complex interaction between the latent time signal and the token distribution.
Specificity: These effects are specific to the $\tau$ direction; random perturbations of the same magnitude do not produce coherent trends in entropy or confidence.
Depth Correction: When perturbations are injected in early layers, the model progressively compensates for them as computation proceeds through deeper layers, restoring the signal to its clean value. However, extreme steering (e.g., $\hat{t}=100$ ) is harder to correct and can lead to generation collapse (premature EOS tokens).

3. Structured Low-Dimensional Geometry

Low Dimensionality: The variance of the mean vectors is captured by a very small number of principal components (often $<3$ ), with the first component explaining over 90% of the variance.
Manifold Structure: The trajectory of mean vectors across denoising steps forms a structured, low-dimensional manifold resembling a parabola. This geometry is consistent across layers in LLaDA and organized into specific blocks in Dream.
Cross-Layer Consistency: In LLaDA, most layers maintain a highly correlated representation of the same $\tau$ step, suggesting a shared model-level basis. In Dream, the representation is organized into distinct computational blocks.
Component Anticorrelation: Within a layer, the self-attention output and the MLP output (specifically the post-MLP and post-attention vectors) exhibit strong anticorrelation. They share a common direction but point in opposite ways, suggesting a mechanism where the model may "cancel out" or refine the signal during processing.

Significance and Claims

The paper claims to provide the first systematic evidence that Diffusion Language Models internally model a latent "clock" representing denoising progress, despite lacking explicit timestep inputs.

Mechanistic Insight: The work reveals that the fraction of unmasked tokens is not just an external metric but is actively tracked and represented within the model's residual stream.
Controllability: The identified signal is causally relevant; steering it allows for predictable modulation of model confidence and entropy, demonstrating that the model's "notion" of progress is a manipulable subspace.
Structural Regularity: The discovery of a low-dimensional, parabolic manifold shared across layers suggests that DLMs process denoising time in a highly structured, geometric manner, distinct from the linear representations often found in autoregressive models for other concepts.
Correction Mechanism: The observation that early perturbations are corrected by deeper layers implies a robust internal mechanism for maintaining the integrity of the denoising trajectory.

The authors conclude that while the signal is functionally relevant and structurally organized, the specific circuitry that computes or updates this representation remains an open question for future work. They caution that the signal is entangled with token distributions (e.g., influencing EOS probability) rather than being a purely content-agnostic counter.