💬 NLP

Hierarchical Latent Prediction for Language Models

This paper introduces Hierarchical Latent Prediction (HiLP), a novel auxiliary objective that leverages higher-level abstract latents to mitigate error accumulation in latent-space rollouts, thereby improving long-horizon reasoning, coding performance, and speculative decoding efficiency in language models.

Original authors: Chang Shi, Tim Pearce, Manan Tomar, Siddhartha Sen, John Langford

Published 2026-08-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Chang Shi, Tim Pearce, Manan Tomar, Siddhartha Sen, John Langford

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to write a story. The standard way to do this is called "Next-Token Prediction." It's like a game of "Mad Libs" where the robot looks at the last word you wrote and guesses the very next one. If you say "The cat sat on the," the robot guesses "mat." Then, you feed it "The cat sat on the mat," and it guesses "soft." This method has built some incredibly smart robots, but it has a flaw. Because the robot only looks one step ahead, it can get lost in long stories. If it makes a tiny mistake early on, that mistake gets bigger and bigger as it keeps guessing, like a snowball rolling down a hill. This is called "exposure bias."

To fix this, scientists have tried teaching the robot to look a few steps ahead or to guess the "vibe" of the next few words instead of just the next letter. But these methods often have their own problems: they either can't see far enough into the future, or they get confused by their own mistakes when they try to plan too far ahead. The big question is: Can we teach the robot to understand the story's structure on multiple levels at once—like knowing the plot of a chapter while also knowing the next sentence—without making the robot too slow or complicated to use?

This is exactly what the researchers at Microsoft Research and the University of Texas at Austin tackled in their new paper, "Hierarchical Latent Prediction for Language Models." They introduced a clever new training method called HiLP (Hierarchical Latent Prediction). Think of the robot's brain as having two layers of thinking. The first layer is the "street-level" view, which focuses on the immediate next word, just like the old method. The second layer is the "helicopter view," which looks at a chunk of words at once to understand the bigger picture, like a paragraph or a scene.

In their experiments, the researchers taught the robot to use this "helicopter view" as a secret training tool. They created a special "abstract" layer that summarizes the last few words into a single, high-level concept. Then, they asked the robot to predict where that high-level concept would be a few steps in the future. This forced the robot to learn how the story should flow over a longer distance, not just from word to word. Crucially, this "helicopter view" is only used during training. Once the robot is finished learning, the researchers throw away the extra layers. When the robot actually writes a story for you, it only uses the simple, fast "street-level" view. This means the robot gets smarter at long-term planning without becoming slower or heavier to run.

The results suggest that this approach works well. When they tested the new robot on coding tasks and multi-step reasoning puzzles, it performed better than previous methods that tried to look ahead. For example, on a coding benchmark called HumanEval, the new method achieved a score of 11.33, beating the standard method (which scored 8.77) and other recent attempts like NextLat (which scored 10.58). The researchers also found that this method made "speculative decoding" more efficient. This is a technique where the robot guesses several words at once to speed things up; with HiLP, the robot's guesses were more accurate, meaning it wasted less time correcting itself.

The paper argues that while looking just one step ahead is good for local details, it fails to capture the long-term structure of language. By adding this hierarchical "helicopter view" during training, the robot learns to keep its long-term goals in mind, reducing the "snowball effect" of errors. The authors suggest that this method offers a way to get the benefits of long-horizon planning without the usual trade-offs of speed or complexity, though they note that the specific "lookahead" distance they used was a manual setting, and future work could make this more flexible. Ultimately, HiLP shows that teaching a language model to see the forest and the trees during practice can help it write better stories, even if it only uses the trees when it's time to perform.

Technical Summary: Hierarchical Latent Prediction for Language Models

Problem Statement

Standard Next-Token Prediction (NTP) with teacher-forced training has enabled the development of highly capable transformer language models. However, this paradigm introduces a training-inference mismatch known as exposure bias. During inference, auto-regressive generation relies on the model's own previous outputs rather than ground truth, leading to compounding errors that degrade long-range generation quality.

Recent attempts to mitigate this, such as Multi-Token Prediction (MTP) and Next-Latent Prediction (NextLat), introduce auxiliary objectives to predict future tokens or latent states. However, these approaches face limitations:

MTP often suffers from compounding errors during multi-step rollouts.
NextLat enforces self-prediction in a latent space but typically predicts only one step ahead. While effective for local transition consistency, it provides only indirect gradient pressure for capturing structure at longer temporal scales, leaving long-horizon dependencies attenuated by sequential dynamics unrolling.

The core challenge is how to shape the representation of a language model to encode multi-scale predictive structures during pretraining without incurring architectural overhead or inference latency.

Methodology: Hierarchical Latent Prediction (HiLP)

The authors propose Hierarchical Latent Prediction (HiLP), a training method that introduces temporal hierarchy into the latent-space self-prediction objective. The architecture consists of four jointly learned parameterized maps: an output head ( $p_\theta$ ), lower-level dynamics ( $p_\psi$ ), upper-level dynamics ( $p_\phi$ ), and a combined NTP head ( $p_\rho$ ).

Architecture Components

Lower-Level Latents ( $h_t$ ): A backbone transformer $G_\theta$ produces lower-level latents $h_t$ from the token prefix $X_{1:t}$ .
Higher-Level Abstract Latents ( $u_t$ ): A sliding-window attention (SWA) module with window size $k$ aggregates the last $k$ lower-level states into a higher-level abstract latent: $u_t = \text{SWA}(h_{t-k+1:t})$ .
Higher-Level Dynamics: A model $p_\phi$ predicts $k$ steps ahead in the abstract space ( $\hat{u}_{t+k}$ ) without token input, encouraging lookahead planning beyond single-step transitions.
Combined NTP Head: The next-token prediction is conditioned on both the lower-level latent $h_t$ and the higher-level latent $\tilde{u}_t$ (where $\tilde{u}_t$ includes a stop-gradient to prevent the combined head from updating the lower latents through the SWA window).

Training Objectives

The total loss is a weighted sum of five terms:

Standard NTP Loss ( $L_{ntp}$ ): Standard cross-entropy for next-token prediction.
Lower-Level Transition Consistency ( $L_h$ ): SmoothL1 loss ensuring the lower dynamics model $p_\psi$ accurately predicts the next lower-level latent state.
KL Divergence ( $L_{KL}$ ): Ensures the distribution of predicted lower latents matches the target distribution.
Higher-Level Transition Consistency ( $L_u$ ): SmoothL1 loss ensuring the higher-level dynamics model $p_\phi$ accurately predicts the abstract latent $k$ steps ahead.
Combined NTP Loss ( $L_{cntp}$ ): Cross-entropy for the combined head, leveraging both $h_t$ and $\tilde{u}_t$ .

Inference Strategy: A critical design feature of HiLP is that the entire hierarchical apparatus (SWA module, dynamics models, and combined head) is discarded at inference time. Only the standard NTP head is used, ensuring zero architectural overhead and no increase in inference latency.

Key Contributions

Hierarchical Representation Training: The introduction of HiLP, which integrates multi-scale self-predictive learning into transformer pretraining via sliding-window attention over latents, a higher-level dynamics model, and a combined NTP head.
Evaluation: Comprehensive benchmarking of the resulting models on downstream accuracy (coding and multi-step reasoning) and speculative decoding efficiency, comparing HiLP against MTP and NextLat.
Pure Training-Time Intervention: Demonstration that the complex hierarchical structure serves purely as an auxiliary training signal, allowing the model to retain the efficiency of standard autoregressive decoding during deployment.

Experimental Results

The authors trained 1B-parameter models on 100B tokens using 8× NVIDIA B200 GPUs.

Benchmark Performance

Coding (HumanEval): HiLP achieved a pass@1 score of 11.33, outperforming standard NTP (8.77), MTP (9.21), and NextLat (10.58).
Multi-Step Reasoning: On symbolic and multi-step benchmarks (DataComp for LLMs), HiLP demonstrated improved performance, attributed to the multi-scale latent prediction.

Speculative Decoding

HiLP showed superior efficiency in speculative decoding tasks:

Code Data: HiLP achieved an average of 3.41 accepted tokens per drafting step, compared to 2.57 for MTP and 3.26 for NextLat.
Nemotron-ClimbMix Data: HiLP achieved 2.59 accepted tokens, slightly outperforming NextLat (2.57) and MTP (2.11).
Latent Rollout Stability: Analysis of latent cross-entropy curves revealed that while HiLP maintains near-term prediction loss comparable to NextLat, it significantly reduces error at longer horizons (up to 8 steps), indicating more stable multi-step latent rollouts.

Efficiency

While HiLP adds auxiliary parameters (totaling ~~1.27B vs. 1.06B for the backbone), the inference path remains identical to the baseline. Training throughput for HiLP was approximately 65% of the NTP baseline, which is higher than MTP (~~49%) but lower than NextLat (~83%), reflecting the cost of the additional hierarchical loss graph.

Significance and Claims

The paper claims that HiLP successfully addresses the "myopic gap" in next-token prediction by introducing temporal hierarchy directly into the latent-space self-prediction objective. Unlike previous hierarchical models that consume complexity at inference time (e.g., Large Concept Models or Byte Latent Transformers), HiLP shapes the representation during pretraining to encode multi-scale predictive structure, which is then removed entirely for inference.

The authors posit that this approach allows the model to learn richer latent transition dynamics and better capture long-horizon dependencies without sacrificing the speed or simplicity of standard token-level autoregressive decoding. The work positions itself at the intersection of pretraining objectives beyond NTP, latent-space language modeling, and hierarchical models, offering a novel method to improve long-horizon reasoning and planning capabilities in language models.