💬 NLP

Text-to-Distribution Prediction with Quantile Tokens and Neighbor Context

This paper introduces Quantile Token Regression, a novel approach that inserts dedicated quantile tokens into input sequences and augments them with retrieval-based neighbor context to enable direct input-output pathways and local grounding, thereby significantly outperforming existing baselines in LLM-based distributional regression tasks.

Original authors: Yilun Zhu, Yuan Zhuang, Nikhita Vedula, Dushyanta Dhyani, Shaoyuan Xu, Moyan Li, Mohsen Bayati, Bryan Wang, Shervin Malmasi

Published 2026-04-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yilun Zhu, Yuan Zhuang, Nikhita Vedula, Dushyanta Dhyani, Shaoyuan Xu, Moyan Li, Mohsen Bayati, Bryan Wang, Shervin Malmasi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to guess the price of a house you've never seen before, just by reading its description.

The Old Way (The "Single Guess" Problem):
Most AI models today act like a nervous real estate agent who gives you just one number. "This house is worth $500,000." But what if the market is crazy? What if it could be $400k or $600k? The old AI doesn't tell you that. It just gives a single point estimate, ignoring the uncertainty.

Even newer AI models that try to guess a range of prices often use a "bottleneck" approach. Imagine the AI reads the house description, compresses all that information into a single, tiny mental note, and then tries to guess 99 different prices (from the cheapest to the most expensive) based only on that one tiny note. It's like trying to paint a masterpiece using only one drop of paint. The connection between the house details and the specific price guesses is weak and indirect.

The New Way (The "Quantile Token" Approach):
This paper introduces a smarter way to teach AI how to predict a full range of possibilities (a distribution) instead of just one number. They use two main tricks:

1. The "Specialized Interns" (Quantile Tokens)

Instead of making the AI guess all 99 prices from one tiny mental note, the authors give the AI 99 specialized interns.

Each intern is assigned a specific job: one guesses the 10th percentile (the "cheap" price), one guesses the 50th (the "median" price), and one guesses the 90th (the "expensive" price).
The Magic: These interns can all talk to the house description directly. The "cheap price" intern looks at the description and thinks, "Oh, this house is small and in a bad neighborhood, so I'll guess low." The "expensive price" intern looks at the same description but thinks, "But it has a great view and new appliances, so I'll guess high."
Why it works: They don't have to squeeze everything into one note. Each intern builds their own specific understanding of the house to make their specific guess. This creates a much clearer, more accurate picture of the whole price range.

2. The "Neighborhood Watch" (Retrieval-Augmented Context)

Humans don't guess prices in a vacuum. If you want to know what a house is worth, you look at similar houses nearby.

The authors teach the AI to do the same. Before guessing, the AI searches its database for 8 houses that look very similar to the one you asked about.
The Twist: It doesn't just look at the price of those similar houses. It looks at the entire history of their prices. It sees that similar houses in this neighborhood usually sell between $450k and $550k, with a few outliers going up to $600k.
Why it works: This gives the AI "local evidence." It grounds its prediction in reality. If the AI sees that similar houses have a wide price range, it knows to predict a wide range for your house too. If similar houses are very stable, it predicts a narrow range.

The Result: Sharper, Smarter Predictions

When they tested this on real-world data (like Airbnb listings and Stack Overflow questions about how long it takes to get an answer), the results were impressive:

More Accurate: The AI's guesses were much closer to the truth.
Tighter Ranges: Instead of saying "The price is between $100 and $1,000,000" (which is technically true but useless), the AI said, "The price is likely between $480k and $520k."
Better for Small Data: This method shined especially when there wasn't a lot of data to learn from. It was like a smart student who could ace a test even with a small textbook because they knew how to use their "interns" and "neighbors" effectively.

The "Loss Function" Lesson (The Scorecard)

The paper also figured out the best way to grade the AI.

Some teachers (loss functions) only care if the AI gets the average right, ignoring the extremes.
The authors found a better "scorecard" (called Wasserstein loss) that rewards the AI for getting the whole shape of the distribution right. It's like grading a student not just on their final exam score, but on how well they understood the entire course material, including the tricky parts.

In a Nutshell

This paper teaches AI to stop guessing single numbers and start predicting full stories of possibility. It does this by giving the AI specialized experts for every part of the story and letting them consult a neighborhood of similar examples to make sure their guesses are grounded in reality. The result is a model that understands uncertainty better and gives you much more useful, precise predictions.

1. Problem Definition

The paper addresses Text-to-Distribution Prediction, a task where a Large Language Model (LLM) must predict a full conditional probability distribution $F_{Y|X}$ given an unstructured text input $X$ , rather than a single point estimate.

Supervision: The training data consists of inputs paired with multiple observed outcomes (e.g., multiple price points for a similar Airbnb listing or multiple response times for similar Stack Overflow questions).
Target Representation: The target distribution is represented as a dense grid of empirical quantiles (e.g., the 1st to 99th percentiles).
Key Challenges:
1. Representation Bottleneck: Existing methods (e.g., Vedula et al., 2025) use a shared hidden state for all quantile predictions, creating an indirect path between input features and specific quantile outputs.
2. Lack of Local Grounding: Models often rely solely on the query text, missing the rich distributional context provided by semantically similar instances (e.g., knowing that similar products have similar price distributions, not just similar average prices).
3. Loss Function Mismatch: Standard quantile regression losses (pinball loss) are designed for raw outcomes, not for empirical quantile targets derived from multiple observations, leading to theoretical inconsistencies and bias.

2. Methodology

The authors propose a novel framework combining Quantile Token Regression and Retrieval-Augmented Distribution Estimation.

A. Quantile Token Regression (QTR)

Instead of using a shared representation with separate linear heads, the authors insert dedicated learnable quantile tokens ( $\langle Q_{\tau_1} \rangle, \dots, \langle Q_{\tau_Q} \rangle$ ) directly into the input sequence.

Mechanism: The input sequence becomes $[x_1, \dots, x_n, \langle Q_{\tau_1} \rangle, \dots, \langle Q_{\tau_Q} \rangle]$ .
Self-Attention: Each quantile token attends to different parts of the input text via the Transformer's self-attention mechanism. For example, a token predicting the 90th percentile (tail risk) might attend to complexity indicators, while a token predicting the 10th percentile attends to popularity signals.
Output: A shared linear regressor maps the hidden state of each specific quantile token to its predicted value. This creates a direct input-output pathway for each quantile level.

B. Retrieval-Augmented Distribution Estimation

To ground predictions in local evidence, the model retrieves $K$ semantically similar neighbor instances from a candidate pool.

Augmentation: Unlike prior retrieval methods that attach only a single point label (e.g., a single price) to neighbors, this method attaches the full empirical distribution of the neighbor (represented as a set of quantiles).
Input Format: The model input includes the query text followed by the retrieved neighbors and their associated quantile distributions. This allows the model to learn dispersion and tail behavior by comparing the query to the distributions of similar items.

C. Theoretical Analysis of Loss Functions

The paper provides a theoretical analysis of loss functions for empirical-quantile supervision:

$\ell_1$ and $\ell_2$ Wasserstein Losses: Treating empirical quantiles as noisy measurements of latent population quantiles. Minimizing element-wise $\ell_p$ loss on the quantile grid is shown to be Fisher-consistent for the target distribution.
Pinball Loss Variants:
- Pinball-Q: Applying standard pinball loss to empirical quantile targets is shown to be biased, targeting the quantile of the estimator rather than the true parameter.
- Pinball-Med: Using only the median as a scalar target discards distributional shape information.
Conclusion: $\ell_1$ Wasserstein loss is theoretically and empirically superior, offering the best trade-off between accuracy and sharpness.

3. Key Contributions

Quantile Token Architecture: The first work to insert dedicated quantile tokens into the input sequence, enabling direct, quantile-specific attention pathways and improving interpretability.
Distributional Retrieval: Extending retrieval-augmented generation (RAG) to distributional prediction by retrieving full empirical distributions (quantiles) rather than single point labels.
Theoretical Clarification: A rigorous analysis proving that Wasserstein matching ( $\ell_1/\ell_2$ on quantiles) is the correct objective for empirical-quantile supervision, correcting the misuse of pinball loss in this specific setting.
Empirical Validation: Comprehensive experiments across diverse datasets and model scales demonstrating that the combination of QTR and neighbor retrieval significantly outperforms baselines.

4. Experimental Results

The method was evaluated on two datasets: Inside Airbnb (price prediction, ~840k samples) and StackSample (response time prediction, ~58k samples), using Qwen3 models ranging from 1.7B to 14B parameters.

Performance Gains:
- Accuracy: The proposed method (Quantile Tokens + Neighbors) consistently reduced the average Mean Absolute Percentage Error (MAPE). On StackSample, it reduced MAPE by 68% compared to the baseline without retrieval and 14% compared to retrieval-only baselines.
- Sharpness (Interval Width): The method produced significantly narrower prediction intervals. On StackSample, the 99% Relative Coverage Interval Width (RCIW) was reduced by 131x compared to the baseline.
- Small Data Regimes: The gains were most pronounced on the smaller, more challenging StackSample dataset, where retrieval provided crucial distributional evidence that the model could not learn from the query text alone.
Ablation Studies:
- Loss Functions: $\ell_1$ Wasserstein loss outperformed Pinball variants, achieving lower MAPE and tighter intervals.
- Neighbor Count: Increasing the number of retrieved neighbors ( $K$ ) improved performance, with diminishing returns observed after $K=8$ to $K=16$ .
- Monotonicity: Post-processing (sorting predicted quantiles) was found to be the most effective method for enforcing monotonicity without sacrificing accuracy or widening intervals excessively.

5. Significance

This work bridges the gap between LLM capabilities and rigorous statistical distributional regression.

Practical Impact: It enables LLMs to provide robust uncertainty estimates (e.g., risk assessment, demand forecasting) rather than just point predictions, which is critical for high-stakes applications.
Architectural Innovation: The "Quantile Token" concept offers a new paradigm for structured prediction in Transformers, moving away from shared bottlenecks toward specialized, attention-driven representations.
Methodological Rigor: By correcting the loss function mismatch in empirical-quantile supervision, the paper provides a theoretically sound foundation for future distributional learning tasks using LLMs.

In summary, the paper demonstrates that combining dedicated quantile tokens with distribution-aware retrieval allows LLMs to learn complex, heterogeneous conditional distributions with high accuracy and sharpness, particularly in data-scarce or high-uncertainty scenarios.