💻 computer science

Task-Dependent Intrinsic Dimensionality Across Transformer Layers: An Exploratory Study of Qwen2.5-1.5B

This exploratory study of Qwen2.5-1.5B reveals that the intrinsic dimensionality of internal representations varies non-uniformly across task categories and transformer layers, with reasoning tasks exhibiting lower dimensionality than factual recall in middle layers but converging toward similar levels in the final layers.

Original author: Mitra Varun Gogulapati

Published 2026-11-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original author: Mitra Varun Gogulapati

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the author. For technical accuracy, refer to the original paper. Read full disclaimer

Inside the digital minds of modern artificial intelligence, there exists a vast, invisible landscape where information is stored and transformed. These systems, known as transformers, process language by converting words into long lists of numbers, creating a high-dimensional space that can have thousands of coordinates. Yet, despite this massive capacity, the actual information needed to solve a specific problem often lives in a much smaller, more focused region of that space. Scientists call this the intrinsic dimension, a measure of how many directions are truly necessary to describe the data at any given moment. Understanding whether this internal space expands or contracts depending on the type of task—such as recalling a fact versus solving a logic puzzle—offers a window into how these machines think. If the geometry of these internal representations changes based on the job at hand, it suggests that the model does not treat all knowledge as a single, uniform block, but rather reshapes its internal world to fit the problem.

A recent study set out to map these shifting landscapes within a specific, smaller artificial intelligence model called Qwen2.5-1.5B. The researchers were not trying to teach the model anything new or check if it could answer questions correctly. Instead, they acted as cartographers, observing how the model's internal state changed as it processed different types of prompts. They gathered 200 distinct requests, divided into four categories: factual recall (specifically about chemistry), arithmetic calculations, logical reasoning, and predicting the output of short computer code. For each request, they captured the model's internal "thought" at every single step of its processing, from the very first layer where the input is received to the final layer where the answer is prepared. This allowed them to trace the shape of the model's thinking as it moved deeper into the network.

To measure the complexity of these internal states, the researchers used several tools that act like rulers for high-dimensional space. They looked at how the energy of the data was distributed across the many coordinates. If the data is spread out evenly, the space is considered high-dimensional; if the data is squashed into just a few directions, the space is low-dimensional. They applied these measurements to the model's responses for each of the four task types. The results revealed a surprising and dynamic pattern. In the early and middle stages of processing, the internal representations for logical reasoning tasks occupied a slightly larger, more complex space than those for simple factual recall. However, as the model moved into its final layers, this relationship flipped. In the deepest parts of the network, the representations for factual recall became significantly more complex and spread out, while the space for reasoning tasks actually shrank and became more concentrated.

This reversal was the most striking discovery of the work. At one specific point near the end of the processing chain, the measure of complexity for factual recall was more than double that of the reasoning tasks. This suggests that the model does not simply get "smarter" or more complex as it goes deeper; rather, it reorganizes its internal geometry in a way that depends entirely on what it is trying to do. For the chemistry facts, the model seems to spread its attention across many directions in the final stages, whereas for the logic puzzles, it tightens its focus into a narrower, more efficient path. The researchers noted that this was a descriptive observation of a specific set of prompts and did not prove that one task is inherently harder or better than the other. It simply showed that the shape of the model's thinking changes as it works.

The study also highlighted the limitations of trying to draw broad conclusions from such a small sample. The researchers used only a dozen prompts for each category in their main analysis, which is a very small number for making definitive claims about how the model works in general. They found that the length of the prompts and the specific way the questions were written could influence the results, and they were unable to perfectly match these factors across all categories. Because of this, they could not say with certainty that the observed differences were caused solely by the type of task, rather than by the structure of the questions themselves. Furthermore, the study did not test whether these geometric changes actually helped the model get the right answers or if they were just a side effect of how the model was built.

Ultimately, this work provides a detailed snapshot of how a language model's internal world is not static. It is a fluid environment that stretches and compresses depending on the task. The finding that reasoning and factual recall follow different paths through the final layers of the network challenges the idea that these models process all information in the same way. While the study does not offer a complete map of the territory, it establishes that the depth of the network matters and that the relationship between different types of tasks is not fixed. The researchers have made their data and methods available so others can verify these observations, acknowledging that much more work is needed to understand the full scope of how these digital minds organize their knowledge.

Technical Summary: Task-Dependent Intrinsic Dimensionality Across Transformer Layers

Problem Statement
This study investigates whether different task categories (factual recall, arithmetic, reasoning, and coding) exhibit distinct intrinsic dimensionality profiles within the internal representations of a fixed pretrained language model. While prior research has established that transformer representations possess depth-dependent geometric structures and that ambient width does not equate to effective dimensionality, this work narrows the focus to a specific question: within a single model (Qwen2.5-1.5B), do prompt sets assigned to different tasks organize their variance differently across layers? The author explicitly distinguishes this descriptive analysis from causal claims regarding model training or reasoning mechanisms.

Methodology

Model and Data: The experiment utilizes the pretrained base model Qwen2.5-1.5B (28 transformer blocks, hidden width 1,536). The dataset consists of 200 prompts (50 per category) generated from eight named template families per task. Tasks are operationalized as:
- Factual Recall: Chemistry domain (symbols, atomic numbers).
- Arithmetic: Explicit calculations with fixed answers.
- Reasoning: Boolean or relational problems.
- Coding: Python expression output prediction.
Partitioning: The data is split into an exploratory set (families 0–5) and a held-out evaluation set (families 6–7). The primary analysis relies on the held-out set, comprising 12 prompts per category (2 families × 4 categories).
Extraction: Representations are extracted from the last input token at all 29 output indices (embedding + 28 transformer blocks, including final RMSNorm). Only input prompts are processed; no generated continuations are included.
Dimensionality Estimators: Four distinct metrics are computed on centered covariance matrices of the hidden states:
1. Participation Ratio (PR): Measures spectral concentration of variance.
2. Entropy Effective Rank (DER): An entropy-based measure of the covariance spectrum.
3. PCA Component Thresholds ( $D_q$ ): The number of components required to explain a target variance ( $q \in \{0.80, 0.90, 0.95, 0.99\}$ ).
4. TWO-NN: A local intrinsic dimension estimator based on nearest-neighbor distance ratios.
Statistical Constraints: Due to the small sample size of the held-out set ( $N=12$ ) and the limited number of families (2 per category), the author explicitly withholds primary family-bootstrap confidence intervals and p-values. The analysis is strictly descriptive, acknowledging that the finite sample rank ( $N-1 = 11$ ) bounds the spectral estimates.

Key Results

Depth-Dependent Contrast: The most significant finding is a non-uniform relationship between task categories across layers. Specifically, the contrast between reasoning and factual recall (Reasoning PR minus Recall PR) is positive in early and middle blocks (1–20) but becomes negative in later blocks (21–28).
- At Block 6, the contrast peaks at +0.384.
- At Block 24, the contrast reverses sharply to -1.472, driven by a large increase in the participation ratio for factual recall (2.869) compared to reasoning (1.397).
- At the final output (Block 28 + Norm), the gap narrows but remains negative (-0.110).
Final Output Profiles: At the final layer, factual recall and reasoning both require 5 components to reach 90% variance, whereas arithmetic and coding require only 2. However, the ordering of tasks differs across estimators (e.g., TWO-NN values do not align with PR rankings), highlighting that local neighbor geometry and global spectral concentration capture different aspects of the representation space.
Confounding Factors: The study notes that token lengths vary significantly by category (median 16 for recall vs. 31 for coding). Due to a lack of shared token-length bins across all four categories, a matched-length control was not possible, leaving the observed contrasts potentially confounded by length and template structure.

Limitations and Scope
The author maintains a modest stance regarding the generalizability of these findings:

Sample Size: The held-out analysis is limited to 12 prompts per category, preventing reliable primary bootstrap inference.
Model Specificity: Results are specific to Qwen2.5-1.5B and do not necessarily generalize to other architectures or sizes.
Descriptive Nature: The paper does not claim that higher dimensionality implies better reasoning or accuracy. No causal mechanisms are identified.
Data Constraints: The dataset covers narrow domains (chemistry, specific Python snippets) and does not represent broad factual knowledge or software development.

Significance and Contributions
The paper's primary contribution is a reproducible analysis workflow and a descriptive characterization of how task-category representation clouds differ in dimensionality across transformer depth. It challenges the notion of a uniform task ordering (e.g., that reasoning is always higher-dimensional than recall) by demonstrating that the relationship is depth-dependent. The study serves as an exploratory benchmark, providing analysis code and numerical artifacts while explicitly delineating the boundaries of what can be inferred from small-scale, non-causal experiments. It emphasizes the importance of reporting controls that the dataset cannot support (such as length matching) to ensure transparency in geometric analyses of LLMs.

Technical Summary: Task-Dependent Intrinsic Dimensionality Across Transformer Layers

More like this