Weber's Law in Transformer Magnitude Representations: Efficient Coding, Representational Geometry, and Psychophysical Laws in Language Models

1. Problem Statement

Recent research on how Transformer language models (LLMs) represent numerical, temporal, and spatial magnitudes has yielded contradictory findings. Some studies suggest logarithmic spacing (AlQuabeh et al., 2025), others linear encoding (Zhu et al., 2025), and others per-digit circular representations (Levy & Geva, 2025).

The authors argue that these disagreements stem from methodological limitations rather than fundamental differences in the models. Existing methods often focus on local recovery (e.g., linear probes) or global structure without a unified theoretical framework. The paper proposes applying the formal tools of psychophysics—specifically Weber's Law and Efficient Coding theory—to resolve these contradictions.

Core Research Questions:

• Is the internal geometry of magnitude representations logarithmic, linear, or something else?
• Does this geometry translate into ratio-dependent behavioral discrimination (Weber's Law)?
• Is the representational structure causally relevant to magnitude comparison tasks?
• Do data statistics alone (without biological constraints) suffice to produce log-compressive geometry?

2. Methodology

The study employs a four-converging-paradigm framework across three magnitude domains (numerical, temporal, spatial) and three instruction-tuned models (Llama-3-8B, Mistral-7B, and post-registration Qwen-2.5-7B) spanning different architecture families.

A. Experimental Paradigms

1. Representational Geometry (RSA):
   • Method: Representational Similarity Analysis (RSA) comparing the model's Representational Dissimilarity Matrix (RDM) against theoretical models (Linear, Weber/Logarithmic, Stevens/Power Law).
   • Metrics: Cosine and Euclidean distances in hidden states.
   • Significance: Assessed via Mantel permutation tests to account for non-independence of pairwise distances.
2. Behavioral Discrimination:
   • Task: Forced-choice magnitude comparison (e.g., "Which is larger: 3 dozen or 28?").
   • Metrics: Accuracy, Psychometric functions, and Weber Fraction (WF) estimation (the ratio $\Delta S/S$ at threshold).
3. Precision Gradients:
   • Method: Measuring local representational density ($1/||h(n+1) - h(n)||$) to test if precision decreases as magnitude increases (a prediction of efficient coding).
4. Causal Intervention:
   • Method: Activation patching along a learned "magnitude direction" vector to test if modifying these representations causally shifts comparison probabilities ($\Delta p$).

B. Models and Controls

• Models: Llama-3-8B-Instruct, Mistral-7B-Instruct-v0.3, and Qwen-2.5-7B-Instruct.
• Corpus Analysis: Analyzed 500k OpenWebText documents to verify the input distribution follows a power law (precondition for efficient coding).
• Robustness Controls: Included digit-boundary diagnostics, tokenization controls, unit-boundary checks, and frequency-matched noun controls to rule out confounds like string edit distance or token frequency.

3. Key Contributions

1. Psychophysics-Grounded Framework: First application of the full psychophysical toolkit (Weber fractions, Stevens exponents, RSA with Mantel tests, causal patching) to LLM magnitude representations.
2. Resolution of Conflicting Literature: Demonstrates that logarithmic geometry is the dominant structure, reconciling previous "linear" findings as artifacts of local probe linearity on a smooth, globally logarithmic manifold.
3. Three Critical Dissociations:
   • Geometry vs. Competence: Models can possess perfect logarithmic geometry but fail behavioral discrimination tasks.
   • Model-Specific Readout: Different models with identical geometry exhibit different behavioral capabilities.
   • Layer Inversion: The layers with the strongest geometric structure are not the layers causally responsible for comparison tasks.

4. Key Results

A. Representational Geometry is Consistently Logarithmic (H1 Supported)

• Finding: Across all 96 model–domain–layer cells tested, the representational geometry was log-compressive.
• Evidence: Weber/Logarithmic models significantly outperformed Linear models in RSA correlations ($\rho = .68$ to $.96$) and AIC scores. Linear geometry was never preferred.
• Stevens Exponent ($\beta$): Converged to $\approx 0.01$ (indicating near-pure logarithmic compression), which is even more compressed than human biological systems ($\beta \approx 0.33 - 3.5$).
• Corpus Analysis: Confirmed the efficient coding precondition; integer frequencies in training data follow a power law ($\alpha = 0.77$), close to the $1/s$ prior required for logarithmic encoding.

B. Geometry Does Not Guarantee Behavioral Competence (H2 Mixed)

• Numerical Domain:
  • Llama: Showed ratio-dependent discrimination with a Weber Fraction (WF) of 0.20, falling within the human range (0.10–0.25).
  • Mistral: Failed to show ratio dependence (WF = 0.53); performance was driven by absolute difference.
  • Qwen: Showed strong ratio dependence (WF = 0.179) despite near-exact processing confidence.
• Temporal/Spatial Domains: Both models performed at chance on discrimination tasks despite possessing strong logarithmic geometry for these domains.
• Cause: The "Unit-Boundary" control revealed that models fail to abstract across mixed units (e.g., "minutes" vs. "hours"), treating them as distinct semantic tokens rather than unified magnitudes.

C. Causal Layer Dissociation (E5)

• Finding: A counter-intuitive inversion was observed.
  • Early Layers: Where geometric structure is weakest, activation patching had a high causal impact (4.1× specificity) on comparison tasks.
  • Late Layers: Where geometric structure is strongest (peak RSA), patching had negligible causal impact (1.2× specificity).
• Interpretation: Early layers perform the operational computation for comparison, while late layers encode magnitude as a structured summary for next-token prediction but are not causally engaged in the comparison mechanism.

D. Instruction Tuning as the Switch

• Base vs. Instruct: The base Llama model possessed strong logarithmic geometry (similar to the instruct version) but zero behavioral competence (50% accuracy).
• Conclusion: Pretraining establishes the representational geometry; instruction tuning establishes the functional readout mechanism required for behavioral tasks.

5. Significance and Implications

• Theoretical: The study supports the Efficient Coding Hypothesis in artificial systems. It suggests that data statistics alone (specifically power-law distributions) are sufficient to induce log-compressive geometry in Transformers, even without biological constraints like metabolic costs or firing-rate limits.
• Interpretability: It challenges the assumption that strong representational structure implies functional utility. A model can "know" a concept geometrically without being able to "use" it behaviorally.
• Methodological: It establishes psychophysics as a rigorous framework for mechanistic interpretability, moving beyond simple probing to test functional forms, causal relevance, and behavioral thresholds.
• Limitations: The causal link between input distribution and geometry is correlational (manipulating training distributions is future work). The behavioral null in temporal/spatial domains may be due to unit-conversion issues rather than a lack of magnitude processing.

Conclusion

The paper concludes that Transformer models naturally develop logarithmic magnitude geometry due to the statistical properties of their training data. However, this geometry is dissociated from behavior: behavioral competence depends on specific instruction-tuned readout mechanisms and is not guaranteed by the presence of the geometric structure itself. Furthermore, the functional computation for magnitude comparison occurs in early layers, while the most geometrically refined representations in later layers serve different predictive functions.