📊 statistics

A mathematical framework for parameter recovery in large language models via a joint Euclidean mirror

This paper proposes a mathematical framework that maps large language model response distributions to a low-dimensional Euclidean space via a "joint Euclidean mirror," enabling the asymptotically consistent estimation and recovery of underlying tuning parameters from observed model outputs.

Original authors: Maximilian Baum, Aranyak Acharyya, Tianyi Chen, Avanti Athreya, Youngser Park, Francesco Sanna Passino, Carey E. Priebe, Zachary Lubberts

Published 2026-04-09

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Maximilian Baum, Aranyak Acharyya, Tianyi Chen, Avanti Athreya, Youngser Park, Francesco Sanna Passino, Carey E. Priebe, Zachary Lubberts

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a "black box" machine that writes stories, answers questions, or creates art. This is a Large Language Model (LLM). The problem is, we don't know exactly how the machine works inside, and we can't see the knobs and dials (the parameters) that the engineers used to tune it.

For example, there's a "temperature" knob that controls how creative or random the answers are. There's also a "bias" knob that steers the model toward certain topics. If you see a story written by this machine, can you figure out exactly which settings were used to write it? Usually, the answer is "no."

This paper introduces a clever mathematical trick to solve that problem. Here is the explanation using simple analogies:

1. The "Fingerprint" of a Response

Instead of looking at the text itself (which can be long and messy), the authors treat every answer the AI gives as a cloud of possibilities.

The Analogy: Imagine the AI is a dartboard. If you set the "temperature" to low, the darts (answers) cluster tightly in the center. If you set it to high, the darts scatter all over the board.
The paper says: "Let's stop looking at the individual darts and look at the shape of the whole cloud."

2. The "Mirror" Concept

The core idea is something called a Joint Euclidean Mirror.

The Analogy: Imagine you are in a dark room with a complex, 3D sculpture (the AI's behavior). You can't see the sculpture, but you can see its shadow on a flat wall.
The "Mirror" is a special mathematical surface that acts like a shadow-casting device. It takes the complex, high-dimensional "clouds" of AI answers and flattens them onto a simple, low-dimensional map (like a 2D piece of paper).
Crucially, this map preserves the distance. If two sets of answers are very different (like a serious news report vs. a silly joke), they appear far apart on the map. If they are similar, they appear close together.

3. Mapping the Knobs to the Map

The researchers discovered that this "shadow map" isn't random. It has a structure that matches the knobs on the machine.

The Analogy: Imagine the map is a topographical map of a mountain.
- Moving the "Temperature" knob moves you North-South on the map.
- Moving the "Bias" knob moves you East-West.
Even though the AI is a complex black box, its behavior follows a smooth, predictable path on this map.

4. The "Reverse Engineering" Trick (Parameter Recovery)

This is the most exciting part. Usually, we know the settings and ask, "What will the AI say?"

The Paper's Innovation: They ask the reverse: "We have a story the AI wrote, but we don't know the settings. Can we find the settings?"
How it works:
1. Take the new story and turn it into a "cloud" of data.
2. Project that cloud onto the Mirror Map.
3. Look at where it lands on the map.
4. Since we know that "North" means "High Temperature" and "East" means "High Bias," we can read the coordinates and guess the exact settings used to create that story.

5. Why This Matters

Think of it like a forensic tool for AI.

Detecting Secrets: If a model suddenly starts acting strangely, we can check its "shadow" on the map to see if someone secretly changed the temperature or fed it sensitive data.
Predicting Behavior: If we want to know what the AI will do with a setting we've never tried, we can look at the smooth curve of the map and predict the outcome without actually running the expensive computer simulation.
Comparing Models: It gives us a standardized way to compare different AI models, not by their code (which is often secret), but by how their "shadows" look on the map.

Summary

The paper builds a mathematical GPS for Large Language Models. It translates the chaotic, high-dimensional behavior of AI responses into a simple, smooth map. Once you have the map, you can navigate the AI's behavior, predict what it will do, and even reverse-engineer the hidden settings used to create a specific output. It turns a "black box" into a transparent, navigable landscape.

1. Problem Statement

Large Language Models (LLMs) are often treated as "black boxes" where internal parameters (e.g., temperature, logit bias, training data access) are unobserved or inaccessible. While comparing models based on their weights is often impossible due to proprietary constraints, comparing them based on their output responses is feasible.

The core problem addressed is:

Understanding Sensitivity: How do changes in tuning parameters affect the distribution of LLM responses?
Parameter Recovery: Given a set of unlabeled LLM responses, can we infer the specific tuning parameters (e.g., temperature, prompt weighting) used to generate them?
Geometric Representation: How can we map the complex, high-dimensional space of response distributions into a low-dimensional Euclidean space that preserves the geometric relationships (distances) between them?

2. Methodology

The authors propose a framework treating LLM responses as realizations from high-dimensional probability distributions indexed by a parameter vector $x$ . They introduce the concept of a Joint Euclidean Mirror to map these distributions into a low-dimensional Euclidean space.

A. Theoretical Framework

Space of Distributions ( $\mathcal{F}$ ): Let $\mathcal{F}$ be a family of distributions parameterized by $x \in \mathcal{X} \subset \mathbb{R}^d$ . For an LLM, $F_x$ represents the distribution of responses to a specific query given parameters $x$ .
Dissimilarity Metric ( $D$ ): A distance metric (e.g., Wasserstein distance) is defined on $\mathcal{F}$ to measure the difference between two response distributions.
Exact Euclidean Realizability: The pair $(\mathcal{F}, D)$ is exactly Euclidean $c$ -realizable if there exists a continuous function $f: \mathcal{X} \to \mathbb{R}^c$ such that the Euclidean distance between mapped points equals the distributional distance:
$\|f(x) - f(x')\| = D(F_x, F_{x'})$
This function $f$ is termed the Joint Euclidean Mirror.

B. Estimation Procedure (Algorithm 1)

Since the true distributions are unknown, the method estimates the mirror using observed samples:

Distance Matrix Estimation: For $m$ known parameter settings $\{x_1, \dots, x_m\}$ , collect $n$ samples from each. Compute the empirical dissimilarity matrix $\hat{\Delta}$ using the Wasserstein-1 distance (or other metrics like MMD) between the empirical distributions.
Mirror Value Estimation: Apply Classical Multidimensional Scaling (CMDS) to $\hat{\Delta}$ to obtain a matrix $\hat{\Psi} \in \mathbb{R}^{m \times c}$ . The rows of $\hat{\Psi}$ serve as estimates of the mirror values $f(x_i)$ up to a rigid Euclidean transformation (rotation/translation).
Surface Fitting: Construct a continuous estimator $\hat{f}: \mathcal{X} \to \mathbb{R}^c$ by performing multivariate surface fitting (e.g., Delaunay triangulation interpolation or B-splines) on the points $\{(x_i, \hat{\Psi}_i)\}$ .

C. Parameter Recovery (Algorithm 2)

To recover an unknown parameter $x^*$ from unlabeled samples $S_{x^*}$ :

Include the unlabeled samples in the distance matrix calculation alongside the labeled samples.
Perform CMDS on the expanded $(m+1) \times (m+1)$ matrix to get an estimated mirror value $\hat{f}^*$ for the unknown sample.
Solve the optimization problem:
$\hat{x} = \arg\min_{s \in \mathcal{X}} \| \hat{f}(s) - \hat{f}^* \|$
This finds the parameter $s$ whose estimated mirror value is closest to the observed unknown sample's mirror value.

3. Key Contributions

Formalization of Parameter Recovery: This is the first work to formalize the problem of recovering latent generative parameters from LLM responses within a principled mathematical framework using a joint Euclidean mirror.
Generalization of Euclidean Mirrors: Extends the concept of Euclidean mirrors (previously used for dynamic graphs) to a multivariate setting where distances are parameterized by multiple features (e.g., temperature and prompt weight simultaneously).
Statistical Consistency Proofs: The paper provides rigorous asymptotic proofs showing that:
- The estimated distance matrix converges to the true distance matrix as sample size $n$ and number of parameters $m$ increase (Proposition 1).
- The estimated mirror values converge to the true mirror values (Theorem 1).
- The surface-fitted mirror function converges uniformly to the true mirror (Theorem 2).
- The parameter recovery procedure yields consistent estimates of the unknown parameters (Theorem 3), provided the mirror is invertible and the Jacobian is well-behaved.
Metric Flexibility: While the proofs utilize the Wasserstein distance (which captures differences in location, spread, and shape), the framework is agnostic to the specific metric used, allowing for Total Variation, KL-divergence, or MMD.

4. Experimental Results

The authors validate the framework through simulations and real-world LLM experiments.

Simulated Data (Gaussian Distributions):
- Mirror Estimation: Demonstrated that as sample size $n$ increases, the estimated mirror surface converges to the true parabolic shape of the underlying function.
- Parameter Recovery: A leave-one-out analysis showed that as $n$ increases, the recovery error approaches zero, confirming the theoretical consistency.
LLM Application (ChatGPT):
- Setup: Generated responses using varying temperature ( $t \in [0.1, 0.9]$ ) and prompt weight ( $w \in [10\%, 90\%]$ ) on a prompt about R.A. Fisher.
- Embedding: Responses were embedded using NomicEmbedv1.5.
- Latent Structure: The eigenvalue spectrum of the distance matrix suggested a low-dimensional structure ( $c=5$ ).
- Visualization: The estimated 2D mirror showed that temperature and weight variations shifted the response distributions along distinct, roughly orthogonal axes. This indicates that these parameters control independent aspects of the output distribution.
- Recovery: The parameter recovery algorithm successfully estimated the temperature and weight of unseen responses with a robust linear relationship to the ground truth, despite some bias and variance.

5. Significance and Impact

Interpretability: Provides a tractable way to visualize and understand how specific tuning parameters influence LLM behavior, moving beyond black-box comparisons.
Security and Forensics: Offers a method to infer hidden attributes of a model's generation process. Potential applications include:
- Detecting if a model was trained on sensitive or proprietary data.
- Identifying the specific "tuning" or "persona" used to generate a text.
- Isolating key features of prompts that drive specific model behaviors.
Predictive Modeling: Enables the prediction of response distributions for new, unobserved parameter values by interpolating on the learned mirror surface.
Theoretical Rigor: Establishes a solid statistical foundation for embedding probability distributions in a way that preserves their geometric structure, bridging the gap between abstract distribution theory and practical LLM engineering.

In summary, this paper presents a mathematically grounded, statistically consistent method to "reverse engineer" the tuning parameters of Large Language Models by analyzing the geometry of their output distributions.