📊 statistics

Statistical Embeddings for Similarity, Retrieval, and Interpretable Alignment of Numeric Tabular Datasets

This paper proposes a framework that embeds structured statistical descriptors of numeric tabular datasets into a shared vector space using sentence transformers and penalized Canonical Correlation Analysis to enable interpretable cross-dataset similarity retrieval, alignment, and privacy-preserving integration without requiring shared variable definitions.

Original authors: M. Ross Kunz, John Merickel, Keith Wilson

Published 2026-05-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: M. Ross Kunz, John Merickel, Keith Wilson

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a librarian in a massive, chaotic library. But instead of books, this library is filled with thousands of different spreadsheets (tables of numbers) from all over the world. Some spreadsheets track wine quality, others track steel strength, and some track nuclear graphite.

The problem? These spreadsheets speak different "languages." One might list "alcohol content" in percentages, while another lists "sugar" in grams. They have different column names, different sizes, and different units. If you ask a standard computer, "Find me a spreadsheet similar to this wine one," the computer gets confused because it can't match the column names. It's like trying to find a book by asking for a title that doesn't exist in the other books.

This paper proposes a clever new way to organize this library so a computer (specifically, a Large Language Model, or AI) can understand and find similar spreadsheets, even if they look completely different on the surface.

Here is how they did it, broken down into simple steps:

1. The "Fingerprint" Instead of the Raw Data

Instead of trying to feed the raw numbers into the AI (which is messy and hard to compare), the authors first take a "fingerprint" of each spreadsheet.

The Analogy: Imagine you have a bag of marbles. Instead of showing the AI the bag, you write a short story describing the bag: "There are 150 marbles. Most are red. The average size is 2 inches. There are no giant marbles, but a few are very small."
The Method: The computer runs a standard statistical check (called Exploratory Data Analysis) on the numbers. It counts the rows, finds the average, checks for patterns, and sees how the numbers spread out. It turns all these math facts into a list of natural language sentences.

2. Turning Math into Stories

Once the computer has these descriptions, it turns them into sentences.

The Analogy: It's like translating a secret code into English.
- Math: "Kurtosis = 2.4."
- Sentence: "The distribution is flat with light tails, like a normal curve but flatter."
The Magic: Because these are now sentences, the AI (which is very good at understanding language) can read them. The AI turns each sentence into a "vector" (a mathematical point in space). Think of this as placing a dot on a map for every spreadsheet. If two spreadsheets have similar "stories" (similar statistics), their dots end up close together on the map, even if one is about wine and the other is about steel.

3. Finding the Match (The "Similarity" Test)

Now that every spreadsheet has a dot on the map, how do we know which ones are truly similar?

The Analogy: Imagine two groups of people standing in a room. You want to know if Group A and Group B are related. Instead of just looking at one person from each group, you look at the entire group's posture and movement together.
The Method: The authors use a technique called Canonical Correlation Analysis (CCA). It's a sophisticated way of asking: "Do the patterns in Group A's dots line up with the patterns in Group B's dots?" If they do, the spreadsheets are similar.

4. The "X-Ray" for Interpretability

Usually, when AI says "These two are similar," it's a black box—you don't know why. This paper adds a special feature: Penalized CCA.

The Analogy: It's like an X-ray that highlights exactly which bones are matching. Instead of just saying "These two people look alike," it says, "They look alike because they both have the same height and the same eye color, but their hair is different."
The Result: The system can tell you exactly which statistical facts made the match. For example, it might say, "These two steel datasets are similar because they both have high 'fatigue strength' and 'manganese content,' even though one dataset called it 'Mn%' and the other called it 'Manganese Weight'."

5. Privacy Protection

The authors also showed that you can add a little bit of "static" or noise to the math facts before turning them into sentences.

The Analogy: It's like wearing a disguise. You can still recognize the person's general shape and walk, but you can't see their specific face details.
The Benefit: This allows the system to compare sensitive data (like nuclear plant data) without ever seeing the actual raw numbers, protecting privacy while still finding matches.

What Did They Find?

They tested this on 15 different datasets, ranging from general benchmarks to very specific nuclear and materials science data.

The Score: When they asked the system to find the "nearest neighbor" (the most similar dataset) for a given spreadsheet, it was right 90% of the time.
Robustness: Even when they added privacy noise or removed some details, the system still found the right matches.
Real-World Proof: It successfully matched a dataset about "Red Wine" with "White Wine" (even though they were separate tables) and matched different ways of measuring steel strength, proving it understands the concept of the data, not just the labels.

Summary

This paper gives us a new way to organize the world's numeric data. By turning math into stories, it lets AI understand that a spreadsheet about "steel" and a spreadsheet about "alloys" are cousins, even if they use different words. It helps scientists find the right data to compare their work, and it does so in a way that is private and explains why the matches were made.

Technical Summary: Statistical Embeddings for Similarity, Retrieval, and Interpretable Alignment of Numeric Tabular Datasets

Problem Statement
Numeric tabular datasets are the dominant data format in scientific practice, yet organizing and comparing them across heterogeneous feature spaces remains a significant challenge. Existing dataset search tools typically rely on schema matching (shared variable names) or produce human-readable summaries that lack the vector representations necessary for modern retrieval systems. Large Language Models (LLMs) and their embedding spaces offer a pathway for cross-domain similarity, but they lack native mechanisms to represent numeric datasets meaningfully without shared variable definitions. Current approaches either focus on predictive modeling within a single dataset or fail to provide interpretable alignment between datasets with disjoint feature spaces.

Methodology
The proposed framework addresses these limitations through a three-stage pipeline: Statistical Fingerprinting, Semantic Embedding, and Interpretable Alignment.

Statistical Fingerprinting via EDA:
The methodology begins by characterizing a numeric tabular dataset $D$ through a structured Exploratory Data Analysis (EDA) pipeline. This process generates a comprehensive set of descriptors without requiring access to raw observations at comparison time, supporting privacy-preserving retrieval.
- Preprocessing: Missing values are imputed using soft-thresholding on the Singular Value Decomposition (SVD) to preserve the rank structure.
- Differential Privacy (Optional): To protect sensitive data, Laplace noise is added to scalar descriptors based on their global sensitivity before serialization.
- Descriptor Computation: The pipeline computes:
  - Multivariate descriptors: Matrix norms, spectral quantities (singular values), and variable importance via SCAD (Smoothly Clipped Absolute Deviation) penalized regression.
  - Univariate descriptors: Bounds, central moments, quantiles, entropy, and robust statistics (median, MAD).
  - Structural analysis: Change point detection (via PELT) to identify non-stationarity, and frequency analysis via FFT.
  - Categorical handling: Frequency distributions and partition-level analysis for categorical columns.
- Serialization: Each numeric descriptor is paired with a natural language sentence template (e.g., "Kurtosis is 2.4103; the distribution is platykurtic..."). Variable names or the token "matrix" are prepended to provide context, resulting in an ordered collection of sentences $S$ .
Semantic Embedding:
The collection of descriptor sentences $S$ is encoded into a fixed-length vector space using a pretrained sentence transformer (specifically all-MiniLM-L6-v2). For a dataset $D_i$ , this produces an embedding matrix $V_i \in \mathbb{R}^{d_E \times M_i}$ , where $M_i$ is the number of descriptor sentences. This transforms the dataset into a set of vectors compatible with LLM semantic spaces.
Similarity and Interpretability via CCA:
Cross-dataset similarity is quantified using Canonical Correlation Analysis (CCA) between the embedding matrices of two datasets.
- Similarity Metric: The similarity score $s_{ij}$ is defined as the mean of the canonical correlations between the two embedding spaces.
- Retrieval and Clustering: Pairwise distances ( $1 - s_{ij}$ ) are used to construct a distance matrix for hierarchical clustering (Ward $D^2$ linkage) and nearest-neighbor retrieval.
- Interpretable Alignment: To identify which specific descriptors drive similarity, a penalized CCA formulation is employed. By imposing an $\ell_1$ norm penalty on the canonical weight vectors, the method yields sparse solutions. The non-zero entries in these sparse vectors map directly back to specific descriptor sentences and variable-level quantities, revealing the statistical properties responsible for cross-dataset alignment without requiring shared variable names.

Key Contributions

Structured EDA Pipeline: A formalized mapping from raw tabular data to a collection of embedding vectors via sentence-serialized statistical descriptors, with optional differential privacy integration.
Cross-Dataset Similarity Framework: A retrieval system based on mean canonical correlation that operates effectively across datasets with heterogeneous and disjoint feature spaces.
Interpretable Penalized CCA: A novel application of penalized matrix decomposition to CCA, producing sparse canonical loadings that identify the specific statistical descriptors driving alignment, a capability absent in standard CCA.
Empirical Validation: Evaluation across 15 datasets spanning general-purpose benchmarks (UCI), materials informatics, and nuclear-grade graphite characterization.

Results
The framework was evaluated on 15 datasets with varying domains and feature spaces.

Retrieval Accuracy: The system achieved a total $P@1$ (Precision at 1) score of 0.9 for nearest-neighbor retrieval. This high accuracy was maintained across various embedding conditions, including multivariate-only ablations and differentially private embeddings with privacy budgets $\epsilon \in \{0.1, 1, 10\}$ .
Robustness: Hierarchical clustering results demonstrated that the broad cluster structure remained robust even under high noise injection or when univariate details were removed, though variable names and statistic values improved cluster resolution.
Interpretability: Penalized CCA successfully recovered physically meaningful correspondences. For example, it aligned alloying elements across datasets using different units (atomic vs. weight percent) and matched mechanical strength measures (e.g., fatigue strength vs. yield strength) between independent steel datasets, despite the absence of shared variable names.

Significance
The paper posits that this framework provides a principled pathway for integrating heterogeneous numeric data into Retrieval-Augmented Generation (RAG) pipelines while preserving statistical context. By converting tabular data into statistically grounded, semantically interpretable embeddings, the method enables:

Data-Driven Algorithm Selection: Identifying candidate modeling approaches for unknown datasets based on their statistical similarity to cataloged datasets with known outcomes (connecting to meta-learning and AutoML).
Simulation Initialization: Informing the selection of constitutive models or simulation parameters based on the statistical characteristics of empirical calibration data.
Privacy-Preserving Discovery: Enabling similarity comparisons on sensitive data without exposing raw observations, as the comparison relies solely on differentially private statistical descriptors.

The authors conclude that EDA-derived statistical fingerprints offer a compact, semantically interpretable basis for organizing heterogeneous numeric data within LLM-compatible spaces, addressing a critical gap in scientific data management where variable definitions and schemas are often incompatible.