📈 economics

Making Interpretable Discoveries from Unstructured Data: A High-Dimensional Multiple Hypothesis Testing Approach

This paper introduces a statistically principled framework that leverages AI interpretability to map unstructured data into high-dimensional concept embeddings, enabling robust, interpretable, and reproducible discovery through high-dimensional multiple hypothesis testing with selective inference.

Original authors: Jacob Carlson

Published 2026-05-06

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jacob Carlson

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery, but instead of looking at a few clues, you are handed a library containing millions of books, thousands of hours of audio, and millions of photos. You don't know what you are looking for. You just know there are important patterns hidden inside, but you can't possibly read every single page or listen to every second of audio manually.

This is the problem social scientists face when they try to analyze "unstructured data" like text, video, or audio. They want to discover new insights, but if they try to guess what to look for, they might miss the most important things (the "streetlight effect") or accidentally trick themselves into finding patterns that aren't really there (the "data snooping" problem).

Jacob Carlson's paper proposes a new, automated detective kit to solve this. Here is how it works, broken down into four simple steps using everyday analogies:

1. The "Universal Translator" (Creating Concept Embeddings)

Imagine you have a super-smart robot librarian (an AI model) that has read the entire internet. This librarian doesn't just read words; it understands the ideas behind them.

The paper suggests using a special tool called a Sparse Autoencoder (SAE). Think of this as a high-tech filing cabinet with 100,000 drawers. Each drawer represents a specific, human-understandable "concept" or "idea" (like "mentioning politics," "expressing uncertainty," or "talking about money").

When you feed a piece of text (like a survey response) into this system, the robot librarian instantly checks all 100,000 drawers. It pulls out a binary checklist: "Did this text mention politics? Yes (1). Did it mention money? No (0)."

The Result: You turn a messy paragraph of text into a clean, organized list of 100,000 "Yes/No" switches. This list is called a Concept Embedding.

2. The "Massive Roll Call" (Formulating Hypotheses)

Now, imagine you have two groups of people: Group A (who got a special treatment) and Group B (who didn't). You want to know if the treatment changed what they talked about.

Instead of guessing what to ask, you ask the robot librarian to check every single one of the 100,000 drawers for both groups.

"Did Group A talk about politics more than Group B?"
"Did Group A express more uncertainty than Group B?"
"Did Group A mention money more than Group B?"

You are now running 100,000 tiny tests at once. This is the "discovery" part: you aren't guessing; you are letting the data tell you which drawers were opened more often in one group than the other.

3. The "Smart Filter" (High-Dimensional Multiple Hypothesis Testing)

Here is the tricky part. If you flip a coin 100,000 times, you will get some "Heads" just by pure luck. If you look at 100,000 concepts, you will find some that look different between Group A and Group B just by random chance. If you report all of them, you are lying to yourself.

The paper introduces a statistical filter (based on advanced math called k-FWER control).

The Analogy: Imagine you are looking for a needle in a haystack, but you have a metal detector that beeps 100,000 times. Most beeps are just noise (random luck).
The Solution: The paper's math acts like a very strict bouncer. It says, "We will only let you keep the top 5 'bumps' in the data, and we guarantee that at least 4 of them are real needles, not just noise."
This allows the researcher to find many interesting things (discoveries) without getting fooled by random chance, even when looking at thousands of possibilities at once.

4. The "Human Translator" (Automatic Interpretation)

So far, the computer has told you: "Drawer #4, Drawer #101, and Drawer #5030 were opened more often in Group A." But what do those numbers mean? "Drawer #4" is useless to a human.

The paper uses a second AI (a "Explainer LLM") to translate these numbers back into English.

The Process: The computer shows the Explainer AI the top 10 texts that triggered "Drawer #4." The AI reads them and says, "Ah, this drawer seems to activate when people talk about politics."
The Quality Check: The paper doesn't just trust the AI blindly. It sets up a test: It gives the AI a new set of texts and asks, "Does this text talk about politics?" If the AI's guess matches the robot librarian's original "Yes/No" switch, the translation gets a high "Quality Score." If it guesses wrong, the score is low, and the researcher knows to be skeptical.

Why This Matters

The paper argues that the old way of doing this—where a human researcher reads a few examples, guesses a topic, and then counts it—is flawed because humans are biased and can't read enough data.

This new framework is like a fully automated, unbiased factory:

It scans the whole library (no missing clues).
It checks every single possibility (no guessing).
It uses strict math to filter out luck (no false alarms).
It translates the results into plain English and grades the quality of the translation (no confusion).

The author shows this works by re-analyzing two real studies (one about political dissent and one about inflation opinions). In both cases, the automated system found the same things the human researchers found, plus many new, interesting insights that the humans missed, all in a matter of minutes on a standard computer.

In short: This paper gives researchers a way to let AI do the heavy lifting of "reading" unstructured data, while using strict math to ensure the discoveries are real and the explanations are accurate.

Technical Summary: Making Interpretable Discoveries from Unstructured Data

1. Problem Statement

Social scientists increasingly utilize unstructured data (text, audio, video) to uncover empirical insights, particularly in "unsupervised" settings where the goal is discovery rather than the prediction of pre-specified outcomes. Traditional approaches to analyzing such data often rely on manual "coding" processes where researchers iteratively define concepts and topics. The paper argues this status quo suffers from four critical flaws:

The Streetlight Effect: Researchers may only measure obvious or easily accessible concepts, missing scientifically important "unknown unknowns."
Researcher Degrees of Freedom: Discretionary definition of concepts creates risks of motivated data mining and cherry-picking.
Post-Selection Inference: Defining hypotheses and testing them on the same dataset leads to invalid statistical conclusions if not handled correctly.
Replicability: Manual coding is costly, difficult to reproduce, and sensitive to arbitrary researcher choices.

While sample splitting (using one dataset for discovery and another for testing) addresses inference validity, it reduces statistical power and does not fully solve the issues of researcher discretion or the "streetlight effect."

2. Methodology

The paper proposes a four-step framework for statistically principled, interpretable discovery from unstructured data. The framework minimizes researcher degrees of freedom by automating concept extraction and evaluation.

Step 1: Creating Concept Embeddings

Unstructured data points ( $Z_i$ ) are mapped to high-dimensional, sparse, binary vectors called concept embeddings ( $Y_i \in \{0, 1\}^p$ ).

Mechanism: This is achieved using dictionary learning models, specifically Sparse Autoencoders (SAEs) trained on Large Language Models (LLMs).
Output: Each element $Y_{ij}$ is a binary indicator for the presence of a specific "concept" $j$ in the data point. These concepts are "monosemantic" (interpretable) and catalog a vast space of features the underlying LLM learned during pretraining.
Properties: The vectors are high-dimensional ( $p \gg n$ , often $p \approx 10^5$ ) and sparse.

Step 2: Formulating Concept-Level Hypotheses

The researcher formulates $p$ hypothesis tests, one for each concept indicator.

Hypotheses: For each concept $j$ , a null hypothesis $H_{0,j}$ is tested (e.g., zero average treatment effect on the probability of concept $j$ appearing).
Flexibility: The framework allows for various estimands (means, regression coefficients) derived from the concept indicators and covariates.

Step 3: High-Dimensional Multiple Hypothesis Testing

To handle the $p \gg n$ setting and complex dependencies between concepts, the framework employs high-dimensional selective inference.

Error Control: Instead of controlling the Familywise Error Rate (FWER) for any false rejection (which is overly conservative), the method controls the $k$ -FWER (the probability of making $k$ or more false rejections).
Algorithm: It adapts the step-down procedures of Romano and Wolf [2007].
Theoretical Foundation: The validity of these procedures in high dimensions is established using new results in high-dimensional central limit theory (CLT) for the $k$ -th largest coordinate of sums of random vectors (extending work by Ding et al. [2025] and Belloni et al. [2018]).
Bootstrap: Critical values are derived using a Gaussian multiplier bootstrap to account for dependence structures without assuming independence.

Step 4: Generating and Evaluating Automatic Interpretations

The output of Step 3 is a set of rejected hypotheses (indices of concepts), but these indices lack natural language descriptions.

Generation: An "explainer LLM" is prompted with examples of data points where a specific concept activates highly to generate a natural language description (e.g., "discussion of politics").
Evaluation: The quality of these descriptions is evaluated using a statistical inference approach to "detection scoring." A held-out dataset is used to train a classifier (via an LLM) to predict the presence of the concept based on the generated description. The A-score (accuracy) is estimated with confidence intervals, providing a quantitative measure of description quality.

3. Key Theoretical Contributions

The paper makes several theoretical contributions to econometrics and statistics:

High-Dimensional $k$ -FWER Control: It extends the Romano and Wolf [2007] step-down algorithms to the high-dimensional setting ( $p \gg n$ ), proving they asymptotically control the $k$ -FWER for small, fixed $k$ under mild conditions.
High-Dimensional CLT for Order Statistics: It develops new high-dimensional CLTs for the $k$ -th largest coordinate of sums of independent random vectors, extending the "many approximate means" framework of Belloni et al. [2018] and the order statistic results of Ding et al. [2025].
Inference on "Approximate Means": It validates these procedures for estimators that are asymptotically linear (e.g., OLS coefficients, studentized statistics) rather than just exact sample means.
Statistical Formalization of Autointerp: It provides a formal statistical framework for evaluating automatic interpretation quality (A-scores), treating the evaluation as a conditional inference problem with valid confidence intervals.

4. Empirical Results

The framework is applied to reanalyze two recent studies in empirical economics:

Reanalysis of Bursztyn et al. [2023] (Information Treatment RCT):
- Context: Analyzing open-ended responses to a social cover treatment regarding police defunding.
- Result: Using $k=5$ (allowing up to 5 false discoveries), the framework identified 9 significant concepts, compared to only 2 under strict FWER control.
- Insights: Beyond confirming the original finding (references to articles/evidence), it discovered new nuances: the treatment increased mentions of "prestigious author sources" and decreased expressions of "uncertainty/modality," suggesting the mechanism relies on perceived credibility and reduces speculation.
- Efficiency: The analysis was performed automatically in ~1 hour on a single GPU.
Reanalysis of Stantcheva [2024] (Inflation Attitudes Survey):
- Context: Analyzing open-ended responses to "High inflation is caused by..."
- Result: The framework automatically identified 744 interpretable discoveries.
- Insights: It recovered known themes (monetary policy, greed, supply/demand) without manual keyword lists. It also surfaced high-frequency concepts regarding "subjective uncertainty" and "macroeconomic indicators" with high A-scores.
- Scalability: The method handled $n=453$ and $p=3826$ non-degenerate features, demonstrating feasibility for small-sample social science data.

5. Significance and Claims

The paper claims to provide a general, flexible, and low-cost framework for discovery from unstructured data that addresses the scientific and statistical pitfalls of traditional manual coding.

Principled Discovery: By leveraging pre-trained SAEs, the framework shines a "powerful streetlight" on the data, reducing the risk of missing important concepts (unknown unknowns).
Reduced Researcher Degrees of Freedom: The process is automatic; the researcher does not manually define concepts, thereby mitigating motivated data mining. The few choices required (e.g., model selection) can be preregistered.
Statistical Validity: It solves the post-selection inference problem within a single dataset using high-dimensional selective inference, avoiding the power loss associated with sample splitting.
Interpretability and Evaluation: It bridges the gap between high-dimensional AI features and human understanding by generating natural language descriptions and, crucially, providing statistically valid scores for the quality of those descriptions.
Replicability: The framework is designed to be easily reproducible and allows for fast sensitivity analysis regarding model choices and hyperparameters.

The authors position this framework not as a replacement for all existing methods, but as a complementary tool that enables rigorous, open-ended discovery in settings where traditional hypothesis generation is difficult or prone to bias.