💬 NLP

The Proxy Presumption: From Semantic Embeddings to Valid Social Measures

This paper addresses the "Proxy Presumption" in Computational Social Science by introducing the Construct Validity Protocol (CVP) and Counterfactual Neutralization to rigorously validate semantic embeddings, ensuring they measure social constructs rather than confounding attributes like topic or style.

Original authors: Baishi Li, Ta Yu, Kelvin J. L. Koa, Ke-Wei Huang

Published 2026-05-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Baishi Li, Ta Yu, Kelvin J. L. Koa, Ke-Wei Huang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to measure the "creativity" of a new invention. You decide to use a high-tech ruler that measures the distance between the new invention and old ones. If the distance is huge, you declare the invention "highly creative."

This paper argues that in the world of Artificial Intelligence (AI), researchers are doing exactly this, but they are making a dangerous mistake. They are assuming that because two things are "far apart" in the AI's math, they must be "different" in the real world. The authors call this the "Proxy Presumption."

Here is a simple breakdown of their argument, their solution, and their findings.

1. The Problem: The "Confused Ruler"

The authors say that AI models (specifically the ones that turn text into numbers, called "embeddings") are like a kitchen blender.

The Target (C): This is what you actually want to measure, like "creativity," "bias," or "novelty."
The Noise (Z): This is everything else mixed in, like the topic of the text, the author's writing style, the length of the sentence, or the specific words used.

When you put a text into an AI blender, it spits out a single number (a vector). The problem is that this number is a smoothie of both the Target and the Noise.

The Analogy:
Imagine you want to measure how "spicy" a soup is. You use a thermometer. But the thermometer is also sensitive to how "hot" the soup is (temperature).

If the soup is hot and spicy, the thermometer reads high.
If the soup is hot but bland, the thermometer also reads high.

If you just look at the high reading and say, "Aha! This soup is very spicy!" you are wrong. The thermometer is actually measuring heat, not spiciness.

In AI, researchers often look at the "distance" between two texts and say, "This text is very creative!" But the paper argues that the AI might just be measuring that the text is about a different topic or written in a different style, not that it is actually more creative. They are mistaking the "heat" (noise) for the "spiciness" (the real concept).

2. The Mathematical Proof: Why You Can't Just "Guess"

The paper uses some heavy math to prove a simple point: You cannot separate the spice from the heat just by looking at the soup.

Unless you have a special tool to filter out the heat, you can never be sure if the high temperature on the thermometer is because of the chili peppers or just the stove. Similarly, without special steps, an AI cannot know if a text is "creative" or just "about a different topic." The math proves that the AI's "ruler" is fundamentally confused.

3. The Solution: The "Validity Protocol" (CVP)

To fix this, the authors propose a new set of rules called the Construct Validity Protocol (CVP). Think of this as a quality control checklist for anyone trying to measure social concepts with AI.

Instead of just saying, "Here is a score for creativity," researchers must now prove their ruler actually measures creativity and not just noise. The protocol has three main steps:

Step 1: Define the Recipe. Clearly explain what "creativity" means and what it doesn't mean.
Step 2: Clean the Ingredients. Before measuring, use tools to remove the "noise." For example, if you want to measure "political bias," you might use an AI to rewrite the text to remove the specific topic (like "taxes") so you are only measuring the attitude, not the subject.
Step 3: The "Validity Card." This is a report card that must be attached to every study. It includes:
- Stability Test: If you change the font or the order of words slightly, does the score stay the same? (If not, the ruler is broken).
- The "Different Topic" Test: Does the score change if you talk about a totally different subject? If the score changes just because the topic changed, your ruler is measuring the topic, not the concept.
- The "Real World" Test: Does this score actually predict real-world outcomes?

4. The "Forensic" Investigation

The authors didn't just talk about theory; they acted like detectives. They looked at 17 recent, famous papers that claimed to measure things like "bias," "ideology," and "creativity" using AI.

What they found:

Most papers (10 out of 17) gave a good definition of what they were measuring.
Almost all papers showed some examples to prove it looked right.
BUT, almost NO papers did the hard work of proving their ruler wasn't just measuring the "noise."
- Only 1 paper proved their measure was different from other similar measures.
- Zero papers proved their measure wasn't just tracking the topic or the writing style.
- Zero papers used the strict math required to separate the "spice" from the "heat."

The Verdict: The authors call this the "Jangle Fallacy." This is when two different studies use the same name (e.g., "Bias") but are actually measuring two completely different things because they didn't check if their rulers were clean. One study might be measuring "rude words," while another is measuring "political topics," but they both call it "Bias."

5. The Conclusion

The paper concludes that we cannot just trust AI to measure complex human ideas like "creativity" or "bias" by simply looking at the distance between words.

If we want AI to be a useful tool for social science, we need to stop treating the AI's math as a magic answer. We need to use the Validity Protocol to:

Clean the data of distractions (like topic and style).
Prove that the score actually changes when the concept changes.
Prove that the score doesn't change when the distractions change.

Until researchers start doing this, the paper argues, we are just measuring the "heat" of the soup and pretending we know how "spicy" it is.

Technical Summary: The Proxy Presumption: From Semantic Embeddings to Valid Social Measures

1. Problem Statement: The Proxy Presumption

The paper identifies a fundamental validity crisis in the application of Natural Language Processing (NLP) to Computational Social Science (CSS). While NLP is increasingly used to measure latent social constructs (e.g., novelty, creativity, bias, ideology) via semantic embeddings, the field relies on a "Proxy Presumption." This presumption assumes that geometric properties of embedding spaces (e.g., cosine distance or similarity) serve as direct, valid measures of theoretical social concepts without sufficient empirical validation.

The authors argue that this approach constitutes a non-identification problem. Unsupervised embeddings compress multiple generative factors into a single representation, entangling the target construct ( $C$ ) with nuisance attributes ( $Z$ ), such as topic, style, register, authorship, and length. Without explicit disentanglement or design-based controls, standard metrics often capture method variance (e.g., topic shifts or vocabulary changes) rather than the intended social phenomenon. The paper formalizes this as an ill-posed inverse problem where recovering a latent construct from observed text is theoretically impossible without structural assumptions or interventions.

2. Methodological Framework

2.1 Theoretical Formalization

The authors model document generation as a stochastic process $p_\theta(D | c, z)$ , where $D$ is the document, $c$ is the scalar target construct, and $z$ is a vector of nuisance attributes.

Rotational Ambiguity: Drawing on causal representation learning (Locatello et al., 2019; Schölkopf et al., 2021), the paper proves that unsupervised objectives maximizing data likelihood cannot distinguish between the true latent basis and an arbitrarily rotated basis. Consequently, the learned embedding dimension used for measurement is an inextricable mixture of $c$ and $z$ .
Metric Indeterminacy: Even with a perfectly disentangled encoder, applying geometric functions (like cosine similarity or Euclidean distance) to full embeddings fails because the metric may be dominated by nuisance variance ( $z$ ) rather than construct variance ( $c$ ). For instance, high cosine similarity may reflect shared topics rather than conceptual similarity.

2.2 Proposed Interventions

To mitigate entanglement, the paper outlines a three-level intervention strategy:

Input Disentanglement: Preprocessing text to reduce dependence on $Z$ (e.g., extracting only construct-relevant spans via LLMs, stylistic normalization, or entity masking).
Representation Disentanglement: Training encoders to suppress $Z$ while preserving $C$ using adversarial removal, nullspace projection, or contrastive learning.
Methodological Disentanglement: Designing scoring functions that cancel nuisance contributions. The authors propose Counterfactual Neutralization, where a measurement is defined as the difference between an observed embedding and a counterfactual "neutralized" embedding (where the construct signal is minimized but $Z$ is preserved).

2.3 The Construct Validity Protocol (CVP)

The core contribution is the Construct Validity Protocol (CVP), a standardized pipeline adapted from psychometrics and causal inference to validate embedding-based measures. It consists of three phases:

Phase 1: Conceptualization: Establishing face and content validity through expert scrutiny. This involves defining the construct, enumerating nuisance dimensions, and creating a "known-groups" anchor set (high, low, and borderline examples) to audit decision boundaries.
Phase 2: Operationalization: Designing instruments that reduce entanglement using the interventions described above.
Phase 3: The Validity Suite: A structured reporting framework (a "Validity Card") requiring five specific types of evidence:
1. Reliability/Stability: Testing robustness against perturbations (encoder variants, prompt paraphrasing) using Intraclass Correlation Coefficients (ICC).
2. Convergent Validity: Correlation with an independent, external "gold-standard" measure of the same construct.
3. Discriminant & Incremental Validity: Proving the measure is distinct from nuisance factors (e.g., topic, length) and adds explanatory power beyond them for an external validation label.
4. Known-Groups Validity: Demonstrating the measure separates pre-specified high and low construct groups.
5. Criterion-Related Evidence: Predicting downstream outcomes in the theorized direction while controlling for strong nuisance variables.

3. Key Results and Empirical Analysis

3.1 Worked Example (GoEmotions)

The authors applied the CVP to the GoEmotions dataset to measure "gratitude."

Reliability: By varying encoders, pooling methods, and normalization, they achieved high stability (ICC(2,1) = 0.8467).
Discriminant Validity: A regression analysis showed that the proxy score was largely recoverable from topic features ( $R^2 = 0.7762$ ), indicating significant entanglement with nuisance variables.
Incremental Validity: Despite the entanglement, adding the proxy to a model containing only nuisance features significantly improved the prediction of human labels (AUC increased from 0.9658 to 0.9831), demonstrating that the construct signal exists but is mixed with noise.
Error Analysis: The audit revealed specific failure modes, such as false positives where "thank you" was used sarcastically, highlighting the need for the "known-groups" audit in Phase 1.

3.2 Forensic Literature Review

The authors conducted a forensic analysis of 17 influential papers (2020–2025) proposing computational metrics for social constructs.

Findings: While most papers provided construct definitions and face/content evidence (often via qualitative examples), they largely lacked rigorous identification evidence.
- Discriminant Validity: 0 papers fully demonstrated discriminant validity (ruling out nuisance surrogacy).
- Convergent Validity: Only 1 paper established convergence against an independent instrument at the same unit of analysis.
- Causal Identification: None of the sampled papers used causal identification methods to isolate constructs from confounds; reliance was placed on heuristics or associative controls.
Conclusion: The literature frequently documents that a proxy is defined and stable enough to compute, but rarely demonstrates that it isolates the intended construct. This creates a risk of the "Jangle Fallacy," where different papers claim to measure the same construct but are actually tracking different mixtures of signal and noise.

4. Key Contributions

Non-Identification Result: Formalizing why unsupervised metrics cannot reliably recover latent constructs without explicit assumptions or interventions, proving that statistical independence does not imply disentanglement.
Methodological Solutions: Proposing Counterfactual Neutralization and connecting it to existing debiasing tools (adversarial removal, nullspace projection).
The Construct Validity Protocol (CVP): Providing a community-facing reporting standard that moves from conceptualization to a structured Validity Card, ensuring measures are scientifically defensible.
Forensic Evidence: Documenting the prevalence of the Proxy Presumption in recent literature, showing a systematic gap between geometric proxies and valid social measures.

5. Significance and Claims

The paper positions itself as a position and synthesis contribution, not a final standard. Its primary significance lies in:

Shifting the Burden of Proof: Arguing that "measurement by renaming" (equating geometric distance with social concepts) is insufficient. Validity must be empirically demonstrated, not assumed.
Enabling Cumulative Science: By standardizing validity reporting (via the Validity Card), the protocol aims to make results comparable across studies and reduce the risk of topic/style leakage masquerading as social signals.
Defending Downstream Inference: Emphasizing that if variable construction lacks validity, no downstream causal ML methods can rigorously establish causality.

The authors explicitly state they do not aim to prohibit the use of proxies but to make them testable. They acknowledge limitations, noting that their coding reflects reported practices (which may undercount unpublished analyses) and that the CVP itself requires further iteration and prospective empirical case studies to quantify practical costs and failure modes. The goal is to transform heuristic proxies into robust, scientifically defensible instruments for Computational Social Science.