📈 economics

Measuring What Cannot Be Surveyed: LLMs as Instruments for Latent Cognitive Variables in Labor Economics

This paper establishes a theoretical framework for using Large Language Models as valid measurement instruments for latent cognitive variables in labor economics, demonstrating through the construction and validation of the Augmented Human Capital Index that LLMs can reliably quantify occupational task content to reveal distinct dimensions of AI exposure and correct for measurement error in economic estimation.

Original authors: Cristian Espinal Maya

Published 2026-04-06

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Cristian Espinal Maya

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to measure the "secret sauce" of a job.

In economics, we know that some jobs are just about following a recipe (like a factory worker tightening bolts), while others require a chef's intuition (like a nurse diagnosing a complex illness or a lawyer crafting a unique argument). We call this "latent cognitive content." It's real, it matters, but you can't see it, weigh it, or ask a worker to fill out a survey about it easily. It's like trying to measure the flavor of a soup without tasting it.

For decades, economists have tried to measure this "flavor" using two main tools:

Expert Panels: Hiring a bunch of expensive, slow human experts to read job descriptions and guess the "flavor." (Slow, costly, and humans disagree a lot).
Keyword Counting: Looking for specific words like "computer" or "math." (Too simple; it misses the nuance).

The Big Idea: Using AI as a "Super-Taster"
This paper proposes a radical new tool: Large Language Models (LLMs). Think of an LLM not as a chatbot, but as a super-smart, tireless, and incredibly well-read "taster" that has read every job description, textbook, and manual ever written.

The author, Cristian Espinal Maya, asks: Can we trust this AI "taster" to give us accurate measurements for economic research?

To answer this, he treats the AI not just as a tool, but as a scientific instrument (like a thermometer or a scale). Just as a thermometer must be accurate and consistent, an AI scoring a job must meet four strict rules:

The Four Rules of the "AI Thermometer"

Semantic Exogeneity (The "No Cheating" Rule):
The AI must only look at the description of the job, not the result of the job.
- Analogy: If you are measuring how "spicy" a dish is, you can't ask the AI, "How much money does the chef make?" The AI must judge the ingredients (the text), not the paycheck. If it looks at the paycheck, it's cheating.
Construct Relevance (The "Does it Make Sense?" Rule):
The AI's score must actually match what we think the job is.
- Analogy: If the AI says a "Nurse" is 90% "Spicy" (augmentable by AI) and a "Data Entry Clerk" is 10%, that makes sense. If it got them backwards, the instrument is broken.
Monotonicity (The "More is More" Rule):
If a job gets a higher score, it must actually have more of that quality.
- Analogy: If Job A is rated "Very Spicy" and Job B is "Mild," Job A must actually be spicier. The scale can't be random.
Model Invariance (The "Agreement" Rule):
If you ask two different AI models (like Claude Haiku and Claude Sonnet) to taste the same soup, they should agree on the ranking, even if they disagree on the exact number.
- Analogy: One taster might say "This is a 7 out of 10" and another says "This is an 8 out of 10." As long as they both agree that this soup is spicier than the one rated a 3, the instrument is valid.

The Experiment: The "Augmented Human Capital Index"

The author put this theory to the test. He fed the AI 18,796 different job tasks (from the O*NET database) and asked it to score them on two things:

Augmentation: How much can AI help a human do this better? (The "Chef's Assistant" effect).
Substitution: How much can AI replace the human entirely? (The "Robot Replacement" effect).

The Results:

It Works: The AI scores matched up very well with other existing (but imperfect) measures.
It's Distinct: The AI successfully separated jobs that are augmented (helped) by AI from jobs that are substituted (replaced) by AI. Previous tools often mixed these up.
It's Reliable: When two different AI models scored the same jobs, they agreed 76% of the time. That's better than many human experts!
It's Cheap: Scoring 18,000 jobs took 6 hours and cost about $5. A human expert panel would take months and cost thousands.

The "Measurement Error" Fix

Here is the cleverest part. The author admits the AI isn't perfect. Sometimes Model A says "7" and Model B says "8." This is "noise" (measurement error).

In statistics, noise usually makes your results look weaker than they really are (a problem called "attenuation bias"). But because the author used two AI models, he could use a statistical trick called ORIV (Obviously Related Instrumental Variables).

Analogy: Imagine you are trying to measure the height of a building, but your tape measure is slightly stretched. If you use a second, slightly different tape measure, you can mathematically calculate exactly how much the first one was stretched and "fix" the number.
The Result: By using two AI models to cross-check each other, the author was able to "fix" the measurement error and get a truer picture of how AI affects wages.

Why This Matters

This paper is a game-changer for economics. It proves we can use AI to measure things that were previously unmeasurable, cheaply and quickly.

For Policymakers: We can finally see which jobs will be helped by AI (and need training) versus which will be replaced (and need safety nets).
For Researchers: We can study "cognitive skills" in a way we never could before.
For Everyone: It turns the "black box" of AI's impact on work into something we can actually see, measure, and understand.

In short: The author built a new, super-fast, super-cheap "AI ruler" to measure the invisible parts of our jobs, proved it's accurate, and showed us how to use it to fix old mistakes in economic research.

1. The Problem: Measurement of Latent Cognitive Variables

Economics faces a persistent challenge in measuring latent variables—constructs that are theoretically central but cannot be directly observed in surveys or administrative data. A prime example is the cognitive content of occupational tasks (e.g., distinguishing between routine information processing and contextual judgment, or determining the degree to which a task is augmentable by AI versus substitutable).

Existing measurement approaches suffer from significant limitations:

Expert Panels (e.g., O*NET): Expensive, slow, and prone to low inter-rater reliability for novel constructs.
Crowdsourcing: Faster but introduces non-expert noise.
Patent/Keyword Matching: Provides indirect proxies based on technological capability rather than direct semantic task content, failing to capture nuances like "augmentation" vs. "substitution."

The paper addresses the need for a scalable, cost-effective, and theoretically valid method to quantify the semantic content of occupational tasks.

2. Methodology: LLMs as Econometric Instruments

The author proposes a framework treating Large Language Models (LLMs) not just as scoring tools, but as econometric instruments for latent variables. The core innovation is the formalization of four validity conditions under which LLM-generated scores can be used in regression analyses.

The Four Validity Conditions

Semantic Exogeneity: The LLM prompt must reference only the semantic content of the task (descriptions, skills) and strictly exclude labor market outcomes (wages, employment). This ensures the measurement error is orthogonal to the outcome equation error ( $Cov(\eta, \epsilon) = 0$ ).
Construct Relevance: The LLM score must correlate with the latent variable. This is established via convergent validity (correlation with existing measures), predictive validity, and face validity.
Monotonicity: Higher LLM scores must correspond to a higher degree of the latent variable ( $\partial \hat{H}/\partial H^* > 0$ ), ensuring regression signs are interpretable.
Model Invariance: The ranking of occupations must remain consistent across different LLM models, even if the absolute levels (means) differ.

The Application: Augmented Human Capital Index (AHCo)

Data: 18,796 unique task statements from O*NET (version 30.2).
Scoring: Tasks were scored by Claude Haiku 4.5 on two dimensions:
- Augmentation Potential ( $a_{ok}$ ): How much AI can help as a complementary tool.
- Substitution Risk ( $s_{ok}$ ): How much AI can fully replace the human.
Aggregation: Occupation-level indices are importance-weighted averages of task scores.
Validation Sample: A 20% subsample ( $n=3,666$ ) was independently re-scored by Claude Sonnet 4 to test inter-rater reliability.

3. Key Contributions

Theoretical Formalization: The paper establishes the first rigorous set of conditions (Semantic Exogeneity, Construct Relevance, Monotonicity, Model Invariance) required to treat LLM outputs as valid econometric instruments.
Empirical Validation: It provides a comprehensive assessment of these conditions using the AHCo, validating it against six existing AI exposure indices (Felten, Eloundou, Webb, Frey-Osborne, Eisfeldt).
Econometric Characterization: It demonstrates that LLM-generated scores follow a classical measurement error structure. This allows for the application of Obviously Related Instrumental Variables (ORIV) estimation to correct for attenuation bias, a standard technique usually reserved for human survey data.

4. Key Results

Validity and Reliability

Convergent Validity: The AHCo shows strong correlation with established indices, particularly Eloundou GPT-γ ( $r=0.85$ ) and Felten AIOE ( $r=0.79$ ).
Discriminant Validity: The index correlates weakly with direct automation measures ( $r=0.21$ with Eloundou $\alpha$ ), confirming that "augmentation" and "substitution" are distinct constructs.
Dimensionality (PCA): Principal Component Analysis of 11 indices reveals two distinct dimensions:
- PC1 (62% variance): General "Cognitive AI Relevance."
- PC2 (15% variance): "Augmentation vs. Substitution." The AHCo loads negatively on PC2, confirming it specifically captures the augmentation dimension.
Inter-Rater Reliability:
- Between Claude Haiku and Sonnet: Pearson $r = 0.76$ , Krippendorff's $\alpha = 0.71$ (mean-adjusted).
- Level Bias: Sonnet scores were systematically higher (+8.6 points) than Haiku, but this bias was constant across the score range (Bland-Altman plot), satisfying Proposition 1 (level bias does not affect standardized regression coefficients).
- Prompt Sensitivity: While prompt wording affects the absolute scale, task-level variance dominates prompt-induced variance, and rankings remain robust.

Econometric Properties

Measurement Error Correction: The disagreement between models is consistent with classical attenuation.
ORIV Estimation: Using Sonnet scores as an instrument for Haiku scores in an augmented Mincer wage equation:
- The ORIV coefficient ( $\beta = 0.100$ ) was 25% larger than the OLS coefficient ( $\beta = 0.080$ ), successfully correcting for attenuation bias.
- The result aligns with external IV estimates from previous literature, validating the LLM approach.
Predictive Power: In a "horse race" regression, adding AHCo to existing indices (like Frey-Osborne) increased the $R^2$ by an additional 0.86 percentage points, demonstrating that AHCo captures unique, complementary information.

5. Significance and Implications

Cost-Quality Frontier: The methodology offers a massive efficiency gain. Scoring 18,796 tasks cost approximately $5 and took 6 hours, compared to the months and high costs required for traditional expert panels.
Generalizability: The framework extends beyond labor economics to any domain requiring the quantification of semantic content in documents, policies, or institutions (e.g., contract analysis, policy evaluation, judicial reasoning).
Best Practices: The paper provides a concrete protocol for practitioners, emphasizing the use of multiple models, standardized scoring, prompt sensitivity analysis, and ORIV estimation to mitigate measurement error.

Conclusion: The paper successfully demonstrates that LLMs can serve as valid, reliable, and cost-effective instruments for latent economic variables, provided specific validity conditions are met and measurement error is appropriately corrected using econometric techniques like ORIV.