Technical Summary: The Open Source Economic Index of AI Adoption and Capability

Problem Statement

Despite significant global investment in generative AI and widespread organizational adoption, empirical evidence regarding the granular impact of AI on specific labor tasks remains opaque. Existing methods for measuring AI's economic impact suffer from critical limitations:

• Surveys: Prone to under-reporting bias due to social desirability.
• Controlled Experiments: Often limited to single occupations, lacking a broad economic picture.
• Prior Chat Analysis: Previous studies mapping user-LLM chats to occupational tasks (e.g., Handa et al., Chatterji et al.) rely on proprietary data, hindering reproducibility. Furthermore, simply categorizing chats does not measure the actual performance or capability of the AI in executing those tasks.
• Capability Benchmarks: Existing benchmarks (e.g., GDPval, Remote Labor Index) often rely on custom-generated tasks or human-graded deliverables, introducing ambiguity and limiting the scale of evaluation.

There is a need for an open-source, reproducible framework that simultaneously measures AI adoption rates across occupations and the functional capability of AI agents to execute the specific tasks observed in real-world usage.

Methodology

The authors propose a two-part system to address these gaps: an Open Source Economic Index for adoption and an Economic Capability Benchmark for performance.

1. Open Source Economic Index (Adoption Measurement)

This system quantifies which occupational sectors utilize AI and the depth of that utilization using publicly available data.

• Data Source: The study utilizes the WildChat-4.8M dataset, a collection of non-toxic human-LLM interactions.
• Filtering Pipeline:
  1. Language/Uniqueness: Filters for unique English chats.
  2. Occupational Relevancy: Uses gpt-oss-120b to classify chats as "occupationally relevant" (related to work/professional tasks). This yields ~789,000 relevant chats.
  3. Semantic Mapping: Maps relevant chats to O*NET (Occupational Information Network) tasks.
     • Step 1: Summarize chat into a single request.
     • Step 2: Generate embeddings using Qwen3-Embedding-0.6B and perform nearest-neighbor search against O*NET tasks (2,087 Detailed Work Activities, 332 Intermediate, 41 Work Activities).
     • Step 3: Use gpt-4o-mini as a filter to retain only tasks where the core skill or activity overlaps with the chat.
• Aggregation: Chats are weighted by task count and aggregated to the occupation level to create an index of representation and penetration depth.

2. Economic Capability Benchmark (Performance Measurement)

This system evaluates whether AI agents can autonomously or collaboratively execute the tasks identified in the adoption index.

• Scenario Generation:
  • Grounded in O*NET tasks and MCP (Model Context Protocol) servers from Smithery.
  • The system crawls Smithery servers, validates tool quality, and semantically matches O*NET tasks to available tools using embeddings.
  • Scenarios are generated as (occupation, task, server) tuples, prompting an LLM to create realistic workplace requests with specific tool-call answer keys.
• Evaluation Modes:
  • Single-Turn: Autonomous execution of a workflow.
  • Multi-Turn: Collaborative execution where a simulated user agent withholds critical parameters, requiring the AI to ask clarifying questions (role inversion).
• Evaluation Dimensions (LLM-as-Judge):
  • Tool Call Accuracy: Binary check for correct tool and arguments (functional equivalence).
  • Workflow Completion: 1–5 scale assessing multi-step sequencing and data flow.
  • Grounding: 1–5 scale assessing if claims are supported by tool outputs vs. hallucinations.
  • Autonomy & Follow-up Quality: Specific to multi-turn scenarios, measuring independence and information gathering.

Key Results

Adoption Findings

• Sector Concentration: AI adoption is highest in Arts, Design, Entertainment, Business & Financial, and Computer & Mathematical sectors.
• Low Adoption: Legal services show surprisingly low LLM adaptation compared to other white-collar fields.
• Depth of Usage: While adoption is visible, deep penetration is limited. Only 0.4% of occupations show AI usage for at least 75% of their tasks (compared to ~4% in prior proprietary studies).
• Skill Usage: Cognitive skills (writing, programming, critical thinking, reading comprehension) dominate the index.

Capability Findings

The authors tested the Kimi-k2.5 model across 9 high-frequency occupations (e.g., Programmers, HR Specialists, Journalists) using ~5,000 generated scenarios.

• Macro vs. Micro Performance: AI agents demonstrate strong capability in high-level workflow completion (mean score ~4.08/5) but frequently err in granular details.
• Tool Call Accuracy: Only ~60% of required tool calls are executed correctly. Errors often involve hallucinated arguments or missing required parameters.
• Grounding: Mean grounding scores are ~3.7/5, indicating a non-negligible rate of hallucination where claims are not supported by tool outputs.
• Human-in-the-Loop: In multi-turn scenarios, user collaboration improves workflow completion but often degrades tool call accuracy and grounding, suggesting that human oversight remains critical for granular steps.
• Discrepancy: Theoretical capability (what agents can do in benchmarks) often outpaces the depth of current real-world adoption, yet complete occupational replacement is unlikely as collaborative workflows outpace purely autonomous execution.

Significance and Contributions

The paper claims the following contributions to the field of AI economics and evaluation:

1. Reproducibility: By utilizing the open WildChat dataset and open-source mapping pipelines, the authors provide a transparent alternative to proprietary economic indices, allowing for independent verification of AI adoption trends.
2. Granular Capability Assessment: The benchmark moves beyond "win-rate" against humans or static tasks. It evaluates agents on O*NET-grounded, tool-mediated workflows, distinguishing between autonomous execution and collaborative steering.
3. Nuanced Labor Market Insights: The study challenges the narrative of immediate, total automation. It suggests that while AI can handle macro-workflows, the "jagged frontier" of labor impact is defined by errors in granular tool usage and the necessity of human oversight for accuracy and grounding.
4. Methodological Innovation: The integration of MCP servers with O*NET taxonomies creates a scalable, dynamic benchmark that reflects real-world software ecosystems rather than static, custom-built tasks.

The authors conclude that while AI adoption is growing, the risk of significant job displacement is currently mitigated by the gap between high-level workflow success and the frequent need for human intervention in tool execution and error correction.