Technical Summary: Artificial Effort

Problem Statement

Real-effort tasks are a foundational methodological tool in experimental economics, designed to measure human effort, performance, fatigue, and responsiveness to incentives by requiring participants to perform cognitively costly activities. The validity of these tasks rests on a critical, often implicit assumption: that the agent completing the task is human.

With the rapid diffusion of Large Language Models (LLMs) and their increasing capability in multimodal reasoning, this assumption is increasingly insecure, particularly in unsupervised online environments (e.g., online labor markets, web-based experiments). If LLMs can solve canonical real-effort tasks with high accuracy at negligible marginal cost, these tasks may cease to function as measures of human effort. Instead, they could inadvertently measure access to capable AI models, compromising construct validity and potentially enabling malicious "task-milling" for revenue. While prior literature has explored AI in survey generation, there is a lack of direct evidence regarding whether state-of-the-art LLMs can effectively solve the specific real-effort tasks routinely used by economists.

Methodology

The authors conducted a comprehensive benchmark evaluating the ability of 23 state-of-the-art LLMs from three major providers (OpenAI, Google, and Anthropic) to solve eight canonical real-effort tasks.

Tasks:
The study utilized eight tasks available in the Otree repository, covering a spectrum of perceptual, arithmetic, and reasoning demands:

1. Addition: Summing three integers.
2. Counting Zeros: Counting zeros in a 15×10 binary table.
3. Letter Decoding: Decoding a string using a letter-to-digit mapping table.
4. Number Pair Summation: Identifying a pair of numbers in a 4×4 grid that sum to 10.0.
5. Sequence Completion: Predicting the next element in a mathematical sequence.
6. Sudoku: Solving a partially filled 6×6 grid.
7. Distorted Text: Transcribing a 12-character alphanumeric string from a distorted image.
8. String Entry: Reproducing a 9-character string of special characters from an image.

Experimental Setup:

• Models: 23 models spanning lightweight (e.g., Haiku, Flash Lite, Nano) to frontier (e.g., Opus, Pro, GPT-5.4) tiers. All models support multimodal inputs.
• Procedure: Tasks were rendered as HTML pages with images. Inputs were sent via the OpenRouter API. Each model performed 20 independent runs per task.
• Scoring: Answers were scored as correct only if they exactly matched the ground truth; no partial credit was awarded.
• Treatments: A 2×2 design tested the sensitivity of LLMs to verbal incentives and persona framing:
  • T0 (Control): Neutral prompt.
  • T1: Standard persona, no monetary incentive.
  • T2: Standard persona, verbal monetary incentive ($0.50 per correct answer).
  • T3: Human persona, no monetary incentive.
  • T4: Human persona, verbal monetary incentive.
• Cost Analysis: The study calculated the total API cost per task based on input/output token usage and provider pricing, comparing it against typical online-experiment piece rates ($0.25 per correct answer).

Key Contributions and Results

1. Task Difficulty and Automation Susceptibility

Performance varies significantly by task type.

• Highly Automatable: Tasks involving small, structured inputs and elementary operations (Addition, Number Pair Summation, Letter Decoding, Sequence Completion) are solved with near-perfect accuracy (90–100%) by most models.
• Resistant Tasks: Tasks requiring complex visual search, enumeration, or combinatorial reasoning (Sudoku, Counting Zeros, String Entry) remain challenging, with accuracies ranging from 25% to 37% even for top-tier models.
• Intermediate: Distorted Text Transcription falls in the middle (77.4%), limited by perceptual noise rather than pure reasoning.

2. Model Performance and Evolution

• Tier vs. Provider: Performance is determined largely by model tier (e.g., Opus/Pro vs. Sonnet/Flash) rather than the provider. Top-tier models achieve >80% average accuracy, while mid-tier models achieve ~60%.
• Rapid Improvement: Newer releases consistently outperform predecessors within the same tier. The most significant gains are observed in mid-range tiers, suggesting that the set of models capable of automating these tasks is expanding rapidly and becoming more accessible.
• Cherry-Picking: A strategic agent selecting the best-performing model for each specific task could achieve an average accuracy of 93.1%, significantly higher than using a single model (82.5%). This implies that even "resistant" tasks can be solved at high rates if the agent has the knowledge to select the optimal tool.

3. Economic Viability of Substitution

The substitution of human effort with LLMs is economically dominant.

• Cost Efficiency: Even the most expensive models tested cost less than $0.40 per task (20 runs). Cheaper models cost under $0.05.
• Net Gain: At a conservative piece rate of $0.25 per correct answer, every tested model generates a positive net gain (ranging from ~$2.66 to $3.92 per task).
• Conclusion: Replacing human participants with LLMs is not only feasible but highly profitable for a profit-seeking agent.

4. Insensitivity to Incentives and Persona

• No Effect: Verbal offers of monetary incentives or instructions to "behave as a human" had no statistically significant effect on LLM performance.
• Mechanism: Unlike humans, LLMs do not have an "effort margin" to adjust based on marginal returns. Their computational process is fixed by architecture and prompt; the text of the prompt does not alter the underlying inference procedure.
• Diagnostic Potential: The absence of learning effects, fatigue, or incentive responsiveness in LLMs provides a potential diagnostic tool. A flat accuracy profile over repeated trials, combined with stable variance, may signal automated rather than human completion.

Significance and Recommendations

The paper establishes a boundary condition for the use of real-effort tasks in unsupervised settings: when participants can cheaply outsource task completion to an LLM, observed performance may no longer reflect genuine human effort. The authors argue that the construct validity of these tasks is compromised in the era of AI.

Based on these findings, the authors offer three practical recommendations for researchers:

1. Task Selection: Prefer tasks requiring perceptual skills (e.g., counting zeros, string entry) over purely computational ones (e.g., addition) when participant authenticity is a concern, as current LLMs struggle more with the former.
2. Incentive Screening: Use incentive language as a screening device by asking participants to perform similar tasks with and without monetary incentives; divergent behavior may help distinguish humans from LLMs.
3. Dynamic Monitoring: Monitor within-session dynamics, as the absence of characteristic human patterns of learning and fatigue may help identify automated responses.

The study concludes that as LLMs continue to improve, particularly in mid-tier models, the set of real-effort tasks vulnerable to automation will likely expand, necessitating a re-evaluation of how effort is measured in experimental economics.