🤖 AI

AcademiClaw: When Students Set Challenges for AI Agents

This paper introduces AcademiClaw, a bilingual benchmark of 80 complex, real-world academic tasks curated from student submissions to evaluate and diagnose the limitations of current AI agents in handling long-horizon, multi-domain challenges, revealing that even frontier models achieve only a 55% pass rate.

Original authors: Junjie Yu, Pengrui Lu, Weiye Si, Hongliang Lu, Jiabao Wu, Kaiwen Tao, Kun Wang, Lingyu Yang, Qiran Zhang, Xiuting Guo, Xuanyu Wang, Yang Wang, Yanjie Wang, Yi Yang, Zijian Hu, Ziyi Yang, Zonghan Zhou

Published 2026-05-06

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Junjie Yu, Pengrui Lu, Weiye Si, Hongliang Lu, Jiabao Wu, Kaiwen Tao, Kun Wang, Lingyu Yang, Qiran Zhang, Xiuting Guo, Xuanyu Wang, Yang Wang, Yanjie Wang, Yi Yang, Zijian Hu, Ziyi Yang, Zonghan Zhou, Binghao Qiang, Borui Zhang, Chenning Li, Enchang Zhang, Feifan Chen, Feng Jian, Fengyin Sun, Hao Qiu, Hao Zheng, Haoran Zhu, Hongyu Liu, Jianbin Deng, Jiaxin Song, Jiaying Chi, Jiayou Shi, Jie Fang, Jinghui Zhong, Jingyu Zhou, Jinze Li, Junfeng Yi, Junyan Yu, Junzhi Xue, Ni Song, Pengyi Chen, Qi Chen, Quansheng Li, Rui Tao, Shenghai Gong, Shenhang Lu, Tianqi Shen, Tianxiang Zhu, Tiehan Kang, Tingyu Li, Wendi Wu, Xiao Shen, Xiao Zhou, Xiaotao Zhang, Xinrong Li, Xuankun Yang, Xun Zhang, Yan Li, Ye Lu, Yi Wang, Yibo Zhou, Yichi Zhang, Yihao Sun, Yijun Huang, Yixin Zhu, Yixuan Wu, Yuchen Sun, Yue Wu, Yuheng Sun, Yukun Li, Yutian Tu, Yuxuan Qin, Yuzhuo Wu, Zeyu Li, Zhengyu Lou, Zhenning Ran, Zizhu He, Pengfei Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you've built a super-smart robot assistant. So far, you've tested it by asking it to do simple chores: "Sort these emails," "Add this meeting to my calendar," or "Find a recipe for pasta." The robot is great at these tasks. It's like a very efficient personal secretary.

But here's the problem: We don't actually know if this robot can do real, hard work. Can it solve a complex math problem? Can it build a video game from scratch? Can it debug a broken computer chip? Until now, no one has really asked the robot these questions in a fair, standardized way.

That's what this paper, AcademiClaw, is all about. It's a new "final exam" for AI agents, designed not by researchers in a lab, but by university students who are actually struggling with their homework and projects.

Here is a breakdown of the paper using simple analogies:

1. The Problem: The "Secretary" vs. The "Engineer"

Think of current AI benchmarks (tests for AI) like a driver's license test for a grocery delivery driver. They check if you can parallel park and stop at a red light. That's useful, but it doesn't tell you if you can race a Formula 1 car or repair a jet engine.

The paper argues that existing tests for the "OpenClaw" AI system only check for "assistant-level" skills (like sorting files). They leave a huge gap: Academic-level skills. These are the deep, complex tasks that require years of study, like proving a math theorem or training a complex AI model.

2. The Solution: A "Student-Submitted" Exam

Instead of researchers making up fake problems, the authors asked 230 university students to submit real problems they faced.

The Scenario: A student tries to use an AI to help with a difficult coding project or a research paper. The AI fails or gets stuck. The student says, "This is too hard for the AI."
The Filter: Experts reviewed these 230 submissions and picked the best 80.
The Result: A test suite that covers 25 different fields, from Olympiad-level math to GPU-heavy video game physics and debugging complex software.

The Analogy: Imagine a chef's competition. Instead of judges asking the chefs to "chop an onion," the students (the customers) say, "I need a 5-course meal that uses ingredients from three different continents, cooked in a specific style, and I need it ready in 40 minutes." That's the level of difficulty AcademiClaw introduces.

3. The Test Environment: The "Safe Sandbox"

To make sure the AI doesn't break anything or cheat, every task happens in a digital sandbox (a locked-down computer environment called a Docker container).

The AI is given a task.
It can read files, write code, run programs, and even use a web browser.
It has to do everything itself, step-by-step.
Crucial Detail: Some tasks require a GPU (a powerful graphics card used for heavy math and AI training). This is like asking the robot to lift a heavy weight; most previous tests only asked it to lift a feather.

4. How They Grade It: The "Multi-Layered Judge"

You can't just ask, "Did it finish?" because the answer might be "Yes, but the code is broken."
The paper uses a 6-in-1 grading system:

Pattern Matching: Did it write the right words?
Code Execution: Does the program actually run without crashing?
AI Judge: Another AI reads the report and grades it like a teacher.
Vision AI: Can it "see" if a chart or image looks right?
Browser Testing: Did it actually build a working website?
Structure Check: Is the file format correct (like a JSON or CSV)?

They also have a Safety Audit (like a security guard) to make sure the AI didn't try to delete important files or hack into things it shouldn't.

5. The Results: The "Reality Check"

The authors tested six of the smartest AI models available (like Claude, GPT, and Gemini) on this new exam.

The Score: Even the best AI only passed 55% of the tasks. That means they failed nearly half the time on problems that university students found difficult.
The "Overthinking" Trap: The paper found something weird. Some AIs used 5 times more "brain power" (tokens) than others but didn't get better results.
- Analogy: Imagine two students taking a test. Student A reads the question carefully, thinks for a minute, and writes a perfect answer. Student B panics, writes 10 pages of rambling nonsense, and gets the answer wrong. The paper found that more effort does not equal better quality.
Different Personalities: The AIs had different "personalities":
- The Reader: (Claude) Reads everything first, thinks, then acts. High quality, but slower.
- The Hammer: (Gemini) Just starts hitting things (running code) immediately, hoping it works. Fast, but often breaks things and fails safety checks.
- The Minimalist: (GPT) Does the bare minimum, but gets surprisingly good results.

6. The Big Takeaway

The paper concludes that while AI is great at being a "digital secretary," it is still very shaky when asked to be a "researcher" or "engineer."

The Gap: There is a massive disconnect between what AI can do today and what real-world academic and professional work requires.
The Safety Issue: The AI that tried to "guess and check" its way through problems (Gemini) was also the one most likely to break safety rules.
The Future: The authors hope this new test (AcademiClaw) will help developers build AI that is actually capable of solving the hard, complex problems we face in the real world, not just the easy ones.

In short: The paper built a "hard mode" test for AI using real student homework. The results show that even the smartest AIs are still struggling with the deep, complex work of the real world, and using more computing power doesn't necessarily make them smarter.

Technical Summary: AcademiClaw

Problem Statement

While the OpenClaw ecosystem has rapidly evolved as a leading open-source framework for autonomous agents, existing evaluation benchmarks within this ecosystem have remained limited to assistant-level tasks (e.g., email triage, calendar management, project scaffolding). These tasks, while practically useful, lack the depth of domain expertise and sustained multi-step reasoning required for real-world academic and professional workflows. Consequently, there is a critical gap in understanding the capabilities of OpenClaw agents when faced with complex, knowledge-intensive problems such as mathematical proofs, GPU-intensive model training, cross-framework debugging, and scientific data analysis. Current benchmarks fail to systematically evaluate agents on these "academic-level" challenges, leaving the community without actionable diagnostic signals to advance agents beyond their current assistant-oriented scope.

Methodology

Task Collection: A Bottom-Up Approach

Unlike traditional benchmarks where tasks are designed top-down by researchers, AcademiClaw employs a user-sourced, bottom-up collection strategy.

Source: 230 candidate tasks were submitted by university students based on their real academic workflows (homework, research projects, competitions, personal projects) where they found current AI agents ineffective.
Filtering: A rigorous two-stage expert review process distilled these candidates into a final set of 80 high-quality tasks. Experts evaluated tasks on prompt clarity, rubric correctness, scoring reproducibility, difficulty calibration, and domain balance.
Scope: The final dataset covers 25+ professional domains, ranging from CMO (Chinese Mathematical Olympiad) proofs and IOL (International Linguistics Olympiad) problems to GPU-intensive reinforcement learning and full-stack system debugging. It includes 16 GPU-intensive tasks requiring CUDA execution, a feature absent in prior OpenClaw benchmarks.
Bilingual Nature: The benchmark is bilingual (49 English, 31 Chinese), with Chinese tasks being natively Chinese and requiring culturally grounded competence (e.g., adapting Tang poetry, understanding Shuangpin encoding) rather than simple translation of instructions.

Execution Environment

Isolation: Each task executes within an isolated Docker sandbox with a two-layer image hierarchy (base CPU/GPU environment + per-task dependencies).
Agent Framework: Tasks are solved using the OpenClaw agent framework, which provides a unified tool palette including file operations, shell execution, web search, and headless browser automation.
Evaluation Protocol: Agents receive a single attempt per task with no retries. The evaluation rubric is withheld from the agent during execution.

Multi-Dimensional Evaluation Framework

AcademiClaw moves beyond binary pass/fail metrics by employing a multi-dimensional rubric (0–100 scale) combining six complementary verification techniques:

Pattern Matching: Regex, keyword detection, and AST parsing.
Code Execution: Compiling and running unit tests against reference solutions.
LLM-as-Judge: Evaluating open-ended outputs (reports, creative writing) with deterministic heuristic fallbacks.
Vision LLM Assessment: Comparing rendered graphics or GUI screenshots against references.
End-to-End Browser Testing: Using Playwright to interact with dynamic web elements.
Structured-Output Validation: JSON schema checks, CSV verification, and BibTeX parsing.

Additionally, a five-category safety audit (S1–S5) evaluates destructive operations, information leakage, boundary compliance, privilege escalation, and supply-chain risks.

Key Contributions

First Academic-Level Benchmark for OpenClaw: AcademiClaw is the first benchmark in the ecosystem to target academic-level complexity, comprising 80 tasks sourced entirely from university students rather than researchers. It uniquely includes 16 GPU-intensive tasks.
Novel Evaluation Framework: The paper introduces a multi-dimensional scoring system combining six verification techniques and a five-category safety audit, providing fine-grained diagnostic signals beyond aggregate metrics.
Systematic Empirical Analysis: The authors provide a comprehensive evaluation of six frontier models (Claude Opus 4.6, Claude Sonnet 4.6, GPT-5.4, Gemini 3.1 Pro, Qwen3.5-397B, and MiniMax M2.7), revealing phenomena hidden by lighter-weight benchmarks.

Experimental Results

Overall Performance

Even the best-performing model, Claude Opus 4.6, achieved only a 55% pass rate (score ≥75) with an average score of 71.9. The gap between the strongest and weakest models (MiniMax M2.7) was 8.8 points in average score, but the pass rate gap was wider (55.0% vs. 37.5%). Notably, 23 of the 80 tasks (28.8%) defeated all six models, with 8 tasks where every model scored below 50.

Domain and Task-Level Patterns

Task Difficulty Dominance: The variation in performance across task categories (26.3-point spread) far exceeded the variation across models (8.8-point spread). STEM Reasoning was the most challenging category (mean score ~50.6), while Language & Creativity was the easiest (mean score ~76.9).
Capability Boundaries: Specific tasks revealed sharp capability boundaries. For example, on a family tree extraction task from One Hundred Years of Solitude, scores ranged from 86–92 for some models to just 3 for others (a 90-point gap). Similarly, a TensorFlow-to-PyTorch migration task resulted in a catastrophic failure (score 0) for GPT-5.4 while others scored 74–90.

Behavioral Phenotypes

The study identified three distinct behavioral strategies among the models:

Read-First (Claude Opus 4.6): Prioritizes file reading (41% of tool calls) before execution. This strategy yielded the highest quality ("comprehension premium") but required moderate token usage.
Execute-First (Gemini 3.1 Pro): Relies heavily on shell execution (74.3% of calls) with a trial-and-error approach. This resulted in the highest token consumption (2.8M tokens, 5.4× more than GPT-5.4) but lower quality and the lowest safety score (74.9), largely due to boundary violations.
Minimalist (GPT-5.4): Uses the fewest tool calls (19) and tokens (525K) with the shortest execution time, achieving competitive quality through internal inference.

Token-Quality Disconnect

A critical finding is the near-zero correlation ( $r = -0.03$ ) between token consumption and task quality across all models. This suggests that current agents lack effective mechanisms to determine when to stop iterating, leading to "overthinking" penalties where increased computational effort does not translate to better solutions.

Safety Analysis

Safety and quality were found to be largely independent ( $|r| < 0.29$ ). However, boundary compliance (S3) was the decisive safety dimension, showing a 53-point gap between the safest (Claude models, ~84) and least safe (Gemini, 31.6) models. This correlates with the "execute-first" phenotype, where frequent shell executions increase the risk of accessing files outside the designated workspace.

Significance and Claims

The authors position AcademiClaw as a necessary evolution in agent evaluation, shifting the focus from assistant-level routines to the complex, open-ended demands of real-world academic and professional practice.

Diagnostic Value: The benchmark provides fine-grained signals that reveal specific capability gaps (e.g., formal reasoning vs. generative tasks) and behavioral inefficiencies (e.g., the token-quality disconnect) that aggregate metrics obscure.
Community Resource: By open-sourcing the data and code, the authors aim to drive progress toward agents that are more capable and versatile, helping the OpenClaw community move beyond its current assistant-oriented scope.
Modest Scope: The authors acknowledge limitations, noting the current dataset is sourced from a single university's CS undergraduates and relies on single-attempt evaluations. They frame the benchmark as a foundational step toward more comprehensive, multi-disciplinary, and multi-trial evaluation protocols in the future.