AgentDS Technical Report: Benchmarking the Future of Human-AI Collaboration in Domain-Specific Data Science

1. Problem Statement

While Large Language Models (LLMs) and AI agents have demonstrated impressive capabilities in automating code generation and executing standard machine learning pipelines, a critical gap remains: the extent to which autonomous AI agents can match human experts on domain-specific data science tasks.

Current research and benchmarks often focus on generic code generation or tabular data, neglecting the nuanced, domain-specific knowledge required for real-world problems. It is unclear whether AI agents can effectively leverage domain insights, handle multimodal data (images, text, structured files), or make strategic decisions regarding model selection and generalization. The paper addresses the question: To what extent do human experts outperform autonomous AI agents on domain-specific tasks, and in which aspects does human expertise provide a decisive advantage?

2. Methodology: The AgentDS Benchmark

To evaluate this, the authors introduced AgentDS, a benchmark and competition designed to test both autonomous AI agents and human-AI collaboration.

A. Benchmark Design

• Scope: The benchmark consists of 17 challenges across six industries: Commerce, Food Production, Healthcare, Insurance, Manufacturing, and Retail Banking.
• Core Principles:
  1. Domain-Specific Complexity: Challenges are designed so that generic pipelines fail. Success requires understanding specific features and processing steps unique to the industry.
  2. Multimodal Integration: Unlike standard tabular benchmarks, AgentDS challenges include additional modalities (images, PDFs, JSON, text) that encode complementary information. Effective solutions require extracting insights from these non-tabular sources.
  3. Real-World Plausibility: Data is synthesized but mirrors genuine industry relationships, constraints, and correlations, validated against domain literature.
• Data Curation: A four-stage process involving domain research, data generation (embedding latent variables into additional modalities), performance bound calibration, and documentation.

B. Competition Setup

• Participants: 29 teams (80 participants) competed over 10 days.
• Rules: Teams could use any AI tools, allowing for a spectrum of workflows from fully autonomous to human-guided.
• Evaluation:
  • Metrics: Domain-specific metrics (e.g., Macro-F1, RMSE, NDCG@10).
  • Scoring: A quantile-based scoring system normalizes performance across different metrics and scales (0 to 1), ensuring fair comparison.
  • Aggregation: Scores are averaged across challenges within a domain, then across all six domains to produce an overall capability score.

C. Baselines

To establish a performance floor, two AI-only baselines were evaluated without human intervention:

1. Direct Prompting (GPT-4o): Generates end-to-end code in a single prompt.
2. Agentic Coding (Claude Code): Operates autonomously, iteratively writing, executing, and refining code within a fixed time budget.

3. Key Contributions

1. AgentDS Benchmark: The first comprehensive benchmark specifically targeting domain-specific reasoning and multimodal integration in data science, moving beyond generic code generation.
2. Empirical Comparison: A systematic comparison of three paradigms: Pure AI (Direct Prompting), Pure AI (Agentic), and Human-AI Collaboration.
3. Qualitative & Quantitative Analysis: Detailed analysis of participant submissions and AI baselines, identifying specific failure modes of AI and the irreplaceable value of human expertise.
4. Open Resources: Release of the benchmark datasets, code, and competition results to the community.

4. Key Results

The competition yielded three primary findings:

A. AI Agents Struggle with Domain-Specific Reasoning

• Performance Gap: The AI-only baselines performed near or below the median of human participants.
  • GPT-4o (Direct Prompting): Ranked 17th/29 (Quantile Score: 0.143), performing at or below the median in all domains.
  • Claude Code (Agentic): Ranked 10th/29 (Quantile Score: 0.458), significantly better than GPT-4o but still well below the top human teams.
• Failure Modes:
  • Inability to Leverage Multimodal Signals: AI agents failed to extract useful features from images or text, often ignoring these modalities entirely.
  • Over-reliance on Generic Pipelines: Agents defaulted to standard preprocessing and gradient-boosted models, failing to adapt to domain-specific constraints.
  • Autonomy Limits: Several teams abandoned fully autonomous frameworks due to high costs and diminishing returns, shifting to human-guided workflows.

B. Human Expertise is Irreplaceable

Human data scientists provided value in areas where AI failed:

• Strategic Diagnosis: Humans identified structural weaknesses (e.g., distribution shifts, miscalibrated peaks) that AI missed.
• Domain Knowledge Injection: Humans engineered features based on external knowledge (e.g., medical normal ranges, credit risk thresholds) not inferable from data alone.
• Filtering AI Suggestions: Teams actively rejected AI-generated pipelines that looked complex but reduced validation scores.
• Generalization Judgment: Humans made strategic model selection decisions to avoid overfitting, prioritizing robustness over immediate score optimization.

C. Human-AI Collaboration is Superior

• Top Performance: The strongest solutions came from Human-AI collaboration, not AI alone.
• Workflow: Successful teams used a "Human for Direction, AI for Acceleration" model. Humans guided problem formulation, feature engineering, and strategy, while AI handled code generation, experimentation, and iteration.
• Feedback Loops: Effective collaboration involved tight, human-initiated feedback loops where humans diagnosed failures and directed the AI to refine hypotheses.

5. Significance and Implications

• Challenging the Automation Narrative: The results challenge the assumption that AI will soon enable fully autonomous data science. Instead, the future lies in augmented intelligence where AI supports, rather than replaces, human experts.
• Redefining AI Development: The findings suggest that the next generation of AI for data science should focus on collaborative tools that support human reasoning and domain integration, rather than striving for full autonomy in complex, domain-specific tasks.
• Benchmarking Standard: AgentDS provides a rigorous framework for evaluating future AI agents, emphasizing that true capability requires handling multimodal data and applying domain-specific logic, not just generating syntactically correct code.

In conclusion, the AgentDS report demonstrates that while AI agents can accelerate implementation, human expertise remains the critical bottleneck and differentiator in solving complex, real-world data science problems. The most effective path forward is a symbiotic partnership where human strategic reasoning guides AI computational power.