💬 NLP

AgentExpt: Automating AI Experiment Design with LLM-based Resource Retrieval Agent

The paper presents AgentExpt, a framework that automates AI experiment design by leveraging a large-scale dataset of 100,000 papers to train a collective perception-enhanced retriever and a reasoning-augmented reranker, significantly outperforming prior baselines in recommending suitable baselines and datasets.

Original authors: Yu Li, Lehui Li, Qingmin Liao, Fengli Xu, Yong Li

Published 2026-03-24

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yu Li, Lehui Li, Qingmin Liao, Fengli Xu, Yong Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef trying to create a new, revolutionary dish. You have a brilliant idea for a flavor combination, but before you start cooking, you face two massive hurdles:

The Recipe Problem: Which famous chefs (baselines) should you try to copy or improve upon to prove your dish is special?
The Ingredient Problem: Which specific market or farm (dataset) should you buy your vegetables from to ensure your ingredients are fresh and comparable to others?

In the world of AI research, scientists face this exact problem every day. There are thousands of "recipes" (algorithms) and "farms" (datasets). Choosing the wrong ones can make your research look bad, even if your idea is genius. Usually, scientists have to spend weeks manually reading papers to figure out what worked for others.

Enter "AgentExpt": The AI Sous-Chef.

This paper introduces a new system called AgentExpt. Think of it as a super-smart, tireless research assistant that doesn't just search for keywords; it understands the culture and history of scientific cooking. Here is how it works, broken down into simple steps:

1. The Massive Cookbook (The Knowledge Base)

First, the creators built a giant library. They didn't just look at recipe books; they looked at 108,000 actual research papers published in the last decade. They read the "Experiments" section of every single one to see exactly which ingredients and recipes the authors actually used.

The Analogy: Imagine scanning every restaurant review in the world to build a list of exactly which chefs and which farms the top restaurants actually use, rather than just what they say they use on their website.

2. The "Gossip" Network (Collective Perception)

Most search engines only look at a recipe's title or description (the "self-description"). But AgentExpt looks at what other people say about it.

The Analogy: If you want to know if a specific tomato variety is good, you don't just read the seed packet (the self-description). You ask the other chefs: "Hey, did anyone use this tomato in a spicy sauce? Did it fall apart?"
AgentExpt gathers these "gossip" stories (citation contexts) from thousands of papers. It creates a "Collective Perception" profile. It knows that "Dataset X" is great for small groups but terrible for big crowds, not because the dataset says so, but because the community says so.

3. The "Chain of Trust" (Reasoning-Augmented Reranking)

After the assistant finds a short list of good candidates, it needs to pick the absolute best ones. It does this by looking at chains of connections.

The Analogy: Imagine you want to make a pizza. You find a great cheese. But how do you know it's the right cheese? You look for a chain: "Chef A used this cheese with this specific flour, and Chef B used that flour with this specific oven, and they both made amazing pizzas."
AgentExpt traces these invisible threads (Paper → Dataset → Paper → Baseline). It uses an AI brain to reason: "These two things have been used together successfully many times before. They are a perfect match for your new idea."

Why is this a big deal?

Before AgentExpt, picking the right tools for an AI experiment was like guessing in the dark. You might pick a popular tool that doesn't actually fit your specific problem.

It's faster: It cuts down weeks of research to seconds.
It's smarter: It doesn't just match words; it matches intent and history.
It's transparent: It doesn't just give you a list; it explains why it chose them, showing you the "chain of trust" (e.g., "I chose this because 50 other successful papers used it with your specific setup").

The Result

When they tested it, AgentExpt was much better at finding the right tools than any previous system. It found the "hidden gems" that other search engines missed because it understood the community's real-world experience, not just the official marketing brochures.

In short: AgentExpt turns the chaotic, overwhelming world of AI research into a guided tour, showing you exactly which tools and ingredients the community trusts, so you can focus on the creative part of your discovery.

1. Problem Statement

In modern AI research, selecting appropriate baselines (comparison methods) and datasets is a critical, high-stakes decision that determines the validity and reproducibility of empirical conclusions. However, this process faces three major challenges:

Limited Coverage: Existing recommendation systems often rely on curated portals (e.g., Papers With Code) which do not reflect the actual, diverse usage of resources across the broader scientific literature.
Context-Dependent Suitability: Suitability is rarely explicit in metadata. A baseline or dataset suitable for one task may be misleading for another. Current methods often rely on surface-level semantic similarity (first-person descriptions) and fail to capture how artifacts are actually used in practice (third-person evidence).
Decoupled Selection: Baseline and dataset selection are typically treated as independent tasks, ignoring the crucial joint compatibility and interaction chains between them that often define a successful experimental setup.

2. Methodology: The AgentExpt Framework

AgentExpt is a comprehensive framework designed to automate baseline and dataset recommendation through a two-stage pipeline: Collective Perception-Augmented Retrieval and Reasoning-Augmented Reranking.

A. Dataset Construction (The Knowledge Base)

To address the coverage gap, the authors constructed a large-scale, literature-grounded knowledge base:

Scale: Aggregated 108,825 accepted papers from ten flagship AI venues (NeurIPS, ICML, ICLR, ACL, etc.) spanning 2015–2025.
Extraction Pipeline: An automated pipeline using MinerU for parsing and LLMs for extraction identified 116,970 baselines and 68,316 datasets.
Validation: A two-stage filtering process (rule-based + LLM verification) resolved naming variations and linked papers to the specific components they actually used, creating ~543k paper-baseline and ~326k paper-dataset links.

B. Module 1: Collective Perception-Augmented Retrieval

This stage aims to improve recall by capturing "community usage signals" beyond simple text matching.

Dual-View Representation: For each candidate (baseline/dataset), the system constructs a representation $x_t$ $x_{t}$ by concatenating:
1. Self-Description: The artifact's own abstract or introduction (First-person).
2. Collective Perception (CP): A summary of how the community uses the artifact, distilled from citation contexts in downstream papers (Third-person).
CP Synthesis: Using GPT-4o, the system aggregates thousands of citation contexts into a concise profile describing when and why the artifact is used (e.g., specific metrics, splits, or co-usage patterns).
Training: A dense retriever (based on Qwen3-Embedding) is fine-tuned using contrastive learning to align the query (research idea) with these dual-view representations.

C. Module 2: Reasoning-Augmented Reranking

This stage addresses the decoupling of baselines and datasets by leveraging their interaction history.

Interaction Chains: The system constructs a heterogeneous graph of Papers, Baselines, and Datasets. It extracts interaction chains (e.g., $Paper \to Dataset \to Paper' \to Baseline$ ) to identify latent co-usage patterns.
Reasoning Prior: For a given query and candidate, the system retrieves the top-3 interaction chains and their co-usage counts.
Reranking: A Large Language Model (LLM) is fine-tuned via Supervised Fine-Tuning (SFT) to act as a reranker. It takes the query and the interaction chain evidence as input, generating interpretable justifications and a refined ranking. This allows the model to reason about experimental suitability rather than just semantic similarity.

3. Key Contributions

AgentExpt Framework: A unified system that automates baseline and dataset recommendation by integrating community-level perception and interaction-chain reasoning.
Large-Scale Knowledge Base: The curation of a high-quality dataset linking 108k+ papers to their actual experimental components, significantly outperforming existing portal-based datasets in coverage and accuracy.
Collective Perception Mechanism: A novel method to augment self-descriptions with third-person citation contexts, effectively modeling the "community consensus" on resource suitability.
Reasoning-Augmented Reranker: A novel approach that encodes baseline-dataset interaction chains as reasoning traces, enabling the LLM to provide experimentally grounded, interpretable recommendations.

4. Experimental Results

The framework was evaluated against seven strong baselines (including BM25, SciBERT, DataFinder, and graph-based methods) on both Paper-Baseline and Paper-Dataset recommendation tasks.

Performance Gains: AgentExpt achieved state-of-the-art performance. Compared to the strongest baseline (SymTax), it yielded:
- +5.85% average gain in Recall@20.
- +7.90% average gain in HitRate@10.
- Specific improvements included +7.23% Recall@20 for baselines and +8.27% HitRate@10 for datasets.
Ablation Studies:
- Collective Perception: Removing CP caused the largest performance drop (~17.8% Recall@20 loss), proving that third-person usage signals are critical.
- Interaction Chains & Reasoning: Removing the reasoning-augmented reranker significantly reduced HitRate, confirming that understanding the relationship between baselines and datasets is essential for precision.
Efficiency: Despite the two-stage process, the inference latency is 0.367s per query on an A800 GPU, well within the threshold for real-time interactive research assistance.

5. Significance and Impact

Scientific Rigor: AgentExpt moves beyond "popularity" or "keyword matching" to recommend resources based on actual experimental usage patterns, enhancing the reproducibility and comparability of AI research.
Automation of Research Workflows: It fills a critical gap in automated scientific discovery (which currently focuses on idea generation or paper writing) by automating the crucial experimental design phase.
Interpretability: Unlike black-box retrieval, AgentExpt provides reasoning traces explaining why a specific baseline or dataset was chosen, making the decision process auditable for researchers.
Infrastructure: The released knowledge base serves as a foundational resource for future research into scientific literature mining and automated experimental design.

In summary, AgentExpt demonstrates that leveraging community perception and structured interaction reasoning via LLMs can significantly automate and improve the reliability of experimental design in AI research.