🤖 machine learning

ArtifactLinker: Linking Scientific Artifacts for Automatic State-of-the-Art Discovery

The paper introduces ArtifactLinker, a two-stage framework that models scientific artifacts as a graph to automatically discover state-of-the-art models for datasets by ranking potential links with GNNs or LLMs and verifying them via LLM-based coding agents, validated by the new ArtifactBench benchmark.

Original authors: Haofei Yu, Jiaxuan You, Peter Clark, Bodhisattwa Prasad Majumder, Kyle Richardson

Published 2026-05-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Haofei Yu, Jiaxuan You, Peter Clark, Bodhisattwa Prasad Majumder, Kyle Richardson

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the world of scientific research as a massive, chaotic library. In this library, there are millions of books (datasets) and millions of different tools or machines (models) that people have built to read and understand those books. The problem? No one knows exactly which machine works best with which book. Researchers have to guess, try, and fail, wasting a lot of time and energy.

This paper introduces a new system called ArtifactLinker (and a new "test library" called ArtifactBench) designed to solve this problem automatically. Think of it as a super-smart librarian who can instantly figure out the best machine-book pairings without having to physically test every single combination.

Here is how it works, broken down into simple steps:

1. The Map: The "Artifact Graph"

First, the researchers turned the messy library into a giant, structured map (a graph).

The Nodes (Dots): Each dot is either a Model (a tool), a Dataset (a book), a Paper (a recipe), or Code (the instructions).
The Edges (Lines): The lines connecting them show relationships. For example, a line connects a Model to a Dataset if someone has already tested them together. The line has a "score" on it (like a grade of 0.85) showing how well they worked.

The goal is to find the missing lines on this map. Specifically, they want to find a Model and a Dataset that haven't been tested together yet, but if they were, the Model would get a better score than any other Model currently used on that Dataset. This is called finding the new "State-of-the-Art" (SOTA).

2. The Two-Stage Detective: "Rank and Verify"

Since there are too many possible combinations to test one by one (it would take forever), the system uses a two-step detective process:

Step 1: The Ranker (The Intuition)
Imagine a detective who looks at the map and uses their experience to guess which missing connections are most likely to be successful.

This part uses Graph Neural Networks (GNNs). Think of these as a system that learns the "shape" of the library. It knows that if Model A is similar to Model B, and Model B works well on Book X, then Model A might also work well on Book X.
It doesn't run any code yet; it just looks at the patterns and creates a "Top 10" list of the most promising Model-Dataset pairs to check.

Step 2: The Verifier (The Reality Check)
Once the Ranker picks the top candidates, the system switches to a team of AI Agents (robotic assistants).

These agents actually write and run the computer code to test the Model on the Dataset.
The "Self-Evolving" Trick: If an agent gets stuck (e.g., the code crashes because of a weird file format), it doesn't just give up. It writes down the solution in a "memory book." The next time an agent faces a similar problem, it checks the memory book and uses the solution immediately. This stops them from making the same mistakes twice.

3. The Results: What Did They Find?

The researchers tested this system on a specific type of task called Natural Language Inference (NLI), which is basically asking a computer to understand if one sentence logically follows another.

Better than guessing: The "Ranker" (the GNN) was surprisingly good at predicting which pairs would work, often doing better than just asking a large language model (like a very smart chatbot) to guess based on text alone.
Saving time: By using the Ranker to filter out the bad guesses, the system only had to run the expensive, time-consuming code tests on a tiny fraction of the possibilities.
Real discoveries: In their final test, the system actually found a Model that had never been tested on a specific Dataset before. When they ran the test, it got a score that was nearly the best ever recorded for that task, proving the system could find new, high-quality results automatically.

The Big Picture

The paper claims that by treating scientific research like a connected map and using a "guess-then-check" strategy with smart AI agents, we can automate the discovery of the best tools for the job. It turns the slow, manual process of "trying everything" into a fast, targeted search that finds the winners without wasting resources on the losers.

In short: They built a system that looks at the history of scientific experiments, predicts the next big breakthrough, and then automatically runs the experiment to prove it's real.

Technical Summary: ArtifactLinker

Problem Definition

The paper addresses the challenge of automatic State-of-the-Art (SOTA) discovery within the rapidly expanding ecosystem of scientific artifacts, specifically focusing on the HuggingFace Hub. While platforms like HuggingFace host vast numbers of models and datasets, researchers face a combinatorial explosion in potential model–dataset pairs, making exhaustive manual evaluation infeasible.

The authors formalize this challenge as a link prediction task on a heterogeneous "artifact graph." In this graph:

Nodes represent models, datasets, papers, and codebases.
Edges represent relationships such as fine-tuning, referencing, and evaluation.
Edge Attributes include performance metrics (e.g., F1 scores).

The core objective is to identify missing links (unobserved model–dataset pairs) that, upon verification, yield a performance score strictly exceeding the current SOTA for a given dataset. This task is complicated by two primary factors:

Task Ambiguity: The lack of a formal definition and rigorous benchmarks for "automatic discovery."
Scalability: The search space is too large for exhaustive code execution; verifying every potential pair is computationally intractable.

Methodology: ArtifactLinker

The authors propose ArtifactLinker, a two-stage framework designed to efficiently navigate the artifact graph and discover new SOTA results.

1. Stage One: Link Ranking (Graph Modeling)

The first stage filters the vast search space by ranking promising unobserved model–dataset pairs without executing code.

Graph Construction: A heterogeneous graph is constructed from HuggingFace, enriched with metadata from arXiv and GitHub. Nodes are initialized with semantic embeddings derived from LLM-summarized documentation (model cards, READMEs), with a strict protocol to redact quantitative metrics to prevent data leakage.
Addressing Selection Bias: The authors identify a selection bias problem where standard predictors trained only on observed pairs estimate $E[Y|S=1]$ (performance given the pair was evaluated), ignoring the probability that a pair is fundamentally incompatible ( $S=0$ ).
Factorized Prediction: To correct this, the ranking function is formulated as the product of two components:
$\hat{f}(m, d) = P(S_{md}=1) \cdot E[Y_{md} | S_{md}=1]$
- Link Predictor ( $\phi_l$ ): Estimates the compatibility probability $P(S_{md}=1)$ .
- Attribute Predictor ( $\phi_a$ ): Estimates the conditional performance score $E[Y_{md} | S_{md}=1]$ .
Architecture: A shared Graph Neural Network (GNN) encoder (specifically GATv2) processes the graph topology. Two specialized heads (bilinear for link prediction, regression for attribute prediction) are trained jointly. This joint training allows the attribute predictor to learn from the compatibility signals of the link predictor, providing structural knowledge about viable boundaries.

2. Stage Two: Link Verification (Self-Evolving Multi-Agent)

The top-ranked candidates from the first stage undergo automated verification via code execution.

Multi-Agent Framework: The system employs a Planner and an Executor. The Planner drafts an error-aware evaluation blueprint, while the Executor implements the code using the CodeAct workflow and HuggingFace API tools.
Self-Evolving Memory: To address the "combination problem" (where agents repeatedly fail on the same artifact-specific idiosyncrasies), the system incorporates a cross-instance memory loop. An LLM reviewer analyzes execution logs to distill root causes and successful workarounds into structured memories. These memories are injected into the system prompt for subsequent iterations, allowing the agent to apply proven fixes to new combinations involving previously encountered artifacts.

Key Contributions

ArtifactBench: The authors introduce a new benchmark comprising 14,053 artifacts and 51,337 relations. It defines concrete prediction, ranking, and verification tasks for SOTA discovery, evaluated under both transductive (all nodes seen) and inductive (unseen nodes) settings.
ArtifactLinker Framework: A novel rank-and-verify pipeline that combines GNN-based structural reasoning with LLM-based coding agents. It establishes new baselines on ArtifactBench.
End-to-End Discovery: A case study on Natural Language Inference (NLI) tasks demonstrating the framework's ability to uncover unobserved SOTA results and provide research insights (e.g., capability non-monotonicity in newer LLMs).

Experimental Results

Ranking Performance: Contrary to the intuition that LLMs are universal solvers, GNN-based methods outperformed LLM-based approaches in link and attribute ranking tasks when graph information was available.
- In link ranking (Transductive), GATv2 achieved an MRR of 0.307, surpassing the best reranker-based method (Jina-v2+graph) at 0.246.
- In attribute prediction, GNNs like BUDDY achieved an MAE of 0.061, outperforming GPT-5.2+graph (0.093).
- GNNs maintained competitive performance in inductive settings, suggesting they learn generalizable structural features rather than merely memorizing topology.
Verification Efficiency: The joint ranking score significantly reduced verification costs. The system recovered 50% of existing SOTA performances after verifying only the top 10 candidates, whereas a biased baseline required over 60 attempts.
Agent Performance: The self-evolving multi-agent verification system achieved a 72.6% success rate on a curated set of 263 pairs. Ablation studies showed that removing multi-turn interaction or tool-use capabilities drastically reduced success rates (to 28.9% and 56.7%, respectively).
Case Study (NLI): The system successfully identified a new SOTA score for the model sileod/deberta-v3-large-tasksource-nli on the SNLI dataset (0.9212), which was within 1 percentage point of the leaderboard SOTA, despite the model card not reporting this specific score.

Significance and Claims

The paper claims that ArtifactLinker demonstrates the feasibility of treating the HuggingFace Hub as an autonomous research engine. By modeling scientific artifacts as a graph and employing a rank-and-verify strategy, the framework transforms the ill-defined problem of "automatic discovery" into an evaluable objective.

The authors position ArtifactBench as a rich testbed for iteratively improving automatic research methods. They emphasize that their approach does not replace human oversight but serves as a scalable tool to efficiently navigate large discovery spaces, surfacing promising model–dataset interactions that might otherwise remain hidden due to the sheer volume of available artifacts. The work highlights that structural graph information is a critical, often underutilized resource that can outperform semantic priors alone in predicting scientific outcomes.