AutoDataset: A Lightweight System for Continuous Dataset Discovery and Search

AutoDataset is a lightweight, automated system that continuously monitors arXiv to detect, extract, and index newly released datasets from research papers, enabling real-time discovery and significantly improving search efficiency by up to 80%.

The Gist

Imagine you are a chef trying to find the perfect new recipe for a dish you want to cook. In the past, you might have relied on a few well-known cookbooks or asked your friends to tell you about new recipes they found. But now, imagine that every single day, thousands of new recipe books are being printed and dropped into a massive, chaotic library. Most of these books are just about cooking techniques, but a few contain brand-new, amazing recipes.

The problem? Finding those specific recipe books in that massive library is a nightmare. You'd have to read the title of every single book, skim the introduction, and then flip through hundreds of pages just to see if a recipe is actually inside. By the time you find one, it might be weeks old, and you might have missed the best ones entirely.

This is exactly the problem researchers face with Machine Learning Datasets. Every day, scientists publish papers introducing new data sets (the "ingredients" for AI). But there are so many papers, and they are published so fast, that researchers can't keep up. They waste hours manually searching for these datasets.

Enter AutoDataset. Think of it as a super-fast, robotic librarian that never sleeps.

How AutoDataset Works (The "Robot Librarian")

Instead of a human reading every book, AutoDataset uses a clever, two-step robot process to scan the library (which is actually arXiv, a giant online archive of research papers):

Step 1: The "Sniff Test" (The Gatekeeper)
Imagine a robot standing at the entrance of the library. It doesn't read the whole book. It just glances at the Title and the First Paragraph (the Abstract).

• It asks: "Does this book sound like it has a new recipe inside?"
• If the answer is "No," it tosses the book aside in 11 milliseconds (faster than you can blink).
• If the answer is "Yes," it flags the book and sends it to the next station.
• Why this matters: This step is incredibly fast and accurate, filtering out 99% of the noise so the system doesn't get overwhelmed.

Step 2: The "Deep Dive" (The Extractor)
For the books that passed the sniff test, a second robot gets to work. It opens the full book and reads it carefully to find the exact page where the recipe is described.

• It pulls out a short, clear summary of the recipe (the dataset description).
• It hunts for the link to the actual ingredients (the URL where the data is hosted). Sometimes the link is hidden in the footnotes or the back of the book, so this robot is smart enough to check the "source code" of the book (the LaTeX file) to make sure it doesn't miss it.

Step 3: The "Smart Search Engine"
Once the robot has the summary and the link, it files the information away in a special index. Now, when a researcher asks, "I need a dataset for training AI to recognize cats in rain," the system doesn't just show a list of links. It understands the meaning of your question and instantly pops up the perfect recipe card with the link ready to go.

Why Is This a Big Deal?

Before AutoDataset, finding a new dataset was like hunting for a needle in a haystack while wearing blindfold. Researchers had to:

1. Search Google.
2. Open 10 different PDFs.
3. Read through them to find the "Download" button.
4. Realize the button was broken or led to the wrong place.
5. Repeat this for hours.

With AutoDataset:

• It's like having a magic wand. You say what you need, and the system instantly hands you the exact ingredient you need, verified and ready to use.
• The paper claims this cuts the time researchers spend searching by 80%. Instead of spending 10 minutes hunting, it takes them 1 minute.

The Bottom Line

AutoDataset is a lightweight, automated system that acts as a real-time radar for new AI data. It doesn't wait for people to upload data to a website; it goes directly to the source (the research papers), reads them instantly, and organizes the findings so researchers can stop searching and start building.

It turns a chaotic, manual treasure hunt into a smooth, instant retrieval experience, ensuring that the newest and best "ingredients" for AI are available the moment they are discovered.

Technical Summary

Here is a detailed technical summary of the paper "AutoDataset: A Lightweight System for Continuous Dataset Discovery and Search."

1. Problem Statement

The rapid expansion of machine learning (ML) is driven by the continuous scaling of training data. However, the discovery of newly released, task-specific datasets has become a bottleneck.

• Limitations of Existing Platforms: Current platforms (e.g., Hugging Face, OpenML, Google Dataset Search) rely heavily on manual curation or community uploads. This leads to inconsistent metadata, maintenance gaps, and significant latency between a paper's publication and the dataset's availability on these platforms.
• Volume and Velocity: The number of papers introducing new datasets has exploded (a ~95-fold increase from 2014 to 2025), overwhelming manual tracking methods.
• Inefficiency of Current Search: Researchers currently spend significant time manually scanning PDFs, verifying links, and navigating between repositories to find the correct dataset for a specific paper.
• LLM Overhead: While Large Language Models (LLMs) can extract information, their high computational cost and inference latency make them impractical for the continuous, real-time ingestion required to monitor preprint servers like arXiv.

2. Methodology: The AutoDataset Pipeline

AutoDataset is a "paper-first" automated system designed to continuously monitor arXiv, detect dataset-introducing papers, and index them with near-zero latency. It employs a lightweight, modular, four-stage pipeline:

A. Dataset Detection (BERT-GATE)

• Function: Acts as a high-speed filter to identify papers that introduce new datasets based solely on titles and abstracts.
• Architecture: A lightweight binary classifier based on bert-base-uncased.
• Performance: Achieves an F1 score of 0.94 and an inference latency of ~11 ms per paper.
• Mechanism: It processes concatenated titles and abstracts. Papers scoring above a 0.5 threshold are passed to the next stage.

B. Description Extraction (BERT-DESC)

• Function: Extracts concise, human-readable dataset descriptions from the full text of positively identified papers.
• Architecture: A fine-tuned sentence-level classifier.
• Innovation: To handle descriptions that span multiple sentences, the system uses a class-conditional sliding window approach. It feeds a symmetric window of surrounding text to the model, using smaller strides for positive labels and larger strides for negative ones to maintain context while reducing redundancy.
• Performance: Outperforms heavy long-context transformers (Longformer, BigBird) and API-based LLMs (Qwen-Plus) in both accuracy (F1 0.858) and speed (9.39 ms).

C. Link Extraction (Hybrid Approach)

• Challenge: URLs in PDFs are often detached from context (e.g., in footnotes), making simple regex extraction unreliable.
• Solution: A robust hybrid method:
  1. PDF Parsing: Extracts links from the PDF.
  2. LaTeX Source Analysis: If PDF extraction fails or is ambiguous, the system fetches the paper's LaTeX source to extract hyperlinks, which preserves semantic context better.
  3. Scoring & Verification: A rule-based scoring system filters generic links. An optional lightweight LLM step verifies the primary URL; if it fails, the system falls back to the highest-scoring rule-based candidate.

D. Indexing and Retrieval

• Function: Consolidates structured records (Paper Link, Dataset URL, Description, Timestamp) into a searchable index.
• Technology: Uses a dense semantic retriever (Alibaba-NLP/gte-large-en-v1.5).
• Process: Dataset descriptions are pre-encoded offline. User queries are embedded in real-time and matched via cosine similarity to retrieve the most relevant datasets.

3. Key Contributions

1. AutoDataset System: The first fully automated, paper-first system that continuously monitors arXiv to surface and index dataset-introducing papers with near-zero latency.
2. Efficient Two-Stage BERT Pipeline: A highly optimized architecture (BERT-GATE + BERT-DESC) that replaces computationally expensive LLMs, enabling scalable, real-time processing with minimal inference time (~11ms).
3. Robust Link Extraction: A novel hybrid method combining PDF parsing and LaTeX source analysis to ensure high-fidelity extraction of primary dataset URLs.
4. Interactive Web Demo: A live, configurable web application that allows users to monitor the ingestion loop and perform natural language searches on the dataset index.

4. Results and Evaluation

• Detection Accuracy: BERT-GATE achieves 0.97 Accuracy and 0.94 F1 on a held-out test set of 7,789 annotated papers.
• Extraction Efficiency: BERT-DESC achieves an F1 of 0.858, significantly outperforming baselines (e.g., +13.3% over the best baseline) while being 96.5% faster than heavy transformers.
• Data Efficiency: Ablation studies show both models reach performance saturation with relatively small annotated datasets, validating the approach's feasibility without massive human annotation costs.
• Retrieval Performance: The dense semantic retriever consistently ranks the correct dataset at the top of search results, effectively bridging the gap between informal user queries and structured metadata.
• Workflow Efficiency: In a case study involving multimodal document retrieval, AutoDataset reduced the time to find a valid dataset link from 4–9 minutes (manual) to 0.5–2 minutes, representing a 60–80% reduction in time-to-first-valid-link.

5. Significance

AutoDataset addresses a critical bottleneck in the ML research lifecycle. By shifting from manual curation to automated, literature-grounded discovery, it:

• Accelerates Research: Drastically reduces the time researchers spend locating new benchmarks and datasets.
• Ensures Freshness: Provides access to datasets immediately upon publication, long before they appear on community platforms.
• Scalability: Demonstrates that lightweight, targeted models can outperform heavy generative models for specific extraction tasks, offering a blueprint for scalable, real-time scientific literature mining.
• Standardization: Anchors dataset discovery to canonical papers, naturally avoiding the redundancy and deduplication issues common in community-hosted repositories.