An Extreme Multi-label Text Classification (XMTC) Library Dataset: What if we took "Use of Practical AI in Digital Libraries" seriously?

This paper introduces a large bilingual (English/German) corpus of catalog records annotated with the Integrated Authority File (GND) and a machine-actionable GND taxonomy to enable ontology-aware multi-label classification and agent-assisted cataloging, aiming to develop transparent, authority-anchored AI tools that enhance the efficiency and scalability of subject indexing in digital libraries.

The Gist

Imagine a massive, global library that never sleeps. Every day, thousands of new books, research papers, and reports arrive. To make sure you can find the right book when you search for "how to bake sourdough" or "the history of quantum physics," librarians must tag every single item with specific subject labels.

In the old days, a human librarian would read the title and abstract, think hard, and pick the perfect tags. But now, the library is growing so fast (and in so many languages) that humans can't keep up. It's like trying to sort a mountain of incoming mail by hand while the mountain keeps getting taller.

This paper introduces a new tool to help: TIB-SID. Think of it as a "training gym" for Artificial Intelligence (AI) to learn how to be a super-librarian.

Here is the breakdown of what the authors did, using some everyday analogies:

1. The Problem: The "Long Tail" of Knowledge

Libraries use a giant, strict dictionary of approved topics called the GND (Integrated Authority File). It has over 200,000 terms.

• The Head: Some topics are super popular (like "Mathematics" or "History").
• The Tail: Many topics are very rare (like "18th-century Icelandic fishing nets" or "a specific type of algae").

Most AI systems are great at the popular stuff but terrible at the rare stuff. They are like a student who aces the math test but fails the obscure history question because they've never seen it before. This dataset is designed to teach AI how to handle both the popular and the obscure.

2. The Dataset: A Bilingual Training Camp

The authors created a massive dataset of 136,000 library records in English and German.

• The Input: The title and summary of a book or paper.
• The Output: The correct "tags" (subject terms) from the official GND dictionary.

They didn't just dump the data; they organized it like a school curriculum:

• Training Set: The homework the AI studies.
• Dev/Test Sets: The final exams to see if the AI actually learned.

They also cleaned up the "dictionary" (the GND taxonomy) and turned it into a format computers can easily read, complete with definitions and synonyms. It's like turning a dusty, handwritten encyclopedia into a searchable, hyper-linked digital database.

3. The Challenge: Extreme Multi-Label Classification

In tech-speak, this is called Extreme Multi-label Text Classification (XMTC).

• Simple version: "Is this email spam or not?" (Two choices).
• This version: "Here is a book. Pick the top 20 correct tags out of 200,000 possible tags."

It's like playing a game of "Guess the Topic" where you have to pick the right 20 words from a dictionary the size of a small city, and many of those words only appear once or twice in the whole game.

4. The Experiment: Three AI "Students"

The authors tested three different AI strategies to see which one could be the best librarian assistant:

• Student 1 (The "Similarity Seeker"): This AI looks at the new book and asks, "What other books in the library look like this?" It then steals the tags from those similar books.

  • Result: Good at finding general themes, but sometimes grabs tags that are technically correct but irrelevant (like tagging a book about "Apple the fruit" with "Apple the tech company" because they both have the word "Apple").

• Student 2 (The "Chatbot"): This AI uses a Large Language Model (like a super-smart chatbot). It reads the book, asks the chatbot to suggest keywords, and then tries to match those keywords to the official dictionary.

  • Result: Very creative and good at understanding context, but sometimes gets confused by the strict rules of the dictionary. It might suggest a perfect word that doesn't exist in the official list.

• Student 3 (The "Hybrid Veteran"): This is the winner. It combines the old-school, rule-based math methods with the new, smart chatbot tricks. It translates everything, trains specific models for English and German, and then uses a smart AI to double-check and rank the results.

  • Result: It got the highest score. It's the most reliable, but it's also the most complex.

5. The Big Takeaway: AI as a Co-Pilot, Not a Captain

The most important lesson from this paper isn't that AI is perfect. It's that AI is a great assistant, but it needs a human in the loop.

• The Analogy: Imagine a GPS. It's amazing at finding the fastest route (the AI). But if the road is closed for construction or there's a parade, the GPS might get confused. You (the human librarian) need to look at the GPS's suggestion and say, "Actually, take the side street."
• The Goal: The authors want to build "AI Co-pilots" that do the heavy lifting—scanning thousands of books and suggesting the top 20 tags—so the human librarian can just spend 30 seconds verifying them. This saves time and keeps the library organized without losing the human touch.

Summary

This paper is a gift to the library world and the AI community. It says: "Here is a real-world, messy, bilingual dataset. Here are three ways to solve the problem. And here is the proof that while AI is getting smarter, we still need humans to make the final call to ensure the library remains a trustworthy place for discovery."

It's not just about making computers faster; it's about making the library more useful for everyone, everywhere.

Technical Summary

Here is a detailed technical summary of the paper "An Extreme Multi-label Text Classification (XMTC) Library Dataset: What if we took 'Use of Practical AI in Digital Libraries' seriously?"

1. Problem Statement

Subject indexing is critical for the discovery and interoperability of library collections but is increasingly unsustainable due to the volume, multilingual nature, and diversity of modern catalog records. While Large Language Models (LLMs) and agentic pipelines offer potential solutions, current approaches face significant challenges in library contexts:

• Lack of Grounding: Generic text classification models often fail to map free text to authoritative, controlled vocabularies (e.g., GND) required by libraries.
• Evaluation Mismatch: Standard XMTC metrics (like accuracy or F1) do not capture library-specific needs such as hierarchical coherence, explainability, or the "usefulness" of suggestions to human catalogers.
• Data Scarcity: There is a lack of large-scale, bilingual (English/German), machine-learning-ready benchmarks that pair real library records with structured, ontology-grounded subject annotations.

2. Methodology and Dataset Construction

The authors introduce TIB-SID (TIB Subject Indexing Dataset), a comprehensive resource designed to bridge the gap between extreme multi-label text classification and library science.

A. The Dataset (TIB-SID)

• Source: Derived from the TIB (Leibniz Information Centre for Science and Technology) open data, comprising 136,569 catalog records.
• Languages: Bilingual (English and German), with records spanning diverse domains (STEM, Social Sciences, Humanities).
• Annotation: Records are annotated with subjects from the GND (Gemeinsame Normdatei), the German Integrated Authority File.
  • Taxonomy: The GND Sachbegriff (subject terms) contains ~207,001 unique concepts. The dataset uses a machine-actionable JSON version derived from MARC 21, enriched with alternate names, related subjects, and definitions.
  • Statistics: The dataset exhibits a classic long-tail distribution: 41,218 unique subjects appear in the dataset, but many are rare (few-shot/zero-shot scenarios).
• Splits: Predefined train/dev/test splits (90k/20k/26k records) balanced by record type (Book, Article, Thesis, etc.) and language.
• Format: Records are provided in JSON-LD, encoding bibliographic metadata (title, abstract, creators) and linking to GND URIs.

B. Statistical Analysis

The authors conducted a deep statistical analysis of the dataset to inform model design:

• Long-Tail & Sparsity: Significant overlap exists between splits (Jaccard ~0.36), but distributional divergence (KL divergence, Chi-squared) confirms that training and test sets have different frequency profiles, necessitating robust handling of rare labels.
• Polysemy: Analysis of string identity and semantic embeddings (using Gemma, E5, and Jina models) revealed that true polysemy (ambiguous terms) in GND is rare (<1%). Most "duplicate" strings are terminological reuses across related entities rather than genuine semantic ambiguity.

C. Experimental Systems

Three representative systems were evaluated on the dataset (based on SemEval 2025 and GermEval 2025 tasks) to benchmark performance:

1. System 1 (LA2I2F): A semantic retrieval approach using Sentence-Transformers (all-mpnet-base-v2). It combines ontological reasoning (query vs. subject embeddings) and analogical reasoning (retrieving similar training records and transferring their labels).
2. System 2 (KIFSPrompt): A Retrieval-Augmented Generation (RAG) pipeline. It retrieves few-shot examples, uses an LLM (Ministral-8B) to generate free-text keywords, maps them to GND terms via hybrid search (HNSW + BM25), and re-ranks using another LLM (Llama-3.1).
3. System 3 (Annif): A hybrid ensemble approach. It uses LLMs to translate data and generate synthetic training samples, then trains traditional XMTC models (Omikuji Bonsai, MLLM, XTransformer) on monolingual data. Predictions are merged and re-ranked by an LLM.

3. Key Contributions

• TIB-SID Dataset: The first large-scale, bilingual, ML-ready benchmark for library subject indexing, featuring stable links to a controlled vocabulary (GND) and predefined splits.
• Ontology-Aware XMTC: The resource shifts the focus from generic classification to authority-grounded classification, where the goal is mapping text to specific, trusted concepts.
• Qualitative & Quantitative Evaluation: The paper moves beyond standard metrics by introducing librarian-in-the-loop evaluation. It assesses not just recall/precision but "usefulness" (distinguishing between technically correct but irrelevant suggestions vs. truly correct ones).
• Error Analysis Framework: Detailed categorization of errors (e.g., False Negatives due to missed concepts vs. synonyms; False Positives due to analogical over-transfer or mapping ambiguities) provides a roadmap for future system improvements.

4. Results

Performance was measured using nDCG@k (Normalized Discounted Cumulative Gain), prioritizing the ranking quality of the top-k predictions.

• Quantitative Performance:
  • System 3 (Hybrid/ANNIF) achieved the highest scores (nDCG@5 = 0.6020, nDCG@20 = 0.6652), demonstrating that combining traditional XMTC algorithms with LLM-based data augmentation and re-ranking is currently the most effective approach.
  • System 2 (LLM-only RAG) performed well (nDCG@5 = 0.4919) without fine-tuning, proving the viability of few-shot prompting for this task.
  • System 1 (Semantic Retrieval) scored lower (nDCG@5 = 0.3639), struggling with the dominance of analogical reasoning which often over-predicted labels from similar documents.
• Qualitative Findings:
  • System 3 excelled in Economics and Mathematics but struggled with low-frequency (long-tail) subjects due to the limitations of its supervised components.
  • System 2 showed strong performance in German but often produced "plausible but overly broad" terms.
  • Human Evaluation: Librarians noted that while LLMs capture topical proximity well, they struggle to distinguish between semantically distinct but related subjects (e.g., "deception" vs. "spoofing" in a military context), leading to "technically correct but irrelevant" suggestions.

5. Significance and Future Work

• Practical Impact: The dataset and evaluation framework provide a realistic testbed for developing AI co-pilots for librarians. It emphasizes that successful AI in libraries must be transparent, auditable, and grounded in controlled vocabularies.
• Research Directions:
  • Long-Tail Handling: Future work must address the prediction of rare subjects (zero-shot) more effectively, potentially by better integrating embedding-based retrieval with supervised learning.
  • Vocabulary Evolution: The analysis suggests that error patterns (e.g., mapping errors for unknown concepts) can be used to identify gaps in the GND taxonomy, suggesting new terms or synonyms.
  • Human-in-the-Loop: The results advocate for hybrid workflows where AI handles candidate generation and ranking, while human experts perform final validation, rather than full automation.

In conclusion, this paper establishes a new standard for evaluating AI in digital libraries, demonstrating that while LLMs show promise, the most robust solutions currently combine traditional machine learning with LLM-based augmentation and rigorous, domain-specific evaluation.