📊 statistics

MajinBook: An open catalogue of digitally mediated world literature

This paper introduces MajinBook, an open catalogue that links metadata from shadow libraries with Goodreads data to create a high-precision, machine-readable corpus of over 539,000 digitally mediated English-language books, while addressing traditional corpus biases and discussing the project's legal permissibility for research under EU and US frameworks.

Original authors: Antoine Mazières, Thierry Poibeau

Published 2026-05-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Antoine Mazières, Thierry Poibeau

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Building a Better Library Map

Imagine you want to study how people read and what stories matter to the world. To do this, you need a massive map of books.

For a long time, researchers used HathiTrust, which is like a giant, official library built by universities. It has millions of books, but it has two big problems:

It's locked: Many books are under copyright, so you can't read the full text; you can only look at them through a secure, restricted window.
It's messy: Many books were just scanned from physical paper. The computer often misreads the letters (like confusing an 'o' for a '0'), making the text hard to use for advanced computer analysis.

The authors of this paper wanted a better map. They turned to "Shadow Libraries" (like Library Genesis and Z-Library). Think of these as massive, underground digital bazaars where millions of books are shared freely. They have way more books than the official library, but they are chaotic. The "shelves" are disorganized, the labels are often wrong or missing, and it's hard to tell which book is which.

MajinBook is the project that cleaned up this chaos. It created a high-quality, organized catalogue of over 539,000 English books (plus millions of others in French, German, and Spanish) by connecting the messy "shadow library" files with the organized data from Goodreads (the social network for book lovers).

The Ingredients: How They Built It

1. The "Digital-Only" Rule (The Sieve)

The shadow libraries have books in many formats. Some are PDFs (digital scans of paper pages), and some are EPUBs (native digital files, like a clean webpage).

The authors decided to throw away all the PDFs.

The Analogy: Imagine you are trying to study the taste of a soup. The PDFs are like taking a photo of the soup and trying to taste the picture. It's blurry and inconsistent. The EPUBs are like the actual soup in a bowl; you can taste every ingredient clearly.
The Result: By only keeping the "clean" digital files (EPUBs), they lost some very old books (because they weren't digitized yet), but they gained a dataset that is much cleaner, easier for computers to read, and surprisingly diverse in languages.

2. The "Work vs. Edition" Problem

In the world of books, there is a difference between a Work (the story itself, like The Odyssey) and an Edition (a specific version, like the 1990 translation or the 2020 audiobook).

The Problem: Shadow libraries might have 50 different files for The Odyssey. If you count them all as 50 different books, your data is skewed.
The Solution: The authors used Goodreads as a "skeleton" or a "filing system." Goodreads already knows that all 50 versions belong to the same "Work." They used this knowledge to group the messy shadow library files together, linking them to the correct story and its original publication date.

3. The Matching Game (Finding the Needle in the Haystack)

How do you connect a messy file from a shadow library to the correct entry on Goodreads?

The Strategy: They didn't try to match the exact cover art or page count (which often changes). Instead, they looked at identifiers (like ISBN numbers) and titles.
The Fuzzy Logic: Sometimes a title is slightly misspelled in the shadow library (e.g., "Harry Pottter"). The authors used a computer trick called "fuzzy matching" to realize, "Hey, that's probably Harry Potter."
The Safety Check: They tested this method by having humans check 200 random matches. They found that if the computer gave a "confidence score" of 80 or higher, it was almost always correct. They set their final rule at that 80-point threshold to ensure their catalogue is highly accurate.

The Results: What Did They Find?

A Massive, Clean Collection: They created a list of 539,000 English books that are ready for computers to analyze.
A Time Bias (But a Useful One): Because they only kept "native digital" files, their collection has fewer books from the 1800s and more from the 2000s.
- The Paper's Take: The authors admit this isn't a perfect history of all literature. Instead, it's a perfect history of 21st-century digital culture. It shows us what books are popular right now in the digital age.
More Languages: Surprisingly, by filtering for digital files, they actually got a more diverse mix of languages. The "PDF" pile was mostly English, but the "EPUB" pile included a healthy amount of French, German, Spanish, and others.

The Legal "Fine Print"

The paper addresses a tricky question: Is it legal to use these shadow libraries for research?

The Argument: The authors argue that they aren't selling the books or distributing the stories. They are only publishing a list of facts (titles, authors, dates).
The Analogy: It's like a librarian making a card catalog. The card catalog doesn't contain the story; it just tells you where the story is and who wrote it. The authors claim that creating this "card catalog" for research purposes falls under legal exceptions for "Text and Data Mining" in both the US and EU.

Summary

MajinBook is a tool that takes the chaotic, massive, and often messy world of "shadow libraries" and organizes it using the social wisdom of Goodreads. The result is a super-clean, computer-friendly list of books that helps researchers study culture, literature, and reading habits in the digital age, all while navigating the complex legal landscape of copyright.

Technical Summary of MajinBook: An Open Catalogue of Digitally Mediated World Literature

Problem Statement
The paper addresses a critical bottleneck in Computational Social Science (CSS) and cultural analytics: the lack of high-quality, machine-readable, and precisely identified book corpora derived from "shadow libraries" (e.g., Library Genesis, Z-Library). While these archives contain tens of millions of items, their metadata is often poor, hindering the precise sampling and identification of specific works required for diachronic analysis. Conversely, established institutional corpora like HathiTrust suffer from significant limitations, including Optical Character Recognition (OCR) errors, representational biases favoring large research institutions, and restrictive access controls (the "data capsule" model) that inhibit collaborative tool development. Furthermore, the legal and technical challenges of processing PDFs from scanned corpora often result in inconsistent text quality. The authors argue that while Large Language Models (LLMs) have begun utilizing these shadow libraries as bulk training data, their application in rigorous CSS research remains underdeveloped due to the inability to link specific digital files to their corresponding abstract "works" and first publication dates.

Methodology
The authors propose a methodology that prioritizes natively digital formats to ensure machine-readability and constructs a "bibliographic scaffold" by linking shadow library metadata with structured data from the social reading platform Goodreads.

Data Selection and Filtering:
- Source: The study aggregates metadata from Library Genesis (LibGen) and Z-Library (accessed via Anna's Archive) as of early 2025, totaling over 77 million items.
- Format Filter: A decisive methodological choice was made to discard all PDF files (77.9% of the source data) due to parsing difficulties and OCR inconsistencies. The study focuses exclusively on natively digital formats (primarily EPUB, including .mobi, .azw, .fb2), which constitute 19.2% of the source. This "productive sieve" introduces a temporal bias toward recent publications and commercially viable re-issues but ensures high text quality and linguistic diversity.
- Goodreads Crawl: To build the bibliographic scaffold, the authors performed a comprehensive crawl of Goodreads. Starting with 1.2 million seed editions derived from tags, shelves, genres, and awards, they recursively expanded the dataset via author-based and recommendation-based discovery. The crawl terminated when new item discovery plateaued, resulting in a dataset of 4.78 million works, 28.1 million editions, and 2.15 million authors.
Linkage Strategy:
- Preprocessing: Shadow library EPUB files were converted to raw text. A 128-element MinHash signature was computed for each item, and Locality-Sensitive Hashing (LSH) was used to cluster near-duplicates (Jaccard similarity $\ge$ 0.8), grouping different editions of the same work in a specific language.
- Matching: The strategy linked shadow library clusters to Goodreads "Works" (abstract entities) rather than specific editions. Matching relied on the intersection of book identifiers (ISBN, ASIN) found in both the shadow library cluster and the Goodreads work's editions.
- Verification: A secondary verification step compared titles using a partial ratio fuzzy match. A human evaluation study on 200 English candidates determined the optimal threshold for title scores. A threshold of 80 was selected to maximize precision (nearly 1.0) while maintaining a robust dataset size.
Dataset Construction:
- The final primary dataset focuses on English-language matches, resulting in a high-precision catalogue of 539,530 references.
- Secondary datasets were released for French (47,960), German (35,559), and Spanish (30,169) based on volume and title score distribution similarity, though these lack the same level of human-verified precision.

Key Results

Corpus Scale and Quality: The resulting MajinBook corpus contains over 539,000 English-language references spanning three centuries. It is enriched with first publication dates, genres, and popularity metrics (ratings, reviews).
Linguistic Diversity: By filtering for EPUBs, the authors achieved a significantly more linguistically diverse corpus than the PDF subset of shadow libraries or the HathiTrust collection. The EPUB subset has a normalized Herfindahl Index (HI) of 0.24, compared to 0.74 for the PDF subset and 0.32 for HathiTrust. English content drops from 85.97% in the PDF set to 47.93% in the EPUB-based MajinBook corpus, enhancing the dataset's representativeness for cultural analytics.
Temporal Bias: The corpus exhibits a "super-exponential" growth curve toward the present, reflecting the availability of natively digital formats. The authors frame this not as a flaw but as a defining characteristic of the "digitally mediated canon."
Matching Accuracy: The human evaluation confirmed that the chosen title score threshold of 80 yields a precision of nearly 1.0. Even for visually ambiguous files (lacking title pages), the title score proved a robust predictor of correct matches.

Significance and Claims
The paper claims to introduce the first open catalogue designed specifically to resolve the metadata challenges of shadow libraries for computational social science. Its significance lies in:

Bridging the Gap: It connects vast, crowd-sourced archives with structured bibliographic data, enabling researchers to navigate between abstract cultural works and their specific digital instances.
Methodological Innovation: It demonstrates that discarding the majority of a corpus (PDFs) in favor of natively digital formats (EPUBs) yields a dataset that is cleaner, more structured, and linguistically more diverse, suitable for high-precision cultural analytics.
Legal and Ethical Positioning: The authors situate their work within the legal frameworks of the EU (Text and Data Mining exceptions) and the US (Fair Use), arguing that their output is a metadata-only index of factual data (titles, authors, dates) rather than a distribution of copyrighted text. They emphasize the non-commercial, academic nature of the project and its role in fostering public knowledge.
Open Science: The project releases the primary catalogue, secondary language datasets, and the underlying raw metadata and MinHash signatures openly, facilitating future research in CSS and digital humanities.

The authors modestly note that the corpus does not represent "world literature" in its historical entirety but rather a specific, contemporary selection of culture as it is curated and consumed in the 21st century through digital channels.