Invisible to humans, visible to machines: a preregistered audit of Unicode fidelity across four biomedical bibliographic APIs

Technical Summary: Invisible to Humans, Visible to Machines

Problem Statement
Biomedical text mining, scientometric analysis, and the construction of training corpora for Large Language Models (LLMs) rely on a foundational, yet rarely tested, assumption: that abstract text retrieved via bibliographic APIs faithfully reproduces the published abstract. Between the author's manuscript and the researcher's tool lies a chain of transformations (manuscript preparation, publisher deposit, API serialization) that can silently discard information at the character level. While lexical features composed of basic Latin letters are generally preserved, rare Unicode characters—such as typographic punctuation, mathematical symbols, Greek letters, and special whitespace—may be altered or lost. These changes are often invisible to human readers but critical for machine text-reading pipelines, where tokenization, embedding generation, and character-level indicators (e.g., for detecting AI-assisted writing) depend on specific code points. Prior work has addressed metadata completeness or document classification but has not systematically quantified character-level fidelity across major APIs against a common ground truth under a pre-registered design.

Methodology
This study conducted a pre-registered audit (OSF osf.io/269b5) comparing four widely used public biomedical APIs: PubMed E-utilities, Crossref, OpenAlex, and Semantic Scholar. The ground truth was established using JATS XML files from the PubMed Central (PMC) Open Access subset for the publication year 2024.

• Sampling: A simple random sample of 4,000 English-language research articles was drawn from a complete enumeration of the PMC OA subset (≈700,000 records).
• Target Characters: Fifty-one Unicode code points were tracked, grouped into six classes. Four primary classes were analyzed: typographic punctuation (e.g., em-dashes, curly quotes), mathematical/scientific symbols, Greek letters, and special whitespace (e.g., non-breaking spaces). Two exploratory classes (subscripts/superscripts and miscellaneous) were also noted.
• Estimands: Two co-primary metrics were calculated for each API × character class cell:
  1. Conditional Preservation (RQ1a): The share of papers preserving at least one character of the class among eligible papers (those with the character in JATS and a non-empty abstract from the API). This isolates the API's character-handling layer.
  2. End-to-End Preserved Availability (RQ1b): The share among all JATS-positive papers where the API returns a non-empty abstract and the character is preserved. This combines availability with fidelity.
• Decision Rules: A cell was classified as "systematic loss" if the upper bound of the 95% Wilson confidence interval was below 5%, "faithful" if the lower bound exceeded 90%, and "partial" otherwise.
• Mechanism Audit: For cells meeting the systematic loss criterion, a blinded mechanism audit was performed on 50 discordant records. A language model (Claude Haiku), acting as a blinded auditor, identified the cause of loss (e.g., substitution, serialization loss). Intra-rater reliability was measured via Cohen's kappa.

Key Results
The audit revealed that character-level fidelity is highly API-dependent and often undocumented.

1. Two Systematic, Deterministic Losses:

   • PubMed (Typographic Punctuation): The <AbstractText> field preserved Unicode typographic punctuation in only 0.6% of eligible abstracts (95% CI 0.3–1.0%). The audit confirmed this is a deterministic substitution of characters (e.g., em-dashes, curly quotes) with ASCII equivalents (e.g., hyphens, straight quotes).
   • OpenAlex (Special Whitespace): OpenAlex preserved special whitespace in 0.0% of eligible abstracts (95% CI 0.0–0.4%). This loss is structural; the abstract_inverted_index stores only word tokens and positions, reconstructing the text with a single standard space (U+0020), making non-standard whitespace unrecoverable.

2. Faithful Preservation of Semantic Characters:

   • All four APIs preserved mathematical/scientific symbols and Greek letters with high fidelity (>95% conditional preservation). The lower bounds of the 95% confidence intervals exceeded 90% for all eight cells in these categories.

3. Partial Preservation:

   • OpenAlex showed partial preservation for typographic punctuation (34.0%), as some characters within tokens survived tokenization while others did not.
   • Semantic Scholar showed partial preservation for special whitespace (59.6%), performing better than OpenAlex but worse than PubMed and Crossref.

4. Coverage Gaps (Crossref):

   • Crossref returned no abstract for 24.6% of the sampled papers (75.4% coverage). This gap is publisher-dependent, with Elsevier and the American Chemical Society depositing 0% of abstracts, while MDPI, Frontiers, and others deposited 100%. This is a publisher-deposition issue, not a character normalization issue.

5. Mechanism Confirmation:

   • The blinded audit confirmed both systematic losses as deterministic. PubMed's loss was 100% substitution; OpenAlex's loss was 100% non-ambiguous (42% serialization loss, 8% substitution), clearing the ≥90% threshold.

Significance and Claims
The paper claims that character-level bibliometrics is source-dependent. Without API-level validation, a measured stylistic signal may reflect the metadata pipeline's normalization policies rather than the literature itself.

• Invisible to Humans, Visible to Machines: The transformations (e.g., replacing an em-dash with a hyphen) are indistinguishable to human readers but fundamentally alter the byte sequence, tokenization, and embeddings processed by machines.
• Consequences for AI-Assisted Writing Detection: Studies inferring AI authorship from character-level signals (e.g., em-dash frequency) are compromised if the API normalizes these characters. For instance, measuring em-dash frequency in PubMed effectively measures NCBI's normalization policy, as the character is lost in 99% of cases.
• Consequences for Corpora and Search: Corpora built from Crossref abstracts carry a publisher-correlated selection bias (omitting Elsevier and ACS). Furthermore, the same paper yields different tokens and embeddings depending on the API, affecting deduplication, semantic search, and benchmark reproducibility.
• Methodological Note on Auditing: The study demonstrates that in the age of AI, "auditors" for character-level tasks may need to be language models operating on code points rather than humans, as the characters in question are often visually indistinguishable to the human eye (e.g., NBSP vs. space).

The study concludes that for typographic punctuation, researchers should rely on Crossref or Semantic Scholar; for mathematical and Greek characters, any of the four APIs suffices; and for special whitespace, inverted-index sources like OpenAlex should be avoided. The findings underscore the necessity of validating metadata pipelines before drawing conclusions from character-level signals.