💬 NLP

sciwrite-lint: Verification Infrastructure for the Age of Science Vibe-Writing

The paper introduces sciwrite-lint, an open-source, local verification tool that uses AI to automatically validate scientific manuscripts by checking reference integrity and citation support, while also proposing an experimental "SciLint Score" to computationally assess the philosophical robustness of scientific arguments.

Original authors: Sergey V Samsonau

Published 2026-04-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sergey V Samsonau

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are walking through a massive, chaotic library where millions of new books are being printed every second. Some of these books are brilliant masterpieces, but many are filled with lies, made-up facts, and references to books that don't even exist.

For a long time, we relied on two ways to check if a book was good:

The "Gatekeeper" System: A few famous librarians (journal editors) tried to read every book before letting it into the library. But they are slow, tired, and often just check if the author is famous, not if the facts are true.
The "Honor System": Anyone can put a book on the shelf. We just have to trust the author didn't lie.

Now, imagine a robot that can write a whole book in seconds. It's so fast that the Gatekeepers can't keep up, and the Honor System is overwhelmed by thousands of fake books. The robot is great at sounding smart, but it's terrible at checking its own facts. It often invents citations (references to other books) that look real but are actually ghosts.

Enter sciwrite-lint: The "Fact-Checking Robot" for Your Desk.

This paper introduces a new tool called sciwrite-lint. Think of it not as a librarian who judges your book, but as a personal detective that runs on your own computer.

Here is how it works, using simple analogies:

1. It's a "Local" Detective (Privacy First)

Most AI tools send your secret manuscript to a giant cloud server to be checked. That's like handing your private diary to a stranger.
sciwrite-lint is different. It runs entirely on your own computer (your "local machine"). You don't have to send your unpublished work anywhere. It's like having a private investigator in your living room who never leaves the house.

2. The "Reference Chain" Detective

When a scientist writes a paper, they say, "As Smith (2020) proved..."

Old Way: A human reviewer might just glance at the list of names to see if they look real.
sciwrite-lint Way: It doesn't just check the name; it goes out and reads the book Smith actually wrote.
- Does Smith's book exist?
- Did Smith's book get retracted (cancelled) because it was a lie?
- Does Smith's book actually prove what the author claims?
- It even checks the books Smith cited! It follows the "paper trail" one step deep to make sure the whole chain of evidence is solid.

3. The "Vibe-Check" vs. The "Hard Truth"

The paper calls the current problem "Science Vibe-Writing." This is when AI writes a paper that sounds confident and flows beautifully, but the facts are made up. It's like a smooth-talking salesman selling you a car that doesn't have an engine.
sciwrite-lint ignores the "vibe." It doesn't care if the writing is pretty. It cares if the engine exists. It checks:

Integrity: Are the numbers in the text the same as the numbers in the table? (Did the author lie about the math?)
Consistency: Did the author promise to solve a problem in the introduction but forget to solve it in the conclusion?
Evidence: If you say "This drug cures cancer," does the cited paper actually say that, or did the author just guess?

4. The "Report Card" (The SciLint Score)

Instead of just saying "Pass" or "Fail," the tool gives the paper a Score.

The Integrity Score: How honest is the evidence? (Are the references real? Are the claims supported?)
The Contribution Score (Experimental): Does the paper actually do anything new? Is it a boring rehash of old ideas, or does it solve a real problem?

Think of it like a nutrition label for science.

A paper with high integrity but low contribution is like a healthy apple: It's real and safe, but maybe not very exciting.
A paper with low integrity is like a plastic apple: It looks perfect, but it's fake and useless.
sciwrite-lint tells you immediately if you are holding a plastic apple.

Why Do We Need This?

The paper argues that the old systems are broken.

Journals are too slow and biased.
AI is making too many fake papers too fast.
Humans can't read every paper to check the facts.

sciwrite-lint is the solution because it automates the "boring but necessary" work of checking facts. It runs on a standard computer (like the one you use for work), uses free databases, and doesn't need a supercomputer.

In short:
If science is a giant game of "Telephone" where information gets passed down, sciwrite-lint is the tool that stops the game from turning into a game of "Fake Telephone." It ensures that when we say "Science says X," we can actually prove that Science really said X, and that X is true.

1. Problem Statement

The paper identifies a critical failure in the current scientific quality assurance ecosystem, exacerbated by the rise of AI-assisted writing ("science vibe-writing").

Journal Gatekeeping: Peer review is described as slow, biased, and ineffective at detecting fabrication (e.g., fabricated citations passing review at top venues like NeurIPS). It measures prestige rather than integrity.
Open Science: Preprints and open repositories lack quality filters, relying solely on author integrity.
The AI Amplification: AI tools generate papers faster than human review can absorb. Crucially, local AI models (used for privacy) hallucinate references and metadata at high rates, while cloud models expose unpublished data to third parties.
Broken Citation Practices: Citations are often copied without reading, contain errors, or reference retracted papers. Current metrics (h-index, impact factor) measure attention, not the validity of evidence chains.

Core Gap: There is no widely adopted tool that verifies the paper itself (integrity of evidence and soundness of arguments) rather than its proxies (citations, journal rank).

2. Methodology: The sciwrite-lint Pipeline

The authors propose sciwrite-lint, an open-source, fully local verification pipeline. It runs on a single consumer GPU (16GB+ VRAM) using open-weights models and free public databases, ensuring no manuscript data leaves the researcher's machine.

Architecture & Workflow

The pipeline processes a manuscript (LaTeX or PDF) through a structured, non-agentic workflow:

Figure & Text Consistency (Local): A Vision-Language Model (VLM, e.g., Qwen3-VL) extracts figure descriptions. Deterministic checks and LLMs verify internal consistency (e.g., numbers in text vs. tables, abstract vs. body alignment).
Reference Verification (Network + Local):
- Identifiers (DOI, arXiv ID) are checked against canonical APIs (CrossRef, OpenAlex, etc.).
- Retraction Check: DOIs are cross-referenced against the Retraction Watch database.
- Matching Engine: For references without IDs, a fuzzy matching engine scores candidates based on title, author, and year, avoiding strict filters that might exclude papers with metadata errors.
Deep Traversal (The "One-Level" Deep):
- The pipeline downloads and parses cited papers (via GROBID) into structured markdown.
- It verifies the cited paper's own consistency (figures, internal logic).
- It checks the cited paper's bibliography for hallucinations and metadata errors.
Claim Verification:
- Claims in the manuscript are embedded and matched against relevant sections of the cited paper using KNN retrieval.
- An LLM (Qwen3 8B) determines if the cited text actually supports the claim.
Aggregation: Signals from metadata, consistency, claim support, and bibliography checks are aggregated into a Per-Reference Reliability Score.

Key Technical Design Choices

Structured vs. Agentic: The LLM acts as a semantic engine, not an autonomous agent. It receives structured inputs and produces JSON outputs. This ensures reproducibility and auditability.
Local-First: Uses open-weights models (Qwen3, Snowflake Arctic Embed) to prevent data leakage.
Optimization: Uses vLLM with automatic prefix caching (APC) and sequential model loading to maximize a single consumer GPU.

3. Key Contributions

A. The Verification Pipeline

First End-to-End Graph Traversal: Unlike existing tools that only check if a reference exists, sciwrite-lint downloads cited papers, parses them, and verifies their internal consistency and their own bibliographies.
Local Execution: A fully local pipeline that runs on consumer hardware, addressing privacy concerns and the "hallucination" risks of local AI writing.

B. SciLint Score (Integrity & Contribution)

The paper introduces a multiplicative scoring formula:
$S(p) = I(p) \times \text{Referencing Quality} \times \text{Contribution}$

Integrity Component ( $I(p)$ ): Measures internal consistency and reference reliability.
Referencing Quality: Aggregates per-reference scores based on metadata accuracy, retraction status, and claim support.
Contribution Component (Experimental): Operationalizes five frameworks from the Philosophy of Science into computable metrics:
1. Popper: Empirical content (falsifiability).
2. Lakatos: Progressiveness (novel predictions vs. ad-hoc fixes).
3. Kitcher: Explanatory unification (bridging research communities).
4. Laudan: Problem-solving effectiveness.
5. Mayo: Test severity (ablations, baselines).
  Note: Novelty is explicitly excluded as it is unreliable for cross-sectional assessment.

C. Evaluation Framework

Error Injection: Synthetic errors injected into papers to test recall.
Real-World Corpus: Evaluation on 30 unseen papers from arXiv and bioRxiv.
Calibration: Ranking known high-quality (Nobel) vs. retracted/fraudulent papers to validate the scoring logic.

4. Results

Error Detection: The pipeline achieved 98.5% recall on injected errors (dangling citations and broken references) with zero false positives on deterministic text checks.
Real-World Performance:
- On 30 unseen papers, the pipeline generated 1,755 findings.
- False Positive Analysis: Initial runs had a high false positive rate (60%) on PDF papers lacking DOIs due to naive title matching. The introduction of the Matching Engine (Section 3.4) and structured ID extraction significantly mitigated this.
- Scoring: Scores ranged from 0.071 to 0.533.
Calibration Success:
- The system correctly ranked Nobel Prize papers (e.g., LIGO 2016, Graphene 2004) in the top quartile and retracted/fraudulent papers (e.g., LK-99, LaCour 2014) in the bottom quartile.
- The "Bold Claims Penalty" correctly identified LK-99 as having high progressiveness but near-zero problem-solving effectiveness.
Performance: Initial runs take ~30 minutes (dominated by claim verification); cached runs complete in minutes.

5. Significance and Impact

Paradigm Shift: Moves scientific QA from "gatekeeping by journal prestige" to "verification of the paper's structural integrity."
AI Resilience: Provides a necessary defense against the "vibe-writing" phenomenon where AI generates fluent but factually hollow or fabricated content. It specifically targets the "hallucination cascade" where AI cites AI.
Privacy & Sovereignty: By running locally, it solves the dilemma of researchers needing AI assistance without exposing unpublished data to cloud providers.
New Metric for Science: The SciLint Score offers a potential alternative to impact factors, focusing on the quality of the argument and evidence chain rather than citation counts.
Community Tool: As an open-source, auditable tool, it allows the community to inspect, improve, and adapt the verification logic, preventing the centralization of scientific truth in a "black box" algorithm.

Conclusion

The paper argues that the current scientific ecosystem is broken and unable to scale with AI-generated content. sciwrite-lint offers a third path: a local, automated, and auditable infrastructure that verifies the integrity of evidence and the soundness of arguments directly from the manuscript, independent of journal prestige or author reputation. While the contribution scoring component is experimental, the integrity verification pipeline is presented as a robust, immediate solution for the "Age of Science Vibe-Writing."