📄 other

Observability of False Negatives in Preprint Moderation: A Framework for Auditable Scientific Gatekeeping

This paper proposes a framework for "auditable scientific gatekeeping" to address the critical measurement gap in preprint moderation, arguing that while decision criteria are often transparent, the inability to observe the outcomes of rejected submissions prevents the empirical assessment of false-negative rates and necessitates new reporting standards to make substantive screening empirically testable.

Original authors: Atsushi Otsuka

Published 2026-10-07

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Atsushi Otsuka

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Science relies on a simple promise: that new ideas can be seen, tested, and debated by anyone. For decades, the primary way researchers shared their early work was through preprint servers, digital libraries where scientists post their papers before they have been formally checked by a journal. These platforms act as a rapid highway for discovery, letting the world see a claim or a method the moment it is written, rather than waiting months or years for the slow, traditional process of peer review. However, these servers are not just open filing cabinets; they are curated spaces. Before a paper appears on the public list, a team of volunteers reads it to decide if it belongs there. They can reject a submission if it seems to lack originality, if the science is not significant, or if the work is too incomplete to be useful. This gatekeeping role is necessary to keep the library from being flooded with nonsense, but it creates a difficult problem. When a gatekeeper says "no," how can anyone know if that decision was right? If a paper is hidden away, and the author never publishes it elsewhere, the world never sees the result. The paper might have been a brilliant discovery that was mistakenly buried, or it might have been a dead end that was correctly stopped. Without a way to look back at the rejected pile, the system cannot tell the difference between a mistake and a success.

This is the central puzzle that independent researcher Atsushi Otsuka tackles in a new study. He does not argue that the current system is broken or that the volunteers are biased. Instead, he asks a more fundamental question about measurement: does the system have a way to see its own errors? Otsuka examines the rules and public reports of arXiv, the largest and most famous preprint server, to see if it is possible to track what happens to the papers that get rejected. He finds that while the server is very good at telling us how many papers it receives and how many it turns away, it does not tell us what happens to the ones it turns away. The server knows its own rejection rate, but it does not track whether those rejected papers later get published in other journals, whether they are cited by other scientists, or whether the author improved them and tried again. Without this follow-up information, the system cannot calculate how often it makes a mistake. It is like a security guard who knows exactly how many people they stopped at the door, but has no record of whether any of those stopped people were actually innocent visitors who were wrongly turned away.

Otsuka's work builds on a long history of similar studies in the world of traditional academic journals. For years, researchers have tracked rejected journal manuscripts to see if they eventually get published elsewhere or if they go on to become highly cited, important works. These studies have shown that even the best journals sometimes miss brilliant papers, and that the process of rejection itself can change an author's behavior, perhaps discouraging them from finishing the work. Otsuka applies this same logic to preprint servers. He points out a crucial asymmetry in how we judge these decisions. If a rejected paper later gets published in a top journal, that is strong evidence that the preprint server made a mistake. But if a rejected paper never gets published, that does not prove the server was right. The rejection itself might have caused the author to give up, or the paper might have been rejected for a reason that was actually wrong. Because the rejection changes the future of the paper, you cannot simply look at the final outcome to decide if the initial "no" was correct.

To solve this, Otsuka proposes a new framework for making scientific gatekeeping "auditable." He breaks the problem down into three levels. The first level is decision transparency, which means simply reporting how many papers were rejected and why. The second is outcome observability, which means tracking what happens to those rejected papers later. The third and most important level is error auditability. This requires the platform to declare in advance how it will judge whether a decision was wrong. For example, if a paper is rejected because it needs "significant revision," the platform must have a clear rule for checking later whether the author actually made those revisions before publishing elsewhere. Otsuka's audit of arXiv's current public records shows that the server is strong on the first level but weak on the others. The server reports its total numbers and its general policies, but it does not publish data linking specific rejections to later outcomes, nor does it provide a method for checking if a rejected paper was actually a mistake.

The study does not claim that arXiv is rejecting good science, nor does it claim that the volunteers are making errors. It simply states that the current public information is not enough to know. The server might be perfect, or it might be making many mistakes; the public data structure simply cannot tell. Otsuka suggests that for a system to be truly trustworthy, it must be able to measure its own accuracy. He recommends that platforms start collecting and sharing specific data: how many rejected papers later appear in journals, how often authors appeal and win, and whether the reasons given for rejection match the final fate of the work. He also suggests that platforms should be careful about using an author's past record or their university affiliation as a reason for rejection, unless they can prove that these factors do not lead to unfair errors. The goal is not to eliminate all rejections, which is impossible, but to create a system where the rate of error can be seen and studied.

Ultimately, this research shifts the conversation from arguing about whether gatekeeping is good or bad to asking how we can measure its quality. The paper argues that if a system has the power to hide scientific work from the world, it has a responsibility to show us how often it gets that decision wrong. By following the rejected papers and connecting them to their later lives, the scientific community can move from guessing about the quality of its gatekeepers to actually knowing. This approach does not require a perfect definition of what "good science" is. It only requires a clear, honest record of what happens after a "no" is said. If the system can show us that record, it can prove that it is working as intended. If it cannot, then the system remains a black box, where decisions are made in the dark, and the cost of those decisions is paid by the scientists whose work never gets seen.

Technical Summary: Observability of False Negatives in Preprint Moderation

Problem Statement
Preprint platforms, such as arXiv, occupy a dual role in scholarly communication: they serve as rapid dissemination infrastructures while simultaneously exercising substantive gatekeeping functions (e.g., declining submissions for lack of originality, novelty, or significance). While the normative debate regarding the desirability of such moderation exists, this paper identifies a distinct measurement problem: Can a gatekeeping system observe when its own negative decisions are erroneous?

The core issue is that while the probability of a negative decision ( $P(D=0)$ ) may be observable (e.g., aggregate rejection rates), the probability that such a decision is a criterion-relative error ( $P(E=1 | D=0)$ ) is often unobservable. Unlike journal peer review, where rejected manuscripts are frequently tracked to later publication and citation outcomes to validate editorial decisions, substantive preprint moderation currently lacks a public mechanism to link negative decisions to subsequent outcomes. This absence prevents the estimation of false-negative rates, even as negative decisions themselves alter author behavior and visibility, creating an evidential asymmetry where non-publication does not symmetrically validate the original rejection.

Methodology
The paper employs a documentation-based observability audit combined with a formal theoretical framework:

Formal Framework: The author defines a mathematical structure for "auditable scientific gatekeeping."
- Variables: $D$ (dissemination decision), $K$ (declared criterion/reason), and $O_t$ (observable post-decision trajectory).
- Audit Map ( $A_K$ ): A declared function mapping outcomes to decision validity (contradict, consistent, unresolved).
- False Negative Definition: A criterion-relative false negative ( $E_K=1$ ) occurs when $D=0$ and the audit map indicates the outcome contradicts the original reason.
- Propositions: The paper proves that estimating error rates requires four elements: a record of the negative decision, the specific criterion, a linkable follow-up outcome, and a declared audit map. It further establishes that outcome observability is asymmetric; observing a contradiction falsifies a decision, but the absence of a contradiction does not validate it due to potential censoring effects (e.g., authors abandoning work after rejection).
Documentation Audit: The author reviews public arXiv documentation (policies, 2025 annual report, appeal guidelines) and external reporting (e.g., Science, Data & Society) to inventory observable quantities.
- Scope: The audit is limited to public materials and does not infer the contents of internal databases.
- Criteria: For each quantity relevant to Proposition 1 (decision, reason, outcome, audit map), the paper categorizes availability as "Reported," "Partial," or "Not Identified."
Literature Synthesis: The paper synthesizes existing methodologies from journal gatekeeping literature (e.g., Ray et al., Bornmann & Daniel, Siler et al.) that utilize rejected-manuscript follow-ups, citation analysis, and re-evaluation studies to demonstrate that outcome auditing is an established, feasible method.

Key Contributions

Conceptual Distinction: The paper separates decision transparency (reporting what was decided), outcome observability (linking decisions to later events), and error auditability (defining how outcomes validate or contradict decisions). It argues that transparency without auditability is insufficient for evaluating gatekeeping accuracy.
Formalization of Criterion-Relative Error: It defines false negatives not as a universal failure of "scientific quality" but as a specific contradiction between a declared decision criterion (e.g., "lack of novelty") and a later observable outcome, avoiding the need for a universal ground truth.
Identification of Evidential Asymmetry: The paper clarifies that rejection itself is a causal factor in subsequent outcomes (e.g., reduced visibility, author discouragement). Therefore, the failure of a rejected manuscript to appear elsewhere cannot be used as proof of the rejection's correctness, whereas a successful publication elsewhere can serve as evidence of a potential error.
Empirical Audit of arXiv: The study provides a specific inventory of arXiv's public reporting, finding that while submission volumes and general rejection trends are reported, reason-specific decline counts, appeal outcomes, and linked follow-up data are not identified in public materials. Consequently, external observers cannot estimate criterion-relative false-negative rates.

Results
The documentation audit reveals a gap in the public data structure required for error estimation:

Observable: Total submission volumes, general moderation criteria, and qualitative trends in rejection rates (e.g., increases attributed to AI-generated content).
Not Identified:
- Reason-coded counts for substantive declines.
- Aggregate appeal volumes and reversal rates.
- Linked follow-up series for declined submissions (e.g., later publication in journals).
- Declared revision-distance metrics for auditing "significant revision" decisions.
- Criterion-relative false-negative rates.

The paper concludes that while arXiv may possess internal data to perform these audits, the public data structure currently prevents the estimation of decision error. The system reports the throughput of gatekeeping but not the accuracy of its substantive judgments.

Significance and Claims
The paper modestly claims to propose a framework for auditable scientific gatekeeping rather than accusing arXiv of specific biases or inaccuracies. Its significance lies in:

Shifting the Debate: Moving the focus from whether moderation is "too strict" or "biased" to the fundamental question of whether the system's error rates are measurable.
Portability: The framework applies to any scientific selection system (journal desk rejection, conference triage, grant screening) that makes substantive pre-publication decisions.
Institutional Design: It argues that if a platform exercises the authority to deny visibility on substantive grounds, it incurs an obligation to expose a measurable route by which the correctness of those decisions can be examined.
Practical Recommendations: The paper proposes a minimum reporting package (Level 3: Error Auditability) including standardized reason codes, periodic random-sample follow-ups of declined manuscripts, declared audit maps for revision criteria, and aggregate reporting of contradiction/reversal rates.

The paper explicitly states it does not estimate arXiv's false-negative rate, nor does it claim that journal acceptance equates to scientific truth. Instead, it asserts that without the structural linkage of decision, reason, outcome, and audit map, the probability of error remains unidentifiable, rendering the gatekeeping system empirically opaque regarding its own mistakes.

Technical Summary: Observability of False Negatives in Preprint Moderation

More like this