💻 computer science

Computer Science Conferences Should Require Nonrepudiable Experimental Results

This position paper argues that computer science conferences must adopt tamper-evident, nonrepudiable protocols to cryptographically bind reported experimental results to actual computations, addressing current verification gaps through a formal threat model and a reference implementation called K-Veritas.

Original authors: Mamadou K. Keita, Christopher Homan

Published 2026-05-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mamadou K. Keita, Christopher Homan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Core Problem: "Trust Me, Bro" Science

Imagine you are a judge at a cooking competition. A chef submits a recipe and claims their cake is the best in the world. They hand you a photo of the cake and a written description of how they made it.

The current system is like this: The chef signs a checklist saying, "Yes, I used real eggs," and "Yes, I baked it for 30 minutes." They might even show you a video of them baking. But here's the catch: The chef controls the video. They could have edited the video to show a real cake while actually serving you a cardboard cutout. Or, they could have baked 50 cakes, picked the one that looked perfect, and told you that was the only one they made.

In computer science, specifically in Machine Learning (AI), this is exactly what happens. Researchers submit papers with tables of numbers (results) and checklists. Reviewers (the judges) are busy and can't re-run the experiments. They have to take the numbers on faith. The authors of this paper argue that this system is broken because it relies on honesty rather than proof.

The Proposed Solution: The "Black Box" Receipt

The authors propose a new rule for computer science conferences: Experimental Nonrepudiation.

Think of this as a tamper-evident receipt for a scientific experiment.

Nonrepudiation is a fancy security word meaning: "You cannot deny you did this, and you cannot change the record later."
The goal is to bind the numbers in the paper to the actual moment the computer did the work, in a way that the author cannot fake or alter.

How It Would Work (The "K-Veritas" Tool)

The authors built a prototype tool called K-Veritas to show this is possible. Here is how it works, using a metaphor:

Imagine the researcher is a chef, and the computer is the kitchen.

The Wrapper: Instead of just telling the computer to "bake the cake," the researcher uses a special tool (K-Veritas) that wraps around the baking process.
The Silent Witness: This tool acts like a silent, independent witness standing in the kitchen. It doesn't touch the ingredients (the data) and doesn't change the recipe (the code). It just watches.
The Snapshot: While the computer works, the witness takes snapshots of:
- The exact code being used.
- The time it took.
- The electricity and heat used (to prove the machine was actually working hard).
- The final numbers the computer printed out.
The Sealed Envelope: Once the experiment is done, the witness seals all these snapshots into a digital "envelope" and signs it with a special key that only the witness has. The researcher never gets this key.
The Result: The researcher submits their paper along with this signed envelope. The conference reviewers can open the envelope and verify: "Did the computer actually run for 40 minutes? Did the code match? Do the numbers match what was printed?"

If the researcher tries to change a number in the paper later, the seal breaks, and the signature becomes invalid.

Why Current Methods Fail

The paper explains why the current "safety nets" don't work:

Checklists: Asking a chef to sign a paper saying "I used real eggs" doesn't prove they didn't use plastic eggs.
Code Sharing: Just because the chef shares the recipe doesn't mean they baked the cake they claimed to. They could share a working recipe but report results from a different, better cake they baked secretly.
Logging Tools: Tools that track progress are like a diary written by the chef. If the chef wants to hide a mistake, they can just edit the diary.
Pre-registration: This is like the chef promising before they start, "I will bake a chocolate cake." It stops them from changing the plan, but it doesn't prove they actually baked the cake or that the numbers they report are real.

The "Fake Review" Connection

The paper makes a clever comparison:

Top conferences (like ICML) now ban reviewers from using AI to write reviews. Why? Because the community can't tell if a review is real human thought or a robot's fake text.
The authors argue: If we can't trust a fake review, we definitely can't trust a fake result. If a review needs proof of humanity, a scientific result needs proof of computation.

What This Tool Can and Cannot Do

The authors are honest about the limits of their tool (K-Veritas):

It stops: Cheating with text editors, faking numbers, editing logs after the fact, or claiming you ran a huge experiment when you didn't.
It doesn't stop: A super-sophisticated hacker who can trick the computer's operating system or hardware itself (like a fake GPS signal).
The Trade-off: Even if it doesn't stop every possible cheat, it raises the cost of cheating. Right now, cheating just requires a text editor. With this system, you'd have to actually run the expensive computer experiment or hack the computer's brain.

The Path Forward

The authors aren't saying K-Veritas is the final answer. They are calling it a "testbed" (a prototype). They propose a three-step plan for the community:

Voluntary: Let researchers use it if they want to get a "Verified" badge.
Expected: Make it standard practice, like sharing code is becoming standard.
Required: Eventually, require it for all papers with experimental results.

The Bottom Line

Science is built on trust, but trust needs evidence. This paper argues that computer science conferences should stop asking researchers to "please be honest" and start requiring digital receipts that prove the numbers in the paper came from a real, unaltered computer run. It's about moving from "Trust me" to "Show me the receipt."

Technical Summary: Computer Science Conferences Should Require Nonrepudiable Experimental Results

Problem Definition
The paper identifies a structural "verification gap" in computer science conferences, particularly in machine learning (ML). Current reproducibility measures—such as self-reported checklists (e.g., NeurIPS), optional artifact evaluation, author-controlled logging platforms (e.g., Weights & Biases), and pre-registration—fail to bind reported numerical results to the actual execution of the code that produced them. These mechanisms verify the intention to disclose or the usability of code, but they cannot answer the fundamental question: "Did the training run described in this paper actually produce the results this paper reports?"

The authors define this specific failure mode as experiment nonrepudiation. Borrowing from security literature, they define it as the property where an author cannot later alter, deny, or misrepresent the record of what their computation actually produced. The current system relies on trust in self-reporting, leaving the community vulnerable to text-level fabrication, log manipulation, and selective reporting of favorable runs.

Methodology and Proposed Solution
To address this, the authors propose a protocol for generating tamper-evident, nonrepudiable attestations of experimental results. The core of their methodology is a protocol that must satisfy six specific security properties:

Passivity: The observer must not modify the computation.
Data Blindness: The observer must never access the dataset $D$ , ensuring proprietary data remains private.
Execution-Binding: Reported metrics must be cryptographically linked to the specific execution instance, verified via runtime telemetry (CPU/GPU usage, memory).
Tamper-Evidence: Any modification to the report (metrics, hyperparameters, timestamps) must invalidate the cryptographic signature.
Author-Key Separation: The signing key must be held by an independent attestation service, not the author, preventing the author from generating arbitrary attestations.
Independent Verifiability: Any third party must be able to validate the attestation using a public key without trusting the author.

To demonstrate the tractability of this problem, the authors built K-Veritas, a reference implementation in Go. K-Veritas functions as a standalone binary wrapper that authors invoke via a command prefix (e.g., kveritas run -- python train.py). It operates as follows:

Capture: It captures stdout/stderr non-blockingly, parses metrics, and hashes source files before and after execution.
Telemetry: A background sampler records hardware utilization (CPU time, memory, GPU usage) at intervals.
Sealing: Upon session close, it computes a canonical SHA-256 digest of the entire session (file hashes, stdout streams, metrics, hardware samples).
Signing: It sends only this 64-character digest to a remote attestation service. The service, holding the private key, returns an RSA-PSS signature. The author never sees the private key.
Output: The system generates a signed PDF report and a signed archive containing the source code present at execution.

Results and Demonstrations
The paper presents two primary forms of evidence:

The "Untrustworthy Table" Exercise: The authors demonstrate that two tables of results—one from a real run and one generated by a language model to be plausible—are indistinguishable to a human reviewer. This illustrates that plausibility is not a substitute for authenticity.
K-Veritas Testbed: The authors executed K-Veritas on two distinct training runs: a synthetic Keras LSTM and a RoBERTa-base fine-tuned on SST-2. The system successfully captured:
- Hardware fingerprints (GPU/CPU models).
- Training duration and final metrics (loss, accuracy).
- Source code and stdout hashes.
- A "Hardware-Metric Consistency (HMC)" score, which flags anomalies (e.g., a metric reported without corresponding hardware activity).
- Valid RSA-PSS signatures.

The authors explicitly note that K-Veritas is a testbed, not a finished solution. It successfully prevents casual and moderate fabrication (text editing, log tampering, naive selective reporting) but acknowledges it cannot defeat privileged adversaries with kernel or firmware access (e.g., OS-level telemetry spoofing).

Key Contributions

Conceptual Definition: The paper formally defines "experiment nonrepudiation" as a distinct problem class, separating it from reproducibility (can it be rerun?) and replicability (does it yield the same result?).
Threat Model: It outlines a comprehensive threat model, distinguishing between attacks software-only protocols can defeat (fabrication, log editing) and those they cannot (kernel/hardware compromise).
Protocol Specification: It defines the necessary security properties for any compliant protocol, emphasizing the separation of the signing key from the author.
Implementation: It provides K-Veritas as a proof-of-concept that such a protocol can be implemented without requiring authors to share sensitive data or modify their core training logic.

Significance and Claims
The paper argues that nonrepudiation should be treated as a first-class requirement for computer science conferences, similar to the prohibition of AI-generated reviews at ICML 2025. The authors claim that just as the community recognizes fabricated reviews as a threat requiring policy intervention, fabricated results should be treated with equal severity.

The significance of the work lies in shifting the burden of proof from "trust the author" to "verify the execution." The authors propose a three-phase adoption path (Voluntary $\to$ Expected $\to$ Required) over a 3–8 year horizon. They emphasize that while the system is not perfect against state-level adversaries, it fundamentally changes the economics of fabrication: moving the cost from a simple text editor to the requirement of running real computation or compromising a kernel.

The paper concludes by calling for an independent, non-profit organization to maintain the standard, ensuring that no single entity controls the verification of scientific legitimacy. The framework is presented as generalizable beyond ML to any field relying on computational pipelines for empirical claims.