💻 computer science

Convergent assessment cannot be policed: a residual-entropy bound on authorship attribution and its consequences for integrity in quantitative disciplines

This paper argues that authorship attribution is structurally impossible for convergent assessment formats like multiple-choice questions because correct responses contain zero residual entropy, rendering current AI-integrity policies based on essay detection unsound for quantitative disciplines and necessitating modality-specific controls.

Original authors: TANZIM ISLAM KHAN

Published 2026-08-04

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: TANZIM ISLAM KHAN

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery: "Who wrote this?" In the world of schools and universities, this is the ultimate question of academic integrity. For years, the main tool detectives have used is a special kind of "fingerprint scanner" designed for long, flowing stories—essays. These scanners look for tiny, unique habits in how a person writes, like how they use commas or structure their sentences. But here is the twist: AI chatbots are getting so good at writing these stories that the scanners are struggling. Schools are trying to fix this by applying the same "essay scanner" logic to everything else students do, like math problems, multiple-choice quizzes, and coding tasks.

To understand why this might be a dead end, we need to look at two simple ideas. First, think of "information" as the amount of unique detail in a message. A long essay is like a messy, unique fingerprint; it has tons of tiny details that can tell you who made it. A math answer, like the number "42," is like a perfect, smooth marble. If two people solve the same problem correctly, they both produce the exact same "42." There is no messiness, no unique fingerprint, and no extra detail to inspect. Second, think of "authorship attribution" as the act of using those unique details to prove who did the work. If there are no unique details to begin with, no amount of fancy technology can magically invent them. This paper asks a bold question: What happens when we try to use a fingerprint scanner on a smooth marble?

The Paper's Big Discovery: The "Smooth Marble" Problem

This research article, titled "Convergent assessment cannot be policed," argues that schools are making a fundamental mistake. They are trying to use essay-based rules to police math, science, and multiple-choice tests, but the author says this is impossible. It's not that the AI detectors are bad or that the tools need to be improved; it's that the type of work itself doesn't contain enough information to identify the author.

The author introduces a concept called "residual entropy." Imagine you are baking cookies. If you ask a student to "write a story about a cookie," they might write a poem, a horror story, or a recipe. There are millions of ways to do it, and each version has a unique "fingerprint" of the writer's style. This is high entropy. But if you ask a student to "solve for X" and the answer is "5," there is only one correct way to write that answer. Whether a human genius, a struggling student, or a super-intelligent robot wrote it, the result is the exact same "5." The author calls this zero residual entropy. Once the answer is correct, there is zero variation left to analyze.

The paper proves mathematically that you cannot extract more information than what is actually there. If the answer is just a single letter or number, there is no "signal" to find. Trying to use AI detectors on these formats is like trying to find a fingerprint on a perfectly smooth, blank piece of glass. No matter how powerful your microscope is, you won't find a print because the glass doesn't have one.

The Three Zones of Risk

The author used data from 19 different studies across 13 subjects to map out where different school tasks fall. They discovered three distinct "zones" or regimes:

The Exposed Zone (The Danger Zone): This includes multiple-choice questions, short numerical answers, and standard problem sets. In this zone, AI is incredibly good (often scoring over 96% on multiple-choice), but the answers have zero authorship signal. The paper argues that in this zone, you simply cannot tell if a human or a robot did the work. It's not a matter of "hard to tell"; it's that the question "Who did this?" has no answer.
The Latent Zone (The Waiting Room): This includes things like short numeric problems where AI currently isn't quite good enough to pass easily. These are "safe" only because the AI isn't smart enough yet. But the author warns this is a ticking clock. Once AI gets better at these, the "safety" disappears instantly because there is still no authorship signal to fall back on.
The Detectable Zone (The Safe Zone): This includes long essays, reflective writing, and complex coding projects. Here, the answers are so varied and unique that AI detectors can work. The paper notes that research into detecting AI in code is actually successful because code has enough "fingerprint" details, unlike a simple math answer.

The Shocking Numbers

When the author applied this logic to a typical science and engineering university program, the results were startling. They found that 38% of the final grades in these programs come from tasks in the "Exposed Zone" (like multiple-choice and problem sets). In contrast, only 5% of humanities programs fall into this trap.

If you raise the standard of proof to what is needed for a criminal court case (a very high bar), the number of exposed tasks in science and engineering jumps to 52%. This means that for more than half of a science student's final grade, the school has no way to prove who actually did the work, and no tool exists that can ever do it.

What This Means for Schools

The paper explicitly rules out the idea that we can just buy better software to fix this. The author states clearly that no improvement in detection technology can solve the problem for multiple-choice or math questions because the information simply isn't there to be found. Trying to accuse a student of using unauthorized AI on a math quiz based on "AI detection" is not just ineffective; the author argues it is mathematically impossible to prove.

Instead, the paper suggests schools need to change their strategy entirely:

Stop trying to police the unpoliceable: Don't use authorship detectors on multiple-choice or short-answer math. It's a waste of time and leads to unfair accusations.
Change the game: If you want to know who did the work, you need to change the task. Use supervised exams (where a teacher watches you), ask students to explain their thinking in person, or design questions that are specific to the local classroom so AI can't just look up the answer.
Accept the limits: For the "Exposed Zone," the only real security is preventing the AI from being used in the first place (like in a locked room), not trying to catch it after the fact.

In short, the paper concludes that the current "one-size-fits-all" approach to academic integrity is broken. You can't use a tool designed for essays to police math problems. For a huge chunk of science and engineering education, the question isn't "How do we catch the cheaters?" but rather "How do we design our tests so that the question of unauthorized AI use doesn't even make sense?"

Technical Summary: Convergent Assessment Cannot Be Policed

Problem Statement
Current academic integrity policies for generative artificial intelligence (AI) are structurally flawed because they are designed around the essay and applied indiscriminately to all assessment formats. While detection tools, process-evidence requirements, and security guidance function on the assumption of extended prose artefacts, they are routinely applied to convergent formats such as multiple-choice items, numerical problems, and short answers. This paper argues that this generalization is unsound not due to a lack of technological sophistication, but due to a fundamental information-theoretic limitation. In convergent formats, correct responses are often byte-identical regardless of the author, rendering authorship attribution mathematically undefined rather than merely difficult.

Methodology
The paper employs an information-theoretic framework to analyze assessment security, introducing the concept of residual entropy.

Theoretical Bound: The author defines residual entropy, $H(R|C)$ , as the entropy of a response distribution conditional on the response being acceptable (correct). They establish that the mutual information between a response and its author, $I(R; A|C)$ , is bounded by this residual entropy. Consequently, any evidentiary standard requiring a specific posterior probability demands a minimum number of "authorship bits" ( $b(p, \pi)$ ). If the residual entropy of a task falls below this threshold, authorship attribution is impossible regardless of the instrument used.
Modality Characterization: Ten assessment modalities were placed on a logarithmic residual-entropy scale based on task structure (number of acceptable responses and freedom of expression). Estimates utilized standard entropy values for English prose where applicable.
Capability Corpus: The study assembled 19 published measurements of AI attainment across 13 disciplines. These scores were normalized to the share of available marks to ensure comparability. The corpus included data ranging from knowledge-based multiple-choice items (96.3% attainment) to locally specified clinical coding (7.0% attainment).
Regime Classification: A three-regime model was constructed by crossing two dimensions: AI capability (defined as >60% attainment) and the residual-entropy floor (the minimum entropy required to meet an evidentiary standard).

Key Contributions

Formal Bound: The paper derives a formal proof (Proposition 1) demonstrating that authorship-identifying information is strictly bounded by the residual entropy of the response. It establishes that convergent assessment lies below the entropy floor required for any defensible evidentiary standard.
Modality Characterization: It provides a principled ordering of ten assessment modalities based on their residual entropy, distinguishing between formats that carry an authorship signal and those that do not.
Capability Corpus: It compiles a cross-disciplinary dataset of AI performance, highlighting the vast variation in AI attainment depending on assessment type.
Three-Regime Model: The paper partitions assessment into three distinct regimes:
- Exposed: AI-tractable but below the entropy floor (no authorship signal).
- Latent: Secure only because current models lack capability, but lacking an authorship signal for when they improve.
- Detectable: Above the entropy floor, supporting authorship-based controls.

Results

Entropy Floor: Under a "clear and convincing" standard with a central signal yield estimate, the entropy floor is approximately 39.6 bits.
Modalities Below the Floor: Multiple-choice items (0 bits), short numeric answers (1 bit), problem sets (18 bits), and short-answer prose (24 bits) fall below the threshold. In these formats, authorship attribution supplies no evidence.
Modalities Above the Floor: Programming tasks (150 bits) and open design projects (8,000 bits) exceed the floor, allowing for potential detection.
AI Attainment: AI performance is highest in constrained formats (e.g., 96.3% on multiple-choice) and lower in open-ended or locally specified tasks.
Programme Exposure: In a representative science and engineering programme, 38% of summative assessment falls into the exposed regime (AI-tractable but undetectable), compared to only 5% in a humanities programme. When including latent tasks, 56% of the science/engineering assessment is below the entropy floor. Under a stricter "criminal" evidentiary standard, the exposed share rises to 52%.
Literature Confirmation: The model predicts that authorship detection research should be absent for formats below the entropy floor. The literature confirms this: extensive research exists for prose and code (above the floor), but no studies attempt authorship attribution for multiple-choice or numerical answers because the artefacts contain no signal to detect.

Significance and Claims
The paper claims that the current approach to academic integrity in quantitative disciplines is fundamentally misaligned with the nature of the assessment. The primary significance of the work is the reframing of the problem: for a substantial portion of the curriculum (particularly in STEM), the question of "which detector to buy" is invalid because the information required for detection does not exist.

The author argues that integrity controls must be matched to the modality. For the exposed regime, authorship-based controls (stylometry, process evidence, output classification) are not just ineffective but generate unsubstantiated accusations. Security in these regimes must instead rely on conditions (supervision, interactive verification) or design (task individualization, locally specified content). For the latent regime, security is temporary and expires as model capabilities improve.

The paper concludes that integrity policy must shift from a one-size-fits-all essay-based model to a modality-specific approach that acknowledges the information-theoretic limits of authorship attribution. The goal is not to improve detection tools for convergent tasks, but to redesign assessment to ensure that summative weight is carried by formats where security is coherent and verifiable.

Technical Summary: Convergent Assessment Cannot Be Policed

More like this