💻 computer science

Can Humans Detect AI? Mining Textual Signals of AI-Assisted Writing Under Varying Scrutiny Conditions

This study demonstrates that while human judges can significantly distinguish between AI-assisted writing produced under the threat of detection and writing produced without such a warning, traditional computational text features fail to capture the subtle signals that allow humans to make this distinction.

Original authors: Daniel Tabach (Georgia Institute of Technology)

Published 2026-04-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Daniel Tabach (Georgia Institute of Technology)

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The "Invisible Ink" Experiment: Can We Spot an AI Ghostwriter?

Imagine you are a teacher grading essays. You have two students, Alex and Sam. Both used an AI assistant to help them write their papers. To the naked eye, both essays look identical: the vocabulary is similar, the sentences are the same length, and they both use the same facts.

But there’s a catch: Alex knew he was being watched by a high-tech "AI Detector," while Sam had no idea.

A researcher named Daniel Tabach conducted a study to see if that simple knowledge—the "threat" of being caught—changes how people write, and if humans can sense that change even when computers can't.

The Setup: The Two Types of Writers

The study split people into two groups:

The "Chill" Group (Control): They were given an AI chatbot and told to write an opinion piece. They felt free to copy-paste or use the AI however they wanted.
The "Under the Microscope" Group (Treatment): They were given the same chatbot, but with a bright yellow warning: "Heads up! An AI detector is going to scan this."

The Goal: Would the "Under the Microscope" group try harder to "act human" to avoid being flagged?

The Mystery: The "Ghost in the Machine"

Here is where the study gets weird. The researcher looked at the essays using every "digital magnifying glass" available—tools that measure word variety, sentence rhythm, and how much text was copy-pasted from the AI.

The result? The essays were identical.

On paper, the "Under the Microscope" group didn't change their style at all. They still copy-pasted large chunks of text, and their sentence structures were just as "robotic" as the first group. If you ran these essays through a computer program, the computer would say, "These are both AI-assisted. No difference here."

But then, the humans stepped in.

The researcher showed these essays to a panel of 251 judges and asked a simple question: "Which one sounds like it was written by a human?"

Even though the computer couldn't see a difference, the humans could. The judges picked the essays from the "Under the Microscope" group as "human" significantly more often than the other group.

The Analogy: The "Polished Mirror" Effect

Think of it like this:

Imagine two people are cleaning a window.

The first person (the "Chill" group) just wipes it quickly. They leave some streaks and smudges because they don't care.
The second person (the "Under the Microscope" group) knows a professional inspector is coming. They don't necessarily use a different soap or a different cloth (the "textual features" are the same), but they scrub a little differently. They might spend more time on the corners or move the cloth in a more deliberate way.

To a machine measuring the amount of soap used, the two windows look the same. But to a human eye looking through the glass, the second window feels "clearer" or "more natural."

What Does This Actually Mean?

The study suggests something profound about the "vibe" of writing:

The "Human" Signal is Subtle: When we know we are being watched, we don't just change what we say; we change the energy of how we interact with our tools. The "warned" writers spent more time chatting with the AI, iterating, and tweaking.
Humans are better than "Feature-Counters": Current AI detectors look for math patterns (like how many times you use the word "the"). But humans are sensing something deeper—perhaps a sense of intent, a specific rhythm, or a "soul" in the way the ideas are paced—that math hasn't learned to measure yet.

The Bottom Line: Even if you use an AI to write your essay, if you know you're being watched, you might subconsciously leave behind a "human fingerprint" that a computer will miss, but a person will feel.

Technical Summary: Can Humans Detect AI? Mining Textual Signals of AI-Assisted Writing Under Varying Scrutiny Conditions

1. Problem Statement

As Large Language Models (LLMs) become increasingly proficient at generating human-like text, concerns regarding academic integrity and authenticity have surged. While many institutions have adopted automated AI detection tools, it remains unclear whether the threat of detection actually alters human behavior when using AI. Furthermore, there is a critical gap in understanding whether human intuition can detect subtle shifts in writing style caused by such scrutiny—shifts that might bypass current feature-based computational detection methods.

2. Methodology

The researcher employed a two-phase controlled experiment using a custom-built Streamlit application integrated with the Anthropic Claude (Sonnet) API.

Phase 1: Behavioral Study (The Writers)
- Participants: 21 writers were randomly assigned to two groups.
- Conditions: The Control Group received no warning. The Treatment Group received a "conceptual prime"—a yellow banner stating their work would be scanned by an AI detection tool.
- Task: Participants wrote two opinion pieces (one "For" and one "Against" remote work) using an AI chatbot.
- Metrics: The study tracked interaction metrics (turns, duration, tokens), an engineered AI Overlap Score (measuring verbatim/near-verbatim reuse of AI output), and various stylometric features (lexical diversity, sentence length variance, pronoun usage, etc.).
Phase 2: Perceptual Study (The Judges)
- Participants: 251 independent judges.
- Task: A deceptive "forced-choice" evaluation. Judges were presented with 1,999 paired comparisons (one control document and one treatment document of the same stance). They had to select which document they believed was "written by a human."
- Design: Judges were unaware that both writers had access to AI; they were essentially testing if a human could distinguish between a writer who was "warned" and one who was not.

3. Key Contributions

Behavioral Insight: The study identifies that the threat of detection does not necessarily reduce AI reliance; instead, it polarizes behavior (users either iterate more intensely with the AI or abandon it entirely).
The "Detection Gap": The paper highlights a disconnect between computational stylometry and human perception. It demonstrates that humans can detect "human-like" qualities in AI-assisted text that current NLP features (like TTR, hedging, or contraction rates) fail to capture.
Experimental Framework: A robust methodology for testing the "observer effect" in AI-assisted writing through a deceptive, two-stage experimental design.

4. Results

Phase 1 (Writers):
- The treatment group showed higher engagement: they took more conversation turns, spent more time on tasks, and used more tokens.
- Crucially, the text was indistinguishable: There were no statistically significant differences between the groups in terms of AI overlap scores or stylometric features (lexical diversity, sentence structure, etc.).
Phase 2 (Judges):
- Despite the lack of measurable textual differences, judges selected the warned (treatment) documents as "human" 54.13% of the time, compared to 45.87% for the control group.
- This result was statistically significant ( $p = 0.000243$ ), rejecting the null hypothesis of random guessing.
- The effect was consistent across both writing stances (For/Against) and was amplified by judges with higher confidence and longer reading times.

5. Significance

The study concludes that the threat of scrutiny changes the "rhythm" or "feel" of AI-assisted writing in a way that is perceptible to humans but invisible to current automated detection algorithms.

This suggests two major implications:

For AI Detection: Current feature-based detection tools are likely missing subtle, high-level stylistic cues that humans use to judge authenticity.
For Human Behavior: The mere presence of an AI detector acts as a psychological catalyst that prompts users to "curate" or "re-frame" their AI interactions, potentially making the resulting text more "human-passing" without actually reducing the underlying reliance on AI.