💬 NLP

EVIL-Detect for NLPCC 2026 Shared Task 6: LLM-Generated Text Detection

This paper presents EVIL-Detect, a multi-signal ensemble framework with conflict-aware fusion that achieved first place in the NLPCC 2026 Shared Task 6 by effectively detecting LLM-generated, human-written, and LLM-refined text in Chinese scenarios.

Original authors: Hongrui Bao, Hangyu Rong, Zhuoshang Wang, Yubing Ren, Yanan Cao

Published 2026-08-12

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Hongrui Bao, Hangyu Rong, Zhuoshang Wang, Yubing Ren, Yanan Cao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the internet as a giant, bustling library where books are written by both humans and a new, incredibly fast kind of librarian: Artificial Intelligence. For a long time, these AI librarians (called Large Language Models) could only write simple notes, but now they can write entire stories, essays, and articles that sound almost exactly like a human wrote them. This is a double-edged sword. On one hand, it's a superpower for creativity; on the other, it makes it hard to tell who actually wrote a piece of text. If a student submits an essay, how do we know if they wrote it, if an AI wrote it, or if they asked an AI to help polish their rough draft? This is the big puzzle scientists are trying to solve: building a "lie detector" for text that can spot the difference between a human, a robot, and a human-robot team-up.

In this story, a team of researchers from China has built a new tool called EVIL-Detect to solve this puzzle. They didn't just build a single detector; they built a "detective squad" that works together to catch the truth. Their system was tested in a major competition called NLPCC 2026, where it faced a tricky challenge: distinguishing between three types of text—purely human writing, purely AI writing, and text that was written by a human but then edited or polished by an AI. While other methods tried to guess the answer with a single brain, EVIL-Detect used a team approach, combining different clues to get the right answer. In the final score, their system achieved a macro-F1 score of 0.8888, ranking first in the official evaluation.

The Problem: The "Three-Way" Mystery

Most old-school detectors were like binary switches: they could only say "Human" or "AI." But the real world is messier. People often use AI to fix their grammar or make their writing sound better. This creates a third category: LLM-refined text. Imagine a student writes a draft (Human), then asks an AI to "make this sound more professional" (Refined), or an AI writes a whole story from scratch (AI). The challenge is that the "Refined" text looks like a mix of both, making it very hard to spot.

The researchers found that if you just try to teach a computer to guess all three at once, it gets confused. In their tests, a standard method that tried to learn all three categories directly performed terribly, scoring only 0.1690 on the test. It was like trying to teach a dog to speak three languages at once; it just gave up.

The Solution: The Detective Squad (EVIL-Detect)

Instead of one big brain, the team built EVIL-Detect (which stands for Edit-aware View-Integrated Learning for Detection). Think of this system as a team of four different detectives, each looking at the text through a different pair of glasses. They don't just vote; they talk to each other and resolve their disagreements to reach a final verdict.

Here are the four detectives and what they look for:

The "Edit Detective" (Supervised Training): This detective is trained to measure how much the text was changed. It looks at the difference between a human's original style and the final version. If the text is pure human, the change is zero. If it's pure AI, the change is huge. If it's "Refined," the change is somewhere in the middle. This detective gives a score on a sliding scale from 0 to 1, telling the team how "edited" the text feels.
The "Zero-Shot Detective" (EchoPrompt): This detective doesn't need to be trained on the specific data. It uses a clever trick: it asks the text, "Does this sound like something a robot would say?" by comparing how likely different AI models are to generate that text. If the text sounds very much like a robot's natural output, this detective raises a red flag for "AI-generated."
The "Word Count Detective" (Lexical Statistics): This detective looks at the small, boring details: how often certain words or letter combinations appear. Humans and robots tend to use different "fingerprints" of words. This detective counts these patterns to see if the text leans more toward human habits or machine habits.
The "Rule Book Detective" (Text Rules): This is the safety net. It looks for obvious giveaways that a human wouldn't make, like leftover computer code (HTML tags like <div> or <html>) or unfinished sentences that look like a robot got cut off mid-sentence. If it sees these, it immediately flags the text as AI.

How They Work Together: The "Conflict-Aware" Meeting

The magic of EVIL-Detect isn't just having four detectives; it's how they handle it when they disagree. Sometimes the "Edit Detective" thinks a text is "Refined," but the "Word Count Detective" thinks it's "AI."

Instead of just averaging their votes (which would be like a democracy where the minority opinion gets ignored), the team uses a Conflict-Aware Fusion strategy. They have a set of rules for resolving arguments:

If two detectives agree, they go with that answer.
If they disagree, they look at the specific clues. For example, if the "Edit Detective" is unsure between "Human" and "Refined," the team checks the "Zero-Shot Detective" to see if there are strong signs of AI.
Finally, the "Rule Book Detective" gets the last word. If the text has obvious computer code in it, the system overrides everything else and calls it AI.

The Results: Winning the Competition

When the team tested their system on the official competition data, which included tricky, unseen examples designed to break detectors, EVIL-Detect performed incredibly well.

On the first test set (testp1), it scored a macro-F1 of 0.8913.
On the second, harder test set (testp2), it scored 0.8888.

The system was particularly good at spotting pure AI text (scoring 0.9219 on that category) and did a solid job with the tricky "Refined" text (scoring 0.8407). The researchers found that the "Edit Detective" was the strongest single member of the team, but the system only reached the top spot because the other detectives helped fix the mistakes the main one made.

What They Learned (and What They Rejected)

The paper also tested some other ideas to see if they worked, and they found that some popular methods actually failed in this specific three-way challenge.

Direct Training Failed: They tried training a single AI model to just "guess" the three categories directly. It was a disaster, scoring only 0.1690. This proved that you can't just throw data at a model and expect it to understand the nuance between "Human," "AI," and "Refined" without help.
Old Detectors Struggled: They also tried a method called "Binoculars," which is good for simple binary detection, but it only scored 0.3928 here. It couldn't handle the complexity of the three classes.

The researchers suggest that the secret to success wasn't a single "super-model," but rather a multi-signal ensemble. By combining the "edit strength" (how much was changed), the "likelihood" (does it sound like a robot?), and the "word patterns," they created a system that is robust enough to handle the messy reality of how people actually use AI today.

In short, EVIL-Detect shows that to catch the truth in a world of AI and humans, you need a team of specialists who can argue, compromise, and double-check each other, rather than relying on a single guess.

Technical Summary: EVIL-Detect for NLPCC 2026 Shared Task 6

Problem Definition
The paper addresses the challenge of detecting text generated or refined by Large Language Models (LLMs) within realistic Chinese scenarios. While existing literature predominantly focuses on binary classification (distinguishing Human-Written Text [HWT] from LLM-Generated Text [LGT]), the NLPCC 2026 Shared Task 6 introduces a more complex, three-class classification problem:

HWT: Purely human-written text.
LGT: Text fully generated by an LLM.
HLT: LLM-refined text (human-written text polished or rewritten by an LLM).

The authors note that this setting reflects real-world AI-assisted writing but presents significant challenges due to strong out-of-distribution (OOD) shifts between training data (derived from CUDRT) and evaluation data (derived from DetectRL-X), which involve unseen domains, generators, and generation schemes. Direct adaptation of binary classifiers or standard supervised fine-tuning (SFT) to this three-class setting was found to be insufficiently robust, with some baseline SFT approaches achieving macro-F1 scores as low as 0.1690.

Methodology: EVIL-Detect
The authors propose EVIL-Detect (Edit-aware View-Integrated Learning for Detection), a multi-signal ensemble framework designed to handle the continuum between HWT, HLT, and LGT. The system architecture integrates four distinct evidence sources through a conflict-aware fusion strategy:

Supervised Training Module (Edit-extent Regression):
- Inspired by EditLens, this module treats detection as a continuous regression problem rather than discrete classification.
- It defines an editing-extent target axis where HWT is 0, LGT is 1, and HLT is a value $d(h, t)$ representing the distance between the human source and the refined text.
- Two variants are implemented:
  - EditLens: Uses surface-level soft character $n$ -gram distances. The optimal configuration (rank044) uses $n$ -gram orders 1–7 with linearly increasing weights.
  - Soft-EditLens: Uses semantic phrase-level soft matching via embeddings to capture preserved expressions despite lexical replacement. It outputs both a continuous regression score and an ordinal bucket index.
- Both variants utilize Qwen models with Low-Rank Adaptation (LoRA).
Zero-Shot Detection Module:
- Utilizes EchoPrompt, a training-free method that computes likelihood-contrast scores between instruction-tuned and base models.
- Instead of standalone predictions, multiple prompt/model branches (using Qwen3.5 and Qwen2.5) generate binary "LGT-support votes" to serve as auxiliary evidence.
Lexical Frequency Statistics Module:
- Constructs label-wise frequency lexicons from the training set to identify discriminative $n$ -grams.
- Computes smoothed label-conditional probabilities and aggregated log-odds scores (e.g., LGT-vs-HWT, AI-vs-HWT) to provide lightweight surface-level evidence.
Fusion and Decision Module:
- Conflict-Aware Integration: The system does not use uniform averaging. Instead, it uses calibrated decision boundaries to convert EditLens scores into base labels. If two calibrated boundaries disagree, a resolver $R(\cdot)$ is triggered.
- The resolver uses Soft-EditLens regression/bucket scores and the aggregated LGT-support votes to resolve conflicts between specific class pairs (e.g., HWT/HLT, HLT/LGT).
- High-Precision Text Rules: As a final safeguard, conservative rules correct specific artifacts, such as HTML/XML structural markup or script residues (indicative of LGT) and explicit rewriting traces (indicative of HLT).

Key Contributions

Analysis of the Chinese Three-Class Setting: The paper analyzes the performance of representative detection methods in the specific context of HWT, LGT, and HLT, demonstrating that direct adaptation of binary methods fails under OOD shifts.
EVIL-Detect Framework: The proposal of a conflict-aware ensemble framework that integrates edit-extent regression, zero-shot likelihood evidence, lexical statistics, and rule-based corrections.
State-of-the-Art Performance: The system achieved a macro-F1 score of 0.8888 on the official test set (testp2), ranking first in the NLPCC 2026 Shared Task 6.

Experimental Results

Performance: The system achieved a macro-F1 of 0.8913 on testp1 and 0.8888 on testp2.
Class-wise Performance: LGT detection was the strongest (F1 $\approx$ 0.92), while HLT remained the most challenging class (F1 $\approx$ 0.84), consistent with its intermediate nature.
Ablation Studies:
- The EditLens component alone achieved 0.8411 macro-F1.
- Adding conflict-aware integration and text rules improved the score to 0.8816.
- The full fusion system (0.8888) significantly reduced confusion errors, specifically reducing LGT $\to$ HLT errors from 59 to 30 and HLT $\to$ LGT errors from 58 to 28 compared to the EditLens-only baseline.
- Alternative designs, such as direct generative SFT classifiers (e.g., GLM-4-9B, Qwen3.5-9B), performed poorly (macro-F1 < 0.19), validating the authors' design choice to use editing-extent modeling as the base signal rather than relying solely on generative classification.

Significance
The paper claims that robust three-class detection in realistic, OOD settings cannot rely on a single signal. Instead, the integration of edit-strength modeling, likelihood-based evidence, lexical cues, and high-precision corrections provides the necessary complementary information to distinguish between human, machine-generated, and machine-refined text. The work highlights the necessity of conflict resolution strategies when fusing heterogeneous signals for fine-grained text detection tasks.