NOTAI.AI: Explainable Detection of Machine-Generated Text via Curvature and Feature Attribution

The paper introduces NOTAI.AI, an explainable framework that combines curvature-based signals, neural, and stylometric features within an XGBoost classifier to detect machine-generated text while using SHAP and an LLM layer to generate structured, natural-language rationales for its decisions.

The Gist

Imagine you are a detective trying to solve a mystery: Did a human write this story, or did a robot (an AI) write it?

In the past, robots were easy to spot because they sounded stiff or made weird grammar mistakes. But today's AI is like a master forger; it writes so smoothly and naturally that even experts get fooled.

The paper introduces NOTAI.AI, a new "super-detective" tool designed to catch these AI forgers. But unlike old tools that just give you a suspicious "yes/no" score, NOTAI.AI acts like a transparent detective who shows you exactly why they think a text is fake, using simple language anyone can understand.

Here is how it works, broken down with some everyday analogies:

1. The Three Eyes of the Detective

Most detectors look at a text with just one pair of eyes. NOTAI.AI puts on three different pairs of glasses to look at the same text from different angles. It combines these three views to make a final decision:

• The "Smoothness" Glasses (Curvature):
  Imagine a human writing a story is like walking through a forest. You might step on a rock, trip over a root, or take a sudden turn. It's a bit bumpy and unpredictable.
  AI, however, is like a robot on a perfectly paved highway. It takes the most direct, "smoothest" path every time.

  • The Trick: NOTAI.AI measures how "bumpy" the text is. If the text is too smooth and predictable (like a highway), it suspects a robot. If it has natural bumps and irregularities, it suspects a human.

• The "Vocabulary" Glasses (Stylometrics):
  This looks at the "fingerprint" of the writing.

  • Repetition: Humans get bored and use different words. AI sometimes gets stuck in a loop, repeating the same phrases or sentence structures.
  • Word Choice: Humans might use rare, weird words (like "hapax legomena" – words you only see once). AI tends to stick to the most common, safe words.
  • Punctuation: How many commas or periods are there? Humans have a unique rhythm; AI often has a robotic, uniform rhythm.

• The "Brain" Glasses (Neural Signals):
  This is a super-smart AI (called ModernBERT) that has read millions of books. It acts like a literary critic. It asks, "Does this sentence feel like something a human would say, or does it feel like something a computer would generate?" It gives a "gut feeling" score based on deep patterns.

2. The "Super-Brain" (The Meta-Classifier)

Once the three pairs of glasses gather their clues, they pass the notes to a Chief Detective (an algorithm called XGBoost).

• The Chief Detective doesn't just listen to one clue. It weighs them all together.
• Example: "The text is very smooth (AI clue), but it uses a very rare word (Human clue). Let's look at the punctuation... ah, the punctuation is too perfect. I'm 96% sure this is AI."

3. The "Translator" (Explainability)

This is the most important part. Old detectors would just say, "95% Chance: AI." That's not helpful if you are a teacher or a journalist trying to prove it.

NOTAI.AI has a Translator (an LLM) that turns the math into a story.

• Instead of: "SHAP value for Feature 4 is 0.45."
• It says: "We think this is AI because the text is unusually smooth, like a highway, and it repeats the phrase 'in conclusion' three times, which is a common robot habit."

It gives you a structured report with a summary and the top 3 reasons why, written in plain English.

4. The "Playground" (Interactive Demo)

The authors built a website where you can play with the tool.

• The "What-If" Game: You can turn off specific clues. For example, you can tell the detective, "Ignore the smoothness test, just look at the vocabulary."
• The Result: You can watch the detective change its mind in real-time. This helps you understand which clues matter the most.

Why Does This Matter?

In a world where AI can write essays, news articles, and emails that look perfect, we need tools that don't just guess, but explain.

• For Teachers: It helps spot AI essays without accusing a student unfairly.
• For Journalists: It helps verify if a news story was written by a human or a bot farm.
• For Everyone: It builds trust. You aren't just told the answer; you are shown the evidence.

In short: NOTAI.AI is like a detective that doesn't just point a finger; it hands you a magnifying glass, points out the specific footprints and fingerprints, and explains the whole story in a way that makes perfect sense.

Technical Summary

Here is a detailed technical summary of the paper "NOTAI.AI: Explainable Detection of Machine-Generated Text via Curvature and Feature Attribution."

1. Problem Statement

The rapid advancement of Large Language Models (LLMs) has made it increasingly difficult to distinguish between human-written and AI-generated text. While existing detectors exist, they face three critical challenges:

• Opacity: Many systems provide only a probability score without explaining why a text was flagged, limiting their utility for educators, journalists, and analysts.
• Robustness: Detectors often degrade significantly under domain shifts, decoding variations, and adversarial perturbations.
• Signal Limitation: Most approaches rely on a single class of signals (e.g., purely statistical, purely neural, or watermark-based) rather than integrating heterogeneous evidence.

2. Methodology

NOTAI.AI is a hybrid, explainable framework that operates on a four-stage pipeline: Extract → Decide → Explain → Present.

A. Feature Extraction (17 Interpretable Features)

The system aggregates three distinct types of signals to capture distributional, structural, and stylistic properties:

1. Neural Signals:
   • Uses a fine-tuned ModernBERT encoder to generate a probability score (bert_ai_score) capturing semantic fluency and generation priors.
2. Curvature-Based Statistical Signals:
   • Integrates Conditional Probability Curvature (CPC) derived from Fast-DetectGPT. This measures the second-order variation in token likelihood under local perturbations.
   • Premise: Human text exhibits irregular, high-variance probability landscapes, while AI text tends to be smoother and more stable.
3. Stylometric and Readability Features:
   • Readability: Flesch Reading Ease, sentence count, average sentence/word length.
   • Lexical Diversity: Type-Token Ratio (TTR) and Hapax Legomena ratio.
   • Surface Cues: Stopword ratio, cliché ratio, n-gram repetition frequency, and punctuation counts (commas, periods, semicolons).

B. Meta-Classification

• All 17 features are fed into an XGBoost (Extreme Gradient Boosting) classifier.
• This supervised meta-classifier learns non-linear interactions between the neural confidence scores, curvature statistics, and stylometric indicators to make the final binary decision (Human vs. AI).

C. Explainability Layer

To bridge the gap between technical attribution and user understanding, the system employs a two-tier explanation mechanism:

1. SHAP (Shapley Additive Explanations):
   • Calculates the contribution of each feature to the specific prediction.
   • Provides local interpretability (instance-level) and global interpretability (dataset-level feature importance).
   • Positive SHAP values indicate evidence for "AI," while negative values support "Human."
2. LLM-Based Structured Rationale Generation:
   • A structured prompt is sent to a Google Gemma-3-27b-it LLM.
   • Input: Raw text, predicted label, probability, and a list of top positive/negative feature values with their SHAP scores.
   • Output: A structured JSON containing a plain-language summary and specific 1-2 sentence explanations for the top features, translating mathematical attributions into natural language.

D. Deployment

• Implemented as an interactive web application (Flask backend, JavaScript frontend).
• Features a real-time analysis dashboard, visual gauges for confidence, and an interactive ablation menu allowing users to disable specific features to observe how the decision changes.

3. Key Contributions

• Hybrid Architecture: Successfully integrates curvature-based signals (Fast-DetectGPT), neural encoders (ModernBERT), and traditional stylometrics into a single supervised framework.
• Explainability by Design: Moves beyond "black box" scores by using SHAP to generate feature attributions and an LLM to convert them into structured, non-technical rationales.
• Interactive Tooling: Provides a user-facing demo that allows for real-time inspection of how specific signals (e.g., removing curvature data) influence the final verdict.
• Reproducibility: The source code, dataset, and demo video are publicly available.

4. Experimental Results

The system was evaluated on a balanced version of the RAID dataset (Dugan et al., 2024), constructed by downsampling AI-generated samples to achieve a 1:1 ratio with human text (to prevent bias toward the majority class).

• Performance Metrics:

  • NOTAI.AI Ensemble: Achieved 96.3% Accuracy and an F1 Score of 0.963.
  • Comparison: This outperformed individual component models:
    • Curvature Only: F1 = 0.888 (High Precision, lower Recall).
    • Stylometric Only: F1 = 0.871.
    • ModernBERT Only: F1 = 0.856.
  • The ensemble improved the best single-signal baseline by +7.5 F1 points.

• Feature Importance Analysis (SHAP):

  • The three most influential features were Conditional Probability Curvature (CPC), Type-Token Ratio (TTR), and the ModernBERT score.
  • CPC: Showed a non-linear relationship; values near zero or negative correlated with AI, while positive values correlated with human text.
  • TTR: A threshold of ~0.56 was observed; higher vocabulary diversity was often interpreted as AI-generated in this specific context.
  • ModernBERT: Acted as a confirmatory signal, contributing significantly only at high confidence levels.

5. Significance and Future Work

Significance:
NOTAI.AI addresses the critical need for transparent AI detection. By combining high-performance ensemble learning with human-readable explanations, it empowers non-expert users to understand the evidence behind a detection, fostering trust and utility in real-world scenarios like education and journalism.

Limitations & Future Directions:

• Robustness: Like all supervised detectors, it may struggle with severe domain shifts, unseen generators, or adversarial paraphrasing.
• Granularity: Current explanations are feature-level (macro). Future work aims to localize influential words and phrases at the token level (micro-level) via interactive heatmaps.
• Scope: Future iterations plan to expand to multilingual support and human-AI co-authored scenarios.