💬 NLP

From Perceptions To Evidence: Detecting AI-Generated Content In Turkish News Media With A Fine-Tuned Bert Classifier

This study presents the first empirical investigation into AI-generated content in Turkish news media by fine-tuning a BERT classifier on 3,600 articles to detect LLM rewriting, revealing that approximately 2.5% of examined content is AI-revised while demonstrating the model's high accuracy and temporal stability.

Original authors: Ozancan Ozdemir

Published 2026-02-17

📖 4 min read☕ Coffee break read

CC0 1.0

Original authors: Ozancan Ozdemir

Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the newsroom as a bustling kitchen. For decades, chefs (journalists) have cooked up stories by hand, chopping vegetables and stirring pots with their own hands. But recently, a new, incredibly fast robot chef (AI) has been hired. This robot can taste a recipe, understand the style of the kitchen, and rewrite the menu in seconds.

The problem? Sometimes, the robot's cooking looks so much like the human chef's that you can't tell the difference. And sometimes, the robot gets confused and puts a pineapple on a pizza that should be pepperoni (a "hallucination").

This paper is like a super-smart food critic who has been trained to taste-test thousands of dishes and figure out: "Was this cooked by a human, or did the robot rewrite it?"

Here is the breakdown of the study in simple terms:

1. The Mission: Catching the Robot Chefs

The author, Ozancan Ozdemir, noticed that while everyone is talking about AI in English news, nobody was checking if Turkish news outlets were using it. So, he built a "lie detector" specifically for the Turkish language.

2. The Training Camp (How the AI Learned)

To teach his detector, he couldn't just guess. He needed a training set, like a cooking class.

The "Human" Dishes: He gathered 1,800 news articles from 2021. Why 2021? Because that was before the big AI boom. He knew these were 100% written by humans.
The "Robot" Dishes: He took those same human articles and fed them into an AI (ChatGPT), giving it strict instructions: "Rewrite this to sound like the newspaper, but don't change the facts or the length."
The Result: He now had a dataset of 3,600 articles: half human, half AI-revised. He used this to train a "brain" (a model called Turkish BERT) to spot the subtle differences between human writing and robot rewriting.

The Analogy: Imagine a master sommelier tasting two wines. One is a vintage from a specific vineyard (Human), and the other is a perfect copy made by a machine (AI). At first, they taste the same. But after tasting thousands of pairs, the sommelier learns to spot a tiny, almost invisible difference in the "aftertaste" that only the machine leaves behind.

3. The Test: Did the Detector Work?

The "sommelier" was tested on a new set of dishes it had never seen before.

The Score: It got a 97% accuracy rate. It was incredibly good at telling the difference.
The Confidence: When the detector said, "This is human," it was 96% sure. When it said, "This is AI," it was just as sure. It wasn't guessing; it was confident.

4. The Big Reveal: How Much AI is in Turkish News?

Once the detector was ready, the author used it to scan over 3,500 real news articles from 2023 to 2026 (the years after AI became popular) from three different types of newspapers:

A neutral/center newspaper.
A pro-government newspaper.
An opposition newspaper.

The Findings:

The "Robot" Presence: On average, about 2.5% of the news articles had been touched by AI.
The Pattern: It wasn't a flood; it was a trickle. Most news (97.5%) was still written by humans.
The Trend: The use of AI spiked right after ChatGPT launched in 2023, dipped a bit, and then started rising again in 2026.
The Bias Check: Interestingly, the opposition newspaper used AI the least, while the government-aligned paper used it the most. But the detector worked equally well on all of them.

5. Why This Matters (The "So What?")

Think of the news as a trust bank. If you don't know if a story was written by a human or a robot, you might stop trusting the bank.

The Risk: AI can accidentally invent facts (like the paper mentions a football team lineup that doesn't exist) or hide biases.
The Solution: This study proves we can build tools to detect this. It's the first step toward making sure that when we read the news, we know who (or what) wrote it.

The Bottom Line

This paper is a warning and a tool. It says: "Hey, robots are quietly helping write the news in Turkey. We can't see them with our eyes, but we built a digital microscope that can spot them."

The author suggests that just like food labels tell us if a product is organic or processed, news articles should have a label saying, "Written by a human," or "Revised by AI." This keeps the public informed and the news trustworthy.

1. Problem Statement

The rapid integration of Large Language Models (LLMs) into journalism has raised concerns regarding transparency, trustworthiness, and the potential for bias and hallucinations in news content. While computational studies exist for English-language media, there is a significant empirical gap in Turkish media research. Existing Turkish studies rely on qualitative interviews or focus on "fake news" detection rather than identifying AI-generated or AI-revised content.

Key challenges specific to this domain include:

Linguistic Complexity: Turkish is an agglutinative language with rich morphology, making models trained on analytic languages (like English) unsuitable for direct transfer.
Lack of Ground Truth: There are no existing datasets or computational methods to quantify the prevalence of AI usage in Turkish newsrooms.
Political Polarization: Turkish media is highly polarized; understanding if AI adoption varies across political spectrums (pro-government, opposition, central) is crucial.

2. Methodology

A. Dataset Construction

The authors constructed a labeled dataset of 3,600 news articles to train a binary classifier (Human-written vs. AI-revised).

Sources: Articles were scraped from three major Turkish outlets representing distinct political orientations: a central outlet, a pro-government outlet, and an opposition-aligned outlet.
Human-Written Class (1,800 articles): Randomly selected from 2021, a period assumed to be pre-LLM adoption in Turkish journalism, ensuring a "pure" human baseline.
AI-Generated/Revised Class (1,800 articles): Created by using ChatGPT to rewrite the 2021 articles.
- Prompt Engineering: The prompt instructed the LLM to act as an editor for the specific newspaper, preserving facts, dates, and length while adapting tone, style, and fluency to match the outlet's editorial voice.
- Validation: The resulting AI texts showed high semantic similarity (cosine similarity $\approx$ 0.9) and comparable word counts to the originals, ensuring the model learns stylistic shifts rather than simple content differences.

B. Model Architecture & Training

Base Model: dbmdz/bert-base-turkish-cased, a Turkish-specific BERT model pre-trained on ~35GB of Turkish text (OSCAR, Wikipedia, OPUS). This model preserves case sensitivity (critical for Turkish dotted/undotted 'I' distinctions) and handles agglutinative morphology.
Fine-Tuning Strategy:
- Task: Binary classification (Human vs. AI).
- Head: A linear classification layer appended to the [CLS] token representation.
- Loss Function: Cross-entropy with Label Smoothing ( $\alpha=0.1$ ) to prevent overconfidence.
- Regularization: Dropout (0.2 on hidden states and attention), Gradient Clipping (norm 1.0), and Early Stopping (patience 2 epochs).
- Hyperparameters: AdamW optimizer, learning rate $2 \times 10^{-5}$ , batch size 16 (accumulated), mixed-precision (FP16) training.
- Data Split: Stratified 80/10/10 split (Train/Val/Test) to maintain class balance.

C. Evaluation Metrics

Performance was measured using Accuracy, Precision, Recall, and F1 Score (primary metric due to balanced costs of false positives/negatives). Additionally, prediction confidence scores were analyzed for external validation.

3. Key Contributions

First Labeled Dataset: Creation of the first dataset specifically for detecting AI-revised Turkish news, addressing a critical resource gap.
Deep Learning Baseline: Establishment of the first transformer-based baseline for this task in Turkish NLP, achieving state-of-the-art performance.
Empirical Measurement: Moving beyond self-reported journalist surveys to provide the first data-driven, computational characterization of AI usage in Turkish media over a multi-year period (2023–2026).
Replicable Framework: Documentation of the full pipeline (scraping, prompt-based augmentation, fine-tuning) for future research in under-resourced languages.

4. Results

A. Model Performance (Internal)

Test Set Performance: The fine-tuned model achieved an F1 score of 97.08% on the held-out test set.
Symmetry: Precision and Recall were nearly identical for both classes (Precision $\approx$ 0.99, Recall $\approx$ 0.98), indicating no systematic bias toward detecting one class over the other.
Convergence: The model converged rapidly, with validation loss reaching a minimum at Epoch 4.

B. External Validation (Real-World Application)

The trained model was applied to 3,500+ unseen articles from the same three outlets spanning 2023–2026.

Prevalence Estimate: Approximately 2.5% of news articles in the examined timeframe were identified as rewritten or revised by LLMs.
Temporal Trends: Usage peaked in 2023 (shortly after ChatGPT's release) and dipped slightly before rising again in early 2026 to 2023 levels.
Political Variation:
- Opposition outlets: Highest proportion of human-written content (~98%).
- Pro-government outlets: Lowest proportion of human-written content (~96.4%).
- Central outlets: Intermediate levels.
Confidence: The model predicted with high confidence (>96%) across all sources and years, suggesting stable decision boundaries regardless of political orientation.

5. Significance and Implications

Scientific Contribution: This study bridges the gap between qualitative perceptions and quantitative evidence in Turkish media studies. It proves that transformer models can effectively detect subtle stylistic shifts caused by LLMs in agglutinative languages.
Media Accountability: The findings highlight a "comfort gap" where audiences distrust AI content, yet a measurable portion of news is AI-assisted. The study argues for mandatory disclosure labels for AI-assisted journalism to preserve public trust.
Future Research: The paper outlines a path for expanding this work to multi-class detection (e.g., distinguishing between human-written, AI-rewritten, and fully AI-generated), analyzing specific news genres (sports, health), and employing long-context architectures to overcome the 512-token limit of standard BERT.

6. Limitations

Sample Size: The dataset (3,600 articles) is modest by modern standards, and the external validation lacks ground-truth labels (relying on model confidence).
Token Limit: The 512-token limit of BERT necessitates truncating longer articles, potentially losing context.
Representativeness: Only three outlets were used, which may not fully capture the diversity of the entire Turkish media landscape.

In conclusion, this paper provides a robust technical foundation for monitoring AI adoption in Turkish journalism, demonstrating that high-accuracy detection is feasible and that AI usage, while currently low (~2.5%), is a persistent and measurable factor in modern news production.