TRNEWS-2025: A Large-Scale, Manually Annotated Turkish News Dataset for Text Classification and NLP Research

Technical Summary: TRNEWS-2025

Problem Statement
The field of Turkish Natural Language Processing (NLP) faces a scarcity of high-quality, up-to-date, and domain-diverse labeled datasets required for effective text classification and machine learning research. While existing resources such as TTC-4900 and TTC-3600 exist, they often suffer from limited category diversity, outdated content, narrow subject coverage, or restricted access. Furthermore, many current datasets focus on short texts (e.g., social media comments) or specific tasks like sentiment analysis, failing to capture the long contextual structures and linguistic diversity inherent in news articles. There is a specific need for a comprehensive, multi-class news dataset that reflects current language usage and social issues up to 2025 to support reproducible research in Turkish NLP.

Methodology
The study introduces TRNEWS-2025, a large-scale, manually annotated Turkish news dataset. The methodology encompasses data collection, annotation, preprocessing, and experimental evaluation:

• Data Collection: News texts were gathered via web scraping from open-access Turkish online newspapers and media portals between 2023 and 2025. The collection adhered to strict inclusion criteria: texts must be in Turkish, contain full content, match a specific category, and be non-duplicate. Exclusion criteria removed advertisements, opinion pieces, user comments, incomplete texts, and automatically generated content.
• Labeling Scheme: The dataset is structured around nine distinct categories: Magazine, Politics, Sports, Arts–Culture, Health, Finance–Economy, Science–Technology, Tourism, and Environment.
• Annotation Process: A multi-labeler approach was employed where each text was reviewed by at least two annotators. Disagreements were resolved under academic supervision. An annotation guideline was established to handle borderline cases (e.g., intersections between economics and politics). Inter-annotator reliability was measured using Cohen's Kappa, yielding a coefficient greater than 0.60, indicating good agreement.
• Preprocessing: Basic cleaning steps included UTF-8 encoding, HTML tag removal, punctuation normalization, lowercasing, and duplicate removal. Notably, advanced linguistic processing (e.g., lemmatization) was intentionally omitted to allow researchers the flexibility to apply their own preprocessing strategies.
• Experimental Evaluation: To assess the dataset's usability and distinctiveness, two distinct model paradigms were trained and evaluated:
  1. BERT: A modern transformer-based model utilizing self-attention mechanisms.
  2. BiLSTM+GloVe: A classic deep learning architecture combining Bidirectional Long Short-Term Memory networks with global word co-occurrence embeddings.
     The models were evaluated using standard metrics including accuracy, precision, recall, and F1-score.

Key Results

• Dataset Statistics: The dataset contains a total of approximately 555,000 news articles. The distribution is multi-class, with "Magazine" being the largest category (315,917 samples) and "Environment" the smallest (15,610 samples). Text lengths vary significantly, supporting both traditional and deep learning approaches.
• Model Performance:
  • BERT: Achieved a validation accuracy of approximately 86%. The model demonstrated strong performance in classes such as Magazine, Sports, Health, and Politics. However, the study noted that training accuracy curves did not follow typical epoch-based convergence patterns, leading the authors to rely on validation performance for evaluation.
  • BiLSTM+GloVe: Showed a more stable learning process with consistent declines in both training and validation losses, indicating good generalization without overfitting. It achieved a test accuracy of 91.6% with significantly lower computational time compared to BERT.
• Confusion Analysis: Both models exhibited confusion between semantically overlapping categories, specifically between Arts–Culture and Magazine, and between Environment and Politics/Finance-Economy. The authors interpret this not as a dataset flaw, but as a reflection of the natural semantic overlaps found in real-world news language.

Significance and Claims
The paper positions TRNEWS-2025 as a significant contribution to Turkish NLP by providing a reliable, up-to-date, and comprehensive source for multi-class news classification. Its primary significance lies in:

1. Recency and Diversity: Unlike older datasets, it captures language usage and social issues up to 2025 across nine diverse domains.
2. Reproducibility: The dataset is open-access and manually annotated with high inter-annotator agreement, facilitating reproducible research.
3. Versatility: The study demonstrates that the dataset is compatible with both modern transformer-based models (BERT) and classical deep learning architectures (BiLSTM), making it a suitable benchmark for comparative model performance evaluations.
4. Real-World Representation: The observed classification confusions between related categories suggest the dataset reflects the complexity of actual news language rather than an artificially rigid structure.

The authors conclude that TRNEWS-2025 serves as a robust infrastructure for various research areas, including Turkish text classification, topic identification, explainable AI (XAI) studies, and language model evaluations.