Layer Wise IndoBERT Representation Evolution for Indonesian Misinformation Detection

Technical Summary: Layer Wise IndoBERT Representation Evolution for Indonesian Misinformation Detection

Problem Statement
The rapid proliferation of misinformation on social media, particularly in low-resource languages like Bahasa Indonesia, presents a critical challenge for automated detection systems. Hoaxes in this context often rely on subtle semantic shifts, emotional framing, and contextual manipulation rather than explicit falsehoods, making them difficult to detect using surface-level lexical cues. While transformer-based models, specifically IndoBERT, have demonstrated strong predictive performance in Indonesian fake news detection, they are predominantly treated as "black boxes." Existing research focuses on accuracy metrics (precision, recall, F1-score) but lacks insight into how task-relevant features are encoded, transformed, and consolidated across the internal layers of the network. Furthermore, the functional differences between different sentence representation strategies (e.g., [CLS] tokens vs. mean-pooled embeddings) in this domain remain unexplored.

Methodology
The study employs a systematic, layer-wise representation analysis framework applied to fine-tuned IndoBERT models (Base, Large, and Lite variants) trained on a binary hoax detection task.

• Dataset: A curated dataset of 11,200 full-length news articles (balanced between hoax and non-hoax) was constructed by integrating data from professional Indonesian news portals and verified fact-checking platforms. The data was split into 80% training, 10% validation, and 10% testing.
• Model Training: Multiple IndoBERT variants were fine-tuned end-to-end using the Adam optimizer (learning rate $2 \times 10^{-5}$, batch size 32) for up to 10 epochs with early stopping.
• Representation Extraction: The study extracts hidden states from all transformer layers ($H_0$ to $H_L$). Two distinct sentence-level representation strategies are analyzed:
  1. [CLS]-based representation: The hidden state of the special classification token.
  2. Mean-pooled representation: The average of all token embeddings in the sequence.
• Analysis Metric: Cosine similarity is computed between sentence-level embeddings of adjacent layers to quantify representational change. Lower similarity indicates significant transformation, while higher similarity suggests stability or convergence.
• Qualitative Analysis: The study includes an inspection of misclassified samples, sensitivity to input length, and the impact of real-world noise (e.g., encoding errors, scraping artifacts) to contextualize quantitative findings.

Key Results

1. Predictive Performance: Fine-tuned IndoBERT models achieved high performance, with IndoBERT-Base reaching 98% accuracy and 99% F1-score, significantly outperforming classical baselines (Naïve Bayes, LSTM) and the multilingual mBERT model.
2. Three-Phase Representational Evolution: The layer-wise similarity analysis revealed a consistent, non-linear progression of representations across all IndoBERT variants:
   • Early Layers (Stability): High similarity between adjacent layers indicates that initial processing preserves pretrained linguistic and syntactic structures.
   • Middle Layers (Transformation): The most substantial drops in cosine similarity occur in the intermediate layers (approximately layers 4–9). This indicates a zone of active representational reorganization where task-specific semantic features are progressively formed and refined.
   • Upper Layers (Convergence): Similarity increases again in the final layers, suggesting a stabilization and convergence toward task-aligned representations suitable for classification.
3. Functional Differentiation of Representations:
   • [CLS] Representations: Exhibit sharper transitions and more aggressive specialization in the middle layers, aligning closely with the classification objective.
   • Mean-Pooled Representations: Show smoother, more gradual semantic transitions across layers, preserving distributed semantic continuity.
4. Robustness Observations: Qualitative analysis revealed that misclassifications are often driven by low-level input distortions (e.g., encoding errors, fragmented tokens) rather than high-level semantic ambiguity. These artifacts disrupt the representational geometry in early and middle layers, preventing the upper layers from forming stable task-aligned embeddings. Additionally, the model showed sensitivity to input length, struggling with truncated texts that lack sufficient contextual cues for middle-layer restructuring.

Significance and Claims
The paper claims that its primary contribution is shifting the analytical focus from purely predictive performance to the internal representational dynamics of transformer models in low-resource misinformation detection.

• Empirical Insight: The study provides empirical evidence that IndoBERT does not refine representations linearly. Instead, it identifies the middle layers as the critical "transformation zones" where hoax-related features emerge and are reorganized, while upper layers serve to consolidate these features.
• Interpretability: By characterizing the distinct roles of [CLS] and mean-pooled embeddings, the work highlights that task-oriented summarization and distributed semantic continuity evolve differently within the same architecture.
• Practical Implication: The findings suggest that the success of IndoBERT in hoax detection is a consequence of this structured, progressive refinement. Consequently, the reliability of the model is tightly coupled to the quality of input representations; low-level noise can destabilize the entire representational hierarchy, even if the final classifier is highly accurate.
• Future Direction: The authors posit that understanding these internal dynamics can inform the development of more efficient, interpretable, and diagnostically grounded transformer-based models, potentially enabling early-exit architectures or better handling of noisy real-world data.

The study concludes that while performance metrics confirm the efficacy of IndoBERT, layer-wise analysis is essential for understanding how the model processes complex linguistic phenomena like ambiguity and discourse manipulation, offering a foundation for more robust and transparent misinformation detection systems.