Leveraging Machine Learning Techniques to Investigate Media and Information Literacy Competence in Tackling Disinformation

This study employs machine learning models on data from 723 education and communication students to predict Media and Information Literacy competencies regarding disinformation, revealing that complex algorithms and factors like academic year and prior training significantly enhance prediction accuracy to guide targeted educational interventions.

The Gist

Imagine the internet as a massive, bustling marketplace. In this marketplace, there are honest vendors selling fresh, truthful information, but there are also clever tricksters selling "fake fruit" wrapped in shiny, attractive packaging. This is disinformation.

For a long time, experts have worried that young people, who spend a lot of time in this digital marketplace, don't have the right tools to tell the difference between the fresh fruit and the fake fruit. They need Media and Information Literacy (MIL)—which is basically a "superpower" that helps you spot the fakes, check the ingredients, and decide what to buy.

This paper is about a group of researchers who decided to build a smart robot to help them understand how well future teachers and journalists (the people who will eventually teach us all) are using this superpower.

Here is the story of their experiment, broken down simply:

1. The Mission: Finding the "Fake Fruit" Detectives

The researchers gathered 723 university students studying to become teachers or communicators. They asked them a series of questions (a survey) to see how good they were at:

• Knowing what fake news looks like.
• Doing the right things to check if news is real.
• Feeling responsible for stopping the spread of lies.

2. The Tool: The "Crystal Ball" Robot

Instead of just looking at the survey answers with a magnifying glass (which is how scientists usually do it), they used Machine Learning (ML).

• Think of ML like a super-smart detective robot. You feed it all the survey data, and it tries to find hidden patterns that humans might miss.
• The robot's job was to do three things:
  1. Guess the Student's Major: Could the robot tell if a student was studying Education or Communication just by looking at their answers?
  2. Find the Clues: Which questions were the most important? Was it knowing the definition of fake news, or was it having taken a class on the topic?
  3. Predict the Score: If we know a student's age, gender, and training, can the robot guess how good they will be at spotting lies?

3. The Results: What the Robot Found

The robot tried on different "hats" (different algorithms) to see which one worked best.

• The "Brainy" Robots Won: The simple robots (like a single decision tree) were okay, but the complex, team-based robots (like Random Forest and Support Vector Machines) were much better. They were like a team of detectives working together, sharing clues, rather than one detective working alone.
• The "Training" Clue: The most important clue the robot found was prior training. Students who had already taken a class on disinformation were much better at spotting the fake fruit. It's like having a map of the marketplace; if you've been there before, you know where the tricksters hide.
• The "Year" Clue: The robot also noticed that students in their final year of university were generally better at this than first-year students. They had been "training" longer.

4. The Big Takeaway

The study proved that complex tools are needed to solve complex problems. You can't just ask, "Do you know what fake news is?" and expect a simple answer. You need to look at the whole picture: their education, their training, and their attitudes.

Why does this matter?
Imagine you are a coach trying to train a sports team. If you only look at one stat (like speed), you might miss that a player is actually great at strategy. This study says that schools need to use these "smart robot" insights to design better classes.

• Don't just teach facts: Teach students how to think and act like detectives.
• Start early: Since training makes a huge difference, we need to start teaching these skills before students even get to university.
• Personalize the training: Since different students have different strengths, we can use these models to create custom lessons for them.

In a Nutshell

The researchers built a digital crystal ball to see how well future teachers can spot lies online. They found that the smartest tools (Machine Learning) can spot patterns that simple math misses. The biggest lesson? Training works. If you teach people how to spot the "fake fruit" in the digital marketplace, they become much better at protecting themselves and others from getting tricked.

Technical Summary

1. Problem Statement

The rapid spread of disinformation in digital environments poses a significant threat to democratic societies, particularly among young people who are highly connected but often lack robust Media and Information Literacy (MIL) training. While MIL is recognized as a critical competency for critical thinking and responsible media engagement, predictive modeling of MIL skills specifically in the context of disinformation remains underexplored.

Existing literature often relies on traditional statistical analyses or descriptive studies. There is a gap in using advanced Machine Learning (ML) techniques to:

• Predict MIL competencies based on sociodemographic and academic variables.
• Identify the specific factors (e.g., prior training, academic year) that most strongly influence a student's ability to detect and counter disinformation.
• Determine whether complex ML models outperform simpler approaches in this educational domain.

2. Methodology

Research Design and Data Collection

• Study Type: Quantitative, cross-sectional study.
• Participants: $N = 723$ university students from Spain, comprising two fields of study: Education ($n=435$) and Communication ($n=288$).
• Instrument: An ad-hoc questionnaire titled "Perception of Future Edu-Communicators on Disinformation."
  • Structure: 25 items on a 4-point Likert scale.
  • Dimensions:
    1. Sociodemographics: Gender, age, academic year, field of study, prior training.
    2. MIL Theoretical Dimensions: Knowledge, Skills, Attitudes.
    3. Responsibility: Perception of professional duty in combating disinformation.
  • Validation: Expert judgment (5 specialists) and psychometric analysis (Cronbach's $\alpha$ = 0.77; KMO = 0.85). Exploratory Factor Analysis (EFA) confirmed a 3-factor structure explaining 66.86% of variance.

Data Preprocessing

• Tools: SPSS v.25 and Python (scikit-learn, LightGBM).
• Processing:
  • Categorical variables were encoded using One-Hot Encoding.
  • Scores were aggregated for four target categories: Knowledge, Skills, Attitudes, and Responsibility.
  • The final dataset contained 30 numerical features and 723 instances.
  • Split: 80% Training / 20% Testing (stratified to maintain class proportions).
  • Handling Imbalance: The "Communication" class was smaller than "Education"; models used balanced class weights, and F1-score was prioritized over accuracy for classification evaluation.

Experimental Tasks

The study executed three distinct ML tasks:

1. Classification: Predicting the student's academic branch (Education vs. Communication).
2. Feature Selection: Identifying the most relevant variables for prediction using four methods:
   • SelectKBest (SKB) - ANOVA scoring.
   • Forward Feature Selector (FFS) - Sequential optimization.
   • Recursive Feature Elimination (RFE).
   • Feature Importance via Decision Tree (DT).
3. Regression: Estimating the specific scores students would achieve in the four MIL dimensions (Knowledge, Skills, Attitudes, Responsibility).

Algorithms Evaluated

• Classification: Support Vector Machine (SVM), Decision Tree (DT), Random Forest (RF), LightGBM, k-Nearest Neighbors (KNN).
• Regression: Linear Regression (LR), Ridge Regression (RR), DT Regressor, RF Regressor.
• Hyperparameter Tuning: Extensive grid search was performed for each algorithm (e.g., kernel types for SVM, tree depth for DT/RF, regularization $\alpha$ for Ridge).

3. Key Results

A. Questionnaire Validation

The instrument demonstrated high reliability. EFA confirmed three distinct factors:

1. Knowledge: Conceptual understanding of disinformation.
2. Skills/Behaviors: Ability to verify sources and discern truth.
3. Attitudes/Responsibility: Ethical commitment to combating misinformation.

B. Classification Performance (Academic Branch)

• Best Model: Support Vector Machine (SVM) with an RBF kernel achieved the highest performance.
  • Test F1-Score: 0.815
  • Test Accuracy: 0.814
• Comparison:
  • Ensemble methods (LightGBM: 0.792 F1; RF: 0.744 F1) performed well but slightly below SVM.
  • Decision Tree (DT) performed the worst (0.679 F1), indicating that single-tree approaches are insufficient for this complexity.
  • KNN showed moderate stability (0.779 F1).
• Conclusion: Complex, non-linear models significantly outperform simpler tree-based or distance-based classifiers in distinguishing between Education and Communication students' MIL profiles.

C. Feature Selection Analysis

• Top Predictors: The Disinformation Training variable (whether the student had prior training) was the most consistently selected feature across all methods (appearing in all 4 selection techniques).
• Other Key Features: Academic Year, specific Knowledge items (K1, K2, K3), and Attitude/Responsibility items (A3, R5).
• Best Selector: Forward Feature Selector (FFS) yielded the best generalization (Test F1: 0.756), suggesting that iterative combination of features is superior to univariate statistical ranking alone.

D. Regression Performance (Competency Prediction)

Different models were optimal for different competency dimensions:

• Knowledge: Best predicted by Linear Regression (LR) (Test RMSE: 2.287), suggesting a linear relationship between inputs and knowledge scores.
• Skills: Best predicted by Random Forest (RF) (Test RMSE: 1.835), indicating non-linear interactions.
• Attitudes: Best predicted by Ridge Regression (RR) (Test RMSE: 1.598), the lowest error overall, benefiting from regularization.
• Responsibility: Best predicted by Random Forest (RF) (Test RMSE: 2.064).

4. Key Contributions

1. Methodological Innovation: This is one of the first studies to apply a comprehensive suite of ML algorithms (classification, feature selection, and regression) specifically to predict MIL competencies against disinformation in higher education.
2. Model Superiority: Demonstrates that complex models (SVM, RF, LightGBM) are necessary for accurately modeling MIL, outperforming traditional linear or single-tree approaches.
3. Variable Identification: Empirically validates that prior training in disinformation and academic year are the most critical predictors of MIL competence, more so than gender or age.
4. Dimension-Specific Modeling: Shows that different MIL dimensions require different modeling approaches (e.g., linear models for knowledge vs. ensemble models for skills), offering a nuanced view of competency assessment.

5. Significance and Implications

• Educational Policy: The findings suggest that educational interventions should not be "one-size-fits-all." Strategies should be tailored based on the student's academic year and prior training status.
• Curriculum Design: Since prior training is a dominant predictor, universities must integrate continuous, structured MIL training into the curricula of both Education and Communication degrees, rather than relying on one-off workshops.
• Predictive Analytics: ML tools can be used by institutions to identify at-risk students (those with low predicted MIL scores) and deploy targeted, personalized interventions.
• Future Research: The study highlights the need for larger, multi-regional datasets and the integration of performance-based assessments (beyond self-reports) to further refine these predictive models.

6. Limitations

• Self-Reported Data: Reliance on student self-perception may introduce social desirability bias.
• Sample Scope: The sample was limited to two specific degrees in a single region of Spain, potentially limiting the generalizability of the results to other disciplines or cultural contexts.
• Data Volume: While sufficient for the study, ML models generally benefit from larger datasets to reduce variance and improve generalization.