A Multi-level Analysis of Factors Associated with Student Performance: A Machine Learning Approach to the SAEB Microdata

Technical Summary: A Multi-level Analysis of Factors Associated with Student Performance

Problem Statement
Identifying the determinants of academic success in Brazilian basic education is a complex challenge due to the country's vast socioeconomic heterogeneity. Traditional statistical approaches often struggle to capture the non-linear, multi-level interplay between individual student characteristics, teacher profiles, school infrastructure, and management practices. While Educational Data Mining (EDM) offers tools for pattern recognition, there remains a gap in applying systemic, multi-level machine learning approaches to the most comprehensive national dataset for basic education (SAEB) to inform equitable public policy. The core problem addressed is distinguishing between isolated individual factors and systemic, school-level determinants that drive student performance.

Methodology
The study employs a structured data mining pipeline using microdata from the System of Assessment of Basic Education (SAEB), administered by INEP. The methodology is characterized by four distinct phases:

1. Multi-level Data Integration: The researchers constructed a hierarchical dataset by joining four distinct data sources via school and class identifiers:

   • Student: Socioeconomic profiles and proficiency scores (Mathematics and Portuguese).
   • Teacher: Professional backgrounds, pedagogical practices, and perceptions of school climate.
   • School: Aggregated indicators, including socioeconomic levels and teacher training rates.
   • Principal: Management profiles, experience, and community engagement practices.
     The final dataset comprised 6,482,168 valid student records (9th grade and high school).

2. Preprocessing and Feature Selection:

   • Data cleaning involved listwise deletion of missing values and conversion of string-based percentages to floats.
   • Categorical variables were encoded using One-Hot Encoding, and features were standardized via StandardScaler.
   • The Boruta algorithm was applied to an initial set of 98 features to statistically identify relevant predictors, confirming 44 features. A final subset of 17 features was selected based on statistical significance and theoretical relevance to educational policy.

3. Modeling and Evaluation:

   • The study framed the task as a binary classification problem: predicting whether a student performs "Above Average" or "Below Average" based on composite proficiency scores.
   • Four tree-based ensemble algorithms were compared: Random Forest, XGBoost, LightGBM, and CatBoost.
   • Models were trained on an 80/20 train-test split. Performance was evaluated using Accuracy, Precision, Recall, F1-Score, and the Area Under the Receiver Operating Characteristic Curve (AUC-ROC).

4. Explainable AI (XAI):

   • To move beyond "black box" prediction, the study applied SHAP (SHapley Additive exPlanations) to the best-performing model. This allowed for both global feature importance analysis and local interpretation of individual predictions.

Key Contributions
The paper makes four primary contributions to the fields of Educational Data Mining and public policy analysis:

1. Novel Multi-level Model: The design and implementation of a predictive model that systematically integrates four distinct data layers (student, teacher, school, principal) from the national SAEB assessment.
2. Rigorous Comparative Evaluation: A comprehensive benchmarking of four powerful ensemble models, demonstrating that a Random Forest model outperforms gradient boosting variants (XGBoost, LightGBM, CatBoost) in this specific context without extensive hyperparameter tuning.
3. Systemic Feature Identification: A data-driven analysis that unequivocally identifies the school's average socioeconomic level as the dominant predictor, surpassing individual student or teacher characteristics in predictive power.
4. Actionable Policy Insights: The translation of complex model outputs into evidence-based insights for policymakers, quantifying the systemic nature of academic achievement and highlighting the potential of targeted interventions to reduce inter-school disparities.

Results

• Model Performance: The Random Forest classifier achieved superior performance with an Accuracy of 90.17% and an AUC of 0.9669. In contrast, the gradient boosting models (XGBoost, LightGBM, CatBoost) yielded significantly lower accuracies (ranging from 62.48% to 64.17%) and AUC scores (0.67–0.69) when run with default parameters.
• Classification Balance: The Random Forest model demonstrated balanced performance across both classes ("Above Average" and "Below Average"), with precision and recall scores consistently around 0.90–0.92, indicating a lack of bias toward either group.
• Feature Importance: The SHAP analysis revealed a clear hierarchy of influence:
  • Dominant Factor: The School's Average Socioeconomic Level was the single most powerful predictor. Higher socioeconomic levels (e.g., Nivel VI) strongly correlated with "Above Average" predictions.
  • Secondary Factors: Institutional variables such as the percentage of teachers with adequate training and student participation rates were highly influential.
  • Individual Factors: While student-level variables (e.g., parental education, number of bedrooms) were significant, their impact was secondary to the broader school ecosystem.
  • Teacher/Principal Variables: Individual teacher experience and principal tenure were relevant but less dominant than aggregate school-level indicators.

Significance and Claims
The paper claims that academic performance in Brazilian basic education is a systemic phenomenon deeply tied to the school's ecosystem rather than the result of isolated individual interventions. The study argues that the strong predictive power of the school's socioeconomic context validates the need for policies that address disparities between schools rather than focusing solely on individual student remediation.

By leveraging Explainable AI, the authors assert that their model serves as a diagnostic tool capable of shifting educational policy from "one-size-fits-all" approaches to targeted, evidence-based interventions. Specifically, the findings suggest that resource allocation (funding, qualified teachers, infrastructure) should be directed toward schools with lower socioeconomic profiles to foster educational equity. The study concludes that while individual characteristics matter, they are profoundly influenced by the collective context of the school, a finding that aligns with sociological theories of social reproduction and cultural capital.

The authors remain modest regarding causality, emphasizing that the model identifies statistical associations rather than deterministic causality. They acknowledge limitations, such as the use of listwise deletion for missing data and the cross-sectional nature of the analysis, suggesting future work could incorporate longitudinal data and advanced imputation techniques.