Technical Summary: Optimizing Machine Learning in Tabular Social Science Datasets

Problem Statement

Machine learning offers powerful tools for identifying complex interactions and nonlinear patterns in social science, yet predictive performance often remains low, particularly when working with small sample sizes common in surveys and studies of rare events. In the context of the Predicting Fertility (PreFer) data challenge, researchers were tasked with predicting births and adoptions using Dutch survey and administrative data. The core challenge addressed in this paper is how to improve predictive accuracy in tabular social science datasets where sample sizes are limited and the number of available features may be insufficient to capture the full complexity of the outcome.

The authors frame the problem through the lens of two sources of prediction error:

1. Learning Error: The gap between predictions and the true conditional mean for identical individuals, caused by the model failing to learn the relationship between features and outcomes. This is often exacerbated by small sample sizes.
2. Irreducible Error: The gap between the true conditional mean and the actual outcome for identical individuals, which can only be reduced by changing the set of features (i.e., adding more relevant information).

Methodology

The authors employed two specific data engineering strategies to effectively create "more" data, addressing both learning error (via sample size expansion) and irreducible error (via feature expansion). These strategies were applied to the Longitudinal Internet studies for the Social Sciences (LISS) panel survey data.

1. Time-Shift Data Augmentation

This strategy addresses learning error by artificially increasing the sample size.

• Concept: Leveraging the longitudinal nature of panel data, the authors repurposed observations from earlier time periods to create additional training cases.
• Implementation:
  • The original dataset was in a "wide" format (one row per individual with multiple years of data). The authors pivoted this to a "long" format where each row represents an entity-time-period (a "person-year").
  • They calculated outcomes for an earlier three-year period (2018–2020) and appended these as new rows to the training data.
  • Temporal Alignment: Features were systematically renamed and shifted forward by three years to maintain the correct temporal relationship between predictors and outcomes. For example, a 2017 fertility intention variable was renamed to match the 2020 variable, as both represented intentions measured one year prior to their respective outcome windows.
  • Value Adjustments: Time-dependent values (e.g., birth years) were adjusted by adding three years. Monetary values (e.g., income) were adjusted using the Consumer Price Index to account for inflation.
  • Drift Mitigation: To address potential "temporal drift" (changes in data patterns over time), a binary feature was added to indicate whether a row originated from the original period or the time-shifted period, allowing the model to learn systematic differences.
• Result: This strategy nearly tripled the sample size from 987 to 2,839.

2. Partner Linkage

This strategy addresses irreducible error by increasing the number of features.

• Concept: Many social science datasets store household members in separate rows, preventing models from utilizing information about spouses or cohabiting partners when predicting an individual's outcome.
• Implementation:
  • The authors identified partners within the LISS survey using household IDs and relationship position labels (e.g., "wedded partner," "unwedded partner").
  • Strict filters were applied to ensure accurate pairing, including cross-referencing self-reports of cohabitation and verifying that reported genders and birth years matched between partners.
  • Once linked, features from the partner's survey responses (e.g., fertility intentions, gynecologist visits, birth year) were merged into the focal person's row.
  • Linkage was performed only when both partners had non-missing outcomes to ensure consistency between training and holdout sets.
• Result: Partner data was successfully linked for approximately 35% of all person-years with partners, adding 14 partner-specific features to the model.

Model and Evaluation

• Algorithm: The primary model used was XGBoost (a gradient boosting framework) with hyperparameter tuning via grid search. Logistic regression was also tested as a proof-of-concept.
• Feature Engineering: Beyond the two main strategies, the authors constructed scales for abstract attitudes (e.g., traditional fertility attitudes) and "coalesced" data over time (selecting the most recent value for repeated measures).
• Validation: The study utilized 5-fold cross-validation. Crucially, to prevent data leakage and temporal drift, time-shifted data was included only in the training folds, not the test folds. Household members were kept within the same fold to prevent information leakage.
• Metrics: Performance was evaluated using $R^2_{CV}$ (cross-validated R-squared), F1 score, and other standard metrics. Confidence intervals were generated via bootstrapping.

Key Results

The study demonstrates that both data engineering strategies independently and jointly improved predictive performance:

1. Baseline Performance: Using the original training set (987 cases), the XGBoost model achieved an $R^2_{CV}$ of 0.543.
2. Individual Strategy Impact:
   • Partner Linkage: Increased $R^2_{CV}$ to 0.557 (+0.013). The authors note this gain was modest partly because partner data could only be merged for ~35% of partnered individuals.
   • Time-Shift Augmentation: Increased $R^2_{CV}$ to 0.581 (+0.037).
3. Combined Impact: Applying both strategies together raised $R^2_{CV}$ to 0.591, an improvement of +0.047 over the baseline.
4. Contextual Magnitude:
   • The combined improvement of +0.047 was roughly equivalent to 30% of the total performance gain achieved by the full winning model over a simple "fertility intentions" baseline.
   • The gain from the combined strategies was larger than the gain achieved by hyperparameter tuning alone (which improved performance by ~0.023).
   • In terms of the primary competition metric (F1 score), the combined strategies improved performance by 0.030, a difference nearly as large as the gap between the winning model and the second-place model.
5. Generalizability: The strategies also improved performance when applied to a logistic regression model with baseline features and a logistic regression model with the full feature set, suggesting the benefits are not specific to tree-based models.

Significance and Claims

The paper claims that the winning performance in the PreFer challenge was driven not by an innovative model architecture (as the authors used a standard XGBoost model), but by creative data engineering that transformed existing data into effectively "more" data.

• Practical Utility: The authors argue that these strategies are broadly applicable to a wide range of social science datasets, including longitudinal surveys and administrative records, particularly where sample sizes are small.
• Addressing Error Sources: The work provides a concrete case study of how to simultaneously address learning error (via sample size expansion) and irreducible error (via feature expansion).
• Data Stewardship: The paper concludes with recommendations for data stewards to structure data in machine-parsable formats (e.g., consistent variable naming across waves, clear household IDs) to facilitate such augmentation techniques.
• Modest Claims: The authors remain modest, acknowledging that the benefits of these strategies depend on the specific prediction task. They note that temporal drift can reduce the utility of time-shift augmentation in rapidly changing contexts, and that partner linkage may lead to overfitting if partner characteristics are irrelevant to the outcome. They suggest that the "curse of dimensionality" associated with adding features might be alleviated by the increased sample size provided by time-shift augmentation.

In summary, the paper posits that social scientists often have more data available than they realize, and that strategic augmentation and linkage can significantly enhance machine learning performance in tabular social science contexts.