📊 statistics

Feature Selection for Multidimensional Poverty Measurement: Taming Indicator Dimensionality

This paper demonstrates that machine learning-based feature selection can significantly reduce the computational and interpretive burden of multidimensional poverty measurement by identifying a small, non-redundant subset of indicators that maintains classification accuracy comparable to full indicator sets, as evidenced by an analysis of India's IHDS-II data.

Original authors: Kan Sun

Published 2026-09-23✓ Author reviewed ⓘ

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kan Sun

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Poverty has long been measured by a single number: how much money a person or family has. If their income falls below a certain line, they are counted as poor. But this narrow view misses the reality of hardship. A family might have enough to buy food but lack clean water, electricity, or access to a doctor. To capture this fuller picture, social scientists and policymakers have turned to multidimensional poverty measurement. Instead of looking at just one thing, they track dozens of different indicators—such as whether children go to school, if the home has a flush toilet, or if adults suffer from chronic illness. By combining these many signals, they can identify who is truly deprived in multiple ways at once. This approach is now the global standard for understanding poverty, used by organizations like the United Nations to guide aid and policy across more than a hundred countries. Yet, as these lists of indicators grow longer, often reaching into the hundreds, a new problem emerges. Collecting and processing so much data becomes incredibly expensive and slow, making it difficult for governments to monitor the poor effectively in real time.

A researcher at Fudan University in China, Kan Sun, set out to solve this practical bottleneck. The question was not whether we should measure many aspects of life, but whether we really need to measure every single one to get an accurate result. Sun asked if there was a way to trim the fat from these massive lists without losing the ability to correctly identify the poor. To find the answer, the researcher turned to tools usually reserved for artificial intelligence and machine learning. Specifically, the study used a set of techniques called "feature selection." In plain terms, these are methods that act like a sieve, sorting through a large pile of data to find the few items that carry the most important information and discarding the rest as redundant. The goal was to see if a small, carefully chosen group of indicators could do the same job as a much larger, unwieldy set.

The study tested these ideas using a massive survey of over 42,000 households in India, which tracked 15 different signs of deprivation across education, health, and living standards. The researcher applied five different machine-learning algorithms to this data. Each algorithm acted like a different judge, ranking the 15 indicators by how useful they were for predicting whether a household was poor. The results were striking. The algorithms unanimously agreed that a very small subset of indicators contained almost all the necessary information. At the very top of the list was female schooling. The data showed that knowing whether the most educated woman in a household had fewer than six years of education was enough to predict poverty with near-identical accuracy to the full set. If a woman in the house lacked this basic education, the household was almost certainly poor across multiple dimensions. If she had it, the household was almost certainly not poor.

Beyond female education, a few other factors like adult health and core living standards also proved highly informative. In contrast, the indicators that seemed important on paper turned out to be useless for sorting the poor from the non-poor. For instance, computer ownership was ranked at the very bottom. This was not because computers are unimportant, but because almost no one in the survey owned one; since nearly everyone was deprived of this item, it could not help distinguish between a poor family and a slightly less poor one. Similarly, electricity was so common that its absence was rare, making it a poor tool for sorting. The study found that by dropping the least useful indicators, the researchers could reduce the list from 15 items down to 8 without losing much accuracy; the system could still identify the poor with near-identical accuracy. However, accuracy does not fall sharply until fewer than 5 indicators remain. This finding held true even when the researchers changed the rules for what counted as "poor" or adjusted how much weight was given to different categories.

To ensure this was not just a quirk of the Indian data, the researcher repeated the exercise with a similar survey from South Africa. The results were nearly identical. In that country as well, education and health formed a small, powerful core that carried the weight of the entire measurement system, while a large number of living-standard indicators turned out to be redundant. The study confirmed that this pattern is not a fluke of one dataset but a structural feature of how poverty manifests in these regions.

It is crucial to understand what this study does and does not say. The research does not argue that we should stop measuring things like computer ownership or wall quality because they are unimportant for human well-being. The choice of which dimensions to include in a poverty measure is a moral and political decision, based on what a society values. This study does not make those choices. Instead, it acts as a practical tool for the people who have already made those choices. It shows them that once they have decided to measure 15 things, they might not need to collect data on all 15 to get a reliable count of the poor. By identifying which indicators are statistically redundant, the study offers a way to make poverty monitoring systems cheaper, faster, and easier to manage without sacrificing accuracy. The final decision on what to measure remains with policymakers, but they can now do so with a clear map of which indicators are essential and which are merely extra.

Technical Summary: Feature Selection for Multidimensional Poverty Measurement

Problem Statement
Multidimensional poverty measurement faces a "practical curse of dimensionality." Real-world monitoring systems often aggregate dozens or even hundreds of welfare indicators to identify the poor. While comprehensive indicator sets are normatively attractive, they impose heavy computational and interpretive costs. Furthermore, broad indicator coverage often includes redundant dimensions that carry little independent information, complicating the identification process without necessarily improving accuracy. The paper addresses the need to streamline these measurement frameworks without altering the fundamental classification of who is poor.

Methodology
The study employs a supervised machine learning approach to feature selection, applied within the Alkire–Foster (AF) framework, the standard method for the global Multidimensional Poverty Index (MPI).

Data Sources:
- Primary Dataset: India Human Development Survey (IHDS-II, 2011–12), covering 42,152 households.
- Validation Dataset: National Income Dynamics Study (NIDS, Wave 1, 2008) from South Africa, covering 7,296 households.
- Indicator Framework: A 15-indicator deprivation framework spanning three dimensions (Education, Health, Living Standards) for India, and a comparable 14-indicator framework for South Africa.
Identification Step:
- The AF method is used to generate binary labels (poor/non-poor) based on a dual-cutoff approach.
- Dimensional deprivation cutoffs are applied to indicators, weighted equally within dimensions, and aggregated against a poverty cutoff ( $k = 1/3$ ).
- The output is a matrix of raw indicators and a binary vector of poverty status.
Feature Selection Techniques:
Five supervised feature selection methods were applied to rank indicators based on their ability to predict the binary poverty label:
- Tree-based Ensembles: LightGBM, XGBoost, and Random Forest (ranking by split gain or Gini impurity reduction).
- Regularization: LASSO (L1-regularized logistic regression), which shrinks coefficients of uninformative features to zero.
- Recursive Elimination: Recursive Feature Elimination (RFE) with logistic regression.
- Note: The methods are distinguished from unsupervised dimensionality reduction (e.g., PCA) because they optimize specifically for the poverty label rather than general variance retention.
Evaluation:
- Indicators were ranked by consensus across the five methods.
- Classification accuracy (using LightGBM) was measured as the least important indicators were progressively removed.
- Robustness checks were performed against alternative poverty cutoffs ( $k=1/4, 1/2$ ) and weighting schemes.
- Results were compared against random elimination baselines to ensure the identified redundancy was not an artifact of an easy classification problem.

Key Results

High Redundancy in Indicator Sets: A small subset of indicators reproduces poverty classifications with near-identical accuracy to the full set.
- In India, retaining only 8 indicators (out of 15) maintained an accuracy of 0.962 (compared to 0.989 with all 15). Accuracy only dropped sharply when fewer than 5 indicators remained.
- In South Africa, retaining 5 indicators (out of 14) maintained an accuracy of 0.959 (compared to 0.995 with all 14).
The Informative Core: Across both countries and all methods, female schooling emerged as the single strongest predictor of household multidimensional poverty. Adult health indicators and core living-standard measures followed in importance.
The Redundant Periphery: Indicators with extreme deprivation rates (e.g., computer ownership where ~93% are deprived, or electricity where ~13% are deprived) consistently ranked at the bottom, contributing almost no discriminative information.
Robustness: The findings held true across different poverty cutoffs ( $k$ ) and weighting schemes. The feature selection ordering significantly outperformed random elimination, confirming that the identified core is structurally informative.

Significance and Claims
The paper positions feature selection not as a replacement for normative judgment, but as a complementary tool for empirical efficiency.

Normative vs. Empirical Distinction: The author emphasizes that the choice of which dimensions constitute poverty is a normative decision settled by theory, public deliberation, and policy. Feature selection does not dictate these choices. Instead, it operates downstream to identify which accepted indicators are empirically redundant.
Operational Efficiency: The study demonstrates that monitoring systems can be streamlined at a fraction of the computational and data collection cost without sacrificing identification validity. For example, in the Indian context, a single, cheaply observed indicator (female schooling) captures most of the information conveyed by a 15-indicator battery.
Methodological Contribution: The paper advances the "statistical" criterion in Alkire's framework for selecting capabilities. It transforms this criterion from a simple correlation check into a reproducible, data-driven procedure that supplies systematic evidence on indicator redundancy.
Generalizability: The replication in South Africa suggests that the structural pattern—a compact, informative core of education and health surrounded by a redundant living-standard periphery—is a generalizable feature of multidimensional poverty in developing contexts, rather than an artifact of a single dataset.

In conclusion, the paper argues that feature selection allows policymakers to separate indicators that are diagnostic of poverty (useful for targeting) from those that are merely descriptive or redundant, thereby enabling more efficient resource allocation in poverty monitoring systems.

Technical Summary: Feature Selection for Multidimensional Poverty Measurement

More like this