Estimating Long Run Welfare Outcome in Rotating Panel with Grouped Fixed Effects: Application to Poverty Dynamics in Peru

This paper proposes applying grouped fixed effects (GFE) to rotating panel data from Peru's ENAHO survey to estimate long-run poverty dynamics, demonstrating that this method outperforms traditional synthetic panel approaches by more accurately capturing observed household transitions and providing a richer, interpretable structure for analyzing poverty persistence and mobility.

The Gist

Imagine you are trying to understand the life story of a thousand families in Peru. You want to know: Who stays poor? Who escapes poverty? And who falls back in?

The problem is, you don't have a movie camera that has been filming these families continuously for 15 years. Instead, you have a rotating panel. Think of this like a relay race where runners are swapped out every few laps. You see Family A for three years, then they drop out and Family B comes in. You see Family B for three years, then they drop out.

The Challenge:
If you only look at the snapshots (the years you see them), you can't tell if Family A is always struggling or if they just had a bad week. If you try to guess their whole life story by grouping them with people who look similar (like "families with three kids"), you might get it wrong because you're ignoring the specific, unique history of each family.

The Solution: The "Grouped Fixed Effects" (GFE) Method
The authors of this paper, Hongdi Zhao and Seungmin Lee, propose a clever new way to solve this puzzle using a method called Grouped Fixed Effects (GFE).

Here is how it works, using a simple analogy:

1. The "Musical Chairs" of Poverty

Imagine a room full of families. In traditional methods, researchers might try to guess a family's future based on static traits (like "they live in a village" or "the dad has a high school diploma").

The GFE method is different. It says: "Let's watch how these families move over the few years we see them, and then sort them into groups based on their movement patterns."

• Group 1 (The Climbers): These families start low but steadily climb up the ladder of wealth every year we see them.
• Group 2 (The Stagnant): These families stay at the same level, stuck in the mud.
• Group 3 (The Rollercoasters): These families jump up and down wildly.
• Group 4 (The Fallers): These families start okay but slowly slide down.

The magic of GFE is that it doesn't just look at who the families are (their demographics); it looks at how they behave over time. It uses the short overlaps (when Family A and Family B are both in the survey for a year) to learn the "rules of the game" for each group.

2. Filling in the Blanks (The "Time Travel" Trick)

Once the computer has sorted the families into these four "behavioral groups," it can do something amazing: It can predict the future.

Even though we stopped watching Family A after Year 3, we know they belong to "Group 1" (The Climbers). We also know that "Group 1" generally gets richer every year. So, the model says: "Even though we didn't see Family A in Year 4, 5, or 6, we can confidently predict they are still climbing."

This allows the researchers to reconstruct a complete movie of poverty dynamics from 2007 to 2019, even though no single family was filmed for the whole time.

3. Why is this better than the old way?

The old method (called "Synthetic Panels") is like trying to predict a movie plot by looking at two different movies and guessing the middle part based on the actors' faces. It often misses the plot twists.

The GFE method is like having a director who understands the genre of the movie. By grouping families by their actual trajectory, the model captures the "vibe" of their economic life.

• Result: When the authors tested their predictions against real data they held back, their "time-travel" guesses were much more accurate than the old methods. They could predict who would stay poor and who would escape with about 83% accuracy.

4. What did they find in Peru?

Using this method on Peru's data, they discovered four distinct "types" of households:

• The Top Tier: These families are well-off, educated, and have access to electricity and water. They stay rich.
• The Bottom Tier: These families struggle with basic needs, have less education, and are more likely to speak indigenous languages. They stay poor.
• The Middle: There are two middle groups. One is slowly improving, while the other is slowly slipping.

Crucially, they found that poverty isn't just a single state; it's a trajectory. Some families are "chronically poor" (trapped in the Bottom Tier), while others are "transiently poor" (just passing through the Middle Tier).

The Big Takeaway

This paper is a toolkit for economists. It shows that even if you don't have a perfect, long-term dataset, you can still tell the long-term story of poverty if you use the right sorting algorithm.

Instead of guessing based on static labels, the GFE method listens to the rhythm of the data. It groups families by how they dance over time, allowing us to see the full choreography of poverty and prosperity, even when we only get to watch a few steps of the dance.

In short: They turned a broken, fragmented puzzle into a clear picture by realizing that the pattern of movement is more important than the snapshot of the moment.

Technical Summary

1. Problem Statement

The paper addresses the challenge of estimating long-term poverty dynamics (persistence, mobility, and chronic poverty) in the absence of long-term panel data.

• Data Limitation: Most developing countries lack true long-term panels due to high costs. Instead, they utilize rotating panel designs (e.g., Peru's ENAHO), where households are observed for a short period (e.g., 3–6 years) before being rotated out and replaced by new households.
• Limitations of Existing Methods:
  • Pseudo-panels and Synthetic Panels: These methods rely on repeated cross-sections and cohort-level aggregation. They discard valuable within-household variation observed in rotating panels. They often rely on strong parametric assumptions (e.g., cohort-level autocorrelation) that may not hold over long horizons, leading to biased point estimates or wide bounds.
  • Standard Fixed Effects (FE): Estimating individual household fixed effects in short, unbalanced panels is noisy and imprecise. Furthermore, individual FE does not provide a mechanism to extrapolate welfare trajectories beyond the observed years.
• The Gap: There is a need for a method that leverages the partial overlap in rotating panels to identify unobserved heterogeneity, allows for time-varying dynamics, and can extrapolate welfare outcomes to estimate long-run poverty transitions.

2. Methodology: Grouped Fixed Effects (GFE)

The authors apply the Grouped Fixed Effects (GFE) estimator (Bonhomme and Manresa, 2015) to the rotating panel structure.

A. Model Specification

The welfare outcome ($y_{it}$), defined as the Inverse Hyperbolic Sine (IHS) transformed per-capita household expenditure, is modeled as:
$$y_{it} = x'_{it}\theta + \alpha_{g_i t} + \mu_{p_i} + \varepsilon_{it}$$
Where:

• $x_{it}$: Vector of observed time-varying and time-invariant covariates (e.g., education, age, urban/rural status).
• $\theta$: Common coefficients across all households.
• $\alpha_{g_i t}$: Group-time fixed effect. Households are assigned to one of $G$ latent groups. Each group shares a specific time-varying unobserved heterogeneity path ($\alpha_{gt}$).
• $\mu_{p_i}$: Time-invariant province fixed effects (an extension of the standard GFE model) to control for persistent geographic differences (prices, infrastructure).
• $g_i$: Latent group assignment for household $i$.

B. Estimation Algorithm

The parameters $(\theta, \alpha, \gamma, \mu)$ are estimated by minimizing the sum of squared residuals over the unbalanced set of observed household-year cells ($d_{it}=1$).

• Algorithm: The authors use a Variable Neighborhood Search (VNS) combined with local search (Algorithm 2 from BM2015).
  • Initialization: Random starts for group assignments.
  • Iteration: Alternates between updating parameters (OLS given group assignments) and updating group assignments (assigning households to the group minimizing their residual sum of squares).
  • Handling Missing Data: The objective function only sums over observed years ($d_{it}=1$), ensuring that households with shorter observation windows are not penalized for missing years.
• Model Selection: The optimal number of groups ($G$) is selected using a combination of Bayesian Information Criterion (BIC) (for in-sample fit) and Root Mean Square Error (RMSE) on a held-out test set (for out-of-sample predictive performance).

C. Extrapolation and Imputation

Once groups and group-time paths are estimated, the model constructs "completed" welfare paths for all years (including unobserved years):

1. Imputation of Covariates: Missing covariates ($x_{it}$) for non-survey years are filled using rules (e.g., carrying forward time-invariant traits, mechanically updating age).
2. Prediction: The model predicts welfare for unobserved years using the estimated group trajectory ($\hat{\alpha}_{g_i t}$) and the filled covariates.
3. Poverty Dynamics: Predicted welfare is compared against poverty lines to generate transition matrices (entry, exit, persistence) over horizons longer than the observed panel.

3. Data and Empirical Implementation

• Dataset: Peru's National Household Survey on Living Conditions and Poverty (ENAHO), 2007–2019.
• Sample: Households observed for at least 3 years with heads aged 25–55.
  • Total observations: ~56,390 household-year observations from 14,886 households.
• Validation Strategy: A one-step-ahead holdout approach. For each household, the final observed year is held out as a test set, while the model is trained on all prior observed years. This tests the model's ability to predict the next year's poverty status based on history.

4. Key Results

A. Model Selection and Fit

• Optimal Groups: The model selection diagnostics (RMSE and BIC) consistently point to $G=4$ latent groups as optimal. While BIC continues to decrease with larger $G$ (overfitting), RMSE bottoms out at $G=4$ and increases thereafter, indicating that additional groups capture noise rather than signal.
• Predictive Accuracy:
  • The GFE model achieves an 83% classification accuracy in predicting poverty status in the held-out final year.
  • Comparison with Synthetic Panels: The GFE approach yields smaller errors (RMSE $\approx$ 0.021) in reproducing observed transition shares compared to the standard Synthetic Panel method (RMSE $\approx$ 0.031). GFE predictions are more stable across years.

B. Latent Group Characteristics

The four identified groups exhibit distinct, economically meaningful profiles:

1. Top Group (~11% of population): High baseline expenditure, high college education, high Spanish language usage, and better access to water/electricity. Shows stable, high welfare.
2. Upper-Mid & Lower-Mid Groups: Intermediate welfare levels with varying educational attainment and living conditions.
3. Bottom Group (~23% of population): Low baseline expenditure, low education, lower access to services.

• Dynamics: The groups show distinct trajectories over time. For instance, the "Bottom" group shows significant improvement in early years followed by stagnation, while the "Top" group remains stable.

C. Poverty Dynamics

• Recovery of Transitions: The GFE-implied transition shares (poor-to-poor, poor-to-non-poor, etc.) closely track the actual data in the holdout years.
• Long-Run Extrapolation: By filling in missing years, the model allows for the estimation of chronic poverty and medium-run mobility (e.g., predicting 2-year or 3-year transitions) which is impossible with raw rotating panel data alone.

5. Robustness Checks

• No Age Restriction: Removing the 25–55 age head restriction (increasing sample size) confirmed that $G=4$ remains optimal and group trajectories remain stable, suggesting the results are not driven by the age filter.
• Longer Observation Window (5+ years): Restricting the sample to households observed for at least 5 years reduced the sample size significantly. While $G=4$ remained optimal, the estimated trajectories became less stable and more sensitive to covariate specifications. This highlights a trade-off: longer observation windows provide more within-household data but reduce the effective sample size and "group-year support," leading to noisier estimates.
• Medium-Run Prediction: The model successfully predicted poverty status two years ahead with >80% accuracy.

6. Significance and Contributions

1. Methodological Innovation: This is the first application of GFE to study poverty dynamics. It bridges the gap between true long panels and repeated cross-sections by exploiting the partial overlap in rotating panels.
2. Superior Performance: The GFE approach outperforms synthetic panel methods in predicting short-run transitions and provides a more interpretable structure (latent types) than point estimates derived from cohort moments.
3. Policy Relevance:
   • Targeting: The latent groups allow policymakers to identify specific "types" of households (e.g., the chronically poor vs. the transiently poor) based on a bundle of characteristics rather than a single snapshot.
   • Long-Run Insight: The method enables the estimation of long-term poverty persistence and chronic poverty incidence even when only short rotating panels are available.
4. Practical Guidance: The paper offers a framework for handling missing covariates in non-survey years and demonstrates that maintaining a broader sample (shorter participation) often yields more stable trajectory estimates than restricting to a small subset of long-duration households.

In conclusion, the paper demonstrates that Grouped Fixed Effects is a robust, data-driven tool for recovering long-run welfare dynamics from short rotating panels, offering a practical alternative to synthetic panels for development economics research.