📊 statistics

HICM: An approach towards Harmonizing Indian Census Migration data and its applications

This paper introduces HICM, a data-centric framework that harmonizes Indian census migration data by identifying and correcting measurement and representativeness biases through principled pre-processing and statistical diagnostics, thereby significantly improving data consistency and reliability for longitudinal migration analysis and policy-making.

Original authors: Nivedita Batra, Chiranjoy Chattopadhyay, Mayurakshi Chaudhuri

Published 2026-04-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Nivedita Batra, Chiranjoy Chattopadhyay, Mayurakshi Chaudhuri

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine India as a giant, bustling city where people are constantly moving from one neighborhood to another. Every ten years, the government takes a giant "snapshot" (a Census) to count who moved where, how long they stayed, and where they came from. This data is like a massive, complex map of human movement.

However, this map has some serious tears and missing pieces. Some neighborhoods weren't counted at all in certain years, some people's addresses are scribbled over and unreadable, and some people forgot to write down how long they've been living in their new home. If researchers try to study this map without fixing the tears, they might draw the wrong conclusions about how India is changing.

This paper, titled HICM, is like a team of expert map restorers who have developed a special toolkit to fix these tears, fill in the missing pieces, and make the maps from 1991, 2001, and 2011 fit together perfectly.

Here is a simple breakdown of what they did, using everyday analogies:

1. The Problem: A Messy Puzzle

The researchers found three main types of "mess" in the data:

The "Ghost" Neighborhoods (Missing Data): In 1991, the state of Jammu & Kashmir was too unstable to count properly, so it was missing from the "In-Migration" list. It's like trying to solve a puzzle but realizing one corner piece is completely gone. If you ignore it, the picture of the whole country is wrong.
The "Blurry" Addresses (Unclassifiable Data): Some people moved, but the census form just said "Unknown where they came from." These are like letters in the mail with no return address. They exist, but we don't know where to put them.
The "Forgotten" Timers (Unknown Duration): Many people didn't write down how long they had been living in their new city. It's like a party where everyone arrives, but no one knows if they've been there for 10 minutes or 10 years. This makes it hard to tell if the movement is temporary or permanent.

2. The Solution: The HICM Toolkit

The authors created a framework called HICM (Harmonizing Indian Census Migration) to fix these issues. Think of it as a three-step repair process:

Step A: Standardizing the Labels (Cleaning the Tags)

First, they realized the labels were inconsistent. In 1991, states were numbered alphabetically (A for Andhra, B for Bihar), but in 2001 and 2011, they were numbered geographically (North to South).

The Fix: They renamed and re-numbered everything so that "Andhra Pradesh" is always "Andhra Pradesh" and "State #1" is always the same state across all three decades. It's like making sure everyone in a group chat uses the same spelling for names so no one gets confused.

Step B: Filling the Ghost Neighborhoods (The Time-Travel Math)

For the missing 1991 data on Jammu & Kashmir, they couldn't just guess. Instead, they used a Time-Travel Math Trick.

The Analogy: Imagine you know how many people moved to a town in 2001 and 2011. You also know how the ratio of people moving there changed between those years. You can use that pattern to "project backward" and estimate what the number likely was in 1991.
They didn't just copy-paste numbers; they calculated the trend of movement to fill the gap smoothly, ensuring the 1991 map looked like a natural part of the 2001 and 2011 maps.

Step C: Sorting the Blurry Letters (Smart Redistribution)

For the people with "Unknown" origins or "Unknown" stay durations, they didn't throw the data away. They used Smart Redistribution.

The "Unknown Origin" Fix: If a person's origin is unknown, they didn't just guess. They looked at the known patterns. If 80% of people in a city came from Village A, and 20% from Village B, they assumed the "unknown" people likely followed that same 80/20 split. They sprinkled these "ghost" people back into the map based on the most likely patterns.
The "Unknown Duration" Fix: They used a logical assumption: Newcomers are more likely to forget how long they've been there. So, they distributed the "unknown time" people mostly into the "short stay" categories (less than a year) and fewer into the "long stay" categories. This is like assuming that if you see a stranger at a party and they don't know how long they've been there, they probably just arrived.

3. The Result: A Clearer Picture

After applying these fixes, the researchers tested their new, "harmonized" maps.

The Smoothness Test: When they compared the 1991, 2001, and 2011 maps, the lines and patterns flowed smoothly. There were no sudden, weird jumps in the data that didn't make sense.
The Network Test: They built a "migration network" (like a subway map of human movement).
- Before: The map was jagged and had missing lines.
- After: The map showed clear "communities" of states that trade people with each other. For example, they could clearly see that Maharashtra is a magnet for workers, while Uttar Pradesh is a major source of workers.

Why Does This Matter?

Imagine you are a city planner trying to build a new hospital. If your population map is missing a whole neighborhood (J&K in 1991) or has blurry data on where people are from, you might build the hospital in the wrong place.

By fixing these data gaps, HICM gives policymakers, scientists, and planners a reliable, high-definition map of India's internal migration. It allows them to:

See real trends over 30 years without being tricked by missing data.
Understand which states are growing and which are shrinking.
Make better decisions about where to build schools, hospitals, and roads.

In short, this paper turns a messy, torn-up, and confusing set of old maps into a single, coherent, and trustworthy guide for understanding how India moves.

1. Problem Statement

The paper addresses the critical issue of systematic biases and inconsistencies in Indian Census migration data (specifically the 1991, 2001, and 2011 decennial records). While the Indian Census provides a rich longitudinal resource for studying population displacement and urbanization, the data suffers from structural gaps that undermine the reliability of temporal and network-based analyses.

The authors identify three primary categories of bias and inconsistency:

Measurement Gaps (Missingness):
- Unclassifiable Migrants: Migrants with unknown "last residence" (approx. <1% of total).
- Unknown Duration of Stay: Migrants with missing duration data (approx. 8–16% of total), which varies by decade.
Coverage Inconsistency:
- The 1991 dataset excludes Jammu and Kashmir (J&K) due to political instability, whereas 2001 and 2011 include it. This creates a "missing place of enumeration" problem, leading to representation bias in longitudinal studies.
Nomenclature and Indexing Inconsistencies:
- Indexing Mismatch: 1991 uses alphabetical indexing for states/UTs, while 2001/2011 use geographical indexing.
- Missing Aggregates: The 1991 dataset lacks explicit "Total" columns per record, complicating cross-decade comparisons.
- Name Variants: Inconsistent state names (e.g., "Orissa" vs. "Odisha") across decades.

2. Methodology: The HICM Framework

The authors propose HICM, a data-centric framework designed to harmonize these datasets through principled pre-processing, mitigation, and validation. The workflow involves three main stages:

A. Nomenclature and Label Harmonization

Standardization: State names are normalized to current identifiers (e.g., "Orissa" $\to$ "Odisha").
Index Alignment: The 1991 dataset is re-indexed to match the geographical ordering of 2001 and 2011 to prevent computational errors in automated pipelines.
Structural Alignment: A "Total" column is added to the 1991 dataset to ensure dimensional consistency with later decades.

B. Mitigation of Coverage Inconsistency (Missing J&K in 1991)

To estimate the missing inflow data for J&K in 1991, the authors employ Temporal Ratio Smoothing (Algorithm 1):

Assumption: Migration flows change gradually rather than abruptly; therefore, ratios of migration over time are more stable than raw flows.
Process:
1. Calculate out-migration ratios between 1991–2001 and 2001–2011 for every source state.
2. Compute a geometric mean of these ratios to create a smoothed backward temporal ratio ( $\tilde{R}_i$ ).
3. Estimate the missing 1991 inflow for J&K by applying this ratio to the known 2001 inflow data: $f^{(1991)}_{i,x} = f^{(2001)}_{i,x} \cdot \tilde{R}_i$ .
Validation: The imputed distribution for J&K in 1991 was compared against ground truth data from 2001/2011, showing smooth replication without abrupt discontinuities.

C. Mitigation of Measurement Gaps (Imputation)

Two distinct strategies are applied to handle missing data while preserving total population constraints:

Redistribution of Unclassifiable Migrants (<1%):
- Migrants with unknown last residence are redistributed proportionally to known places of last residence based on observed migration flows.
- Formula: $\hat{M}_{ij} = M_{ij} + U_j \cdot \frac{M_{ij}}{\sum_i M_{ij}}$
- This preserves the total migrant count and the relative inflow structure of each destination.
Redistribution of "Unknown Duration" Migrants (8–16%):
- Instead of uniform redistribution, the authors apply a sociological assumption: recent migrants are more likely to have missing duration data.
- A fixed weight vector $w$ following an exponential decay is applied to the five duration bins (from <1 year to 20+ years).
- Weights: $w \approx (0.35, 0.30, 0.20, 0.10, 0.05)$ , ensuring that the majority of missing data is allocated to shorter durations ( $w_1 + w_2 + w_3 > 80\%$ ).
- Formula: $\hat{N}_{j,b} = N_{j,b} + w_b \cdot U_j$

3. Key Contributions

Bias Profiling: The first systematic identification and quantification of specific biases (measurement vs. representativeness) in Indian interstate migration data across three decades.
Harmonization Framework: A reproducible, mathematically transparent framework (HICM) that corrects structural gaps without inventing new data, strictly preserving census totals.
Network-Ready Datasets: The production of harmonized datasets specifically optimized for constructing directed, weighted migration networks for longitudinal analysis.

4. Experimental Results and Validation

The authors validated the framework using statistical diagnostics and network analysis:

Distribution Consistency: Violin plots comparing the imputed 1991 J&K data against 2001/2011 data showed no abrupt spikes, confirming temporal consistency.
Statistical Preservation: After redistributing unclassifiable migrants, the statistical summary (Mean, Std Dev, Q1, Q2, Q3) of the 2011 dataset remained nearly identical to the pre-imputation state. Deviations were minimal and confined to the 4th quartile, indicating the redistribution did not distort the underlying data structure.
Network Analysis Application:
- Trends: The average edge weight of migration networks increased by ~30% per decade, indicating rising interstate mobility.
- Community Detection: Using the Louvain algorithm, the harmonized 1991 data revealed 3 communities of states, while 2001 and 2011 stabilized at 4 communities.
- Short-term vs. Long-term: Analyzing migrants with <1 year duration showed different community structures (4 communities in 1991) compared to aggregate stocks, highlighting the value of granular, harmonized data.

5. Significance and Impact

Policy Relevance: The framework provides a robust foundation for policymakers to analyze regional inequalities and migration patterns without the distortion caused by data gaps.
Methodological Advancement: It moves beyond simple imputation by using constraint-preserving strategies that respect the sociological realities of migration (e.g., recent migrants having missing duration data).
Longitudinal Reliability: By fixing indexing and coverage issues, the HICM framework enables valid cross-decade comparisons, which were previously compromised by structural inconsistencies.
Future Applications: The harmonized data supports advanced analyses such as multiplex network modeling, centrality measures, and fairness-sensitive migration models, essential for future urban planning and demographic research in India.