Missing data and cluster graphs: cluster-level missingness vs variable-level missingness

Imagine you are trying to solve a giant jigsaw puzzle to understand how different parts of a system work together—like how mental health, sexual habits, and HIV status might be connected. But there's a problem: some puzzle pieces are missing. In the world of data science, this is called "missing data."

Usually, scientists try to solve this by looking at every single missing piece individually. They ask, "Why is this specific piece missing? Is it because of the piece next to it? Or the one across the room?" This is like having a detailed map of every single street in a city.

However, in real life, we often don't have that level of detail. We might only know that entire neighborhoods (clusters) of information are missing, without knowing exactly which house in the neighborhood is empty. This paper, by Willow Scott and colleagues, asks: Can we still solve the puzzle if we only have a map of the neighborhoods, rather than the individual streets?

The Two Types of Maps

The authors introduce two ways to draw these "neighborhood maps" to handle missing data:

1. The "Detailed Neighborhood" Map (m-C-DMG):
   Imagine you know that the "Sexual Behavior" neighborhood has missing data. On this map, you still keep track of which specific house in that neighborhood is empty. You know that the "Condom Use" house is missing, but the "Number of Partners" house is there. You keep the individual missingness indicators separate, even though you group them into a neighborhood.

   • Analogy: You know the "Downtown" district has a power outage, but you still have a list of exactly which specific streetlights are out.

2. The "Blurred Neighborhood" Map (cm-C-DMG):
   This is a coarser map. Here, you just know that the entire "Sexual Behavior" neighborhood is having issues. You don't know if it's the "Condom Use" house or the "Number of Partners" house that is missing; you just see a big "Missing" sign over the whole district.

   • Analogy: You see a "Downtown District Outage" sign, but you have no idea which specific streetlights are affected.

The Big Discovery: What Can We Still Learn?

The paper explores whether we can still figure out the full picture (the "joint distribution") or the cause-and-effect relationships (like "Does behavior A cause outcome B?") using these blurry maps.

1. The "Blurred" Map is Less Powerful
The authors found that the "Blurred Neighborhood" map (cm-C-DMG) is a bit of a downgrade. Because it merges all the missing details into one big block, it sometimes hides clues that are necessary to solve the puzzle.

• The Metaphor: If you only know that the "Downtown" district is missing data, you might assume the whole district is unreliable. But on the "Detailed" map, you might realize that only one specific house is missing, and the rest of the district is actually fine. The blurred map might make you throw away good data thinking it's all bad, while the detailed map lets you save it.
• The Rule: If you can solve the puzzle using the "Blurred" map, you can definitely solve it with the "Detailed" map. But the reverse isn't true. Sometimes, the "Detailed" map lets you solve a puzzle that the "Blurred" map makes impossible.

2. Solving the Puzzle (Recovering the Joint Distribution)
The paper gives a specific set of rules (a checklist) to see if you can reconstruct the whole picture from the missing data.

• The Rule: You can recover the full picture if the "Missing" signs don't have a direct, sneaky connection to the data they are supposed to be hiding.
• Analogy: Imagine a detective trying to find a stolen watch. If the thief (the missing data mechanism) is standing right next to the watch and whispering secrets to it, the detective can't figure out what happened. But if the thief is in a different room and the only path between them is blocked by a "collider" (a third person who blocks the path), the detective can still solve the case. The paper says: as long as the "Missing" signs aren't directly whispering to the data or connected by a specific type of secret path, you can reconstruct the full story.

3. Figuring Out Cause and Effect (Macro Causal Effects)
The authors also looked at whether we can figure out cause-and-effect (e.g., "Does changing behavior X cause a change in outcome Y?") even when data is missing.

• The Surprise: You don't always need to reconstruct the entire puzzle to know how one piece affects another.
• Analogy: You might not be able to rebuild the entire jigsaw picture of the city, but you might still be able to prove that "Traffic jams cause late arrivals" by looking at just the traffic and clock pieces.
• The paper shows that even if the "Blurred" map makes it impossible to know the full state of the system, it might still be possible to figure out specific cause-and-effect relationships using a set of logical rules (called "do-calculus").

Why Does This Matter?

In the real world, scientists often don't have perfect data. They might know that "Mental Health" and "Sexual Behavior" are linked, but they don't know the exact relationship between every single survey question.

This paper tells us:

• It's okay to group data into clusters if we have to, but we need to be careful.
• If we group too much (using the "Blurred" map), we might lose the ability to answer certain questions.
• However, even with a blurry map, we can often still answer important "What if?" questions about cause and effect, provided we follow the right logical steps.

In short, the paper provides a toolkit for scientists to know exactly how much detail they need to collect. If they have a "Blurred" map, they now know which questions they can still answer and which ones require them to go back and get a "Detailed" map.