TDA Engine v2.1: A Computational Framework for Detecting Structural Voids in Spatially Censored Epidemiological Data with Temporal Classification and Causal Inference

The TDA Engine v2.1 is a topological framework that mathematically distinguishes structural data voids from stochastic gaps in censored epidemiological data by integrating temporal classification and causal inference to guide public health investigations.

The Gist

Imagine you are looking at a map of a city, but instead of showing you where the houses are, the map only shows you the empty spaces between them. In public health, this is a common problem: health officials have maps of where clinics are, but they often don't know why some areas are completely empty. Is it because no one lives there? Or is it because people are sick, but the system is broken, and no one is reporting it?

This paper introduces a new digital tool called the TDA Engine v2.1 (Topological Data Analysis Engine). Think of it as a "Silence Detector" for health maps.

Here is a simple breakdown of how it works, using everyday analogies:

1. The Problem: The "Ghost" on the Map

Traditional maps are like flashlights. They shine a light on where data exists (where people are reporting sickness). But if a flashlight beam hits a dark spot, the flashlight just says, "It's dark here." It doesn't tell you why. Is it a cave? Is it a blackout? Or is it just a shadow?

Old methods tried to smooth over these dark spots, pretending they were just low-density areas. This paper argues that we need to stop smoothing and start measuring the shape of the silence.

2. The Solution: Measuring the "Distance to the Nearest Friend"

The core of this new engine is a concept called Distance-to-Measure (DTM).

• The Analogy: Imagine you are standing in a crowded park. If you are in the middle of a group of friends, you are close to everyone. If you are standing alone in a huge empty field, you are far from your nearest friend.
• How the Engine Works: The engine calculates the distance from every empty spot on the map to the nearest health clinic.
  • If the distance is short, it's probably just a small gap (like a gap between two trees).
  • If the distance is huge (like standing in the middle of a desert with no trees for miles), the engine flags it as a "Structural Void." This is a geometric anomaly—a "hole" in the data that shouldn't exist if the system were working perfectly.

3. The Upgrade: What's New in Version 2.1?

The original version just found the holes. Version 2.1 is like adding a detective and a time machine to the engine. It asks three new questions:

A. "Is this silence a habit or a fluke?" (Temporal Classification)

Sometimes a clinic is closed for a week because of a storm (a fluke). Sometimes it's closed for years because the road is washed out (a habit).

• The Tool: The engine looks at data over time (like checking a diary). It uses a mathematical trick called a Hidden Markov Model (think of it as a weather forecaster for silence).
• The Result: It labels the silence as:
  • Structural: "This is a permanent problem. Send a team to fix it."
  • Intermittent: "This happens sometimes. Keep an eye on it."
  • Stochastic: "This is just random noise. Ignore it."

B. "Why is this silence happening?" (Causal Taxonomy)

Once the engine finds a "Structural Void," it tries to guess the cause, acting like a Sherlock Holmes. It looks at clues on the map:

• BORDER: Is it right next to the country line? (Maybe people are crossing over to get help).
• ACCESS: Are there no roads? (Maybe people can't get there).
• INFRASTRUCTURE: Is there a huge population but zero clinics? (Maybe the clinic was never built).
• SYSTEM: Are there clinics nearby, but no data is coming in? (Maybe the computers are broken or the staff isn't reporting).
• UNKNOWN: We don't know yet. Go investigate.

C. "How bad is the situation?" (O/E Completeness)

The engine doesn't just count empty spots; it calculates how many sick people are missing from the report.

• The Analogy: If a town of 1,000 people usually has 10 cases of malaria, but the report says "0," the engine calculates the "missing" 10 cases. It then ranks these missing cases by severity, telling officials: "Fix the malaria gap in this village before the gap in that village."

4. The Proof: Did it Work?

The authors tested this engine on real data from Kenya. They created a "fake" scenario where they secretly hid data from a specific area (like a game of "Where's Waldo?").

• The Result: The TDA Engine found the hidden area 82% of the time with high precision.
• Comparison: Old methods (like standard heat maps) only found it about 45% of the time. The new engine was much better at spotting the "ghosts."

5. The Big Takeaway

This paper isn't about finding a magic cure for disease. It's about finding the blind spots in our knowledge.

• Before: Health officials looked at a map and saw a blank space. They guessed, "Maybe no one lives there."
• Now: The TDA Engine points to the blank space and says, "This is a Structural Void. It's been silent for 6 months. It's likely because there are no roads (ACCESS) or the data system is broken (SYSTEM). And based on population numbers, we are missing about 500 reported cases here. Go investigate this specific spot."

In short: This tool turns "empty space" on a map into a to-do list for health officials, helping them direct their resources to the places where the silence is loudest and most dangerous.

Technical Summary

1. Problem Statement

Public health surveillance systems often suffer from data silence (absence of data), which creates a critical interpretative ambiguity: does an empty region on a map represent a healthy population with no disease, or a marginalized community with no access to reporting mechanisms (systemic suppression)?

Traditional spatial epidemiological tools fail to resolve this:

• Kernel Density Estimation (KDE): Smooths over gaps, treating them as low-probability gradients rather than distinct features.
• Voronoi Diagrams: Suffer from severe "edge effects" at boundaries.
• Spatial Scan Statistics (e.g., SaTScan): Designed to detect clusters of excess cases, not absences.
• Limitation: These methods conflate stochastic sparsity (natural low density) with structural voids (systematic reporting failures) and rely on arbitrary parameter tuning.

2. Methodology

The paper proposes a shift from density-based smoothing to topological inference using Topological Data Analysis (TDA). The core framework, TDA Engine v2.1, consists of the following components:

A. Geometric Foundation: Distance-to-Measure (DTM)

• Coordinate Projection: Raw GPS coordinates are projected from WGS84 to UTM Zone 36S to ensure Euclidean distance calculations represent true ground meters.
• DTM Filtration: Instead of using the standard distance to the nearest neighbor (sensitive to outliers), the method uses the Distance-to-Measure (DTM) function.
  • $d_{m_0}(x)$ is the quadratic mean of the distances to the $k$ nearest neighbors, where $k = \lceil m_0 \cdot n \rceil$.
  • The mass parameter $m_0 = 0.05$ (5%) is chosen to align with WHO standards for 95% reporting completeness. This makes the metric robust to up to 5% of outliers (censored data).
• Adaptive Thresholding: To eliminate arbitrary constants (e.g., fixed km cutoffs), the Kneedle algorithm is applied to the sorted DTM distribution. It identifies the "elbow" point (maximum curvature) to define the threshold $T_{adaptive}$ where distance growth accelerates, purely based on data geometry.

B. Statistical Validation

• Null Hypothesis: Complete Spatial Randomness (CSR).
• Test Statistic: Instead of the unstable maximum DTM, the framework uses Total Void Area as the test statistic.
• Significance: A Monte Carlo permutation test ($N=999$) generates p-values with Wilson score confidence intervals to confirm that detected voids are not random artifacts.

C. Temporal Classification (New in v2.1)

To distinguish persistent structural silence from transient stochastic fluctuations, the engine analyzes time-series data using two complementary methods:

1. Fano Factor: Measures dispersion ($Var/\bar{x}$) to identify compressed variance (consistent suppression) vs. high variance (episodic drops).
2. Hidden Markov Model (HMM): A two-state model (Silent vs. Reporting) estimated via Baum-Welch EM and decoded via Viterbi.

• Output: Units are classified as STRUCTURAL (persistent, $P \ge 0.60$), INTERMITTENT, or STOCHASTIC.

D. Causal Taxonomy & O/E Engine

• Causal Classification: A decision-tree classifier maps voids to five probable failure mechanisms based on covariates (road density, facility density, border proximity, O/E ratio):
  • BORDER, ACCESS, INFRASTRUCTURE, SYSTEM, UNKNOWN.
• Observed-to-Expected (O/E) Engine: Calculates severity by comparing observed cases against WHO-standard incidence rates for seven diseases (e.g., Malaria, HIV, Cholera). This allows for severity triage (Critical, Moderate, Mild) independent of raw case counts.

3. Key Contributions

The paper outlines eight specific methodological contributions:

1. Adaptive Thresholding: Replaces arbitrary constants with the geometrically derived Kneedle algorithm.
2. Quantitative Validation: Introduces a censoring simulation framework with known ground truth to calculate Jaccard indices and centroid errors.
3. Topological Stability: Demonstrates that the parameter $m_0=0.05$ lies within a stable plateau $[0.03, 0.07]$, ensuring robustness.
4. Enhanced Inference: Uses total void area and Monte Carlo p-values with confidence intervals.
5. Temporal Classification: Combines Fano factors and HMMs to distinguish structural vs. stochastic silence.
6. Causal Taxonomy: Provides actionable investigative leads (5 classes) rather than just descriptive maps.
7. O/E Completeness Engine: Standardizes severity triage across seven disease conditions using WHO benchmarks.
8. Scope Delineation: Explicitly frames structural voids as geometric anomalies consistent with suppression, requiring contextual validation, not proof of suppression.

4. Results

The framework was validated using public Kenyan health facility data (Nyanza region) and synthetic censoring simulations.

• Spatial Accuracy:
  • Jaccard Index: TDA Engine achieved 0.82 (95% CI: 0.74–0.89), significantly outperforming KDE (0.45) and Relative Risk surfaces (0.38).
  • Centroid Error: 342 meters (IQR: 187–512 m), indicating precise localization of voids.
  • Recovery Rate: 79% of true suppressed areas were successfully detected.
• Statistical Significance: Permutation tests yielded p = 0.003 (95% CI: 0.001–0.008), confirming voids are not due to Complete Spatial Randomness.
• Temporal Accuracy: The classifier correctly labeled 91% of structurally silent units across six-period datasets.
• Causal Accuracy: The causal taxonomy achieved 78% accuracy (Kappa = 0.71) against analyst-adjudicated ground truth.
• Case Study Application: Applied to the Nyanza Basin, the engine identified 3 structural voids in peri-urban corridors (high population, zero facilities) classified as "Infrastructure" and "Access" gaps, distinct from 5 stochastic voids in uninhabited forest/lake areas.

5. Significance and Conclusion

TDA Engine v2.1 represents a paradigm shift in epidemiological surveillance:

• From Description to Action: It transforms void detection from a descriptive cartographic exercise into an actionable triage tool. By classifying voids by persistence (temporal) and probable cause (causal), it directs field investigation resources efficiently.
• Mathematical Rigor: It replaces subjective parameter tuning with geometric derivation (Kneedle) and provides statistical confidence intervals for detection significance.
• Operational Utility: The open-source R/Shiny application allows public health officials to distinguish between natural gaps and potential suppression, prioritizing areas based on epidemiological burden (O/E ratio) rather than just geometric size.

Limitations & Ethics: The authors emphasize that the tool identifies candidates for investigation, not definitive proof of suppression. It requires accurate geocoding and sufficient data density. Ethical deployment requires community partnership, data sovereignty, and the understanding that these are geometric anomalies, not legal evidence of malfeasance.

Future Work: Includes integrating cost-distance (travel time) instead of Euclidean distance, real-time DHIS2 integration, and multi-modal data fusion with satellite imagery and conflict data.