Eigenvector Spatial Filters Nuclear Norm Matrix Completion with Application to Air Quality Data

Technical Summary: Eigenvector Spatial Filters Nuclear Norm Matrix Completion (ESFNNMC)

Problem Statement
Reliable reconstruction of missing observations in environmental panel datasets is critical for accurate exposure assessment and policy analysis. Air quality monitoring data often suffer from missing entries due to sensor malfunctions or calibration periods. While traditional nuclear norm matrix completion methods effectively impute missing entries in low-rank matrices, they frequently overlook the spatial dependence inherent in air quality processes. Standard fixed-effects approaches (e.g., Athey et al., 2021) account for unit-specific intercepts but do not explicitly model the spatial autocorrelation between monitoring stations, potentially limiting imputation accuracy in spatially correlated environments.

Methodology
The paper proposes the Eigenvector Spatial Filters Nuclear Norm Matrix Completion (ESFNNMC) method. This approach extends the fixed-effects nuclear norm matrix completion framework by replacing unit-specific fixed effects with a structured spatial component based on Eigenvector Spatial Filters (ESF).

1. Model Formulation:
   The observed matrix $M \in \mathbb{R}^{n \times T}$ is decomposed as:
   $$\hat{M} = \hat{L} + A\hat{\alpha}\mathbf{1}_T^\top + \mathbf{1}_n\hat{v}^\top$$
   Where:

   • $\hat{L}$ is a low-rank matrix capturing latent spatio-temporal factors.
   • $\mathbf{1}_n\hat{v}^\top$ represents time fixed effects.
   • $A\hat{\alpha}\mathbf{1}_T^\top$ replaces the standard unit fixed effects ($\hat{u}\mathbf{1}_T^\top$). Here, $A$ is a matrix of exogenous spatial basis functions (Moran eigenvectors), and $\hat{\alpha}$ are the coefficients to be estimated.

2. Spatial Filter Construction:

   • A spatial weights matrix $W$ is symmetrized and centered.
   • Eigendecomposition is performed on the transformed matrix to obtain Moran eigenvectors.
   • Eigenvectors are selected based on their ability to capture positive spatial autocorrelation (measured by Moran's $I$). The selection retains eigenvectors explaining a user-specified threshold (e.g., 90%) of the total positive spatial autocorrelation signal.
   • This reduces the number of parameters from $n$ (unit effects) to $q$ (spatial filters), where $q \ll n$, while ensuring orthogonality and avoiding multicollinearity.

3. Estimation Algorithm:
   The model is estimated using a Block-Coordinate Descent (BCD) approach for multiconvex optimization. The algorithm iterates through three steps until convergence:

   • Update Spatial Coefficients ($\alpha$): Solves a weighted least-squares problem to fit the spatial component to row-level residuals.
   • Update Time Effects ($v$): Computes time fixed effects by removing average temporal residuals.
   • Update Low-Rank Component ($L$): Applies soft-thresholded Singular Value Decomposition (SVD) to the residual matrix to enforce the low-rank structure.
   • Regularization: The regularization parameter $\lambda$ is selected via cross-validation to minimize the Root Mean Square Error (RMSE) on a validation set.

Key Contributions

• Methodological Integration: The paper unifies three strands of literature: low-rank matrix completion via convex regularization, panel data fixed-effects estimation, and spatial filtering techniques.
• Efficient Parameterization: By substituting unit fixed effects with a parsimonious set of orthogonal spatial eigenvectors, the method captures spatial autocorrelation dynamics without the overfitting risks associated with estimating $n$ separate parameters.
• Computational Efficiency: The proposed BCD algorithm maintains computational costs comparable to standard fixed-effects nuclear norm completion, as demonstrated in simulation runtimes.

Results
The paper evaluates ESFNNMC through extensive simulations and an empirical application to PM10 data in Lombardy, Italy.

• Simulation Findings:

  • Across various matrix dimensions, missingness patterns (2%–25%), and autocorrelation levels, ESFNNMC consistently demonstrates improved imputation accuracy (measured by Median MAPE) compared to standard Fixed-Effects Nuclear Norm Matrix Completion (FENNMC).
  • The performance gains are most pronounced in scenarios with high spatial autocorrelation ($\rho$) and low temporal autocorrelation ($\phi$), particularly in "long" matrices (many spatial units, few time periods).
  • In "wide" matrices (few units, long time series), the advantage of spatial filters diminishes as temporal structure dominates, though ESFNNMC remains competitive.
  • The number of selected spatial filters remains stable (typically 20–30% of the number of rows), confirming the method's parsimony.
  • Computational times for ESFNNMC are nearly identical to FENNMC (e.g., ~0.48s vs. ~0.43s per replication for a 10x10 matrix).

• Empirical Application (Lombardy PM10):

  • Applied to daily PM10 measurements from 64 stations in 2021 (approx. 4% missing data).
  • Seven eigenvectors were selected to explain ~90% of the positive spatial autocorrelation.
  • The reconstructed series successfully reproduced seasonal dynamics and short-term peaks.
  • ESFNNMC produced slightly smoother trajectories than FENNMC, particularly during abrupt fluctuations, by borrowing information from neighboring stations.
  • Note on Validation: The authors acknowledge that direct out-of-sample validation on the real data is challenging because true missing values are unknown. While in-sample MAPE on observed data was calculated, the authors emphasize that a rigorous comparison requires an external validation strategy (e.g., artificial masking) which is proposed as future work.

Significance and Claims
The paper claims that ESFNNMC offers a principled approach to reconstructing high-resolution pollutant fields from incomplete monitoring networks. By explicitly modeling spatial dependence within the matrix completion framework, the method delivers appreciable gains in imputation accuracy for environmental applications, particularly where spatial patterns are strong.

The authors position this work as a contribution to the growing literature on spatial machine learning for environmental data. They modestly note that the current implementation assumes a Missing At Random (MAR) mechanism, whereas environmental data often exhibit Missing Not At Random (MNAR) patterns. Future research directions identified include:

• Incorporating explicit temporal dependence structures (e.g., autoregressive dynamics) rather than simple time fixed effects.
• Modeling MNAR mechanisms.
• Investigating the role of negatively autocorrelated eigenvectors.
• Developing rigorous out-of-sample validation strategies involving artificial masking.

The code is implemented in R but is not yet publicly available at the time of writing.