Bayesian local clustering of functional data via semi-Markovian random partitions

1. Problem Statement

Functional Data Analysis (FDA) often involves clustering curves or surfaces defined over a continuous domain. Traditional clustering methods typically perform global clustering, assigning an entire functional observation to a single cluster. However, functional data often exhibit local heterogeneity, where different segments of the same curve belong to different groups (e.g., curves may behave similarly at the boundaries of a domain but diverge in the center).

Existing approaches to local clustering face limitations:

• Hidden Markov Models (HMMs): Often used to model cluster evolution, but they impose strict first-order Markovian dependencies that may not align with the local support properties of basis expansions.
• Standard Dependent Random Partition Models (e.g., tRPM): While flexible, standard temporal Random Partition Models (tRPM) typically assume first-order dependence. They struggle to capture the specific requirement that a function is locally determined by a block of contiguous basis coefficients (e.g., $d+1$ coefficients for a degree-$d$ B-spline), rather than just a single coefficient.

The authors aim to develop a framework for indirect local clustering that respects the local support properties of B-spline basis expansions, allowing for partially coincident functional behaviors without relying on restrictive Markovian assumptions.

2. Methodology

The proposed framework combines B-spline basis expansions with a novel Semi-Markovian Random Partition Model (smRPM).

A. Functional Representation

Functional observations $y_i(x)$ are approximated using a $d$-degree B-spline basis:
$$y_i(x) \approx \sum_{k=1}^K b_k(x) \theta_{ik}$$
where $\theta_i$ is a vector of basis coefficients.

• Local Clustering Logic: Due to the local support of B-splines, a curve is fully determined within a sub-interval by $d+1$ contiguous coefficients. Therefore, two curves are locally identical if they share $d+1$ contiguous coefficients.
• Latent Structure: Instead of a global cluster label $c_i$, the model uses a matrix of local cluster labels $C = (c_{ik})$, where $c_{ik}$ indicates the cluster assignment for the $k$-th basis coefficient of curve $i$.

B. The Semi-Markovian Random Partition Model (smRPM)

The core innovation is the smRPM, a generalization of the tRPM that introduces higher-order dependence to match the B-spline degree.

1. Partition Evolution: The model defines a sequence of partitions $\rho_1, \dots, \rho_K$ evolving over the basis index $k$.
2. Auxiliary Variables ($\gamma$): To control the evolution, binary latent variables $\gamma_{ik}$ are introduced.
   • $\gamma_{ik} = 1$ implies unit $i$ is "locked" and cannot be reallocated for the next $d_\rho$ transitions.
   • This creates a $d_\rho$-order persistence, ensuring that cluster assignments remain stable over a block of coefficients, matching the B-spline support.
3. Dependence Structure:
   • Partition Dependence ($d_\rho$): The probability of a partition at step $k$ depends on the previous partition and the auxiliary variables from the previous $d_\rho$ steps. This relaxes the first-order Markov assumption.
   • Auxiliary Variable Dependence ($d_\gamma$): The authors propose a logistic autoregressive model for the auxiliary variables themselves. The probability of $\gamma_{ik}$ depends on the previous $d_\gamma$ values of $\gamma_{i, \cdot}$. This captures dependencies in the "locking" mechanism itself.
4. Prior on Coefficients: The basis coefficients $\theta^*_{kj}$ for cluster $j$ at basis $k$ are modeled hierarchically. Their prior mean is a weighted average of coefficients from clusters at $k-1$ that merge into cluster $j$, ensuring smoothness and continuity across cluster boundaries.

C. Inference

Posterior inference is performed via Gibbs Sampling. The authors derive full conditional distributions for:

• The partition labels $c_{ik}$.
• The auxiliary variables $\gamma_{ik}$.
• The hyperparameters governing the dependence of $\gamma$ (using Polson et al.'s data augmentation for logistic regression if $d_\gamma > 0$).
• The basis coefficients and noise variance.

3. Key Contributions

1. Novel smRPM Framework: Introduction of a semi-Markovian random partition model that allows for $d_\rho$-order dependence in partitions and $d_\gamma$-order dependence in auxiliary variables. This is a significant advancement over standard first-order Markovian models (like tRPM) and HMMs.
2. Alignment with B-Spline Properties: The model explicitly matches the dependence order ($d_\rho$) to the degree of the B-spline basis. This ensures that the "local clustering" is mathematically consistent with the definition of B-spline local support (requiring $d+1$ shared coefficients for local identity).
3. Flexible Dependency Modeling: By introducing autoregressive dependence in the auxiliary variables ($d_\gamma$), the model avoids the unrealistic assumption of independence often found in previous dependent partition models.
4. General Applicability: While motivated by functional data, the smRPM is presented as a general tool for any problem involving sequences of dependent partitions where higher-order dependence is required.

4. Results

Simulation Studies

• Performance vs. Dependence Order: The authors conducted simulations comparing smRPM with varying $d_\rho$ and $d_\gamma$.
  • Finding: Models where $d_\rho$ matches the true dependence order of the data significantly outperform Markovian models ($d_\rho=1$).
  • Finding: Setting $d_\gamma = d_\rho$ generally yields the best performance and robustness, outperforming cases where $d_\gamma=0$ (independent auxiliaries).
  • Comparison: The proposed smRPM outperforms the standard tRPM and the Hidden Markov Functional Local Clustering Model (HMFLCM), particularly in scenarios with smaller sample sizes or when the model specification is slightly misspecified (e.g., wrong number of basis functions).
• Robustness: The model remains effective even when the number of basis functions used for estimation differs from the data-generating process, provided the dependence order is correctly specified.

Real Data Application: Venice Lagoon Tides

• Data: Hourly tide level measurements from 11 monitoring stations in the Venice Lagoon.
• Context: The study analyzed periods where the MOSE flood barrier system was active and inactive.
• Findings:
  • Global vs. Local: The model successfully identified that stations behaved globally similarly when MOSE was inactive but exhibited distinct local behaviors when active.
  • Nuance: Unlike global clustering, the model detected a specific station (red cluster) that maintained a unique behavior throughout the interval, while others split into different local groups (blue/yellow) during high-tide events.
  • Uncertainty: The Bayesian framework provided posterior distributions for the number of clusters ($J_k$), offering a probabilistic assessment of clustering stability rather than a single point estimate.

5. Significance and Conclusion

This paper addresses a critical gap in functional data clustering: the inability of standard methods to capture local, partially coincident behaviors without imposing overly restrictive Markovian assumptions.

• Theoretical Impact: The smRPM extends the literature on dependent random partitions by formalizing higher-order dependencies, offering a more flexible prior for complex hierarchical structures.
• Practical Impact: The methodology provides a powerful tool for analyzing functional data where local features are of interest (e.g., medical signals, environmental monitoring, financial curves). The Venice Lagoon application demonstrates its ability to uncover subtle, time-varying patterns that global clustering would miss.
• Future Directions: The authors suggest extending the noise model to Gaussian Processes, incorporating covariates into the auxiliary variable distribution, and exploring alternative Exchangeable Partition Probability Functions (EPPFs) like the Pitman-Yor process for increased flexibility.

In summary, the proposed smRPM offers a statistically rigorous and computationally feasible solution for local clustering of functional data, leveraging the specific geometry of B-spline bases to achieve superior performance over existing state-of-the-art methods.