📊 statistics

Functional Clustering of Survival Data via Smoothed Log-Hazard Trajectories: A Risk-Dynamics Perspective

This paper proposes a novel functional clustering framework for survival data that models smoothed log-hazard trajectories using B-splines and Functional Principal Component Analysis to capture temporal risk dynamics, demonstrating superior interpretability and robustness compared to traditional cumulative-survival-based methods through extensive simulations and real-world clinical applications.

Original authors: Anna De Magistris, Elvira Romano, Fabrizio Maturo

Published 2026-06-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Anna De Magistris, Elvira Romano, Fabrizio Maturo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to understand how different groups of people (or machines, or patients) "age" or face a specific risk over time. Traditionally, statisticians look at a Survival Curve. Think of this like looking at a water tank over time. You see how much water is left in the tank at the end of the day, week, or year. It tells you the total amount of water lost, but it doesn't tell you when the water was leaking out or if the leak was a slow drip or a sudden gush.

This paper proposes a new way to look at the data. Instead of just watching the water level drop (the survival curve), the authors suggest looking at the Hazard Function. Think of this as the instantaneous speed of the leak at any given second. Did the leak start slow and then burst open? Did it stop for a while and then start again? This "leak speed" tells a much richer story about the dynamics of the risk.

Here is a simple breakdown of what the paper does:

1. The Problem: The "Noisy" Leak Speed

If you try to measure the "leak speed" (hazard) directly from real-world data, it's very messy. It's like trying to listen to a conversation in a room with a jackhammer nearby. The data jumps up and down wildly because of random chance or missing information. If you try to group people based on this noisy data, you might group them by the jackhammer's noise rather than the actual conversation.

2. The Solution: Smoothing and "Log-Hazard"

To fix this, the authors use a mathematical "smoother" (like a noise-canceling headphone for data). They take the messy, jagged leak-speed data and turn it into a smooth, flowing curve.

The Trick: They don't smooth the speed itself; they smooth the logarithm of the speed. Imagine taking a picture of a mountain range. If you look at the raw height, the peaks are sharp. If you look at the "log" of the height, the shape is preserved, but the math becomes much easier and more stable. This ensures the "leak speed" never goes below zero (which is impossible in reality) while making the math work better.

3. The Core Idea: Grouping by "Rhythm"

Once they have these smooth curves, they want to group them. But how do you group a curve?

The Old Way: Compare the curves point-by-point (like comparing two songs note-for-note). This is hard because the songs might be slightly out of sync.
The New Way (Functional PCA): The authors use a technique called Functional Principal Component Analysis (FPCA).
- The Analogy: Imagine you have 100 different dance routines. Instead of comparing every single step, FPCA finds the main moves that define the dances. Maybe "Move A" is how high they jump, and "Move B" is how fast they spin.
- The authors find the top few "moves" (principal components) that explain 95% of the differences between all the curves.
- They then ignore the tiny, random wiggles and focus only on these main moves.

4. The Clustering: Finding the Dance Crews

Now that they have reduced the complex curves down to a few simple numbers (the scores for "Move A" and "Move B"), they use standard clustering algorithms (like K-means) to group them.

The Result: They find groups of curves that share the same rhythm of risk.
- Group 1: Maybe these groups have a risk that spikes early and then settles down.
- Group 2: Maybe these groups have a risk that is low at first but slowly climbs up over time.
- Group 3: Maybe these groups have a risk that stays flat and steady.

5. What They Tested

The authors tested this method in two ways:

Simulations: They created fake data where they knew the "true" groups. They showed that their method could find these groups even when the curves were very similar or overlapping, whereas older methods (looking at the total water tank level) often got confused.
Real Data: They applied this to two real medical datasets:
- German Breast Cancer Study: They grouped patients based on how many lymph nodes were affected. Their method found a clear split between "low node count" and "high node count" groups based on the timing of the risk, not just the final outcome.
- Primary Biliary Cirrhosis (Liver Disease): They grouped patients based on disease stage and bilirubin levels. Again, they found distinct patterns in how the risk evolved over time.

6. The Bottom Line

The paper argues that looking at instantaneous risk dynamics (the rhythm of the leak) is often more informative than looking at cumulative survival (the total water lost).

Two groups might end up with the same number of survivors at the end of the study (same water level), but one group might have had a sudden crisis in the middle, while the other had a slow, steady decline.
The authors' method is designed to catch that difference. It groups people by how their risk behaves over time, not just by how many survive in the end.

Important Note from the Paper:
The authors are careful to say that while this method creates clear, interpretable groups, these are exploratory. They are like finding patterns in a map to guide further research, not necessarily a final, perfect medical diagnosis tool. The method works best when you want to understand the story of the risk over time, rather than just the final chapter.

Technical Summary: Functional Clustering of Survival Data via Smoothed Log-Hazard Trajectories

1. Problem Statement

Survival analysis clustering is traditionally dominated by individual-level clustering, which groups subjects based on survival times and covariates. A less explored but potentially valuable perspective is curve-level clustering, which aims to group aggregated survival curves (e.g., Kaplan-Meier curves) to identify broader trends across cohorts, hospitals, or regions.

Existing curve-level methods, such as Survclustcurves, primarily rely on cumulative survival probabilities and log-rank-based dissimilarity matrices. The authors argue that these approaches have a critical limitation: they operate on cumulative information, which can obscure differences in the timing, localization, and intensity of instantaneous risk. When survival curves overlap, cross, or converge, methods based on cumulative probabilities may fail to distinguish cohorts that exhibit fundamentally different underlying risk dynamics. Furthermore, standard non-parametric step functions (like Kaplan-Meier) are mathematically ill-suited for functional data analysis (FDA) due to their discontinuity and sensitivity to discretization noise.

2. Methodology

The paper proposes a Functional Data Analysis (FDA) framework that shifts the analytical focus from cumulative survival probabilities to the instantaneous hazard function. The methodology proceeds through the following stages:

A. Functional Representation of Log-Hazard Trajectories

Instead of clustering raw survival curves, the method models the log-hazard function ( $\log h(t)$ ).

Rationale: The hazard function captures instantaneous risk dynamics (peaks, accelerations, inflection points) that are often flattened in cumulative survival curves. The log-scale ensures positivity after back-transformation and allows for unconstrained smoothing.
Smoothing: Raw empirical hazard estimates are unstable and noisy. The authors apply B-spline smoothing to construct continuous, differentiable trajectories.
Smoothing Selection: Smoothing parameters are selected via a Generalized Cross-Validation (GCV) rule. Crucially, the authors do not select the absolute minimum GCV but choose a smoother within a 20% tolerance ( $\delta = 0.20$ ) of the minimum. Among these "GCV-equivalent" candidates, the one maximizing local variability (average integrated squared second derivative) is selected to preserve interpretable temporal features (e.g., delayed peaks) that overly rigid smoothing might attenuate.

B. Functional Principal Component Analysis (FPCA)

The smoothed log-hazard trajectories are projected into a low-dimensional space using FPCA.

Dimensionality Reduction: The number of retained Functional Principal Components (FPCs), denoted $M$ , is determined before clustering using a conservative 95% cumulative explained-variance rule ( $CPV(M) \geq 0.95$ ). This ensures that the clustering is performed on a representation that retains the main temporal structure of the risk dynamics, rather than optimizing for cluster separation.
Scores: Clustering is performed on unstandardized FPCA scores. Standardization is avoided to preserve the variance structure induced by the eigenvalues, preventing low-variance (potentially noisy) components from dominating the distance metric.

C. Clustering in the Score Space

Standard clustering algorithms (K-means, K-medoids, Ward's hierarchical clustering) are applied to the unstandardized FPCA scores.

Cluster Selection: The optimal number of clusters ( $K$ ) is selected after the FPCA representation is fixed, using internal validity criteria (average Silhouette width) and stability diagnostics (bootstrap resampling).
Interpretation: Distances in the FPCA score space approximate dissimilarities between log-hazard trajectories. Consequently, clusters represent groups with similar relative temporal risk dynamics (multiplicative patterns of hazard ratios) rather than identical absolute hazard values.

3. Key Contributions

Shift in Analytical Object: The paper redefines the object of clustering from cumulative survival curves to smoothed log-hazard trajectories, arguing that the latter encodes richer prognostic information regarding the timing and dynamics of risk.
Robust Functional Pipeline: The integration of B-spline smoothing (with a flexibility-preserving GCV rule) and FPCA provides a stable, differentiable representation of hazard functions, overcoming the instability of raw empirical hazard estimates.
Conservative Dimensionality Reduction: By fixing the FPCA truncation based on a 95% variance threshold independent of clustering performance, the method avoids overfitting and ensures that the clustering reflects genuine functional variability rather than noise.
Distinction from Benchmarks: The framework explicitly contrasts with cumulative-survival-based benchmarks, demonstrating that groups identified by hazard dynamics may differ significantly from those identified by cumulative survival decline.

4. Results

Simulation Studies

Complexity Continuum: The method was tested on simulated data with increasing overlap between latent log-hazard regimes (0% to 50% overlap).
- Performance: K-means applied to unstandardized FPCA scores consistently demonstrated the strongest recovery (Adjusted Rand Index, ARI), maintaining high performance even as overlap increased. It showed a more gradual degradation compared to benchmarks clustering standardized B-spline coefficients or cumulative survival curves.
- Sensitivity: The recovery performance was invariant across FPCA truncation thresholds ranging from 90% to 99%, confirming that the 95% rule does not artificially create the latent structure.
- Outlier Robustness: In stress tests with injected outlier trajectories, the FPCA-based pipeline remained informative, though the selected number of clusters and ARI varied non-monotonically depending on the interaction between outliers and the clustering algorithm.
Case Study: In a scenario with three distinct temporal risk regimes, the FPCA-log-hazard approach achieved perfect recovery (ARI = 1.000), whereas the cumulative survival benchmark failed to recover the time-localized regimes (ARI = 0.351), despite having high internal cohesion.

Real-World Applications

The method was applied to two clinical datasets: the German Breast Cancer Study (GBCS) and the Primary Biliary Cirrhosis (PBC) dataset.

GBCS: The method identified a two-cluster partition based on lymph node involvement (low vs. high). The clusters were driven by a persistent level difference in instantaneous recurrence risk. The partition showed moderate stability (mean Jaccard $\approx$ 0.60).
PBC: The method identified a two-cluster partition separating profiles with higher bilirubin/Stage 4 disease from those with lower bilirubin/Stage 2-3. The separation was driven almost entirely by the first FPC (93.4% variance), representing a global risk-level contrast. Stability was moderate-to-low (mean Jaccard $\approx$ 0.56).
Comparison: In both datasets, the cumulative-survival benchmarks sometimes yielded higher Silhouette values. The authors clarify that this does not imply superiority; rather, it indicates that cumulative curves admit cohesive partitions based on accumulated decline, while the proposed method identifies partitions based on instantaneous risk dynamics. These are distinct inferential questions.

5. Significance and Claims

The paper claims that the proposed framework offers a complementary tool to existing survival clustering methods, particularly when the scientific objective is to understand the temporal evolution of risk rather than just the final survival outcome.

Interpretability: The method provides an interpretable representation of relative temporal risk dynamics, where clusters are defined by shared patterns in the timing and magnitude of risk peaks.
Robustness: The approach is robust to structural complexity and overlapping hazard functions, outperforming benchmarks that rely on cumulative probabilities or raw spline coefficients.
Modesty: The authors explicitly state that the results from real-world data should be viewed as exploratory functional groupings rather than definitive clinical classifications. They acknowledge limitations, including the lack of full uncertainty propagation from individual-level data to the curve-level analysis and the sensitivity of the results to the number of aggregate trajectories available.

In conclusion, the paper argues that by modeling survival data as functional objects on the log-hazard scale, researchers can uncover latent risk dynamics that are geometrically obscured in traditional cumulative survival analyses.