Complexity of Linear Regions in Self-supervised Deep ReLU Networks

Technical Summary: Complexity of Linear Regions in Self-supervised Deep ReLU Networks

Paper Title: Complexity of Linear Regions in Self-supervised Deep ReLU Networks
Authors: Mufhumudzi Muthivhi, Terence L. van Zyl (University of Johannesburg)
Venue: Accepted to CVPR Findings 2026

1. Problem Statement

The complexity of Continuous Piecewise Linear (CPWL) neural networks—specifically those using ReLU activations—is often measured by the number of linear regions (convex polytopes) they partition the input space into. While existing research has extensively analyzed how these regions evolve during supervised learning, there is a significant gap in understanding how Self-Supervised Learning (SSL) affects this geometric structure.

SSL methods, such as contrastive learning and self-distillation, optimize representation spaces differently than supervised classifiers. The authors seek to understand how these distinct objectives influence the local distribution, number, and geometric properties (shape and size) of these linear regions.

2. Methodology

The researchers investigated the geometric evolution of linear regions across three paradigms: Supervised (Classifier, Triplet Loss, SupCon), Contrastive SSL (SimCLR, MoCo), and Self-distillation SSL (SimSiam, BYOL).

• Geometric Metrics: To characterize the polytopes, the study tracks four specific properties:
  • Volume (Area): The extent of the input space covered by a single affine transformation.
  • Eccentricity: A measure of anisotropy (how elongated or "stretched" a region is).
  • Boundaries: The number of unique edges/vertices of the polytope.
  • Number of Regions: The total count of distinct partitions.
• Dimensionality Reduction & Visualization:
  • For 2D datasets (Moons), they used a uniform sampling grid to map activation patterns.
  • For high-dimensional datasets (MNIST, FashionMNIST), they employed SplineCam to project a local neighborhood of the data distribution onto a 2D plane, allowing for the extraction of 2D polytopes.
• Collapse Simulation: To test the utility of these metrics in detecting failure modes, they simulated representation collapse in the SimSiam framework by removing the prediction head.

3. Key Contributions

• Comparative Geometric Analysis: The first systematic comparison of how supervised, contrastive, and self-distillation objectives shape the input space partitioning.
• New Observational Framework: A method to track the evolution of polytopal metrics (volume, eccentricity, boundaries) throughout the training process.
• Early Detection of Collapse: Demonstrating that geometric changes (merging of regions) occur much earlier than traditional statistical measures (standard deviation of features) during representation collapse.

4. Key Results

• Efficiency of SSL: SSL methods achieve comparable accuracy to supervised models but with substantially fewer linear regions. For example, on MNIST, SupCon requires ~11,349 regions for ~85% accuracy, whereas SimCLR achieves ~83% accuracy with only ~4,297 regions.
• Divergent Evolutionary Paths:
  • Supervised/Contrastive: These methods tend to expand the number of regions over time. Contrastive methods use negative pairs to "push" regions apart, creating more isotropic (circular/regular) and smaller regions.
  • Self-distillation: These methods exhibit a "merging" behavior. Without negative pairs to provide a repulsive force, regions consolidate into fewer, larger, and more anisotropic (elongated) partitions.
• Collapse Detection: During simulated collapse, the number of linear regions drops precipitously (retaining only ~5–20% of original regions) well before the standard deviation of the feature representations signals a problem.

5. Significance and Implications

The study concludes that polytopal metrics are reliable indicators of representation quality.

• Model Robustness: The finding that SSL models achieve high performance with lower complexity (fewer regions) aligns with the theory that reduced complexity enhances generalization and robustness.
• Practical Applications:
  • Monitoring: Geometric analysis can serve as an early-warning system for representation collapse during training.
  • Regularization: The authors suggest that future training strategies could explicitly optimize the "polytopal space" (e.g., encouraging specific volumes or eccentricities) to improve transferability.
  • Continual Learning: The distinction between the "expanding" nature of contrastive methods and the "consolidating" nature of self-distillation offers a roadmap for managing capacity when introducing new tasks to a model.