OUI as a Structural Observable: Towards an Activation-Centric View of Neural Network Training

Technical Summary: OUI as a Structural Observable

Problem Statement
Current neural network training diagnostics rely heavily on external metrics such as loss, accuracy, and final calibration. These metrics are often delayed, compressing the internal narrative of training, and fail to reveal the specific structural regimes a model enters during the learning process. While activation functions are the source of a network's expressive power—enabling the partitioning of input space into distinct functional regions via activation patterns—standard diagnostics rarely monitor the evolution of these internal structures. This gap leads to expensive hyperparameter searches, late detection of failures, and a lack of interpretability regarding training dynamics.

Methodology and Definition
The paper centers on the Overfitting–Underfitting Indicator (OUI), redefined here as a structural observable of learning dynamics. The authors adopt a batch-based formulation (Fernández-Hernández et al., 2026) to monitor training online:

1. Activation Masking: For a layer $l$ with $d_l$ units and a batch $B_t$, a binary mask $m^{(l)}_{b,n}(t)$ is defined for each neuron $n$ and sample $x_b$, indicating if the preactivation is positive.
2. Activation Counting: For each neuron, the count $s^{(l)}_n(t)$ of active samples in the batch is computed.
3. Minority Count: The metric $u^{(l)}_n(t) = \min(s^{(l)}_n(t), B - s^{(l)}_n(t))$ is calculated. This value is high when a neuron splits the batch in a balanced manner and low when the neuron is consistently on or off (biased).
4. OUI Calculation: The OUI for the module is the average of these minority counts normalized by $\lfloor B/2 \rfloor$.

Operationally, a high OUI indicates that many units are partitioning the probe batch in a balanced way, reflecting a rich internal activation structure. A low OUI suggests the module has become biased, with units behaving uniformly across inputs.

Key Results and Evidence
The paper synthesizes evidence from three distinct domains to demonstrate OUI's utility as an early, label-free signal:

• Supervised Learning (Regularization): In DenseNet-BC-100 experiments on CIFAR-100, OUI trajectories separated early (around 15% of training) for different weight decay values. High OUI values correlated with better generalization, while low values anticipated underfitting or overfitting regimes before standard validation metrics could distinguish them.
• Reinforcement Learning (PPO Actor-Critic): Reformulated for PPO, OUI successfully discriminated between learning-rate regimes early in training. Crucially, the paper observes a structural asymmetry: optimal critics occupy an intermediate OUI band, while strong actors tend to operate at higher OUI values. This suggests the "good" structural regime is context-dependent.
• Activation-Pattern Convergence: Citing recent work (Pérez-Corral et al., 2026), the paper notes that activation patterns often stabilize significantly earlier than parameter updates. In ViT-B16 training, activation convergence metrics saturated while loss continued to improve, supporting a two-timescale view where networks refine weights within stable activation regimes.

Contributions
The primary contribution of this paper is not a new algorithm, but a conceptual reframing and synthesis of existing evidence:

1. Structural Observable: The paper argues that OUI should be understood as a first practical observable of the internal structural dimension of learning, rather than merely a heuristic for hyperparameter tuning.
2. Activation-Centric View: It proposes an "activation-centric theory of training dynamics," positing that internal activation structures become informative and actionable before convergence, offering a dimension of training not captured by endpoint metrics.
3. Synthesis of Evidence: It unifies findings across supervised learning, reinforcement learning, and online control to show a recurring pattern: activation-based observables can reveal the fate of a training run early.
4. Research Direction: It identifies open questions necessary to advance this view, including the "phase structure" of training (early structural organization vs. late refinement), the "adequacy" of activation regimes relative to specific roles and tasks, and the theoretical connection between these observables and formal notions of stability and generalization.

Significance and Claims
The paper claims that viewing OUI as a structural observable shifts the perspective of training from monitoring delayed external outcomes to monitoring internal structural organization. This has immediate practical implications for early screening in foundation model pretraining, where late-stage failures are costly. Furthermore, it suggests a shift in functional control, allowing training to be regulated by signals closer to what the network is functionally doing to data, rather than just what the optimizer is doing to parameters.

The authors maintain a modest stance, acknowledging that OUI is currently an "empirical foothold" toward a broader theory. They note limitations, such as OUI's natural definition for ReLU-like patterns and the risk of feedback-loop confounds if used as an online control signal. The paper concludes that if the shift from viewing activations as hidden byproducts to explicit objects of study holds, an activation-centric theory of training becomes a necessary step for understanding how neural networks learn.