Activation Function Design Sustains Plasticity in Continual Learning
This paper demonstrates that thoughtful activation function design, specifically through the introduction of Smooth-Leaky and Randomized Smooth-Leaky nonlinearities, serves as a lightweight, architecture-agnostic solution to sustain model plasticity and prevent adaptation loss in continual learning scenarios without requiring additional capacity or task-specific tuning.