Agile Reinforcement Learning through Separable Neural Architecture and Applications

Technical Summary: Agile Reinforcement Learning through Separable Neural Architecture and Applications

Problem Statement
Deep reinforcement learning (RL) is increasingly deployed in resource-constrained environments, such as robotics with limited interaction budgets, edge devices with strict memory constraints, and industrial systems where training interruptions are common. The standard function approximators used in these settings, multilayer perceptrons (MLPs), are often parameter-inefficient. This inefficiency stems from an imperfect inductive bias; MLPs distribute representational capacity uniformly across the input space regardless of where the value function actually varies. This mismatch leads to high sample complexity and poor convergence reliability. While model compression techniques exist, they are post-hoc and do not improve learning efficiency. Recent spline-based architectures, such as Kolmogorov-Arnold Networks (KANs), offer parameter efficiency but suffer from significant computational overhead due to dense edge-wise evaluation, limiting their practical adoption in online RL.

Methodology
This work introduces SPAN (SPline-based Adaptive Networks), a novel function approximation architecture designed for resource-constrained RL. SPAN adapts the KHRONOS framework (Kernel-Expansion Hierarchy for Reduced-Order, Neural-Optimized Surrogates), previously validated for supervised regression, to the online RL setting.

The core architectural innovations are:

1. Learnable Preprocessing Layer: RL observations are arbitrary, non-stationary, and unbounded, whereas KHRONOS requires inputs in the bounded domain $[0, 1]^d$. SPAN integrates a fully connected layer with a sigmoid activation ($\mathbf{z} = \sigma(\mathbf{W}_{pre}\mathbf{s} + \mathbf{b}_{pre})$) to project raw observations into this bounded domain. This layer also serves to standardize input scales and recombine features, conceptually resembling a Koopman operator lifting map that disentangles raw inputs into a latent space where dynamics admit a simpler, separable representation.
2. Separable Tensor-Product Structure: SPAN utilizes B-spline basis functions with local support. By leveraging a rank-$M$ tensor decomposition, SPAN reduces parameter complexity from exponential $\mathcal{O}(N^d)$ (as in full tensor grids) to linear $\mathcal{O}(MdN)$ with respect to input dimension $d$. This overcomes the curse of dimensionality while maintaining the local support property, ensuring that gradient updates from a specific state only affect basis functions in the local neighborhood of that state.
3. Integration: SPAN serves as a drop-in replacement for MLPs in standard actor-critic frameworks. It is integrated into Soft Actor-Critic (SAC) for continuous control, Proximal Policy Optimization (PPO) for discrete control, and Implicit Q-Learning (IQL) for offline RL, leaving training objectives and update rules unchanged.

Key Contributions

1. Architectural Adaptation: The authors present an adaptation of the KHRONOS framework for RL, introducing the learnable preprocessing layer to bridge the gap between stationary normalized inputs and non-stationary RL observations.
2. Theoretical Grounding: The paper establishes a theoretical connection between SPAN’s local support property and the Lipschitz continuity of RL value functions and policies. It argues that because value functions in physically grounded tasks exhibit local smoothness, a function approximator with spatial locality (where capacity is concentrated where the agent’s experience is concentrated) is structurally aligned with the problem.
3. Empirical Validation: SPAN is evaluated across online (PPO, SAC), offline (IQL), and real-world applications, demonstrating improvements in sample efficiency, success rates, and computational cost compared to MLP baselines.

Results

• Online Control (PPO & SAC): Across classic control (CartPole, Acrobot, LunarLander) and MuJoCo continuous control (Hopper, InvertedPendulum, HalfCheetah, Walker2d) tasks, SPAN achieved a 30–50% improvement in sample efficiency and 1.3–9 times higher success rates compared to MLP baselines.
• Computational Cost: Although SPAN incurs a per-step evaluation overhead of 1.2–1.8 times that of MLPs, its superior convergence reliability results in an expected total training cost 1.3–6.3 times lower than MLP baselines when accounting for convergence failures. For instance, on the Hopper task, MLPs failed to converge in most seeds, rendering their expected cost effectively infinite at higher performance thresholds, whereas SPAN maintained high success rates.
• Offline RL (IQL): On the D4RL Adroit Hand Manipulation benchmark, SPAN outperformed MLPs on Expert datasets, achieving improvements of up to 20× on the Door task and 3.3× on the Relocate task.
• Real-World Application (Data Center HVAC): In a Sinergym data center HVAC control simulation, SPAN reduced energy consumption in 9 of 12 months relative to MLP. Simultaneously, it achieved a 1.1–3.4 times reduction in thermal comfort violations across the evaluation year. The paper notes that SPAN’s slightly higher energy consumption in peak summer months coincided with significantly fewer comfort violations, indicating a more robust maintenance of thermal constraints.

Significance and Claims
The paper claims that SPAN is a viable, high-performance alternative to MLPs for learning intrinsically efficient policies in resource-limited settings. By aligning its inductive bias with the Lipschitz structure of RL value functions, SPAN offers superior sample efficiency and convergence reliability. The authors emphasize that while SPAN has higher per-step computational costs, the reduction in required environment interactions and the avoidance of training failures make it more computationally efficient overall. The results suggest that structured function approximators with inductive biases grounded in the mathematical properties of the target functions offer a productive direction for improving the efficiency and reliability of deep reinforcement learning. The paper also acknowledges limitations, noting that SPAN is currently a feedforward architecture and may struggle in sparse-reward settings where value functions change sharply near goals.