Coupled Nonlinear Dynamics of Reservoir Volume and Governance Quality: Stability Analysis and Reinforcement-Learning-Based Management

Technical Summary: Coupled Nonlinear Dynamics of Reservoir Volume and Governance Quality

Problem Statement
Reservoir management traditionally treats institutional factors, such as governance quality and enforcement effort, as fixed exogenous parameters. However, empirical evidence suggests that illegal water extraction and institutional decay are endogenous processes that co-evolve with physical reservoir states. Specifically, the scale of illegal withdrawal responds dynamically to the quality of monitoring and enforcement, creating a feedback loop where weak governance leads to increased losses, which in turn strains the system and potentially further degrades governance capacity. Existing reservoir operation models often fail to capture this socio-hydrological coupling, and while reinforcement learning (RL) has been applied to reservoir control, it has rarely been extended to govern a dynamic "governance quality" state that itself decays and requires active maintenance.

Methodology
The study proposes a coupled nonlinear dynamical model and a reinforcement learning (RL) framework to address these gaps.

1. Mathematical Modeling:

   • State Variables: The system is defined by reservoir volume $V(t)$ and governance quality $s(t) \in [0,1]$.
   • Dynamics: The evolution of $V(t)$ is governed by a stochastic differential equation incorporating inflow, fixed legal outflow, leakage, evaporation, and illegal withdrawal. Crucially, leakage, evaporation, and illegal withdrawal are modeled as nonlinear functions of both $V$ and $s$. For instance, illegal withdrawal increases as governance quality $s$ decreases.
   • Governance Dynamics: The governance state $s(t)$ evolves based on management effort $u(t)$ and a natural decay rate, representing the need for continuous investment to maintain institutional effectiveness.
   • Stochastic Forcing: Inflow and evaporation coefficients are modeled as mean-reverting Ornstein–Uhlenbeck processes to simulate hydrological uncertainty.

2. Analytical Stability Analysis (Open-Loop):

   • For the open-loop system (where $u(t)$ is an exogenous input), the authors establish positivity, boundedness, and the existence/uniqueness of an equilibrium point.
   • Local asymptotic stability is derived via Jacobian linearization, while global asymptotic stability is proven using Lyapunov's direct method and LaSalle's invariance principle.

3. Reinforcement Learning Formulation:

   • MDP Setup: The system is embedded in a Markov Decision Process (MDP) where the agent observes the state (volume, governance, inflow, evaporation) and selects a management effort $u$.
   • Algorithm: The Soft Actor-Critic (SAC) algorithm, an off-policy, maximum-entropy actor-critic method, is used to train a policy $\pi_\phi$. The reward function penalizes deviations from target volume and governance levels, control costs, and depletion events.
   • Closed-Loop Stability: The trained policy is treated as a state-feedback function $u = \pi_\phi(V, s)$. The authors prove that the resulting autonomous closed-loop system admits an equilibrium and satisfies local asymptotic stability conditions, provided the Lipschitz constant of the policy is sufficiently small relative to the natural decay of governance. Practical (Input-to-State) stability is also established under stochastic forcing.

4. Numerical Implementation:

   • The model is calibrated using parameters motivated by the Amir Kabir Dam (Karaj Dam) in Iran, a case study characterized by documented storage decline and governance challenges.
   • Five scenarios are simulated: a reference case, reduced inflow, weakened management capacity, intensified illegal withdrawal, and strengthened management capacity.

Key Results

• Cost Reduction: Across five representative scenarios, the learned SAC policy reduces cumulative control costs by 15–45% compared to a fixed-effort baseline. This efficiency is achieved by moderating management effort as the system approaches the target operating point, rather than maintaining a constant high effort.
• Stability Verification: Numerical verification confirms that the trained policy satisfies the derived Lipschitz and local stability conditions. The policy naturally emerged with a Lipschitz constant low enough to ensure stability without explicit spectral-norm regularization during training.
• The Efficiency-Risk Trade-off: A significant finding is a trade-off between cost efficiency and safety. While the learned policy reduces costs, it results in a higher frequency of excursions below the depletion threshold ($V_{min}$) in most scenarios compared to the fixed-effort baseline. For example, in the reference scenario, depletion excursions increased from 4.0% (baseline) to 8.5% (learned policy).
• Sensitivity Analysis: Increasing the weight of the depletion penalty in the reward function did not monotonically reduce the frequency of depletion excursions, suggesting this trade-off is inherent to the stochastic environment and the exploration-exploitation dynamics of the SAC algorithm, rather than a simple tuning issue.
• Physical Limits: In the reduced-inflow scenario, neither the fixed nor the learned policy could prevent reservoir depletion, confirming that no control policy can compensate for physically infeasible water balances.

Significance and Claims
The paper claims that treating governance quality as an endogenous, dynamically evolving state rather than an exogenous parameter fundamentally alters the predicted long-term behavior of reservoirs and the design of management policies.

• Theoretical Contribution: The study bridges the gap between socio-hydrological modeling and control theory by providing rigorous stability proofs (existence, local, and practical stability) for a closed-loop system where the control input is a learned, state-dependent policy.
• Methodological Contribution: It demonstrates that RL can be successfully applied to coupled human-natural systems where the "human" component (governance) is a state variable subject to decay and recovery, extending RL applications beyond purely physical release schedules.
• Practical Implication: The results offer a decision-support framework for reservoirs facing combined hydrological and institutional pressures. The authors emphasize that while learned policies offer significant cost savings, they may require careful risk management to avoid increased depletion frequency, suggesting that future work should explore risk-sensitive objectives or constrained policy optimization.

The authors remain modest regarding the specific calibration, noting that governance parameters were proxy-calibrated based on qualitative trends rather than direct data fitting, and that the model serves as a proof-of-concept for the coupled dynamics rather than a site-specific predictive tool for the Amir Kabir Dam.