Improving greenhouse fruit-production control by integrating reinforcement learning into short-horizon model predictive control

Technical Summary: Improving Greenhouse Fruit-Production Control by Integrating Reinforcement Learning into Short-Horizon Model Predictive Control

Problem Statement
Greenhouse fruit production control is an economic optimal control problem that seeks to maximize revenue (fruit yield minus operating costs) while adhering to system constraints under external weather disturbances. A fundamental challenge in this domain is the timescale mismatch: climate dynamics operate on minutes, while crop development and fruit yield benefits occur over weeks (40–60 days). Ideally, a controller should optimize over the full production cycle to balance current operating costs against delayed economic benefits. However, solving optimal control problems with such long horizons using large-scale, nonlinear, and stiff models (e.g., GreenLight) is computationally intractable online.

Consequently, practical implementations often rely on Model Predictive Control (MPC) with short prediction horizons (e.g., one hour). While MPC explicitly handles constraints and weather forecasts, short horizons lead to "myopic" behavior, failing to capture the delayed economic benefits of fruit yield. Conversely, Reinforcement Learning (RL) can learn policies offline that consider long-term economic performance but struggles to enforce hard system constraints and often degrades in performance under unseen weather trajectories (poor generalization).

Methodology
The authors propose Trajectory-Selection RL–MPC, a framework that integrates the long-term economic foresight of an RL policy into a short-horizon MPC optimization. The method operates in a closed-loop manner at each time step $t$:

1. RL Rollout: A trained RL policy ($\pi_{RL}$) generates a rollout trajectory ($\tilde{\mathbf{x}}, \tilde{\mathbf{u}}$) over the prediction horizon $N$ starting from the current state $x(t)$ and using forecasted weather.
2. Terminal Ingredients: The terminal state of the RL rollout, $\tilde{x}(t+N)$, is used to construct:
   • A terminal region constraint ($\mathbb{X}_f$): A region around the RL terminal state where the MPC solution must end.
   • A terminal cost ($\hat{V}_f$): A penalty for deviations from the RL terminal state, encouraging the MPC solution to remain close to the RL's long-term plan.
3. MPC Optimization: A nonlinear MPC solves a finite-horizon optimization problem (horizon $N$) using the actual weather forecast and the terminal ingredients. This yields a predicted trajectory $(\mathbf{x}^\star, \mathbf{u}^\star)$.
4. Trajectory Selection: The framework evaluates the finite-horizon planning objective for both the RL rollout trajectory and the MPC-predicted trajectory. It selects the first control input ($u(t)$) from the trajectory with the superior objective value. This mechanism ensures the system does not degrade performance if the MPC optimization fails to improve upon the RL rollout due to local minima or computational tolerances.

The method was applied to GreenLight, a large-scale (27-state), nonlinear, and stiff greenhouse tomato production model. Two experimental settings were used:

• Specialist Policy: Trained on a single weather trajectory to assess if the framework can replicate high-performance RL behavior.
• Generalist Policy: Trained on a broad distribution of weather trajectories (2013–2018) to assess performance on unseen test trajectories (2011, 2012, 2019, 2020, 2023).

Key Results
Simulations were conducted over one-day periods (288 time steps) with a one-hour ($N=12$) prediction horizon.

• Specialist Policy Setting: Trajectory-selection RL–MPC matched the closed-loop performance of the high-performing specialist RL policy (cumulative objective difference of 0.196 vs. 0.194). It significantly outperformed standalone MPC with the same horizon (improvement of $\Delta \mathcal{J} = 0.224$), primarily by increasing Economic Performance Indicators (EPI) through better revenue generation rather than cost reduction.
• Generalist Policy Setting: On 25 unseen weather trajectories, Trajectory-selection RL–MPC outperformed the standalone generalist RL policy by 54% and standalone MPC by 80% in terms of cumulative economic objective.
  • Compared to the RL policy, the framework achieved similar revenue but reduced heating costs by 26.8% and lamp electricity costs by 12.0%, while reducing cumulative penalties (constraint violations) by 47.7%.
  • Compared to MPC, the framework achieved significantly higher revenue, overcoming the myopic behavior of short-horizon MPC.
• Selection Mechanism Analysis: The trajectory selection mechanism was critical. In the specialist setting, the RL rollout was frequently selected during daytime hours when balancing operating costs against future yield was crucial. In the generalist setting under unseen weather, the MPC-predicted trajectory was selected more often, indicating that the short-horizon optimization successfully refined the RL policy's trajectory to handle specific, unseen weather conditions and reduce costs.
• Ablation Study: Removing the selection mechanism caused a 12.2% drop in performance and a 95.2% increase in penalties, highlighting its necessity for maintaining constraint satisfaction and performance stability.

Significance and Claims
The paper claims that Trajectory-Selection RL–MPC successfully bridges the gap between the long-term economic optimization capabilities of RL and the constraint-handling and short-term refinement capabilities of MPC.

• Overcoming Myopia: The framework demonstrates that short-horizon MPC (1 hour) can effectively incorporate long-term economic information (fruit yield delays) via terminal ingredients derived from RL, thereby mitigating the myopic behavior typical of short-horizon controllers.
• Robustness to Unseen Conditions: Unlike standalone RL, which may overfit to training distributions, the proposed framework improves performance on unseen weather trajectories by using MPC to refine the RL policy's trajectory based on actual short-term weather forecasts.
• Scalability: The method is validated on GreenLight, a complex, stiff, large-scale model where long-horizon optimization is computationally infeasible, suggesting the approach is scalable to realistic, high-fidelity greenhouse models.
• Complementary Strengths: The results confirm that the selection mechanism allows the system to leverage the best of both worlds: the long-term foresight of the RL policy and the constraint satisfaction and cost optimization of the MPC.

The authors note that while the results are promising, they are based on closed-loop simulations with perfect state information and weather forecasts. Future work is required to evaluate the framework over full production cycles with explicit fruit-development dynamics and to test robustness under real-world uncertainties such as model mismatch and state estimation errors.