Thesis Code: T11

Advancing water systems control by leveraging multiple forecasts and policy classes via deep Reinforcement Learning

Short description:

In control policy design, the effectiveness relies significantly on the chosen feature representation, which serves as the foundation for decision-making. However, selecting an appropriate feature representation can be challenging, especially in complex systems like water systems management, where multiple interacting processes complicate the understanding of the relevance of specific features for control tasks. Conventional control rules for water reservoirs typically rely on basic information systems (1), but there’s growing recognition of the benefits of more informative representations. Emerging systems leverage forecasts to enhance flexibility and resilience, yet determining the optimal information set for decision-making remains a challenge. This challenge is particularly evident in multipurpose systems with diverse demands and objectives, where the ideal information set may vary with different tradeoffs (2).

In the field of Reinforcement Learning, Direct Policy Search (DPS), and especially Evolutionary Multi-Objective DPS (EMODPS), stands out as a potent approach capable of incorporating multiple exogenous inputs in defining a control policy (3). However, traditional DPS methods often depend on predefined policy classes and fixed inputs, limiting their adaptability across different objective tradeoffs. Neuro-Evolutionary Multi-Objective DPS (NEMODPS) addresses this limitation by expanding the search space to include both policy architectures and coefficients (4). This enables dynamic exploration of policy functional classes and learning the optimal policy representation. Additionally, NEMODPS integrates feature selection to utilize the most informative feature set, further enhancing its effectiveness in control policy design.

The aim of this thesis is the improvement of the NEMODPS deep Reinforcement Learning approach, building on it to better condition the operation of water reservoirs. The student is expected to carry out the following activities on one or more selected case studies (e.g. Lake Como, Italy; Folsom Lake, California; Lake Kariba, Zambia/Zimbabwe):

  1. Literature review: reviewing the state of the art of water reservoir control, with a particular focus on Neuro-Evolutionary deep Reinforcement Learning algorithms (5)
  2. Design of traditional optimal operating policies not informed by any forecast and with fixed policy class
  3. Expand the feature selection including a diverse and more challenging set of input features

References

  1. Hejazi, M. I., Cai, X., & Ruddell, B. L. (2008). The role of hydrologic information in reservoir operation learning from historical releases. Advances in Water Resources, 31(12), 1636–1650. https://doi.org/10.1016/j.advwatres.2008.07.013
  2. M. Zaniolo, M. Giuliani, and A. Castelletti, ‘Policy Representation Learning for Multiobjective Reservoir Policy Design With Different Objective Dynamics’, Water Resources Research, vol. 57, no. 12, p. e2020WR029329, 2021, doi: 10.1029/2020WR029329.
  3. M. Giuliani, J. D. Herman, A. Castelletti, and P. Reed, ‘Many-objective reservoir policy identification and refinement to reduce policy inertia and myopia in water management’, Water Resources Research, vol. 50, Apr. 2014, doi: 10.1002/2013WR014700.
  4. M. Zaniolo, M. Giuliani, and A. Castelletti, ‘Neuro-Evolutionary Direct Policy Search for Multiobjective Optimal Control’, IEEE Transactions on Neural Networks and Learning Systems, vol. 33, no. 10, pp. 5926–5938, Oct. 2022, doi: 10.1109/TNNLS.2021.3071960.
  5. Xu, W., Meng, F., Guo, W., Li, X., & Fu, G. (2021). Deep reinforcement learning for optimal hydropower reservoir operation. Journal of Water Resources Planning and Management, 147(8), 04021045. https://doi.org/10.1061/(asce)wr.1943-5452.0001409

Relevant courses and knowledge:

Natural Resources Management

Number of students:

1 or 2

Requisites:

The student should be comfortable with data handling and programming skills (preferably Python, Matlab).