Dynamic Optimization without Markov Assumptions: application to power systems
0 likes207 views
Ilab METIS is a collaboration between TAO, a machine learning and optimization team at INRIA, and Artelys, an SME focused on optimization. They work on optimizing energy policies through modeling power systems and simulating operational and investment decisions. Their methodologies hybridize reinforcement learning, mathematical programming, and direct policy search to optimize complex, constrained problems with uncertainties while minimizing model error. They have applied these techniques to problems involving European-scale power grids with stochastic renewables.
Dynamic Optimization without Markov Assumptions: application to power systems
- 1. Ilab METIS Optimization of Energy Policies Olivier Teytaud + Inria-Tao + Artelys TAO project-team INRIA Saclay Île-de-France O. Teytaud, Research Fellow, olivier.teytaud@inria.fr http://www.lri.fr/~teytaud/
- 2. Outline Who we are What we solve Methodologies
- 3. Ilab METIS www.lri.fr/~teytaud/metis.html Ilab METIS www.lri.fr/~teytaud/metis.html ● Metis = Tao + Artelys ● TAO tao.lri.fr, Machine Learning & Optimization ● Joint INRIA / CNRS / Univ. Paris-Sud team ● 12 researchers, 17 PhDs, 3 post-docs, 3 engineers ● Artelys www.artelys.com SME - France / US / Canada - 50 persons ==> collaboration through common platform ● Activities ● Optimization (uncertainties, sequential) ● Application to power systems
- 4. Fundings ● Inria team Tao ● Lri (Univ. Paris-Sud, Umr Cnrs 8623) ● FP7 european project (city/factory scale) ● Ademe Bia(transcontinental stuff) ● Ilab (with Artelys) ● Indema (associate team with Taiwan) ● Maybe others, I get lost in fundings
- 5. Outline Who we are What we solve Methodologies
- 6. Industrial application ● Building power systems is expensive power plants, HVDC links, networks... ● Non trivial planning questions ● Compromise: should we move solar power to the south and build networks ? ● Is a HVDC connection “x ↔ y” a good idea ? ● What we do: ● Simulate the operational level of a given power system (this involves optimization of operational decisions) ● Optimize the investments
- 7. ● Planning/control ● Pluriannual planning: evaluate marginal costs of hydroelectricity ● Taking into account stochasticity and uncertainties ==> IOMCA (ANR) ● High scale investment studies (e.g. Europe+North Africa) ● Long term (2030 - 2050) ● Huge (non-stochastic) uncertainties ● Investments: interconnections, storage, smart grids, power plants... ==> POST (ADEME) ● Moderate scale (Cities, Factories) ● Master plan optimization ● Stochastic uncertainties ==> Citines project (FP7) Specialization on Power Systems
- 8. Example: interconnection studies (demand levelling, stabilized supply)
- 9. The POST project – supergrids simulation and optimization European subregions: - Case 1 : electric corridor France / Spain / Marocco - Case 2 : south-west (France/Spain/Italiy/Tunisia/Marocco) - Case 3 : maghreb – Central West Europe ==> towards a European supergrid Related ideas in Asia Mature technology:HVDC links (high-voltage direct current)
- 10. Investment decisions through simulations ● Issues – Demand varying in time, limited previsibility – Transportation introduces constraints – Renewable ==> variability ++ ● Methods – Markovian assumptions ==> wrong – Simplified models ==> Model error >> optimization error ● Our approach ● Machine Learning on top of Mathematical Programming
- 11. Outline Who we are What we solve Methodologies
- 12. A few milestones ● Linear programming is fast ● Bellman decomposition: we can split short term reward + long term reward ● Folklore result: direct policy search ==> we use all of them
- 13. Hybridization reinforcement learning / mathematical programming ● Math programming – Nearly exact solutions for a simplified problem – High-dimensional constrained action space – But small state space & not anytime ● Reinforcement learning – Unstable – Small model bias – Small / simple action space – But high dimensional state space & anytime
- 14. Errors ● Statistical error: due to finite samples (e.g. weather data = archive), possibly with bias (climate change) ● Statistical model error: due to the error in the model of random processes ● Model error: due to system modelling ● Anticipativity error: due to assuming perfect forecasts ● Monoactor: due to neglecting interactions between actor (social welfare) ● Optim. error: due to imperfect optimization
- 15. Plenty of tools ● Dynamic programming based ==> bad modelization of long term dependencies ● Direct policy search: difficult to handle constraints ==> bad modelization of systems ● Model predictive control: bad modelization of randomness ==> we use combined tools
- 16. I love Direct Policy Search ● What is DPS ? ● Implement a simulator ● Implement a policy / controller ● Replace constants in the policy by free parameters ● Optimize these parameters on simulations ● Why I love it ● Pragmatic, benefits from human expertise ● The best in terms of model error ● But ok it is sometimes slow ● Not always that convenient for constraints
- 17. We propose specialized DPS ● A special structure for plenty of constraints ● After all, you can use DPS on top of everything, just by defining a “good” controller ● DP-based tools have a great representation ● Let us use DP-representations in DPS
- 18. Dynamic programming tools Decision at time T = argmax of reward over the T next time steps + V'(state) x StateAt(t0+T) with V computed backwards
- 19. Direct Value Search Decision at time T = argmax of reward over the T next time steps + f(, state) x StateAt(t0+T) with optimized through Direct Policy Search and f a general function approximator (e.g. neural) Using forecasts as in MPC As in DPstyle
- 20. Summary ● Model error: often more important than optim error (whereas most works on optim error) ● We propose methodologies ● Compliant with constraints ● More expensive than MPC ● But not more expensive than DP-tools ● Smallest model error ● User-friendly (human expertise)
- 21. What we propose ● Is ok for correctly specified problems ● Uncertainties which can be modelized by probabilities ● Less model error, more optim. error ● Optim. error reduced by big clusters ● Takes into account the challenges in new power systems ● Stochastic effects (increased by renewables) ● High scale actions (demand-side management) ● High scale models (transcontinental grids)
- 22. What we propose ● Open source ? ● Algorithms are public ● Tools are not ● Data/models are not ● Want to join ? ● Room for mathematics ● Room for geeks ● Room for people who like applications
- 23. Our tools ● Tested on real problems ● Include investment levels – There are operational decisions – There are investment decisions ● Parallel ● Expensive
- 24. Further work ● Nothing on multiple actors (national independence ? intern. risk ?) ● Non stochastic uncertainties: how do we modelize non-probabilistic uncertainties on scientific breakthroughs ? (Wald criterion, Savage, Nash, Regret...)
- 25. Bibliography ● Dynamic Programming and Suboptimal Control: A Survey from ADP to MPC. Bertsekas, 2005. (MPC = deterministic forecasts) ● “Newave vs Odin”: why MPC survives in spite of theoretical shortcomings ● Dallagi et Simovic (EDF R&D) : "Optimisation des actifs hydrauliques d'EDF : besoins métiers, méthodes actuelles et perspectives", PGMO (importance of precise simulations) ● Ernst: The Global Grid, 2013 ● Renewable energy forecasts ought to be probabilistic! Pinson, 2013 (wipfor talk) ● Training a neural network with a financial criterion rather than a prediction criterion. Bengio, 1997 ● Direct Model Predictive Control, Decock et al, 2014 (combining DPS and MPC)