Time Delayed Recurrent Neural Network for Multi-Step Prediction
0 likes217 views
This document describes Kostas Hatalis's research on using recurrent neural networks for multi-step time series prediction of renewable power generation. It discusses using nonlinear autoregressive networks and particle swarm optimization learning in time delayed recurrent neural networks to forecast ocean wave characteristics. Evaluation of the models found the particle swarm optimization approach improved accuracy over nonlinear autoregressive networks, particularly for noisier, longer term predictions.
![Work Done
[1] Hatalis, Kostas, et al. ”Multi-step forecasting of wave power using a
nonlinear recurrent neural network.” 2014 IEEE PES General Meeting—
Conference & Exposition. IEEE, 2014.
[2] Hatalis, Kostas, et al. ”Adaptive particle swarm optimization learning
in a time delayed recurrent neural network for multi-step prediction.”
Foundations of Computational Intelligence (FOCI), 2014 IEEE Symposium
on. IEEE, 2014.
[3] Hatalis, Kostas, et al. ”Swarm Based Parameter Estimation of Wave
Characteristics for Control in Ocean Energy Farms.” In Proceedings of the
IEEE Power Energy Society General Meeting, 2015.
[4] Hatalis, Kostas, et al. ”Particle Swarm Based Model Exploitation for
Parameter Estimation of Wave Realizations.” IEEE Symposium Series on
Computational Intelligence , 2016
Kostas Hatalis Recurrent Neural Network 2014 18 / 18](https://image.slidesharecdn.com/chapter2-190503195453/85/Time-Delayed-Recurrent-Neural-Network-for-Multi-Step-Prediction-18-320.jpg)
Time Delayed Recurrent Neural Network for Multi-Step Prediction
- 1. Time Delayed Recurrent Neural Network for Multi-Step Prediction By Kostas Hatalis hatalis@gmail.com Dept. of Electrical & Computer Engineering Lehigh University, Bethlehem, PA 2014 Kostas Hatalis Recurrent Neural Network 2014 1 / 18
- 2. Renewable Power Forecasting Forecasting is essential to the integration of renewable power generation to the smart grid. Kostas Hatalis Recurrent Neural Network 2014 2 / 18
- 3. General Short Term Time Frames and Methods Resolution of forecasting: Seconds: e.g. turbine/converter control. Minutes: e.g. balancing and transmission. Hours: e.g. storage and scheduling. Days: e.g. farm operations. Two categories of forecasting methods: Physical: meteorological models. Statistical: time series / machine learning. My research focusing on statistical models. Kostas Hatalis Recurrent Neural Network 2014 3 / 18
- 4. Time Series Forecasting Time series forecasts are made as follows: 1 Study features of the series. 2 Remove trend and seasonality to get stationary residuals. 3 Test if IID noise then fit an ARMA model to residuals. 4 Forecast residual series by minimizing the mean squared error of the expected future value. Kostas Hatalis Recurrent Neural Network 2014 4 / 18
- 5. Trend and Seasonal Fitting The classical decomposition model: Kostas Hatalis Recurrent Neural Network 2014 5 / 18
- 6. Fitting ARMA Models For our residual sequence, Yt is an ARMA(p,q) process if: Yt − φ1Yt−1 − ... − φpYt−p = Zt + θ1Zt−1 + ... + θqZt−q where Zt is white noise with mean zero and variance σ2. Once we estimate all the parameters we can forecast future values at time t + k. Eg. of an ARMA(2,1) model forecasting at t + 1 ahead: ˆYt+1 = φ1Yt + φ2Yt−1 + Zt+1 − θ1Zt ARMA is a model for the conditional mean of a process. Kostas Hatalis Recurrent Neural Network 2014 6 / 18
- 7. Time Series Forecasting Other Time Series forecasting models include: Weighted moving average Kalman filtering Exponential smoothing Autoregressive integrated moving average (ARIMA) Seasonal ARIMA (SARIMA) Extrapolation methods Downside? Most assume data is stationary (distribution and parameters do not change overtime), renewables are non-stationary! Kostas Hatalis Recurrent Neural Network 2014 7 / 18
- 8. Evaluation Methods Methods evaluating the predicted values vs the observed value include: Mean Squared Error (MSE) Root Mean Squared Error (RMSE) Median Absolute Deviation (MAD) Mean Absolute Percentage Error (MAPE) Sum of Squared Error (SSE) Mean Absolute Error (MAE) Signed Mean Squared Error (SMSE) Kostas Hatalis Recurrent Neural Network 2014 8 / 18
- 9. Machine Learning Approach Machine learning based forecasting updates it’s model with every new observation Prediction inference based on supervised learning which consists in learning the link between two datasets: the observed data X and an variable y that we are trying to predict, usually called “targets”. Kostas Hatalis Recurrent Neural Network 2014 9 / 18
- 10. Statistical Learning Forecasting Methods A number of machine learning methods have been applied for point forecasting such as support vector machines, Bayesian networks, k-nearest neighbors, etc. For point forecasting Artificial Neural Networks (ANN) are amongst the most popular. Universal Approximation Theorem An artificial neural network with a single hidden layer containing a finite number of neurons can approximate any function provided the activation function f of the hidden neurons are non-linear. Kostas Hatalis Recurrent Neural Network 2014 10 / 18
- 11. NARNET - Forecasting Ocean Wave In my initial forecasting work I used an ANN called nonlinear autoregressive network (NARNET) to forecast wave heights. https://en.wikipedia.org/wiki/Significantwaveheight Kostas Hatalis Recurrent Neural Network 2014 11 / 18
- 12. NARNET - Building a Neural Network When building an ANN its important to define: Architecture: number of layers and nodes. Activation function: sigmoid, tanh, etc. Cost function: mean squared error, etc. Learning: gradient descent backpropagation, etc. Kostas Hatalis Recurrent Neural Network 2014 12 / 18
- 13. NARNET - Model For time series prediction NARNET is a recurrent ANN: Kostas Hatalis Recurrent Neural Network 2014 13 / 18
- 14. NARNET - Results Kostas Hatalis Recurrent Neural Network 2014 14 / 18
- 15. PSONET - Simulated Ocean Waves Short term forecasting of ocean waves must be simulated. Also NARNET has few problems, such as using too much memory and occasionally getting stuck in local minamas. Needed to find new training solution. Kostas Hatalis Recurrent Neural Network 2014 15 / 18
- 16. PSONET - Particle Swarm Optimization The adaptive particle swarm optimization (APSO) algorithm inspired by flock of birds/fish searching for food: −→v i (t + 1) = ω−→v i (t) + c1φ1(−→p i (t) − −→x i (t)) + c2φ2(−→p g (t) − −→x i (t)) −→x i (t + 1) = −→x i (t) + ∆t−→v i (t + 1) Kostas Hatalis Recurrent Neural Network 2014 16 / 18
- 17. PSONET - Results Table: Table of error statistics with higher noisy data. PSONAR 5s 10s 30s 60s MSE 0.1603 0.9153 8.9066 13.4073 RMSE 0.4579 0.9567 2.9843 3.6615 MAPE 0.1734 0.2964 2.0758 3.2429 MAD 0.3470 0.6783 2.4161 2.8644 CC 0.9974 0.9693 0.3743 0.0902 NARNET 5s 10s 30s 60s MSE 0.2097 1.4119 10.8491 15.0918 RMSE 0.4003 1.1882 3.2937 3.7525 MAPE 0.2531 0.7421 1.3229 4.7647 MAD 0.3698 0.8938 2.0769 3.0842 CC 0.9971 0.9165 0.31433 0.12301 Kostas Hatalis Recurrent Neural Network 2014 17 / 18
- 18. Work Done [1] Hatalis, Kostas, et al. ”Multi-step forecasting of wave power using a nonlinear recurrent neural network.” 2014 IEEE PES General Meeting— Conference & Exposition. IEEE, 2014. [2] Hatalis, Kostas, et al. ”Adaptive particle swarm optimization learning in a time delayed recurrent neural network for multi-step prediction.” Foundations of Computational Intelligence (FOCI), 2014 IEEE Symposium on. IEEE, 2014. [3] Hatalis, Kostas, et al. ”Swarm Based Parameter Estimation of Wave Characteristics for Control in Ocean Energy Farms.” In Proceedings of the IEEE Power Energy Society General Meeting, 2015. [4] Hatalis, Kostas, et al. ”Particle Swarm Based Model Exploitation for Parameter Estimation of Wave Realizations.” IEEE Symposium Series on Computational Intelligence , 2016 Kostas Hatalis Recurrent Neural Network 2014 18 / 18