Smooth Pinball based Quantile Neural Network
0 likes404 views
The document presents a novel approach for nonlinear nonparametric probabilistic forecasting using a smooth pinball-based quantile neural network (SPNN), addressing issues like quantile crossover. The proposed method demonstrates enhanced skill, reliability, and sharpness in forecasting wind power using publicly available data from the 2014 Global Energy Forecasting Competition. It introduces new objective functions and constraints to improve quantile estimates and utilizes gradient descent for training the SPNN model.
![Smooth Pinball Neural Network (SPNN)
The objective function for SPNN is then given by
E =
λ1
2NM
W [1] 2
F +
λ2
2NM
W [2] 2
F +
1
NM
N
t=1
M
m=1
...
τm(yt − ˆq
(τm)
t ) + α log 1 + exp −
yt − ˆq
(τm)
t
α
.
Kostas Hatalis Quantile Neural Network 2018 5 / 21](https://image.slidesharecdn.com/chapter5-190503202617/85/Smooth-Pinball-based-Quantile-Neural-Network-5-320.jpg)
![Smooth Pinball Neural Network (SPNN)
Gradient descent with backpropagation can be used to train SPNN.
Example gradients for a 2 layer SPNN:
∂Et
∂W [2]
=
λ2
M
W [2]
+
∂Et
∂ ˆQt
·
∂ ˆQt
∂Z
[2]
t
·
∂Z
[2]
t
∂W [2]
=
λ2
M
W [2]
+
1
M
1
1 + exp yt − ˆQt
α
− T
Ht
With respect to the weights of the first layer W[1] as follows
∂Et
∂W [1]
=
λ1
M
W
[1]
+
∂Et
∂ ˆQt
·
∂ ˆQt
∂Z
[2]
t
·
∂Z
[2]
t
∂Ht
·
∂Ht
∂Z
[1]
t
·
∂Z
[1]
t
∂W [1]
=
λ1
M
W
[1]
+
1
M
1
1 + exp
yt − ˆQt
α
− T
W
[2]
1 − H
2
t Xt
.
Kostas Hatalis Quantile Neural Network 2018 6 / 21](https://image.slidesharecdn.com/chapter5-190503202617/85/Smooth-Pinball-based-Quantile-Neural-Network-6-320.jpg)
![Sensitivity Analysis
Partial Derivatives (PaD) Method:
dt,i,M =
∂ ˆQτm
t
∂xt,i
=
∂ ˆQτm
t
∂Z
[2]
t
·
∂Z
[2]
t
∂Ht
·
∂Ht
∂Z[1],t
·
∂Z
[1]
t
∂xt,i
Sum of Square Derivatives (SSD):
SSDi,M =
N
t=1
(dt,i,M)
Kostas Hatalis Quantile Neural Network 2018 13 / 21](https://image.slidesharecdn.com/chapter5-190503202617/85/Smooth-Pinball-based-Quantile-Neural-Network-13-320.jpg)
![Work Done
Probabilistic Forecasting by Deep Learning
[1] Kostas Hatalis, Alberto J Lamadrid, Katya Scheinberg, and Shalinee
Kishore. Multi quantile estimation based neural network for probabilistic
forecasting of wind power. IEEE Transactions on Neural Networks and
Learning Systems (In-Submission).
Kostas Hatalis Quantile Neural Network 2018 21 / 21](https://image.slidesharecdn.com/chapter5-190503202617/85/Smooth-Pinball-based-Quantile-Neural-Network-21-320.jpg)
Smooth Pinball based Quantile Neural Network
- 1. Smooth Pinball based Quantile Network: Neural Probabilistic Forecasting By Kostas Hatalis hatalis@gmail.com Dept. of Electrical & Computer Engineering Lehigh University, Bethlehem, PA 2018 Kostas Hatalis Quantile Neural Network 2018 1 / 21
- 2. Introduction Inspired by SVQR we want to expand the investigation of nonlinear nonparametric probabilistic forecasting using deep learning. 1 We propose and investigate a new objective function for neural networks. 2 We introduce a way to prevent the quantile crossover problem. 3 We showcase how a multiple quantile based neural network can be used for probabilistic forecasting of wind. 4 We design experiments using publicly available data from the Global Energy Forecasting Competition 2014. 5 We show our method improves the skill, reliability, and sharpness over various benchmarks. Kostas Hatalis Quantile Neural Network 2018 2 / 21
- 3. Smooth Pinball Function Pinball Loss Function ρτ (u) = τu if u ≥ 0 (τ − 1)u if u < 0 Quantile regression optimization problem: min W ,b 1 N N t=1 ρτ (yt − ˆq (τ) t ) (1) The smooth approximation of the pinball function: Sτ,α(u) = τu + α log 1 + exp − u α Smooth quantile regression: min W ,b 1 N N t=1 Sτ,α(yt − ˆq (τ) t ) Kostas Hatalis Quantile Neural Network 2018 3 / 21
- 4. Smooth Pinball Function Kostas Hatalis Quantile Neural Network 2018 4 / 21
- 5. Smooth Pinball Neural Network (SPNN) The objective function for SPNN is then given by E = λ1 2NM W [1] 2 F + λ2 2NM W [2] 2 F + 1 NM N t=1 M m=1 ... τm(yt − ˆq (τm) t ) + α log 1 + exp − yt − ˆq (τm) t α . Kostas Hatalis Quantile Neural Network 2018 5 / 21
- 6. Smooth Pinball Neural Network (SPNN) Gradient descent with backpropagation can be used to train SPNN. Example gradients for a 2 layer SPNN: ∂Et ∂W [2] = λ2 M W [2] + ∂Et ∂ ˆQt · ∂ ˆQt ∂Z [2] t · ∂Z [2] t ∂W [2] = λ2 M W [2] + 1 M 1 1 + exp yt − ˆQt α − T Ht With respect to the weights of the first layer W[1] as follows ∂Et ∂W [1] = λ1 M W [1] + ∂Et ∂ ˆQt · ∂ ˆQt ∂Z [2] t · ∂Z [2] t ∂Ht · ∂Ht ∂Z [1] t · ∂Z [1] t ∂W [1] = λ1 M W [1] + 1 M 1 1 + exp yt − ˆQt α − T W [2] 1 − H 2 t Xt . Kostas Hatalis Quantile Neural Network 2018 6 / 21
- 7. Smooth Pinball Neural Network (SPNN) Kostas Hatalis Quantile Neural Network 2018 7 / 21
- 8. Kostas Hatalis Quantile Neural Network 2018 8 / 21
- 9. Noncrossing Quantiles - Smooth Penalty Constraint The condition 0 < τ1 < ... < τM are defined as the orders of M conditional quantiles to be estimated. To ensure these quantiles do not cross each other the following constraint is needed q (τ1) t ≤ ... ≤ q (τM ) t , ∀t. We define the non-crossing quantile penalty term p as follows penalty = c N t=1 M m=1 max 0, − ˆq (τm−1) t − ˆq (τm) t 2 where ˆq (τ0) t = 0, is the least amount that the two quantile should differ by, and c is the penalty parameter with a high value. Kostas Hatalis Quantile Neural Network 2018 9 / 21
- 10. Empirical Evaluation Datasets Publicly available Global Energy Forecasting Competition 2014 (GEFCom2014) Evaluation Metrics QS, IS, ACE, and Sharpness Benchmark Methods Uniform, Climatology, Persistence, SVQR, QR Case Studies Both studies we test SPNN having one and two hidden layers denoted as SPNN1 and SPNN2. Only use raw data, no feature engineering! 1. Estimate prediction intervals with nominal coverage from 10% to 90% in increments of 10%. Two wind farms. Also looked at QVSS score. 2. We estimate 99 quantiles across all 10 wind farms. Kostas Hatalis Quantile Neural Network 2018 10 / 21
- 11. Quantile Verification Skill Score (QVSS) Quantile verification skill score (QVSS): QVSS = 1 − QSfor QSref QSfor is QS for the forecast method of interest (SPNN in our case) QSref is the QS value for the reference forecast of a benchmark method, which we use linear quantile regression. Measures relative performance between methods. Kostas Hatalis Quantile Neural Network 2018 11 / 21
- 12. Example Forecast Kostas Hatalis Quantile Neural Network 2018 12 / 21
- 13. Sensitivity Analysis Partial Derivatives (PaD) Method: dt,i,M = ∂ ˆQτm t ∂xt,i = ∂ ˆQτm t ∂Z [2] t · ∂Z [2] t ∂Ht · ∂Ht ∂Z[1],t · ∂Z [1] t ∂xt,i Sum of Square Derivatives (SSD): SSDi,M = N t=1 (dt,i,M) Kostas Hatalis Quantile Neural Network 2018 13 / 21
- 14. Sensitivity Analysis Kostas Hatalis Quantile Neural Network 2018 14 / 21
- 15. Sensitivity Analysis, partial derivative of U100 for 90% Kostas Hatalis Quantile Neural Network 2018 15 / 21
- 16. Case Study 1: Zone 1 Kostas Hatalis Quantile Neural Network 2018 16 / 21
- 17. Case Study 1: Zone 2 Kostas Hatalis Quantile Neural Network 2018 17 / 21
- 18. Case Study 2: ACE and Sharpness Kostas Hatalis Quantile Neural Network 2018 18 / 21
- 19. Case Study 2: QS and IS Kostas Hatalis Quantile Neural Network 2018 19 / 21
- 20. Case Study 2: GEFCom2014 Kostas Hatalis Quantile Neural Network 2018 20 / 21
- 21. Work Done Probabilistic Forecasting by Deep Learning [1] Kostas Hatalis, Alberto J Lamadrid, Katya Scheinberg, and Shalinee Kishore. Multi quantile estimation based neural network for probabilistic forecasting of wind power. IEEE Transactions on Neural Networks and Learning Systems (In-Submission). Kostas Hatalis Quantile Neural Network 2018 21 / 21