Modeling Uncertainty For Middleware-based Streaming Power Grid Applications
2 likes407 views
The document describes modeling uncertainty in middleware-based streaming applications for power grids. It presents a discrete-event model built in Ptolemy II to capture uncertainty from sources like middleware latency, network delays, and number of sensor streams. Monte Carlo simulations are run over this model by varying parameters like middleware concurrency and sensor streams. Regression analysis is then used to understand the relationship between these influential parameters and the end-to-end application run time.
![40 60 80 100 120
40
60
80
mmModeling
Uncertainty
For
Middleware-
based
Streaming
Power Grid
Applications
Ilge Akkaya,
Yan Liu,
Edward A.
Lee, Ian
Gorton
Introduction
Modeling
Uncertainty
DE Modeling
Regression
Analysis
Conclusion
Heterogeneous Modeling in Ptolemy II
[Edward A. Lee et al., 2010]
UC Berkeley Introduction Ilge Akkaya 5 / 24](https://image.slidesharecdn.com/mw4ngpaper4-140509123359-phpapp01/85/Modeling-Uncertainty-For-Middleware-based-Streaming-Power-Grid-Applications-5-320.jpg)
![40 60 80 100 120
40
60
80
mmModeling
Uncertainty
For
Middleware-
based
Streaming
Power Grid
Applications
Ilge Akkaya,
Yan Liu,
Edward A.
Lee, Ian
Gorton
Introduction
Modeling
Uncertainty
DE Modeling
Regression
Analysis
Conclusion
Modeling Middleware Architecture
• Middleware modeled
as a thread pool
• Thread pool size is an
uncertainty parameter
• Stochastic delay per
thread processing
[Rician]
• Model trained using
Apache ActiveMQ
TM
benchmark results
UC Berkeley Discrete-Event Modeling in Ptolemy II for Uncertainty Analysis Ilge Akkaya 11 / 24](https://image.slidesharecdn.com/mw4ngpaper4-140509123359-phpapp01/85/Modeling-Uncertainty-For-Middleware-based-Streaming-Power-Grid-Applications-11-320.jpg)
![40 60 80 100 120
40
60
80
mmModeling
Uncertainty
For
Middleware-
based
Streaming
Power Grid
Applications
Ilge Akkaya,
Yan Liu,
Edward A.
Lee, Ian
Gorton
Introduction
Modeling
Uncertainty
DE Modeling
Regression
Analysis
Conclusion
Sample Simulation Traces: Distribution
sensor-to-concentrator
concentrator-to-MW
iteration runtime
0
5
10
15
20
25
30
35
40
45
50
55
60
65
70
75
80
85
0.0 10.0 20.0 122.5
Sample Latency Distribution
time(ms)
EventID
per packet MW delay
0
2
4
6
8
10
12
14
16
18
20
22
24
-0.2 -0.1 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 1.1 1.2 1.3
Middleware Processing Delay
time(ms)
EventID
Partial Latency Distri-
butions for one sam-
ple run of the DE
model. Captures the un-
certainty in HPC com-
putation [green], sen-
sor network [red], mid-
dleware network [blue]
and middleware process-
ing [black]
UC Berkeley Discrete-Event Modeling in Ptolemy II for Uncertainty Analysis Ilge Akkaya 15 / 24](https://image.slidesharecdn.com/mw4ngpaper4-140509123359-phpapp01/85/Modeling-Uncertainty-For-Middleware-based-Streaming-Power-Grid-Applications-15-320.jpg)
Modeling Uncertainty For Middleware-based Streaming Power Grid Applications
- 1. 40 60 80 100 120 40 60 80 mmModeling Uncertainty For Middleware- based Streaming Power Grid Applications Ilge Akkaya, Yan Liu, Edward A. Lee, Ian Gorton Introduction Modeling Uncertainty DE Modeling Regression Analysis Conclusion Modeling Uncertainty For Middleware-based Streaming Power Grid Applications Ilge Akkaya, Yan Liu, Edward A. Lee, Ian Gorton 8thMiddleware For Next Generation Internet Computing Workshop Beijing, China December 9, 2013 UC Berkeley Ilge Akkaya 1 / 24
- 2. 40 60 80 100 120 40 60 80 mmModeling Uncertainty For Middleware- based Streaming Power Grid Applications Ilge Akkaya, Yan Liu, Edward A. Lee, Ian Gorton Introduction Modeling Uncertainty DE Modeling Regression Analysis Conclusion 1 Introduction 2 Uncertainty Models and Parameter-Space Exploration 3 Discrete-Event Modeling in Ptolemy II for Uncertainty Analysis 4 Regression Analysis 5 Conclusion UC Berkeley Introduction Ilge Akkaya 2 / 24
- 3. 40 60 80 100 120 40 60 80 mmModeling Uncertainty For Middleware- based Streaming Power Grid Applications Ilge Akkaya, Yan Liu, Edward A. Lee, Ian Gorton Introduction Modeling Uncertainty DE Modeling Regression Analysis Conclusion Introduction • Cyber-Physical systems rely on the interaction of cyber and physical system components • Contemporary "Cyber" systems are about behavioral correctness and do not have temporal guarantees (e.g.: C code running on an embedded system, middleware) • Cyber systems used to regulate physical plants that may have tight latency requirements ( e.g.: smart grid) • Uncertainty is inevitable in complex system design: network latency, execution time, queuing delays, black-box middleware queues, etc. • Model-based characterization of uncertainty is useful for capturing possible worst-case scenarios UC Berkeley Introduction Ilge Akkaya 3 / 24
- 4. 40 60 80 100 120 40 60 80 mmModeling Uncertainty For Middleware- based Streaming Power Grid Applications Ilge Akkaya, Yan Liu, Edward A. Lee, Ian Gorton Introduction Modeling Uncertainty DE Modeling Regression Analysis Conclusion Introduction • Increasing number of high-throughput sensors (i.e. Phasor Measurement Units (PMUs)) being integrated into the power grid • Wide-area management and control applications need to satisfy • Accuracy • Responsiveness • Scalability requirements • Middleware provides coordination and alignment, at the expense of becoming the bottleneck Data Concentrator Data Concentrator ......... Data Concentrator MIDDLEWARE HPC Cluster UC Berkeley Introduction Ilge Akkaya 4 / 24
- 5. 40 60 80 100 120 40 60 80 mmModeling Uncertainty For Middleware- based Streaming Power Grid Applications Ilge Akkaya, Yan Liu, Edward A. Lee, Ian Gorton Introduction Modeling Uncertainty DE Modeling Regression Analysis Conclusion Heterogeneous Modeling in Ptolemy II [Edward A. Lee et al., 2010] UC Berkeley Introduction Ilge Akkaya 5 / 24
- 6. 40 60 80 100 120 40 60 80 mmModeling Uncertainty For Middleware- based Streaming Power Grid Applications Ilge Akkaya, Yan Liu, Edward A. Lee, Ian Gorton Introduction Modeling Uncertainty DE Modeling Regression Analysis Conclusion Overview: Application Model • An executable discrete-event Ptolemy model for a three-area distributed smart-grid application given above • PMU: Phasor Measurement Unit • PDC: Phasor Data Concentrator • Area: Balancing Authority running on a High Performance Computing (HPC) Cluster UC Berkeley Uncertainty Models and Parameter-Space Exploration Ilge Akkaya 6 / 24
- 7. 40 60 80 100 120 40 60 80 mmModeling Uncertainty For Middleware- based Streaming Power Grid Applications Ilge Akkaya, Yan Liu, Edward A. Lee, Ian Gorton Introduction Modeling Uncertainty DE Modeling Regression Analysis Conclusion Sources of Uncertainty in Communication Architectures • Middleware • Architecture generally does not scale well with increasing number of sensor nodes • Variable data aggregation latency • Distributed applications • Distributed State Estimation example: Computationally expensive, iterative algorithm • Number of iterations is a function of data quality • Network • Link capacity, length, queuing behavior UC Berkeley Uncertainty Models and Parameter-Space Exploration Ilge Akkaya 7 / 24
- 8. 40 60 80 100 120 40 60 80 mmModeling Uncertainty For Middleware- based Streaming Power Grid Applications Ilge Akkaya, Yan Liu, Edward A. Lee, Ian Gorton Introduction Modeling Uncertainty DE Modeling Regression Analysis Conclusion Capturing Uncertainty • Complexity and cost of real testbeds promote model-based performance evaluation of middleware • We perform uncertainty modeling and analysis of end-to-end distributed smart grid applications • Monte Carlo sampling over the parameter space to encapsulate uncertainties in • number of sensor streams • middleware capacity • application run time UC Berkeley Uncertainty Models and Parameter-Space Exploration Ilge Akkaya 8 / 24
- 9. 40 60 80 100 120 40 60 80 mmModeling Uncertainty For Middleware- based Streaming Power Grid Applications Ilge Akkaya, Yan Liu, Edward A. Lee, Ian Gorton Introduction Modeling Uncertainty DE Modeling Regression Analysis Conclusion Case Study: Distributed State Estimation • Distributed State Estimation (DSE) is an algorithm used for estimating power system state • Weighted Least-Squares based algorithm • Distributed version developed to meet tight timing deadlines • Typically run every minute ( deadline=60s), likely to run more frequently on the future power grid UC Berkeley Uncertainty Models and Parameter-Space Exploration Ilge Akkaya 9 / 24
- 10. 40 60 80 100 120 40 60 80 mmModeling Uncertainty For Middleware- based Streaming Power Grid Applications Ilge Akkaya, Yan Liu, Edward A. Lee, Ian Gorton Introduction Modeling Uncertainty DE Modeling Regression Analysis Conclusion Design Workflow for Uncertainty Analysis • Parametrization of uncertainty sources in the model • Monte Carlo sampling of the parameter space • Execution of the parameterized model • Regression analysis • Design refinement of influential parameters UC Berkeley Uncertainty Models and Parameter-Space Exploration Ilge Akkaya 10 / 24
- 11. 40 60 80 100 120 40 60 80 mmModeling Uncertainty For Middleware- based Streaming Power Grid Applications Ilge Akkaya, Yan Liu, Edward A. Lee, Ian Gorton Introduction Modeling Uncertainty DE Modeling Regression Analysis Conclusion Modeling Middleware Architecture • Middleware modeled as a thread pool • Thread pool size is an uncertainty parameter • Stochastic delay per thread processing [Rician] • Model trained using Apache ActiveMQ TM benchmark results UC Berkeley Discrete-Event Modeling in Ptolemy II for Uncertainty Analysis Ilge Akkaya 11 / 24
- 12. 40 60 80 100 120 40 60 80 mmModeling Uncertainty For Middleware- based Streaming Power Grid Applications Ilge Akkaya, Yan Liu, Edward A. Lee, Ian Gorton Introduction Modeling Uncertainty DE Modeling Regression Analysis Conclusion Distribution Fitting Ti ∼ Rice(ν(NPMUi ), σ(NPMUi )) ν(NPMUi ) = 0.0302 log(NPMUi ) + 0.055 σ(NPMUi ) = 0.0007 ∗ NPMUi + 0.0414 where NPMUi is the number of PMU streams at i’th Area real estimate 0 2 4 6 8 10 12 14 16 18 20 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Middleware Per File Delay Rician Distribution Fitting time(s) frequency UC Berkeley Discrete-Event Modeling in Ptolemy II for Uncertainty Analysis Ilge Akkaya 12 / 24
- 13. 40 60 80 100 120 40 60 80 mmModeling Uncertainty For Middleware- based Streaming Power Grid Applications Ilge Akkaya, Yan Liu, Edward A. Lee, Ian Gorton Introduction Modeling Uncertainty DE Modeling Regression Analysis Conclusion Monte Carlo Simulation in Ptolemy II Top-level data flow model generates random parameter values Inner Discrete-Event model is exe- cuted with these parameters for 100 complete iterations ( 6000 s) UC Berkeley Discrete-Event Modeling in Ptolemy II for Uncertainty Analysis Ilge Akkaya 13 / 24
- 14. 40 60 80 100 120 40 60 80 mmModeling Uncertainty For Middleware- based Streaming Power Grid Applications Ilge Akkaya, Yan Liu, Edward A. Lee, Ian Gorton Introduction Modeling Uncertainty DE Modeling Regression Analysis Conclusion Sample Simulation Traces: Time PMU-PDC PDC-MW MW-HPC p2p_in END 0.0 0.5 1.0 1.5 2.0 2.5 x102 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 Area I time(s) events PMU-PDC PDC-MW MW-HPC p2p_in END 0.0 0.5 1.0 1.5 2.0 2.5 x102 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 Area II time(s) events PMU-PDC PDC-MW MW-HPC p2p_in END 0.0 0.5 1.0 1.5 2.0 2.5 x10 2 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 Area III time(s) events UC Berkeley Discrete-Event Modeling in Ptolemy II for Uncertainty Analysis Ilge Akkaya 14 / 24
- 15. 40 60 80 100 120 40 60 80 mmModeling Uncertainty For Middleware- based Streaming Power Grid Applications Ilge Akkaya, Yan Liu, Edward A. Lee, Ian Gorton Introduction Modeling Uncertainty DE Modeling Regression Analysis Conclusion Sample Simulation Traces: Distribution sensor-to-concentrator concentrator-to-MW iteration runtime 0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 0.0 10.0 20.0 122.5 Sample Latency Distribution time(ms) EventID per packet MW delay 0 2 4 6 8 10 12 14 16 18 20 22 24 -0.2 -0.1 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 1.1 1.2 1.3 Middleware Processing Delay time(ms) EventID Partial Latency Distri- butions for one sam- ple run of the DE model. Captures the un- certainty in HPC com- putation [green], sen- sor network [red], mid- dleware network [blue] and middleware process- ing [black] UC Berkeley Discrete-Event Modeling in Ptolemy II for Uncertainty Analysis Ilge Akkaya 15 / 24
- 16. 40 60 80 100 120 40 60 80 mmModeling Uncertainty For Middleware- based Streaming Power Grid Applications Ilge Akkaya, Yan Liu, Edward A. Lee, Ian Gorton Introduction Modeling Uncertainty DE Modeling Regression Analysis Conclusion Monte Carlo Parameter Space Table : Monte Carlo Variables and Respective Probability Mass Functions (range format:initial:increment:final ) Variable Parameter Name PMF Range PMU_Count_1 Uniform 10:10:500 PMU_Count_2 Uniform 10:10:500 PMU_Count_3 Uniform 10:10:500 x1 concurrencyLevel Uniform 2:2:20 x3 numberOfIterations Uniform 1:1:20 x2 max{PMU_Count_1,PMU_Count_2,PMU_Count_3} y maximum end-to-end run time per parameter sample UC Berkeley Discrete-Event Modeling in Ptolemy II for Uncertainty Analysis Ilge Akkaya 16 / 24
- 17. 40 60 80 100 120 40 60 80 mmModeling Uncertainty For Middleware- based Streaming Power Grid Applications Ilge Akkaya, Yan Liu, Edward A. Lee, Ian Gorton Introduction Modeling Uncertainty DE Modeling Regression Analysis Conclusion Exploratory Data Analysis: Middleware Concurrency End-to-end deadline: 60s For maximum number of sensor streams > 280, deadline misses observed at some runs with lower middleware concurrency UC Berkeley Discrete-Event Modeling in Ptolemy II for Uncertainty Analysis Ilge Akkaya 17 / 24
- 18. 40 60 80 100 120 40 60 80 mmModeling Uncertainty For Middleware- based Streaming Power Grid Applications Ilge Akkaya, Yan Liu, Edward A. Lee, Ian Gorton Introduction Modeling Uncertainty DE Modeling Regression Analysis Conclusion Exploratory Data Analysis: Number of DSE Iterations 100 150 200 250 300 350 400 450 Maximum Number of PMUs per area Monte−Carlo Simulation Results For Mean DSE Runtime NumIterations= 1 NumIterations= 2 NumIterations= 3 NumIterations= 4 NumIterations= 5 NumIterations= 6 NumIterations= 7 NumIterations= 8 NumIterations= 9 NumIterations= 10 NumIterations= 11 NumIterations= 12 NumIterations= 13 NumIterations= 14 NumIterations= 15 NumIterations= 16 NumIterations= 17 NumIterations= 18 NumIterations= 19 NumIterations= 20 No observable correlation between number of iterations and run time UC Berkeley Discrete-Event Modeling in Ptolemy II for Uncertainty Analysis Ilge Akkaya 18 / 24
- 19. 40 60 80 100 120 40 60 80 mmModeling Uncertainty For Middleware- based Streaming Power Grid Applications Ilge Akkaya, Yan Liu, Edward A. Lee, Ian Gorton Introduction Modeling Uncertainty DE Modeling Regression Analysis Conclusion Relation Between PMU Count / Concurrency and Maximum Run Time UC Berkeley Regression Analysis Ilge Akkaya 19 / 24
- 20. 40 60 80 100 120 40 60 80 mmModeling Uncertainty For Middleware- based Streaming Power Grid Applications Ilge Akkaya, Yan Liu, Edward A. Lee, Ian Gorton Introduction Modeling Uncertainty DE Modeling Regression Analysis Conclusion Regression Analysis • Goal: to discover the impact of the number of algorithm iterations, middleware capacity and number of sensor streams on the aggregate end-to-end run time • Polynomial regression with independent variables x1, x2, x3 and dependent variable y Max PMU Count Concurrency Level MaxRuntime(s) Figure : Polynomial regression fit with 5% confidence intervals UC Berkeley Regression Analysis Ilge Akkaya 20 / 24
- 21. 40 60 80 100 120 40 60 80 mmModeling Uncertainty For Middleware- based Streaming Power Grid Applications Ilge Akkaya, Yan Liu, Edward A. Lee, Ian Gorton Introduction Modeling Uncertainty DE Modeling Regression Analysis Conclusion Regression Analysis: Results • Number of iterations (x3) was found to have little effect on the dependent variable • Coefficient estimates for x3 in regression tests were found to be unreliable, confidence intervals centered around zero • Data visualization supports regression results: iteration run time quite insignificant; middleware processing time dominates 0 100 200 300 400 500 0 5 10 15 20 0 10 20 30 40 50 60 Number of PMUs Number of Iterations vs Total #PMUs vs Runtime Number of Iterations MaxEnd−to−endRuntime(s) UC Berkeley Regression Analysis Ilge Akkaya 21 / 24
- 22. 40 60 80 100 120 40 60 80 mmModeling Uncertainty For Middleware- based Streaming Power Grid Applications Ilge Akkaya, Yan Liu, Edward A. Lee, Ian Gorton Introduction Modeling Uncertainty DE Modeling Regression Analysis Conclusion Findings • Concurrency level or the deployment architecture of the middleware should be improved to scale • Number of DSE iterations is a minor concern ( some bad data is tolerable at the expense of more iterations) • Model-based Parameter Space Exploration provides pre-deployment results for typical middleware requirements. Future directions include • Replacing network fabric/ middleware and re-simulation for architectural comparisons • Considering more sources of uncertainty • Correct-by-construction uncertainty in systems - no surprises in the end product UC Berkeley Conclusion Ilge Akkaya 22 / 24
- 23. 40 60 80 100 120 40 60 80 mmModeling Uncertainty For Middleware- based Streaming Power Grid Applications Ilge Akkaya, Yan Liu, Edward A. Lee, Ian Gorton Introduction Modeling Uncertainty DE Modeling Regression Analysis Conclusion Conclusion • Distributed power applications are data intensive and algorithm run times/network latency variance is high among different deployments • Pre-deployment analysis on architecture requirements for a variable number of sensor streams need to be tested under varying middleware capacity and latency conditions • Presented a model-based approach for quantifying uncertainty in middleware based applications • Monte Carlo sampled parameter space used in populating an executable heterogeneous model • Discrete-Event simulation carried out to collect end-to-end run time measurements • Regression analysis carried out to account for significant parameters that have the largest influence on limiting behavior UC Berkeley Conclusion Ilge Akkaya 23 / 24
- 24. 40 60 80 100 120 40 60 80 mmModeling Uncertainty For Middleware- based Streaming Power Grid Applications Ilge Akkaya, Yan Liu, Edward A. Lee, Ian Gorton Introduction Modeling Uncertainty DE Modeling Regression Analysis Conclusion Thank You Questions ? UC Berkeley Conclusion Ilge Akkaya 24 / 24