![[Slide courtesy of Randy Shoup, eBay Distinguished Architect] 2
ULS are Everywhere, and they’re HUGE!](https://image.slidesharecdn.com/mesoca2011-160728172418/85/Modeling-the-Performance-of-Ultra-Large-Scale-Systems-Using-Layered-Simulations-2-320.jpg)
Modeling the Performance of Ultra-Large-Scale Systems Using Layered Simulations
- 1. Modeling the Performance of Ultra-Large-Scale Systems 1 Using Layered Simulations
- 2. [Slide courtesy of Randy Shoup, eBay Distinguished Architect] 2 ULS are Everywhere, and they’re HUGE!
- 3. 1. Performance Testing should happen … 4 development process here? here!
- 4. 2. One Model to Rule Them All? 1. How do response time differ between these two mechanism? 5
- 5. 3. Different Stakeholders, Different Interests End User Developer System Engineer System Integrator Sales Rep 6
- 6. An Analogy: The “4+1” View Model Scenario Logical View Process View Development View Physical View 8
- 7. The “3+1” View Model for Performance Modeling Scenario World View Physical View Component View 9
- 8. Component Layer A Layered Simulation Model! 10 World View Layer Physical Layer
- 9. Case Study 1 Bottleneck Detection in RSS Cloud • RSS blogging service providers (e.g., Wordpress) • Evaluation of two notification delivery mechanisms: – Periodic Pull – Real-time Push Q1. How does response time differ? Q2. How does resource usage (e.g., CPU, memory) look like? 11
- 10. Component Layer The RSS Cloud Model 12 World View Layer Physical Layer
- 11. CPU is the Bottleneck! 13
- 12. Case Study 2 Evaluating a Distributed Monitor 1. How do response time differ between these two mechanism? 14
- 14. Distributed vs. Centralized 17 0.1Hz 0.2Hz 0.3Hz 0.1Hz 0.2Hz 0.3Hz faster “cheaper” lower resource usage
- 15. Keeping the Model Up-to-Date • (What-if) Experimenting with different configuration/deployment scenarios • (Digging deeper) Adding new “layers”, as system is understood better • (Design evolution) Updating the simulation models as the system evolves 18
- 16. GH'@&+/) +A 12(&'I &-829'-3) *#%'3144&2'J ' 6! 1) +) 789: ) 02! 9(8; ) <<! 3&+&K' 3&+&F' LH'7$M&+&2,'",1N&3) #%&+-5'7$M&+&2,'O2,&+&-,-' ?2%'= -&+' 7&B&#) 4&+' ". -,&A '' ?29$2&&+' ". -,&A '' O2,&9+1,) +' "1#&-'' 0&4' =! Component Layer' : 'Z1. &+&%'"$A *#18) 2'P ) %&#F' "I ! World View Layer' Physical Layer' 7$-,+$S*,&%'B-H'&2,+1#$d&%! "F! eHG<d' eHL<d' eHQ<d' TABLE IX SIMULATION RESULTS FOR THE ORIGINAL DESIGN OF THE PERFORMANCE MONITOR. DataCollection Response time(s) Cost per Central Monitor Central Monitor Central Monitor Frequency (Hz) transmission ($) Thread Util. (%) CPU Util. (%) RAM Util. (%) 0.1 0.2 2.2 31.0 42.6 37.2 0.2 0.3 2.7 43.7 68.2 47.6 0.3 0.4 3.1 60.2 86.6 59.2 the CPU of the central monitor is close to running at its full capacity, which will likely result in system instability. More- over, whiletheoriginal designhaslower cost per transmission, be validated as information becomes available throug development. Due to the lack of access to production we could only estimate a list of resources and proce eHG<d' eHL<d' eHQ<d' /1-,&+! U(3&14&+X! #) D&+'+&-) *+(&'*-19&! QUESTIONS!
- 17. 20
- 18. Backup Slides 21
- 19. FOUR CHALLENGES The Current Practice of Performance Verification 22
- 20. 1. Too Late in the Development Lifecycle • Design changes are not evaluated until after code is written – Happens at the last stage of a delayed schedule 23 Requirements Design Implementation Verification Maintenance
- 21. 2. Lots of Data • Industrial case studies have 2000> counters • Time consuming to analyze • Hard to compare more than 2 tests at once 24
- 22. 3. No Documented Behavior 25 • Analysts have different perceptions of performance regressions • Analysis may be influenced by – Analyst’s knowledge – Deadline
- 23. 4. Heterogeneous Environments • Multiple labs to parallelize test executions – Hardware and software may differ – Tests from one lab may not be used to analyze tests from another lab 26
- 25. AT THE DESIGN LEVEL Performance Verification 28
- 26. Evaluate Design Changes through Performance Modeling • Analytical models are often not suitable for all stakeholders – Abstract mathematical and statistical concepts • Simulation models can be implemented with support of existing framework – Visualization – No systematic approach to construct models that can be used by different stakeholders 29
- 27. Layered Simulation Model 30 Physical layerComponent layer World view layer Can the current infrastructure support the projected growth of users? Investigate threading model Hardware resource utilization
- 28. Case Studies • We conducted two case studies – RSS Cloud • Show the process of constructed the model • Derive the bottleneck of the application – Performance monitor for ULS systems • Evaluate whether or not an organization should re- architect the software • Our model can be used to extract important information and aid in decision making 31
Editor's Notes
- #3: Slides from eBay to give another taste of scale
- #7: Spend a bit of time about performance concerns. E.g. End user: overall system performance for various configurations Developer: Organization and performance of system modules System Engineer: Hardware resource util, hardware capacity planning System Integrator: Performance after integration of different components Sales Representative: Reliability, costs
- #8: Analytical Modeling: use math equations and statistical concepts to model the performance of a software system. This requires substantial level of expertise. Simulation models (Discrete-event simulation) can created to answer specific performance questions from different stakeholders.
- #10: Explain a bit on what the models are and who those stake holders are
- #11: Discuss the contents in Table III
- #13: Discuss the contents in Table III