Performance Issue? Machine Learning to the rescue!
Download as PPTX, PDF1 like339 views
The document discusses the application of machine learning in performance tuning for software applications, highlighting the importance of measuring various performance indicators like response times and throughput. Traditional tuning methods rely on trial and error, while machine learning can provide insights into which variables significantly impact performance, thereby optimizing the tuning process. It also explores the concept of random forest regression for model creation, offering a method to accurately predict performance outcomes based on different input variables.
![Results
JVM major version
14
Averageresponsetime[ms]](https://image.slidesharecdn.com/performanceissue-machinelearningtotherescue-191213152238/85/Performance-Issue-Machine-Learning-to-the-rescue-14-320.jpg)
![Results
Framework and JVM
15
Averageresponsetime[ms]](https://image.slidesharecdn.com/performanceissue-machinelearningtotherescue-191213152238/85/Performance-Issue-Machine-Learning-to-the-rescue-15-320.jpg)
![Results
Garbage Collection Algorithm
16
Averageresponsetime[ms]](https://image.slidesharecdn.com/performanceissue-machinelearningtotherescue-191213152238/85/Performance-Issue-Machine-Learning-to-the-rescue-16-320.jpg)
Performance Issue? Machine Learning to the rescue!
- 1. Performance Issue? Machine Learning to the Rescue!
- 2. Agenda • Introduction • Measuring performance • What is performance? • How is traditional tuning done? • Now for something completely different! • Analyzing the data • Introduction to terminology • Introduction to the Random Forest • Results • Checking the results • Verifying the model 2
- 3. About me 3 @MaartenSmeetsNL https://nl.linkedin.com/in/smeetsm • Software architect at AMIS / Conclusion • Several certifications SOA, BPM, MCS, Java, SQL, PL/SQL, Mule, AWS, etc • Enthusiastic blogger http://javaoraclesoa.blogspot.com • Frequent presenter at conferences
- 4. What is performance? • Better performance indicates more work can be done using less resources or time • What is work? • Starting an application • Time required to process a single request (response times) • Number of requests which can be processed during a certain period (throughput) • Which resources? • CPU / cores • Memory • Disk • What uses resources to do work? • Application • Application platform
- 5. Traditional performance tuning. Trial and error! • Use informed judgement to manipulate a specific variable • Compare it to a baseline • If it’s good, keep it, else revert • Try another variable • Continue until the performance is sufficient 5 Wouldn’t it help if you could more accurately determine the variable to manipulate before starting a trial and error process? Add a bit of Machine Learning
- 6. What do you need to let the Machine augment ‘informed judgement’ • Lots of performance data! • Different kinds of ‘work’ • response times • throughput • Different kinds of ‘resources’ for load generator and application • Cores • Memory • Variations in the application and platform • JVM • GC algorithm • JVM version • Framework used • A model to process the data • Understands the structure of the data • Is suitable for my data • Can tell me which variables are important 6
- 7. What did I measure? 7 What did I manipulate? Throughput Requests per second Response time Framework JVM (8,11,12,13) Cores Memory Concurrency Garbage Collection
- 8. How does the data look? 8
- 9. How did I measure? 9 Create minimal but comparable implementations for 10 frameworks Java, Kotlin Create a script to loop over scenarios Bash, Python Create and evaluate load generators Python, Node/JavaScript Containerize the implementations Docker Summarize results Bash (awk, sed, curl), Python, Apache Bench Run the setup under various conditions First priming followed by test (weeks of data) Visualize results Python / Jupyter: pandas, numpy, pyplot
- 10. Challenges measuring 10 User processes Load generators JVM process 1 2 3
- 11. Challenges generating load • No more containers Effects of containers are slower startup and Docker NAT overhead No more need to create and clean up containers -> simpler code • Apache Bench for testing No more split individual measures and summary data Proven simple to use tool (also used to test Apache HTTPD) • Python for running the tests / manage the processes Easier than Bash for complex testing • Jupyter for visualization instead of plain Python Displaying graphs and developing the script in small bits is easier Titel van de presentatie 11
- 12. And now for some results! 12
- 13. Disclaimer The performance tests mentioned in this presentation were conducted with intention to obtain information on which variables have most effect for performance. A best effort has been made to conduct an unbiased test. Still, performance depends on many parameters such as hardware, specifics of the implementation, the usage of different back-ends, concurrency, versions of libraries, OS, virtualization and various other factors. Also performance can be measured in many different ways. I cannot provide any guarantees that the same variables are important in other situations or that my selection of variables to look at is suitable for other environments. The use of these results is at your own risk. I shall in no case accept liability for any loss resulting from the use of the results or decisions based on them. 13 Let yourself be inspired and perform your own tests!
- 17. Results response time How do frameworks deal with more CPUs? 17
- 18. Results throughput How do frameworks deal with more CPUs? 18
- 19. Some general observations • Docker has overhead • Network (Docker NAT). Even between containers • Startup times (due to loading of shared libraries and inability to use certain caching features). • GraalVMs native compilation is worthwhile, especially when you pay for resources • Reduced container size, memory usage, startup time • Increased response times and throughput • Certain frameworks do quite well: Quarkus, Vert.x and Helidon SE • Good response times and throughput • And allow native compilation • JVMs • OpenJ9 has the best startup time outside a container and works best at low memory • Oracle JDK and OpenJDK do not differ much 19
- 20. Lots of data But what do I want to know? 20
- 21. What are the most important variables determining my performance? And which are the variables I can (more or less) safely ignore 21
- 22. Can Machine Learning solve this for me? 22
- 24. Machine Learning challenges • Prediction error • Bias • Variance • Overfitting and underfitting • Selection and tuning of a model 24
- 25. Total error Reducible error Prediction error 25 Bias error Variance error Irreducible error Bayes’ error rate Optimum Error rate
- 26. Bias and variance • Bias error • How far is the predicted value from the true value • The systematic error of the model • It is about the model and the data itself • Variance error • The error caused by sensitivity to small variances in the training data set • The dispersion of predicted values over target values with different train-sets • It is about the model sensitivity 26 https://community.alteryx.com/t5/Data-Science-Blog/Bias-Versus-Variance/ba-p/351862
- 27. Overfitting and underfitting • Overfitting occurs when The model captures the noise and the outliers in the data along with the underlying pattern. These models usually have high variance and low bias • Under fitting occurs when The model is unable to capture the underlying pattern of the data These models usually have a low variance and a high bias. 27 https://community.alteryx.com/t5/Data-Science-Blog/Bias-Versus-Variance/ba-p/351862
- 28. The bias variance curse a property of supervised machine learning models • As the complexity of model increases the bias decreases but the variance increases • There is a trade-off between bias and variance You can't get both low bias and low variance at the same time 28
- 29. Model selection • Is it suitable for my type of data? • Does it produce an accurate model? • How does it do bias / variance wise? • Is it sophisticated enough to capture patterns in my data without overfitting? • Is it able to allow determination of feature importances? • Is it easy to use? • I ended up with selecting Random Forest Regression I also tried Support Vector Regression (SVR) (linear kernel to obtain a measure of feature importance, also scaled the data) however was unable to convince it to do accurate predictions 29
- 30. Why a Random Forest? • Accurate predictions • Standard decision trees often have high variance and low bias High chance of overfitting (with ‘deep trees’, many nodes) • With a Random Forest, the bias remains low and the variance is reduced thus we decrease the chances of overfitting • Flexible • Can be used with many variables • Can be used for classification but also for regression • Easy • Feature Importance! 30 200 trees Create a model to predict ytrain based on Xtrain Use the model to do predictions based on Xtest
- 31. Introduction Random Forest • Can be used for classification and regression • A Random Forest can deal with many different features • A Random Forest is a collection of Decision Trees of which results are averaged or majority voted 31
- 32. How is a Random Forest created? • A random forest consists of decision trees. A decision tree consists of • decision nodes the top decision node is called the root node • terminal nodes or leaf nodes • A selection of data and features is used for each tree For every decision tree • a sample of the training data is used • a sample of the features (√nfeatures up to 30 – 40%) is used 32 https://victorzhou.com/blog/intro-to-random-forests/
- 33. For an individual Decision Tree • Find the best split in the data. This is the root node (decision node) • Find in the first branch for that part of the data again the best split. That is the first sub node (decision node) • Continue creating decision nodes until splitting doesn’t improve the situation The average value of the target variable is assigned to the leaf (terminal node) • Continue until there are only leaf nodes left or until a minimum value is reached • Now the decision tree can be used to do a prediction based on the input features 33
- 34. Determine the best split • Determine the mean of all datapoints • Determine the mean square error of all datapoints • Apply a split. Try at every datapoint • Calculate the mean of all datapoints on each side of the split • Calculate the MSE on each side and take a weighted average of MSEs on all sides • Lowest (weighted averaged) MSE is best split 34
- 35. Processing the ensemble of trees called The Random Forest • Take a set of variables • Run them through every decision tree • Determine a predicted target variable for each of the trees • Average the result of all trees 35
- 36. How to evaluate the model? • Split the data in a train set and test set • Train the model using the trainset. • Test predictions using the test set. • Vary the train and test set contents to confirm results • Compare the test set and the train set • Determine the Coefficient of Determination for both sets The proportion of the variance in the dependent variable that is predictable from the independent variable(s) How well are observed outcomes replicated by the model • If R2 is different for both sets, the test or train set is probably biased • Determine the accuracy of the predictions by comparing predicted results of the test set with the actual results of the test set 36
- 37. Back to the data 37
- 38. Model accuracy 38 Coefficient of determination R^2 = 0.99 for both graphs and for train and test sets the proportion of the variance in the measurements that is predictable by the different features using the model Akka on OpenJ9 Not yet been able to determine distinguishing features
- 39. A Random Forest has a cool feature You can determine the relative importance of features in the Decision Trees 39
- 40. Feature Importance • Feature importance is calculated as • the reduction in sum of squared errors whenever a variable is chosen to split • weighted by the probability of reaching that node. • The node probability can be calculated by the number of samples that reach the node, divided by the total number of samples. • The higher the value the more important the feature. • However • the variable importance measures are not reliable in situations where potential predictor variables vary in their scale of measurement or their number of categories 40
- 43. Feature Importance • However • the variable importance measures are not reliable in situations where potential predictor variables vary in their scale of measurement or their number of categories • Permutation Feature Importance does not have these liabilities The decrease in a model score when a single feature value is randomly shuffled. This breaks the relationship between the feature and the target. The drop in the model score is indicative of how much the model depends on the feature. 43
- 44. Permutation Feature Importance • Train the baseline model and record the accuracy • Shuffle values from one feature in the selected dataset (keep the same variance) • Pass the dataset to the model again to obtain predictions and calculate the accuracy. The feature importance is the difference between the benchmark score and the one from the modified (permuted) dataset. • Repeat for all features in the dataset • The resulting score is a measure of how important the feature is. If the accuracy is greatly reduced, the feature is important If the accuracy does not change much by randomizing a feature, the feature is not important 44
- 45. Permutation Feature Importance 45 Throughput Response time Luckily very similar results
- 46. How do you use this information? When you’re having a performance issue • Look at your application and the frameworks and logic used Quarkus, Helidon SE, Vert.x are good • Next try different JVMs • Oracle JDK and OpenJDK are good • SubstrateVM (GraalVM native) might be an option for fast startup less memory • Don’t pay much attention to • Memory • CPU • GC algorithm • JVM version 46 Of course within the scope of the tested variables / code Can this model generalize outside of the tested variables?
- 47. Looking forward • I looked at minimal implementations No database or variations in DB drivers No disk No network No inter or intra framework communication • I only tested at relatively low concurrency Apache Bench is thread based. Many threads cause high CPU and memory • GraalVMs deserves more attention It is not available for every framework Is only recently available for Java 11 47
- 48. Finally • The model has been created based on the supplied variables and data • I had to make choices and scope the test due to time limitations. • Many more variables can be relevant! • How accurate are predictions outside the scope of measurements? • Machine learning can help • Find patterns and generalize data (create a model) • Do predictions • Find out what is important in your data • Provide valuable insights • Performance • is a great use case for machine learning (e.g. data with patterns, optimization challenges) • a fresh look by a machine can help us be more efficient in our tuning efforts 48
- 49. 49
Editor's Notes
- #15: Points differ in Java framework used Assigned memory CPUs assigned for service and test GC algorithm JVM supplier (Zing, OpenJDK, Oracle, OpenJ9)