Technical Deep Dive Into Atlassian's New Apps Performance Testing Framework
2 likes9,003 views
The document details Atlassian's new performance testing framework for Data Center apps, aiming to provide an enterprise-grade ecosystem and facilitate vendor differentiation of applications. It introduces a beta Data Center App Performance Toolkit, designed to simplify testing processes, offering realistic datasets and extensive customization options. Performance testing methodologies and tools, such as JMeter and Selenium, are discussed to support app developers in measuring performance and scalability effectively.
![Upload
Dataset
Restore DB
Upload
attachments
Perform Re-Index
--2019-08-26 05:22:25-- https://centaurus-
datasets.s3.amazonaws.com/jira/8.0.3/large/db.dump
Resolving centaurus-datasets.s3.amazonaws.com (centaurus-
datasets.s3.amazonaws.com)... 52.219.88.116
Connecting to centaurus-datasets.s3.amazonaws.com (centaurus-
datasets.s3.amazonaws.com)|52.219.88.116|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 7340090453 (6.8G) [application/x-www-form-urlencoded]
Saving to: ‘db.dump’
31%
[================================================================>](https://image.slidesharecdn.com/1530-1610platformolekben-190916235931/85/Technical-Deep-Dive-Into-Atlassian-s-New-Apps-Performance-Testing-Framework-93-320.jpg)
![Upload
Dataset
Restore DB
Upload
attachments
Perform Re-Index
[300000/2000000 entries] [1.7 G/11.1 G transferred]
[2289 entries/s] [13.1 M/s bw] [monq 0] [jq 469]
[600000/2000000 entries] [3.3 G/11.1 G transferred]
[1929 entries/s] [11.0 M/s bw] [monq 0] [jq 369]
[900000/2000000 entries] [5.0 G/11.1 G transferred]
[666 entries/s] [3.8 M/s bw] [monq 0] [jq 269]
[1059000/2000000 entries] [5.9 G/11.1 G transferred]
[457 entries/s] [2.6 M/s bw] [monq 0] [jq 216]
[2000000/2000000 entries] [11.1 G/11.1 G transferred]
[385 entries/s] [2.2 M/s bw] [monq 1] [jq 1]](https://image.slidesharecdn.com/1530-1610platformolekben-190916235931/85/Technical-Deep-Dive-Into-Atlassian-s-New-Apps-Performance-Testing-Framework-96-320.jpg)
Technical Deep Dive Into Atlassian's New Apps Performance Testing Framework
- 1. Technical Deep Dive Into Atlassian's New Apps Performance Testing Framework BEN MAGRO | PRODUCT MANAGER | ATLASSIAN | @MAGRO OLEKSANDR METELYTSIA | TEAM LEAD | ATLASSIAN
- 4. BACKGROUND
- 6. FPO
- 9. 1. DELIVER AN ENTERPRISE GRADE ECOSYSTEM FOR OUR CUSTOMERS MISSION CRITICAL DEPLOYMENTS THE PROGRAM’S GOALS
- 10. 1. DELIVER AN ENTERPRISE GRADE ECOSYSTEM FOR OUR CUSTOMERS MISSION CRITICAL DEPLOYMENTS 2. PROVIDE A WAY FOR VENDORS TO DIFFERENTIATE THEIR DC APPS THE PROGRAM’S GOALS
- 11. Enterprise Grade
- 13. Enterprise Grade 1. Architectural Review 2. Performance Testing
- 14. Enterprise Grade 1. Architectural Review 2. Performance Testing Approval
- 15. 620+Data Center Approved Apps
- 16. 93%of the tops apps are approved
- 17. DC APPS LAUNCH SEPTEMBER 2018
- 18. VENDOR FEEDBACK ATLASSIAN CODE BARREL INNVALOG BRICKIT JQL TRICKS BITVOODOO BEECOM ADAPTAVIST SMARTBEAR SNAPBYTES APPFIRE SERVICEROCKET
- 21. VENDOR INTERVIEWS 15PERFORMANCE TESTING TOOK WEEKS CREATING DATASETS
- 22. VENDOR INTERVIEWS 15PERFORMANCE TESTING TOOK WEEKS CREATING DATASETS LIMITED GUIDANCE
- 24. Existing Tools PERFORMANCE TOOLS • Specific Use Case
- 25. Existing Tools PERFORMANCE TOOLS • Specific Use Case • Missing Data Sets
- 26. Existing Tools PERFORMANCE TOOLS • Specific Use Case • Missing Data Sets • Not Cross Product
- 27. Existing Tools PERFORMANCE TOOLS • Specific Use Case • Missing Data Sets • Not Cross Product • Tightly Coupled to Cloud Provider
- 28. Existing Tools PERFORMANCE TOOLS • Specific Use Case • Missing Data Sets • Not Cross Product • Tightly Coupled to Cloud Provider • Time Consuming
- 29. Existing Tools PERFORMANCE TOOLS • Specific Use Case • Missing Data Sets • Not Cross Product • Tightly Coupled to Cloud Provider • Time Consuming • Inconsistant Output
- 30. BETA
- 31. INTRODUCING BETA DATA CENTER APP PERFORMANCE TOOLKIT
- 32. INTRODUCING BETA DATA CENTER APP PERFORMANCE TOOLKIT
- 33. BETA 8.0.X
- 35. GOALS OF THE PERFORMANCE TOOLKIT
- 36. Easy Quick to setup and run. Detailed walkthroughs. GOALS OF THE PERFORMANCE TOOLKIT
- 37. Easy Quick to setup and run. Detailed walkthroughs. Ship with data Ship with realistic enterprise datasets GOALS OF THE PERFORMANCE TOOLKIT
- 38. Easy Quick to setup and run. Detailed walkthroughs. Ship with data Ship with realistic enterprise datasets Decoupled Not tied to deployment type or dataset GOALS OF THE PERFORMANCE TOOLKIT
- 39. Easy Quick to setup and run. Detailed walkthroughs. Ship with data Ship with realistic enterprise datasets Decoupled Not tied to deployment type or dataset Extensible Built to be customised to test your apps GOALS OF THE PERFORMANCE TOOLKIT
- 40. OUR ASK WHAT IT’S NOT
- 41. Carved In StoneSeeking Feedback OUR ASK WHAT IT’S NOT
- 42. Carved In Stone Mandatory Seeking Feedback Recommended OUR ASK WHAT IT’S NOT
- 43. Carved In Stone Mandatory Built only for Atlassian Apps Seeking Feedback Recommended Try to use it for your apps OUR ASK WHAT IT’S NOT
- 44. Carved In Stone Mandatory Built only for Atlassian Apps Abandonware Seeking Feedback Recommended Try to use it for your apps Raise Improvements OUR ASK WHAT IT’S NOT
- 46. BETA
- 48. OUR APPROACH
- 51. Hardware Provisioned hardware, number of application nodes, db power and instance configuration
- 52. Hardware Provisioned hardware, number of application nodes, db power and instance configuration Configuration Performance load profiles, datasets, Product configuration
- 53. Hardware Provisioned hardware, number of application nodes, db power and instance configuration Configuration Performance load profiles, datasets, Product configuration Software Performance tools, load generation scripts, product and tools versions
- 54. Support App Developers By providing tooling and guidance on how to measure the performance and scalability of apps developed for Atlassian’s DC products.
- 55. Start: fix moving parts
- 56. Start: fix moving parts Jira versions Jira Software 8.0.x and ER 7.13.x
- 57. Start: fix moving parts Jira versions Jira Software 8.0.x and ER 7.13.x Deployment AWS Quickstart with defined instance types and parameters
- 58. Start: fix moving parts Datasets Realistic enterprise-scale dataset Jira versions Jira Software 8.0.x and ER 7.13.x Deployment AWS Quickstart with defined instance types and parameters
- 59. Start: fix moving parts Datasets Realistic enterprise-scale dataset Scenarios Performance regression and scalability Jira versions Jira Software 8.0.x and ER 7.13.x Deployment AWS Quickstart with defined instance types and parameters
- 60. OUR GOALS THINGS TO AVOID
- 61. Reinventing the wheelUtilise existing tools/learnings OUR GOALS THINGS TO AVOID
- 62. Reinventing the wheel Custom wrapper Utilise existing tools/learnings JMeter for REST OUR GOALS THINGS TO AVOID
- 63. Reinventing the wheel Custom wrapper Ignore UI Utilise existing tools/learnings JMeter for REST Selenium for UI measures OUR GOALS THINGS TO AVOID
- 64. Reinventing the wheel Custom wrapper Ignore UI Hardcode/magic numbers Utilise existing tools/learnings JMeter for REST Selenium for UI measures Simple yml config OUR GOALS THINGS TO AVOID
- 65. Reinventing the wheel Custom wrapper Ignore UI Hardcode/magic numbers Monolith Utilise existing tools/learnings JMeter for REST Selenium for UI measures Simple yml config Extensible OUR GOALS THINGS TO AVOID
- 66. Taurus automation tool Free and open source Under Apache 2.0 License
- 67. Thin layer An abstraction layer over JMeter and Selenium Taurus automation tool Free and open source Under Apache 2.0 License
- 68. Thin layer An abstraction layer over JMeter and Selenium Platform-independent Runs where Python and Java can run Taurus automation tool Free and open source Under Apache 2.0 License
- 69. Taurus For orchestration and configuration Data Center Application Performance Toolkit
- 70. Taurus For orchestration and configuration JMeter JMeter scripts to generate base load Data Center Application Performance Toolkit
- 71. Taurus For orchestration and configuration JMeter JMeter scripts to generate base load Selenium Selenium PyTest scrips to measure UI response Data Center Application Performance Toolkit
- 72. Taurus For orchestration and configuration JMeter JMeter scripts to generate base load Selenium Selenium PyTest scrips to measure UI response Utils Deployments scripts Datasets upload Reports and charts generation Data Center Application Performance Toolkit
- 74. DEMO
- 76. Instance Setup How to easily set
- 77. Prepare ENV Launch Jira DC Upload Dataset Install Toolkit Configuration Shape usage Please only use circles, rectangles, and rounded rectangles to call attention to a particular part of a screenshot, for the sake of consistency.
- 78. Prepare ENV Launch Jira DC Upload Dataset Install Toolkit Configuration Shape usage Please only use circles, rectangles, and rounded rectangles to call attention to a particular part of a screenshot, for the sake of consistency.
- 79. Prepare ENV Launch Jira DC Upload Dataset Configuration Shape usage Please only use circles, rectangles, and rounded rectangles to call attention to a particular part of a screenshot, for the sake of consistency. Install Toolkit
- 80. Prepare ENV Launch Jira DC Upload Dataset Configuration Shape usage Please only use circles, rectangles, and rounded rectangles to call attention to a particular part of a screenshot, for the sake of consistency. Install Toolkit
- 81. Prepare ENV Launch Jira DC Upload Dataset Configuration Shape usage Please only use circles, rectangles, and rounded rectangles to call attention to a particular part of a screenshot, for the sake of consistency. Install Toolkit
- 82. Prepare ENV Launch Jira DC Upload Dataset Configuration Shape usage Please only use circles, rectangles, and rounded rectangles to call attention to a particular part of a screenshot, for the sake of consistency. Install Toolkit
- 83. Prepare ENV Launch Jira DC Upload Dataset Configuration Shape usage Please only use circles, rectangles, and rounded rectangles to call attention to a particular part of a screenshot, for the sake of consistency. Install Toolkit
- 84. Prepare ENV Launch Jira DC Upload Dataset Configuration Shape usage Please only use circles, rectangles, and rounded rectangles to call attention to a particular part of a screenshot, for the sake of consistency. Install Toolkit
- 85. Launch Jira DC Upload Dataset Configuration Prepare ENV Install Toolkit
- 86. Launch Jira DC Upload Dataset Configuration Prepare ENV Install Toolkit
- 87. Upload Dataset Configuration Prepare ENV Launch Jira DC Generate Eval Use the UI too generate an evaluator license. Contact Atlassian We can provide you 2 time bomb licenses for testing. Ask on your DCHELP ticket. License Setup Install Toolkit
- 88. Launch Jira DC Upload Dataset Configuration Prepare ENV Install Toolkit
- 89. Launch Jira DC Upload Dataset Configuration Restore DB from Postgres SQL Dump By executing populate_db.sh script. Prepare ENV Install Toolkit
- 90. Launch Jira DC Upload Dataset Configuration Restore DB from Postgres SQL Dump By executing populate_db.sh script. Upload attachments By executing upload_attachments.sh script. Prepare ENV Install Toolkit
- 91. Launch Jira DC Upload Dataset Configuration Restore DB from Postgres SQL Dump By executing populate_db.sh script. Upload attachments By executing upload_attachments.sh script. Perform Re-Index Start Re-Index on Jira system preferences page. Prepare ENV Install Toolkit
- 92. $ wget https://dapt_repo/populate_db.sh $ chmod +x populate_db.sh $ ./populate_db.sh Upload Dataset Restore DB Upload attachments Perform Re-Index
- 93. Upload Dataset Restore DB Upload attachments Perform Re-Index --2019-08-26 05:22:25-- https://centaurus- datasets.s3.amazonaws.com/jira/8.0.3/large/db.dump Resolving centaurus-datasets.s3.amazonaws.com (centaurus- datasets.s3.amazonaws.com)... 52.219.88.116 Connecting to centaurus-datasets.s3.amazonaws.com (centaurus- datasets.s3.amazonaws.com)|52.219.88.116|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 7340090453 (6.8G) [application/x-www-form-urlencoded] Saving to: ‘db.dump’ 31% [================================================================>
- 94. Upload Dataset Restore DB Upload attachments Perform Re-Index pg_restore: creating FK CONSTRAINT "public.AO_E8B6CC_PR_TO_COMMIT fk_ao_e8b6cc_pr_to_commit_request_id" pg_restore: creating FK CONSTRAINT "public.AO_E8B6CC_REPO_TO_PROJECT fk_ao_e8b6cc_repo_to_project_repository_id" pg_restore: creating FK CONSTRAINT "public.AO_F1B27B_KEY_COMP_HISTORY fk_ao_f1b27b_key_comp_history_timed_promise_id" pg_restore: creating FK CONSTRAINT “public.AO_F1B27B_KEY_COMPONENT fk_ao_f1b27b_key_component_timed_promise_id" real 68m38.656s user 1m43.237s sys 0m11.933s Step6: Start Jira
- 95. Upload Dataset Restore DB Upload attachments Perform Re-Index $ wget https://dapt_repo/upload_attachments.sh $ chmod +x upload_attachments.sh $ ./upload_attachments.sh
- 96. Upload Dataset Restore DB Upload attachments Perform Re-Index [300000/2000000 entries] [1.7 G/11.1 G transferred] [2289 entries/s] [13.1 M/s bw] [monq 0] [jq 469] [600000/2000000 entries] [3.3 G/11.1 G transferred] [1929 entries/s] [11.0 M/s bw] [monq 0] [jq 369] [900000/2000000 entries] [5.0 G/11.1 G transferred] [666 entries/s] [3.8 M/s bw] [monq 0] [jq 269] [1059000/2000000 entries] [5.9 G/11.1 G transferred] [457 entries/s] [2.6 M/s bw] [monq 0] [jq 216] [2000000/2000000 entries] [11.1 G/11.1 G transferred] [385 entries/s] [2.2 M/s bw] [monq 1] [jq 1]
- 100. Launch Jira DC Upload Dataset Configuration Clone git repository $ git clone dcapt.git Prepare ENV Install Toolkit
- 101. Launch Jira DC Upload Dataset Configuration Clone git repository Follow README.md instructions $ git clone dcapt.git $ pip install bzt Prepare ENV Install Toolkit
- 102. Launch Jira DC Upload Dataset Configuration jira.yml file Prepare ENV Install Toolkit
- 103. Launch Jira DC Upload Dataset Configuration jira.yml file Prepare ENV Install Toolkit
- 104. Launch Jira DC Upload Dataset Configuration jira.yml file env: application_hostname: jira.atlassian.com application_protocol: http application_port: 80 admin_login: admin admin_password: admin concurrency: 200 test_duration: 45m WEBDRIVER_VISIBLE: False Prepare ENV Install Toolkit
- 105. Scenario 1: Performance Testing With No App Data
- 106. Scenario 1: Performance Testing Run 1: Without App Install App Run 2: With App Base line Performance run for fixed configuration without an app: $ bzt jira.yaml Analyse Report
- 107. Scenario 1: Performance Testing Run 1: Without App Install App Run 2: With App Base line Performance run for fixed configuration without an app: JMeter Constant JMeter load. $ bzt jira.yaml Analyse Report
- 108. Scenario 1: Performance Testing Run 1: Without App Install App Run 2: With App Base line Performance run for fixed configuration without an app: JMeter Constant JMeter load. Selenium Measure timings for UI actions. $ bzt jira.yaml Analyse Report
- 113. Install App Load Your App Run 1: Without App Run 2: With App Analyse Report Scenario 1: Performance Testing
- 114. Install App Load Your App INSTALL Load App DATA Setup Data Run 1: Without App Run 2: With App Analyse Report Scenario 1: Performance Testing
- 115. Run 2: With App Run The same run with app installed. $ bzt jira.yaml Run 1: Without App Install App Analyse Report Scenario 1: Performance Testing
- 116. Run 2: With App Run 1: Without App Install App Analyse Report Scenario 1: Performance Testing
- 117. column_name: 90% Line title: Performance Testing runs: - runName: without app resultsPath: results/run1 - runName: with app resultsPath: results/run2 performance_profile.yml Run 2: With App Run 1: Without App Install App Analyse Report Scenario 1: Performance Testing
- 118. column_name: 90% Line title: Performance Testing runs: - runName: without app resultsPath: results/run1 - runName: with app resultsPath: results/run2 performance_profile.yml Run 2: With App Run 1: Without App Install App Analyse Report Scenario 1: Performance Testing
- 119. column_name: 90% Line title: Performance Testing runs: - runName: without app resultsPath: results/run1 - runName: with app resultsPath: results/run2 performance_profile.yml Run 2: With App Run 1: Without App Install App Analyse Report Scenario 1: Performance Testing
- 120. column_name: 90% Line title: Performance Testing runs: - runName: without app resultsPath: results/run1 - runName: with app resultsPath: results/run2 performance_profile.yml $ python csv_chart_generator.py performance_profile.yml Run 2: With App Run 1: Without App Install App Analyse Report Scenario 1: Performance Testing
- 121. Run 2: With App Run 1: Without App Install App Analyse Report Scenario 1: Performance Testing
- 122. INDEX TIMING TEST Now is a great time to run your re-index test
- 123. Extending The Tool To Test Your Apps
- 124. How to extend the Performance Toolkit Modify JMeter Test your app-specific endpoint at high load (e.g. REST APIs). Modify Selenium Test your app with a real browser in parallel (e.g. load times)
- 125. EXTENSION FILES
- 126. Modifying JMeter script main script (jira.jmx) extension script (extension.jmx)
- 127. (extension.jmx) More on modifying JMeter
- 128. Modifying Selenium script main script (jira-ui.py) extension script (extension-ui.py) ... def test_1_selenium_view_issue (webdriver, datasets): view_issue(webdriver, datasets) def test_1_selenium_custom_action (webdriver, datasets): custom_action(webdriver, datasets) ... def custom_action(webdriver, datasets): @print_timing def measure(webdriver, interaction): webdriver.get(f'{URL}/plugin') # WebDriverWait wait condition measure(webdriver, 'selenium_view_report')
- 129. Running tests with your modification main config (jira.yml) bzt jira.yml settings: ... concurrency: 10 test_duration: 5m scenarios: jmeter: script: jmeter/jira.jmx properties: # Workload model total_actions_per_hr: 54500 perc_create_issue: 100
- 130. With App Data Scenario 2: Scalability Testing
- 131. Scenario 2: Scalability Load and Setup Your App Run 3: One node Run 4: Two nodes Run 5: Four nodes Analyze report
- 132. Scenario 2: Scalability Load and Setup Your App INSTALL Load App DATA Setup Data Run 3: One node Run 4: Two nodes Run 5: Four nodes Analyze report
- 133. Run 3: One node Run 4: Two nodes Run 5: Four nodes Analyze report Base line Performance run for fixed configuration with app- specific actions on one node Jira Data Center. $ bzt jira.yaml Scenario 2: Scalability
- 134. Run 3: One node Run 5: Four nodes Analyze report Scenario 2: Scalability Run 4: Two nodes
- 135. Run 3: One node Run 5: Four nodes Analyze report Scenario 2: Scalability Run 4: Two nodes
- 136. Run 3: One node Run 5: Four nodes Analyze report Scale DC to two nodes $ bzt jira.yaml Scenario 2: Scalability Run 4: Two nodes
- 137. Run 3: One node Run 4: Two nodes Run 5: Four nodes Analyze report Scale DC to four nodes Set 4 as Min and Max number of cluster nodes in Jira AWS Quickstart template and update Stack. $ bzt jira.yaml Scenario 2: Scalability
- 138. Run 3: One node Run 4: Two nodes Run 5: Four nodes Analyse report column_name: 90% Line title: Scale Testing runs: - runName: one node resultsPath: results/run3 - runName: two nodes resultsPath: results/run4 - runName: four nodes resultsPath: results/run5 Report yml Scenario 2: Scalability
- 139. Run 3: One node Run 4: Two nodes Run 5: Four nodes Analyse report column_name: 90% Line title: Scale Testing runs: - runName: one node resultsPath: results/run3 - runName: two nodes resultsPath: results/run4 - runName: four nodes resultsPath: results/run5 Report yml Scenario 2: Scalability
- 140. Run 3: One node Run 4: Two nodes Run 5: Four nodes Analyse report column_name: 90% Line title: Scale Testing runs: - runName: one node resultsPath: results/run3 - runName: two nodes resultsPath: results/run4 - runName: four nodes resultsPath: results/run5 Report yml Scenario 2: Scalability
- 141. Run 3: One node Run 4: Two nodes Run 5: Four nodes Analyse report column_name: 90% Line title: Scale Testing runs: - runName: one node resultsPath: results/run3 - runName: two nodes resultsPath: results/run4 - runName: four nodes resultsPath: results/run5 Report yml Scenario 2: Scalability
- 142. Run 3: One node Run 4: Two nodes Run 5: Four nodes Analyse report column_name: 90% Line title: Scale Testing runs: - runName: one node resultsPath: results/run3 - runName: two nodes resultsPath: results/run4 - runName: four nodes resultsPath: results/run5 Report yml $ python csv_chart_generator.py report.yml Scenario 2: Scalability
- 143. Run 3: One node Run 4: Two nodes Run 5: Four nodes Analyse report Scenario 2: Scalability
- 144. Performance Testing 101 Congrats you’ve graduated performance testing 101 with the Data Center App Performance Toolkit
- 145. FEEDBACK
- 148. QUESTIONS
- 150. Thank you! BEN MAGRO | PRODUCT MANAGER | ATLASSIAN | @MAGRO OLEKSANDR METELYTSIA | TEAM LEAD | ATLASSIAN