![Test Setup Configurations
• Management server hosts
– Dual core Intel(R) Xeon(R) CPU, 2.27GHz, ht enabled,
4 processor
– 16 G RAM
• Database
– Quad-Core AMD Opteron(tm), 2.1GHz, ht enabled, 8
processor
– 32 G RAM
• Java heap size
– (5 – 12) GB [Depending on use case]](https://image.slidesharecdn.com/cloudstackperformancetesting-130221152006-phpapp01/85/CloudStack-Performance-Testing-9-320.jpg)
![Baselines For common Use cases
CPU Utilization
DB connections
Steady State Measures – when no external API is
being fired
Time taken for Async job [Deploy VM] to complete
MS failures and time to reconnect all hosts
List API response time, with varied page size
parameters](https://image.slidesharecdn.com/cloudstackperformancetesting-130221152006-phpapp01/85/CloudStack-Performance-Testing-15-320.jpg)
CloudStack Performance Testing
- 1. CloudStack Performance Testing Challenges, Learning, Results Sowmya Krishnan
- 2. Apache CloudStack • Turnkey platform for delivering IaaS clouds • Highly scalable • Flexible – Hypervisor agnostic – Network topologies – Multiple storage options – Scales to tens of thousands of hypervisors
- 3. Typical Deployment Admin Internet Management Server Cluster Primary Router MySQL Backup Load Balancer MySQL L3 Core Switch Top of Rack Switch Object Storage Servers … … … … … Availability Zone 1 Pod 1 Pod 2 Pod 3 Pod N
- 4. Challenge of creating a production scale performance set up What we want… • 30,000 hosts • 100s of Storage nodes • High Network bandwidth • High end switches • … What we have in a QA Lab… Handful of servers One/Two Storage pools Shared switch
- 5. Scalability Challenges Continuous monitoring of the infrastructure Hosts up, VMs up, Routers, Capacity… Orchestration of large number of resources Clustered management servers Outage of a hypervisor host Outage of a management server Outage of network
- 7. Simulator Mgmt. Server Zone User API MySQL Simulator Load Mgmt. Balancer Server Admin API Mgmt. MySQL Server Mgmt. Server
- 8. Simulator • Hypervisor type in itself • Implements commonly used commands of hypervisor host – Start/Stop – List VMs in the host – Program Security Group Rules – Add host to maintenance mode • Deploy Simulator VMs – Using a Simulator Template – Start/Stop – Migrate – Volume attached – Take snapshot of the volume • Storage pools
- 9. Test Setup Configurations • Management server hosts – Dual core Intel(R) Xeon(R) CPU, 2.27GHz, ht enabled, 4 processor – 16 G RAM • Database – Quad-Core AMD Opteron(tm), 2.1GHz, ht enabled, 8 processor – 32 G RAM • Java heap size – (5 – 12) GB [Depending on use case]
- 10. Demo
- 11. Router Check interval Redundant Virtual Router setup 4000 Accounts => 8000 RVRs RouterStatusMonitor task which runs for all routers to monitor each router status Default check interval = 5 mins After about an hour, interval between successive checks shot up to 56 mins Fix: Introduce a new global config to configure the pool size for router checks Pool size = 100 8000/100 = 80 * 3 s (to perform RVR check) = 240 s (4 min) to complete RVR check tasks
- 13. Async Job Response Time As number of VMs increase time taken to deploy instances increased? Not always If Network id is not passed, CS spends significant time searching for the network to place the VM After Fix:
- 14. Deploy VM time
- 15. Baselines For common Use cases CPU Utilization DB connections Steady State Measures – when no external API is being fired Time taken for Async job [Deploy VM] to complete MS failures and time to reconnect all hosts List API response time, with varied page size parameters
- 16. Future work • Enhance Simulator Features • VR performance • Hypervisor specific agents • Storage Metrics • …
- 17. mvn install -Dsimulator cp agent-simulator/tomcatconf/components-simulator.xml.in client//target/cloud-client-ui-4.1.0-SNAPSHOT/WEB- INF/classes/components-simulator.xml Edit client//target/cloud-client-ui-4.1.0-SNAPSHOT/WEB- INF/classes/environment.properties to point to the new xml mvn -P developer -pl developer -Ddeploydb mvn -P developer -pl developer -Ddeploydb-simulator mvn -pl :cloud-client-ui jetty:run -Dsimulator
- 18. Q&A
Editor's Notes
- #6: Typical challenges / questions to be answered as a result of a performance runs of an IaaS productLast three may appear to be more of fault tolerance of a system, it is related to perf since these failures are harder to find in a high scale set up as compared to a smaller one. (Takes more time to identify a failure)
- #13: router.check.interval , CheckRouterCommandSimulator delay8K Routers,