![API test workflow
<?xml&version="1.0"&encoding="UTF98"?>!
<cloudService!name="AWS">!
<excluded>!
<resource!name="kvdb"!/>!
<resource!name="customer">!
<method!name="add"!/>!
<method!name="changeCurrency"!/>!
<method!name="revertCurrency"!/>!
<method!name="setTimeZone"!/>!
<method!name="revertTimeZone"!/>!
</resource>!
</excluded>!
</cloudService>
{"addLoadBalancer":!
[!
{!
"region":{"regionId":REGION_ID},!
"budget":BILLINGCODE_ID,!
"description":"TestNG_Trial_LB",!
"name":DYNAMIC,!
"label":"red",!
"customer":{"customerId":CUSTOMER_ID},!
"dataCenters":[{"dataCenterId":DATACENTER_ID}],!
"owningGroups":[{"groupId":GROUP_ID}],!
"listeners":!
[{"protocol":"HTTP","publicPort":80,"privatePort":80}]!
}!
]!
}
start
Read the cloud
access keys & test
configs
Iterate through the
test resources
Exclude
Resource
?
Run atomic tests
Generate report
Headerconfig.xml
AWS.xml
Read resource/
method inputs
<?xml&version="1.0"&encoding="UTF98"&standalone="no"?>!
<CONFIG>!
<BILLINGCODE_ID>403</BILLINGCODE_ID>!
<USER_ID>50950</USER_ID>!!
<DUPLICATE_BILLING_ID>402</DUPLICATE_BILLING_ID>!
<DUPLICATE_GRP_ID>501</DUPLICATE_GRP_ID>!
<ROLE_ID>55501</ROLE_ID>!
<GROUP_ID>500</GROUP_ID>!!
<ACCOUNT_ID>200</ACCOUNT_ID>!
<CLOUD_ID>1</CLOUD_ID>!
<CUSTOMER_ID>200</CUSTOMER_ID>!
</CONFIG>
Generate
Dependency
resources
<?xml&version="1.0"&encoding="UTF98"&standalone="yes"?>!
<header>!
<csp>AWS</csp>!
<resource>all</resource>!
<accessKey>aws_accesskey</accessKey>!
<secretKey>aws_secretkey</secretKey>!
<apiServer>enstratius_apiserver_ip</apiServer>!
<apiServerPort>enstratius_apiserver_port</apiServerPort>!
<version>enstratius_apiserver_version</version>!
<testForFailure>false</testForFailure>!
</header>
stop
Loadbalancer/add.json
Config_200.xml
Finished iterating
Yes
No](https://image.slidesharecdn.com/multicloudtesting-130619142022-phpapp01/85/Multi-Cloud-testing-24-320.jpg)
Multi-Cloud testing
- 1. Testing in a Multi-Cloud Environment Seema Jethani
- 2. Who am I? Seema Jethani @seemaj Senior Product Manager, Dell ( formerly Enstratius / enStratus ) Blog ★http://seemaj.wordpress.com ★http://cloudfieldnotes.com/author/seema/ Slideshare http://www.slideshare.net/svjethani/ presentations
- 4. Bringing joy
- 5. But also presenting... UNIQUE CHALLENGES
- 6. Speaking their own language
- 7. Capability Terminology Account AWS = Account Terremark = Organization Region AWS = Region Terremark = Environment Data center AWS = Availability Zone Terremark = Compute Pool Machine Image AWS = Machine Image VMware = Template vApp Joyent = Datasets OpenStack = Image CloudStack, Terremark = Template Firewall AWS, OpenStack, CloudStack = Security Group Terremark = Firewall Volume AWS = Elastic Block Storage VMware, IBM, Terremark = Disk OpenStack = Cinder or Nova Volume Partial List
- 8. And are sometimes hard to work with
- 9. Examples of challenging cloud behavior 1. Requiring paperwork / manual processing 2. Requiring calling support to create private images 3. Requiring a new account for new API versions, thus forgetting backward compatibility 4. Using same API version, but continuing to make changes 5. Taking hours to launch a server or launching server requests serially 6. Having a ridiculously low cut-off for throttling API requests e.g. dozens of API calls per second 7. Not providing a way for the VM to discover its own ID
- 10. Each is able to do different things in different ways
- 11. Partial Cloud Feature Matrix Feature AWS Rackspace Terremark Joyent Launch Server S S S S Pause Server S NS S S Make Image S S S NS Create Snapshot S S NS NS Create Volume S S NS NS Create Loadbalancer S S NS NS Create/ Reserve IP S NS S NS
- 12. Examples of the cloud nuances 1.Make Image CloudSigma requires server to be stopped before making an image 2.Attaching a volume e.g. Terremark requires the server to be stopped for for volume to be attached 3.Creating a snapshot e.g. Make a snapshot from the volume fails in CloudCentral if the Volume is not attached to a server 4.Connecting via ssh CloudSigma requires users to access a centos server via vnc first to enable ssh
- 13. Cloud requires us to rethink our testing approach
- 14. We need to learn “cloud speak” Understand differences in behavior But also importantly
- 15. Plan for failure
- 16. How is this different? Customers expect cloud failure but also Expect recovery from cloud behavior to be transparent. They don’t care about root cause infrastructure issues Test for failure , auto-recovery , MTTR
- 17. How is this different? Customers expect cloud failure But don’t expect it to impact their business Test for cloud failure and its impact on customer application performance.
- 19. Enstratius Cloud Management Platform
- 21. build Today Artifacts Installer API tests Github Artifactory Vagrant AWS
- 22. Jenkins ReportNG Report Automated Test Framework (TestNG) GitHub Enstratius API Server Cloud A Cloud Z estrov ant API Test Framework
- 23. Cloud account access keys, test configs HeaderConfig.xml Cloud specific resource/method exclusions JSON/XML Inputs for REST Calls common to all clouds Setup Test bed on Cloud under test Tear Down Test bed on Cloud under test UpdateAdd Delete List Atomic Tests Cloud Specific Inputs for REST Calls (if any) estrov
- 24. API test workflow <?xml&version="1.0"&encoding="UTF98"?>! <cloudService!name="AWS">! <excluded>! <resource!name="kvdb"!/>! <resource!name="customer">! <method!name="add"!/>! <method!name="changeCurrency"!/>! <method!name="revertCurrency"!/>! <method!name="setTimeZone"!/>! <method!name="revertTimeZone"!/>! </resource>! </excluded>! </cloudService> {"addLoadBalancer":! [! {! "region":{"regionId":REGION_ID},! "budget":BILLINGCODE_ID,! "description":"TestNG_Trial_LB",! "name":DYNAMIC,! "label":"red",! "customer":{"customerId":CUSTOMER_ID},! "dataCenters":[{"dataCenterId":DATACENTER_ID}],! "owningGroups":[{"groupId":GROUP_ID}],! "listeners":! [{"protocol":"HTTP","publicPort":80,"privatePort":80}]! }! ]! } start Read the cloud access keys & test configs Iterate through the test resources Exclude Resource ? Run atomic tests Generate report Headerconfig.xml AWS.xml Read resource/ method inputs <?xml&version="1.0"&encoding="UTF98"&standalone="no"?>! <CONFIG>! <BILLINGCODE_ID>403</BILLINGCODE_ID>! <USER_ID>50950</USER_ID>!! <DUPLICATE_BILLING_ID>402</DUPLICATE_BILLING_ID>! <DUPLICATE_GRP_ID>501</DUPLICATE_GRP_ID>! <ROLE_ID>55501</ROLE_ID>! <GROUP_ID>500</GROUP_ID>!! <ACCOUNT_ID>200</ACCOUNT_ID>! <CLOUD_ID>1</CLOUD_ID>! <CUSTOMER_ID>200</CUSTOMER_ID>! </CONFIG> Generate Dependency resources <?xml&version="1.0"&encoding="UTF98"&standalone="yes"?>! <header>! <csp>AWS</csp>! <resource>all</resource>! <accessKey>aws_accesskey</accessKey>! <secretKey>aws_secretkey</secretKey>! <apiServer>enstratius_apiserver_ip</apiServer>! <apiServerPort>enstratius_apiserver_port</apiServerPort>! <version>enstratius_apiserver_version</version>! <testForFailure>false</testForFailure>! </header> stop Loadbalancer/add.json Config_200.xml Finished iterating Yes No
- 25. The extra steps 1. Not all functions are supported by all clouds. An exclusion list needs to be maintained. 2. Cloud specific behavior needs to be coded e.g. Terremark requires the server to be stopped for for volume to be attached 3. Sanity check - Issues found must be verified directly using cloud console / cloud API
- 26. Build Near-term updates Artifacts Launch API testsGithub Artifactory AWS Jenkins Installer Checkout Teardown
- 27. Build Future Artifacts Launch API testsGithub Artifactory AWS Jenkins Installer 1. Checkout 6. Teardown Selenium
- 28. Other tools on the radar