Functest in Depth
2 likes755 views
The document presents an in-depth overview of Functest, a testing framework for validating OpenStack deployments and SDN controller integrations, showcased at the OPNFV Summit 2015. It details various test cases, methodologies, and automation through CI processes, highlighting the use of Docker for test execution and the structure of Functest's configuration. The document also discusses future enhancements and lessons learned from prior projects to improve testing stability and coverage.
Functest in Depth
- 1. FUNCTEST IN DEPTH Jose Lausuch (Ericsson) OPNFV Summit 2015 jose.lausuch@ericsson.com
- 2. Agenda 09/11/2015 Functest in Depth, OPNFV Summit 2015 2 What is Functest? Test cases Repository structure CI & Automation Brahmaputra Release
- 3. OPNFV Projects ecosystem 09/11/2015 Functest in Depth, OPNFV Summit 2015 3 TEST Projects Yardstick Qtip VPerf … FEATURE Projects VPN SFC SDN Contr. Resource mgmt. … PHAROS Testbed OCTOPUS (CI) RELENG INSTALLER Projects Fuel Foreman JOID Compass … Functest TestbedTestbed TestbedTestbed
- 4. What is Functest? 09/11/2015 Functest in Depth, OPNFV Summit 2015 4
- 5. What we do… Focus on verifying the OpenStack deployment and the SDN Controllers Have a full integration and automation mechanism Provide comprehensive testing methodology 09/11/2015 Functest in Depth, OPNFV Summit 2015 5 Base System Functionality Testing
- 6. 09/11/2015 Functest in Depth, OPNFV Summit 2015 6 ETSI NFV Reference WE DON’T: Performance tests HW tests Frameworks
- 7. The gang… 09/11/2015 Functest in Depth, OPNFV Summit 2015 7
- 8. The project… 09/11/2015 Functest in Depth, OPNFV Summit 2015 8 source: Bitergia (http://projects.bitergia.com/opnfv) date: November 2015
- 9. Functest Testcases Arno 09/11/2015 Functest in Depth, OPNFV Summit 2015 9
- 10. Functest Testcases vPing test case Create machines and verify connectivity ODL test case Robot framework, ODL functional testing Rally bench tests Benchmark the OpenStack deployment Tempest test OpenStack native tests (100+ smoke-tests) 09/11/2015 Functest in Depth, OPNFV Summit 2015 10
- 11. Functest Testcases vPing test case Create machines and verify connectivity 09/11/2015 Functest in Depth, OPNFV Summit 2015 11
- 12. VM2 vPing test case 09/11/2015 Functest in Depth, OPNFV Summit 2015 12 VM1 provisional Neutron network
- 13. Functest Testcases ODL test case Robot framework, ODL functional testing 09/11/2015 Functest in Depth, OPNFV Summit 2015 13
- 14. ODL test case 09/11/2015 Functest in Depth, OPNFV Summit 2015 14 Dashboard/API (Horizon) Compute Storage Networking (Neutron) ….. ML2 Plug-in Northbound API ODL – Neutron REST API OVSDB OVS Nodes
- 15. ODL test cases 09/11/2015 Functest in Depth, OPNFV Summit 2015 15 Borrowed from ODL repository 4 basic test cases: Check ODL Restconf Create/Delete/Check Network Create/Delete/Check Subnet Create/Delete/Check Port
- 16. Functest Testcases Rally bench tests Benchmark the OpenStack deployment 09/11/2015 Functest in Depth, OPNFV Summit 2015 16
- 17. Rally 09/11/2015 Functest in Depth, OPNFV Summit 2015 17 Rally is a benchmarking tool that answers the question: “How does OpenStack work at scale?”.
- 18. Rally - scenario example 09/11/2015 Functest in Depth, OPNFV Summit 2015 18 Runners: • Serial: launch scenario N times in one thread • Constant: launch scenario N times in M parallel threads • RPS: generates N threads per second M times Success criteria (SLA) • You can define a maximum time per iteration • It will skip the test if 1 iteration fails Context: • Contexts needed for this scenario (tenants/users)
- 19. Rally - output 09/11/2015 Functest in Depth, OPNFV Summit 2015 19
- 20. Functest Testcases Tempest test OpenStack native tests (100+ smoke-tests) 09/11/2015 Functest in Depth, OPNFV Summit 2015 20
- 21. Tempest test 09/11/2015 Functest in Depth, OPNFV Summit 2015 21 Tempest is a set of integration tests to be run against a live OpenStack cluster Rally performs the tempest installation and generation of tempest.conf
- 22. Functest Structure today 09/11/2015 Functest in Depth, OPNFV Summit 2015 22
- 23. Directory structure 09/11/2015 Functest in Depth, OPNFV Summit 2015 23 Main configuration script. Installs needed tools (Rally, Robot, …) Configuration parameters SDN Controllers test suites Openstack related tests (Rally, Tempest) vPing test case Dockerfile and automation scripts Format results to dashboard ready and other utils vIMS test case
- 24. Configuration file 09/11/2015 Functest in Depth, OPNFV Summit 2015 24 Configuration yaml file Predefined tested values Can be given as an input to Functest Possible to specify commit IDs of the used repositories Versioning control
- 25. CI & Aut 09/11/2015 Functest in Depth, OPNFV Summit 2015 25 tion
- 26. CI & Automation 09/11/2015 Functest in Depth, OPNFV Summit 2015 26 Functest Jenkins job Triggered automatically after a successful deployment One macro per test case You can execute a single test From Jenkins
- 27. CI & Automation 09/11/2015 Functest in Depth, OPNFV Summit 2015 27 Daily verification of OPNFV platform New Features JENKINS BUILD DEPLOYMENT FUNCTEST YARDSTICK ….
- 28. Execution of tests 09/11/2015 Functest in Depth, OPNFV Summit 2015 28 Target POD Jumphost Don’t worry sir, I’ll do it for you jjob
- 29. Brahmaputra release 09/11/2015 Functest in Depth, OPNFV Summit 2015
- 30. Functest in Brahmaputra release Dockerization Test Result collection Dashboard Virtual IMS test case + SDN Controllers tests Feature projects 09/11/2015 Functest in Depth, OPNFV Summit 2015 30
- 31. Docker -ization 09/11/2015 Functest in Depth, OPNFV Summit 2015 31
- 32. Functest Docker 09/11/2015 Functest in Depth, OPNFV Summit 2015 32 $docker run -t opnfv/functest $dir/run_test.sh
- 33. Functest Docker 09/11/2015 Functest in Depth, OPNFV Summit 2015 33 Without Docker With Docker • Have to install needed libraries in your jumphost • Maintain in the documentation the needed libraries and dependencies • Possible OS dependencies • Not portable • Needed libraries preinstalled in the docker image • Update image when there is a new software dependency and push to the OPNFV Docker Hub • No OS dependencies • Portable
- 34. Execution of tests 09/11/2015 Functest in Depth, OPNFV Summit 2015 34 Target POD Jumphost Docker Hub Don’t worry sir, I’ll do it for you jjob
- 35. MRF PCRF CSCF HSS vIMS test case 09/11/2015 Functest in Depth, OPNFV Summit 2015 35
- 36. vIMS test case Full automation of a core Open Source vIMS solution on OPNFV from Clearwater Deployment of an orchestrator (Cloudify) to manage the VNF Run suite of signaling tests on vIMS Successfully tested from Jenkins on LinuxFoundation POD2 09/11/2015 Functest in Depth, OPNFV Summit 2015 36 SIP
- 37. vIMS test case 09/11/2015 Functest in Depth, OPNFV Summit 2015 37 1) Deploy the orchestrator 2) Create the tenant 3) Create the VNF (10 VMs) 4) Launch test suite (more than 100 signaling tests) 5) Collect results 6) Clean-up
- 38. SDN Controllers 09/11/2015 Functest in Depth, OPNFV Summit 2015 38
- 39. SDN Controllers Extension of OpenDaylight test cases Integration of ONOS test framework Open Contrail test 09/11/2015 Functest in Depth, OPNFV Summit 2015 39
- 40. Test results collection 09/11/2015 Functest in Depth, OPNFV Summit 2015 40
- 41. Test results collection 09/11/2015 Functest in Depth, OPNFV Summit 2015 41 Test projects Mongo DB REST API VPerfYardstick QTIP BottlenecksFunctest …
- 42. Test results collection - API Filter per test case – http://<DB_Public_IP>/results?case=vPing Filter per pod – http://<DB_Public_IP>/results?pod=opnfv-jump-2 Filter per project – http://<DB_Public_IP>/results?project=yardstick Combination of filters – http://<DB_Public_IP>/results?project=functest&case=Tempest&pod=opnfv-jump-2 09/11/2015 Functest in Depth, OPNFV Summit 2015 42
- 43. Dashboard 09/11/2015 Functest in Depth, OPNFV Summit 2015 43
- 44. Dashboard Show all the results using simplistic graphs Possible to filter per project/test case/… Comparison of previous results with recent ones … 09/11/2015 Functest in Depth, OPNFV Summit 2015 44
- 45. Dashboard Draft 09/11/2015 Functest in Depth, OPNFV Summit 2015 45
- 46. Beyond Brahmaputra 09/11/2015 Functest in Depth, OPNFV Summit 2015 46 New functional test (vCPE, vHGW, vCDN, …) Results analytics Better Rally usage More SDN controllers test integration Broaden test coverage (security, O&M, …)
- 47. Lessons learned in Arno 09/11/2015 Functest in Depth, OPNFV Summit 2015 47 Need for stable labs some weeks before the release Need to run the tests on different PODs from CI Complex troubleshooting Functest cannot solve all problems
- 48. Challenges 09/11/2015 Functest in Depth, OPNFV Summit 2015 48 Different OpenStack installers Different SDN controllers Many features Lots of combinations to be tested !! … …
- 49. Summary 09/11/2015 Functest in Depth, OPNFV Summit 2015 49 First test project in OPNFV Validate a fresh OPNFV deployment Umbrella for testing projects Umbrella for feature projects We don’t bite, come to us with questions
- 50. 09/11/2015 Functest in Depth, OPNFV Summit 2015 50
- 51. Thank you! 09/11/2015 Functest in Depth, OPNFV Summit 2015 51