NFV Testing
1 like554 views
The document details the agenda and components of NFV (Network Functions Virtualization) testing and integration as part of the OPNFV (Open Platform for NFV) project, emphasizing the role of performance testing, CI/CD methodologies, and various testing tools. It recognizes the complexities involved in benchmarking and testing NFV against traditional networking methods, highlighting the need for specialized performance testing frameworks and metrics. Additionally, it includes an invitation to a community networking event to engage with professionals from various open-source initiatives in the NFV domain.
NFV Testing
- 1. NFV Testing Jose Lausuch (Ericsson) Trevor Cooper (Intel)
- 2. Agenda NFV Testing with Open Source Projects Role of CI Performance Testing
- 5. Enhancements Additions OPNFV – testing and integration project
- 6. PLATFORM SDN COMPONENTS FEATURES SFC BPGPVN IPv6 Hugepages Fault management OPNFV – testing and integration project
- 7. OPNFV – testing and integration project Infrastructure Compute Storage Network Virtualization Cloud Management NFV apps NFV orchestration OSS/BSS
- 8. NFV Testing with Open Source Projects
- 9. Test Ecosystem in OPNFV
- 10. Types of testing Functional Performance Stress Security Infrastructure Features Components VNF MANO … Network Storage Compute Virtual layer Traffic gen. … Network Storage Compute … Upgradability Backup&Restore . . .
- 11. The concept of scenarios scenario X scenario Y scenario Z SFC
- 12. The role of CI Continuous Integration
- 13. Infrastructure – Distributed Pharos Labs 4/3/2017 LF Infra Collab
- 15. Role of CI Jumphost Initiate Run for certain Scenario Download ISO Deploy Functest Yardstick Pull Down Functest Image Pull Down Yardstick Image Labs connected 24x7 using JNLP Google Cloud Storage Docker Hub Docker Hub 4/3/2017 LF Infra Collab Pharos POD (SUT)
- 16. CI workflow yesnew changes? no Clone repo Poll changes Build Build Build Build Server Upload ISO Push Image Google Cloud Storage Docker Hub Patchset Verification Verified +1/-1 Patch
- 17. Working with upstream communities contribute contribute contribute contribute pull from master, deploy, test and/or verify patchset, post feedback
- 18. NFV Performance Testing 1. Meanings 2. Approaches 3. Tools 4. Examples 5. Ambitions “When you can measure what you are speaking about, and express it in numbers, you know something about it; but when you cannot express it in numbers, your knowledge is of a meagre and unsatisfactory kind.” … Lord Kelvin
- 19. Meanings of NFV Performance Testing How do vendor A & B compare? • Can the platform deliver required networking performance? • Evaluating switching, acceleration technology, hardware platform, NFVI, VNF, full solution? What Tools and Methods are needed? • We have relied on trusted proprietary tools • Traditional network performance methodologies are useful but not sufficient NFV => general purpose compute elements in a heterogeneous compute environment • Benchmarking virtual network functions is different to physical functions • System configurations are complex and few deployments are the same (test infrastructures also differ) Performance Testing Tools and Methods are not just for Developers and Test-Experts Performance Testing Infrastructure Test Frameworks Test Dashboards Test Databases Traffic Profiles Test Methodologies KPIs / Metrics Traffic Generators
- 20. Approaches to NFV Performance Testing System Under Test, Workloads and Stimuli • VNF / NFVI • Pre-deployment / In-service • Topology / SW versions / configurations • Data Plane vs Control Plane + Data Plane • Component / sub-system / platform / network service • Traffic profiles, Workloads • Deployment automation vs control Test Objectives • How Fast? How much Latency? How much Capacity? • How many Sessions? • How Quick is Recovery? Test Methods and Metrics • Throughput, back2back, frame-loss • Packet and Frame Delay Distribution • Scalability according to number of flows, active ports, etc. • Stream Type with realistic profiles / flows: L2, L3, L4 • Control and Data Path Coupling • CPU and Memory Consumption • “Soak” tests capture transient changes in performance COVERAGE SPEED ACCURACY RELIABILITY SCALABILITY Activation Operation De-activation IETF NFV Benchmarking Internet Drafts Data Centre Benchmarking Methodology draft-ietf-bmwg-dcbench-methodology-02 Data Centre Benchmarking Terminology draft-ietf-bmwg-dcbench-terminology-05 Benchmarking Methodology for SDN Controller Performance draft-ietf-bmwg-sdn-controller-benchmark-meth-01 Terminology for Benchmarking SDN Controller Performance draft-ietf-bmwg-sdn-controller-benchmark-term-01 Benchmarking VNFs and Their Infrastructure draft-ietf-bmwg-virtual-net-02 Considerations for Benchmarking High Availability of NFVI draft-kim-bmwg-ha-nfvi-01 Benchmarking Methodology for EVPN draft-kishjac-bmwg-evpntest-00 VNF Benchmarking Methodology draft-rosa-bmwg-vnfbench-00 Benchmarking Virtual Switches in OPNFV draft-vsperf-bmwg-vswitch-opnfv-02 ETSI NFV Test Specifications NFV Pre-deployment Testing (Spec) http://www.etsi.org/technologies-clusters/technologies/nfv NFV Interoperability Testing Methodology (Spec) Use cases & recommendations for VNF Snapshot (Spec) NFV Issue Tracker – to report feedback on NFV DRAFTs http://nfvwiki.etsi.org/index.php?title=NFV_Issue_Tracker Path-implementations among NS Endpoints & VNFs https://docbox.etsi.org/isg/nfv/open/drafts/ Capture of VNF data and VNF/VNFC snapshots DevOps and CI/CD - test procedures and metrics Interop test guidelines VNF, MANO, VIM-NFVI Key performance NFVI metrics at different layers
- 21. NFV Performance Testing Tools Test Infrastructure • https://wiki.opnfv.org/display/pharos • https://wiki.opnfv.org/display/pharos/Community+Labs • Video Tutorial: Introduction to Pharos Test Frameworks • Network Service Performance (MANO) • VNF Characterization (Control + Data Plane) • NFVI Performance (Data Plane) • Components and Sub-system Performance • Staging Manager and Bottleneck Analysis • Analytics (post-process test results from CI/CD pipeline) • Integrated statistics / events monitoring Traffic Generators • Vary in measurement capability and suitability • Vary in complexity and cost Traffic Generator IXIA (physical and virtual) Proprietary SW/HW Spirent (physical and virtual) Proprietary SW/HW Xena Proprietary SW/HW Moongen Open Source SW TREX Open Source SW Pktgen Open Source SW LAN Tornado RFC 2544 Proprietary SW Seagull Open Source SW PROX Open Source SW
- 22. NFV Test Tools - VSPERF Modular test framework, combining traffic generation, virtual switches, VNFs, and network configuration test cases IETF Internet Draft: https://tools.ietf.org/html/draft-ietf-bmwg-vswitch-opnfv-01 Test-cases: configure and program vSwitch, deploy and terminate VNFs, execute a traffic generator, modify a VSPERF configuration, execute external commands, etc. http://docs.opnfv.org/en/latest/submodules/vswitchperf/docs/testing/user/userguide/teststeps.html Metrics: Throughput, Packet delay, Packet delay variation, Packet loss, Burst behavior, Packet re-ordering, Packet correctness, Availability and capacity of the DUT http://docs.opnfv.org/en/latest/submodules/vswitchperf/docs/testing/developer/requirements/vswitchperf_ltd.html Results • OPNFV community lab resources https://wiki.opnfv.org/display/pharos/VSPERF+in+Intel+Pharos+Lab+-+Pod+12 • Scenario example http://docs.opnfv.org/en/latest/submodules/vswitchperf/docs/testing/developer/results/scenario.html • CI daily results: https://build.opnfv.org/ci/view/vswitchperf/ https://build.opnfv.org/ci/view/vswitchperf/job/vswitchperf-daily-danube/lastBuild/console • Visualization (proto): http://testresults.opnfv.org/proto/index-vsperf.html Speed Accuracy Reliability Scale Activation • Address Learning Rates and Caching Capacities • Packet Processing Latencies and Variations • Flow Addition • RFC2544. System RecoveryTime • RFC2544. ResetTime • RFC2889. AddressCaching Capacity Operation • RFC2544.PacketLossRatio • RFC2544.PacketLossRate FrmMod • RFC2544.BackToBackFrames • RFC2889.MaxForwardingRate • RFC2889.ForwardPressure • RFC2889.BroadcastFrameForwarding • RFC2889.BroadcastFrameLatencytest • CPU.RFC2544.0PacketLoss • RFC2544.WorstN-BestN • InterPAcketDelayVariation RFC5481 • Frame Filtering • RFC2889. Soak • RFC2889. SoakFrame Modifcation • PacketDelay Variation. RFC3393. Soak • Scalability.RFC25 44.0PacketLoss • MemoryBandwid th.RFC2544.0Pac ketLoss.Scalability
- 23. NFV Test Tools - Yardstick Yardstick is used in for verifying the OPNFV infrastructure KPIs and some OPNFV features • Test configurations in YAML … parsed and converted into an internal model … converted into a Heat template and deployed • Scenario runners execute commands in VMs with SSH • Output written as json records to a file or influxdb, results shown with Grafana Test Cases: http://docs.opnfv.org/en/latest/submodules/yardstick/docs/testing/user/userguide/15-list-of-tcs.html Metrics: http://docs.opnfv.org/en/latest/submodules/yardstick/docs/testing/user/userguide/02-methodology.html#metrics Results: http://testresults.opnfv.org/grafana/
- 24. 1.43 24.90 COLORADO (FUEL) LATEST (KOLLA) DANUBE 25 Test Example - TCP VM Performance
- 25. Test Example – vSwitch Flow Rules • OpenStack deployed with Fuel/Apex uses single logical tenant network (two tenant networks possible with DevStack/Kolla) • Two OVS bridges by default. Br-prv is outward facing bridge. Br-int is inner bridge where VMs are logically connected • VLAN segmentation means additional modify VLAN ID required to and from VMs • OVS flow rules are created by OpenStack Neutron • OVS flow action NORMAL - FDB table updated per packet
- 26. 27 Colorado deployment with OVS normal mode (MAC learning) comparing throughput when UDP traffic has varying source MAC address and varying UDP ports Varying source MAC Varying UDP port 0 1 2 3 4 5 6 70 128 256 512 1,024 1,280 1,518 MillionsofPacketsperSecond 1 Flow 100 Flows 1000 Flows 0 1 2 3 4 5 6 70 128 256 512 1,024 1,280 1,518 MillionsofPacketsperSecond 1 Flow 100 Flows 1000 Flows Test Example - Traffic Flow Comparison
- 27. Stable Integrate Deploy Test Deploy Test 28 Stand-alone Previous Master Future? Developer Testing Test Cases Test Tools Configurations Performance Ranges Reference Scenarios Stress Tests OPNFV Test Suites User Testing Vendor Platform Tests Automated Performance Test Suite Test Data NFV Platform Performance Testing Testing Artifacts Performance Reference Scenarios Good Data + Good Explanations -> Good Configurations -> Good Performance “NFV Performance Standards” Analytics Release Testing
- 28. NFV Testing - Conclusions NFV is more challenging than Cloud Computing … • Just working is not good enough Test areas in evolution are … • Advancing CI and integration test toolchains • Integrating test capabilities with upstream communities • Performance Testing automation
- 29. You’re Invited! Free Beer, Basketball, and Networking • Open Source Community Reception • Tonight from 6:00 – 10:00 PM • Levi’s Stadium, Yahoo! Fantasy Football Lounge (a short walk) • Sponsored by Serro, Juniper, and Inocybe • Meet community members from OPNFV, OpenDaylight, OpenStack, Fd.io and ONAP. Light food and drinks will be provided. • We'll also be showing the NCAA Men’s Basketball Final Game! • RSVP Required via OPNFV website here: • https://www.opnfv.org/event/open-networking-summit