Testing - the gatekeeper for quality in NFV
Download as PPTX, PDF3 likes1,673 views
The document discusses the challenges and strategies associated with Network Functions Virtualization (NFV) testing, emphasizing the need for a comprehensive test strategy to ensure quality and reliability in telecom services. It highlights the evolution of NFV as a cultural transformation in organizations, requiring agile testing approaches and robust service level agreements. Additionally, it outlines compliance with ETSI standards and the importance of automated processes for effective validation and management of NFV components.
Testing - the gatekeeper for quality in NFV
- 1. Testing -The Gatekeeper for Quality in NFV
- 2. 3 Bryon Lowen Vice-President Telco at QualiTest Benny Sand NFV Strategy Lead at QualiTest Webinar Presenters
- 3. Agenda NFV – Present & Future Risks and Challenges in NFV Testing Summary & Questions Creating an NFV test strategy Building SDN / NFV Assurance Process Introduction 4
- 4. Agenda NFV – Present & Future Risks and Challenges in NFV Testing Summary & Questions Creating an NFV test strategy Building SDN / NFV Assurance Process Introduction 5
- 5. About QualiTest Group | World’s 2nd Largest IndependentTesting Company | QualiTest was recognized as the only visionary pure play QA and testing Service Company in the “ Magic Quadrant” 6
- 7. Agenda NFV – Present & Future Risks and Challenges in NFV Testing Summary & Questions Creating an NFV test strategy Building SDN / NFV Assurance Process Introduction 8
- 8. OPEX impact | Shared computing resources | Reduced power, space, cooling requirements | Executing trials of new services without incurring much risk | Dynamic recovery from failures | Proof of Concepts (POCs) and trials can be run faster 9
- 9. CAPEX impact | The use of commodity servers reduces hardware costs | No dependence on specialized hardware | A single, common server architecture can be used within the data center environment | Migration to a distributed cloud architecture allows operators to deploy backup in a cost-effective way | Cycling the hardware more frequently to improve the overall performance of the network 10
- 10. Predications | NFV planning phase is shifting gears | Operators are moving along the NFV curve | Finical forecasts indicate a significant growth in NFV investments | NFV will start to influence budgets and purchasing decisions | NFVAdaption pace is progressing faster then expected 11
- 11. Agenda NFV – Present & Future Risks and Challenges in NFV Testing Summary & Questions Creating an NFV test strategy Building SDN / NFV Assurance Process Introduction 12
- 12. Strategic Challenges | NFV enables accelerated time-to-market | NFV enables organizations to implement services anywhere in the world at any time | NFV involves very complex integration of various software elements | Everything known becomes unknown | NFV is more than a mere shift of technologies; it’s a cultural transformation inTelecom organizations 13
- 13. Testing Challenges | Network testing expertise versusVirtualization and Software testing | Agile orientation will be introduced in Network testing | Variety of NFV infrastructures requires more testing from the carriers | Frequent updates in the Network services will require more management and orchestration testing | More test automation is required to support agility, impact testing and regression testing | NFV test strategies are needed to accelerate and ensure the delivery of NFV solutions with guaranteed quality 14
- 14. Agenda NFV – Present & Future Risks and Challenges in NFV Testing Summary & Questions Creating an NFV test strategy Building SDN / NFV Assurance Process Introduction 15
- 15. Building SDN / NFV assurance process 16 Build strong assurance process to capitalize SDN / NFV potential | Renegotiate contracts | Right Price | Comprehensive Service Level Agreements (SLA) | Increasingly Complex | Not affect Quality of Service (QOS)
- 16. QualiTest NFV validation process QualiTest aims to create a unified end-to-end approach for service assurance and network performance 17 NFV Management Orchestration Infrastructure Migration
- 17. QualiTest NFV validation process Assure full compliance to ETSI ISG Standards. •Define NFV Testing process •Test Requirements •NFV Test Environment •Risk Assessment •Roles and Responsibilities •Team Skills and Training •Test Inputs / Basis Assure full compliance to NFV requirements for any network equipment provider and streamline the ability of vendors and users to be able to implement compatible NFV solutions. Test Environment: simulation tools, network analytics, QoS and QoE probes. Automated Certification Process for each VNF before it is introduced to the network Test management and Orchestration prior to VNF migration, including each component (e.g. Orchestrator, Vnfm, VIM). Test the VNF architecture and implementation to assure smooth transition. NFV Test Policy ETSI NFV ISG Compliance (NFV-Co) Selecting Mano Vendors (NFV-Ven) NFV Staging Environment (NFV-SE) MANO Testing (MANO-T) VNF Migration (VNFM-T) 18
- 18. QualiTest NFV validation process Assure full compliance to ESTI ISG Standards ETSI ISG Standards Test procedure Assuring compliance aims to find the deviations from standards • MANO • VNF/NE • Analyse ISG current standards • Prepare compliance requirements • Analyse each NFV component / int • Define gap points and analyse • Repeat steps 1-6 for each VNF and NE NFV Test Policy ESTI NFV ISG Compliance (NFV-Co) Selecting Mano Vendors (NFV-Ven) NFV Test Environment (NFV-TE) MANO Testing (MANO-T) VNF Migration (VNFM-T) 19
- 19. QualiTest NFV validation process Assure full compliance to NFV requirements for any network equipment provider and streamline the ability of vendors and users to be able to implement compatible NFV solutions. NFV Test Policy ESTI NFV ISG Compliance (NFV-Co) Selecting Mano Vendors (NFV-Ven) NFV Test Environment (NFV-TE) MANO Testing (MANO-T) VNF Migration (VNFM-T) ETSI specifications on Network Functions Virtualization • NFV Infrastructure; Hypervisor Domain • NFV Resiliency Requirements • NFV Architectural Framework • NFV Terminology for Main Concepts in NFV • NFV Infrastructure; Compute Domain • NFV Infrastructure; Network Domain • NFV Service Quality Metrics • NFV Management and Orchestration • NFV Virtual Network Functions Architecture • NFV Security; Security and Trust Guidance • NFV Performance & Portability Best Practices • NFV Proofs of Concept; Framework • NFV Infrastructure; Methodology to describe Interfaces and Abstractions • NFV Security; Problem Statement • NFV Use Cases • NFV Virtualization Requirements 20
- 20. QualiTest NFV validation process Build a comprehensive Test Environment, including Control plane and Data plane workload simulation tools, network analytics, QoS and QoE monitoring probes. NFV Test Policy ESTI NFV ISG Compliance (NFV-Co) Selecting Mano Vendors (NFV-Ven) NFV Test Environment (NFV-TE) MANO Testing (MANO-T) VNF Migration (VNFM-T) NFV provides great flexibility: • Build parallel test environments • Scale up and down compute resources • Copy any existing environment Environment build guidelines: • Management and Orchestration should be similar to the production • NS and VNF Instantiation should be similar to production environment • Same level of resources should be allocated. Test Appliances: • Simulate workload traffic on the client / server • Simulate both data plane & control plane traffic • Measure key metrics data & control plane Virtual Appliances vs. Physical Test Appliances: • Virtual test appliances are more cost effective • Physical test appliances are recommended for highest levels of data- plane performance or microsecond-level timing accuracy 21
- 21. QualiTest NFV validation process Test management and Orchestration prior to VNF migration, including each component (e.g. Orchestrator, VNFM, VIM). NFV Test Policy ESTI NFV ISG Compliance (NFV-Co) Selecting Mano Vendors (NFV-Ven) NFV Test Environment (NFV-TE) MANO Testing (MANO-T) VNF Migration (VNFM-T) The first step is to test NFV Management and Orchestration Architectural Framework NFV Orchestrator VNF Manager (VIM) and NFV Infrastructure (NFVI) VIM– VNF Manager (Vi-Vnfm) Orchestrator – VIM (Or-Vi) The VNF is deployed as one or more Virtual Machines; src ETSI GS NFV 002 v1.2.1 22
- 22. QualiTest NFV validation process Test the VNF architecture and implementation to assure smooth transition. NFV Test Policy ESTI NFV ISG Compliance (NFV-Co) Selecting Mano Vendors (NFV-Ven) NFV Test Environment (NFV-TE) MANO Testing (MANO-T) VNF Migration (VNFM-T) Static testing VNF-Mano Integration Conformance testing VNF Internal functionality Scalability/Performance End-to-end/system test 23
- 23. Agenda Risks in testing NFV & SDN Challenges with NFV – what, when and why to verify with SDN/NFV? Summary & Questions Creating an NFV test strategy Building SDN / NFV assurance process Introduction 24
- 24. NFV framework – Test areas Overall, the NFV framework consists of three main components. 1. Virtualized Network Functions (VNF) 2.NFV Infrastructure (NFVI) 3. Network FunctionsVirtualization Management and Orchestration Architectural Framework (NFV-MANO Architectural Framework) 25
- 25. Example: MMSVNFTest Strategy Multimedia Messaging Service (MMS) is similar to the Short Message Service (SMS) in that a message is passed between an Originating Party (OP) and a Destination Party or Parties (DP). The core Network Element is the Multimedia Messaging Service Centre (MMSC) which acts as a central store and forward server between the OP and DP, as shown below. 26
- 26. MMSVNFTest strategy - Approach | Static testing (drive defect curve) | Scope (requirements/functionality ofVNF) | Shape test (risk based / ISO9126) | Test environment (NFV-TE): | Simulation tools | Network analytics | QoS / QoE probes | Onboarding Automated Certification Process. 27
- 27. Test Phases – FAT and NFVI int Factory AcceptanceTesting Compliance testing for eachVNF introduced to the network: • Step 1: Analyze ISG current standards • Step 2: Prepare compliance requirements • Step 3: Analyze each NFV component and interfaces • Step 4: Define gap points and analyze • Step 5: Prepare correction plan • Step 6: Monitor correction plan execution • Step 7: Repeat steps 1-6 for eachVNF and NE Requirements coverage NFVI –test facility integration Ensure full validation against NFVI in labs 28
- 28. VNF compliance and End2end testing VNF compliance testing: VNF Compliance Criteria – functional criteria: • VNF Design Patterns • VNF Update and Upgrade • VNF's Properties • VNFTopological Characteristics • VNF States andTransitions • VNF Fault Management. End to end testing: • Test service end to end to ensure full integration • TestingVNF performance together with otherVNFs & nonVNF • Validate that introducing the newVNF doesn’t affect otherVNF’s functionality and performance.
- 29. MMSVNFTest strategy – example Metrics 30 1 NFV Service Metrics VM Provisioning Latency, instantiation latency. Time between VM instantiation and first available packet VM Stall (event duration and frequency) VM Scheduling Latency 2 QOS and Data-plane metrics: Latency on each of tens of thousands of data streams Throughput and forwarding rate Frame loss rate Packet-delay variation and short-term average latency Dropped frames and error frames Service Disruption Time for Fail-over Convergence 3 QoE and Control-plane metrics: HTTP: page load time, load time variance Video: MOS-AV score, range = 2-5 with 5 being the best HTML5 video - AS score, 100 % score as the maximum Direct metrics: Peak Signal to Noise Ratio (PSNR) Structural Similarity (SSIM) - compare the original image with the received image Video Quality Metric (VQM) Mean Opinion Score (MOS) - This metric combines delays, perceived jitter at application layer, codec used for communication and packet loss at application layer Indirect metrics: Startup time: Time difference between sending the request for content and the time when the user actually received the content Delivery synchronization - In a multicast many-to-many scenario it is important that the content is received by all participants at the same time. Freshness: The time difference between the time when the content is actually generated and the time when the users receives it. Blocking: When the buffers on the receiver are empty and the user has to wait for content. Connections establishment rate, and transactions per second Total number of connections, round trip time and goodput
- 30. Non Functional -Load and Migration Load / Performance • Ensure auto-scale works properly • Ensure resources consumption is efficient • Measure performance meets SLA in high performance workloads Failover Convergence testing • Convergence measurement involves the measurement of processing time of the trigger event in the control plane and the traffic switchover time. • It is important in a multipleVNF deployment scenario that the convergence time of anyVNF is not impacted by the otherVNFs on the same physical server. Level % of BHTR Used for Low 5% Functional Testing Average 50% Scalability, on-going, fail- over Busy 100% Performance Stress 150% Stress Testing, auto-scaling 31
- 31. Test strategy goals | Build an NFV ecosystem | Drive quality and governance | Continuous integration model | Reduce cost of integration 32
- 32. Agenda NFV – Present & Future Risks and Challenges in NFV Testing Summary & Questions Creating an NFV test strategy Building SDN / NFV Assurance Process Introduction 33
- 33. Summary | NFV allows agile real-time shifting of assets to where they are needed | Testing, in the new world of NFV, changes dramatically and becomes a core discipline that is central to the overall goal of pleasing customers in a cost-effective way | NFV –testing is the change agent from the physical networks to the virtualized ones | NFV –Testing challenges deep interdependencies and lightning fast technology introductions; hence it becomes the quality gatekeeper for NFV | If an operator is to find benefits in the new technology, they primarily want reliable services, at the right price, supported by strong SLAs. | The move to a software-centric architecture means becoming more software-focused (skills pivot) | Move towards network DevOps 34
- 34. Q&A 35
Editor's Notes
- #2: Benny intro says it recorded.. We have a good count of people registered and 100+ on the call today. Then we welcome people, good morning.afternoon,evening, depending on where you are in the world. Before we dive into presentation, i would like to introduce myself and colleague alex. We invite you ask questions via the goto webinar....
- #4: Read the Agenda
- #5: Read the Agenda
- #8: Read the Agenda
- #9: Ami to divide
- #10: Ami to divide
- #11: Benny to create this into 2 slides.
- #12: Read the Agenda
- #13: Benny to update!!!
- #15: HANDOVER TO BRYON SLIDE
- #16: Today, it is clear that the legacy hardware approach cannot cost-effectively handle the huge forecasted growth in data traffic. At MWC said increase x7 Two software technologies are driving the transition: (SDN) (NFV) If an operator is to find benefits in the new technology, they primarily want reliable services, at the right price, supported by strong SLAs. And overall new technology cannot be adopted at the expense of the quality of service (QoS), and if customers do notice a disruption in service, they could seek to renegotiate their contracts. Next I will explore how best to migrate to an SDN/NFV environment, and the assurance process you should look to put in place. This is mainly focused on the NFV environment building on the ETSI standards that have been developed...
- #17: QualiTest aims to create a unified end-to-end approach to service assurance and network performance While new technology is obviously a big part of this transformation, one of our biggest challenges most companies will have will be the cultural shift. The move to a software-centric architecture means becoming more software focused. Companies will need to make a "skills pivot" QualiTest help our customers transform their physical network into a visualized network, with innovative strategies
- #18: The QualiTest validation process is segment into 6 key sub-processes: This approach has been established around the guidelines of the ETSI NFV standards. So I am now going to talk you through some of these key processes.
- #19: The goal of the ETSI effort is to reduce the cost of telecom network infrastructure by allowing the appropriate functions to run on a common, commodity platform that hosts the necessary virtualized environments. The first approach (NFV-Co) has been established around the guidelines of the ETSI NFV standards, Really assuring compliance aims to find the deviations from the standards. Compliance is performed on two levels: MANO VNF/NE Typical test procedure we recommend.
- #20: The next process is NFV-VEN – compliance for vendors. This looks to assure full compliance to NFV requirements for any network equipment provider and really streamline the ability of vendors and users to be able to implement compatible NFV solutions Broken down into 2 key sections: typical test scenarios: Example =MANO Compliance Criteria Such as: VNF Descriptor format – is matching standard, in correct format VNF redundancy model – what is the redundancy model. Validation of the lifecycle management request VNF Package – ensure its build correctly and loads. 2nd is = VNF Compliance Criteria VNF Design Patterns o VNF Internal Structure o VNF Load Balancing Models o VNF Policy Management o Migration operations o VNF Instance Configuration If anyone wants the full list of proposed critieria, please see the NFV testing best practices on our website. Qualitest
- #21: Next process NFV-TE is around NFV test environments.. NFV provides great flexibility when building Test Environment. Using NFV, the end user can: Build parallel test environments to address different needs Scale up and down compute resources Copy any existing environment – live ../ dev / preprod When building Test Environment for NFV, the following guidelines should be kept: Management and Orchestration should be similar to the production, including all elements of NFV (MANO, VNFM, VIM, NFVI) NS and VNF Instantiation should be similar to production environment (hypervisors, computing, storage and network resources) -same level of resources should be allocated. a load environment should be established when running performance and scalability testing Test Appliances Test Appliances are required in order to: Simulate valid workload traffic on the client and server Simulate both data plane and control plane traffic Measure key metrics, both data plane metrics and control plane metrics Virtual Test Appliances vs. Physical Test Appliances Existing physical test appliances may be used, but most test equipment vendors have developed virtual test appliances that work on separated VMs in the NFV Test Environment. Virtual test appliances are usually much more cost effective. Physical test appliances are mainly recommended for the testing virtual environments that require the highest levels of data-plane performance
- #22: This process MANO-T looks at - Test management and Orchestration prior to VNF migration, including each component (e.g. Orchestrator, Vnfm, VIM) So how would the acceptance process look for this area, we have broken this down into some key sections: Type of testing for each area: briefly touch on some of the type of tests: 1 NFV Orchestrator On-boarding of new Network Service (NS) and VNF Packages Setup a Network Service i.e. create a Network Service using the NS on-boarding artefacts Query VNF – retrieve VNF instance state and attributes Disable, enable, update, query, delete VNF package 2 VNF Manager: VNF instance software update/upgrade VNF instance modification VNF instance scaling out/in and up/down 3 Virtualised Infrastructure Manager (VIM) and NFV Infrastructure (NFVI): Resource catalog management NFV Infrastructure faults collection and remediation Create, delete, list, and query VM storage 4 Virtualised Infrastructure Manager – VNF Manager (Vi-Vnfm) Resource allocation requests Virtualised hardware resource configuration 5 Orchestrator – Virtualised Infrastructure Manager (Or-Vi) Resource reservation and allocation Virtualised hardware resource configuration
- #23: Finally – Migration Static testing – validate requirements.. Each VNF deployed in the NFV environment requires independent testing. VNF-Mano Integration: eg VNF Component Re-Use / VNF Update and Upgrade The key is… Testing VNF performance together with other VNFs and validate the introduction of the new VNF, end 2 end testing.
- #25: Virtualized Network Functions (VNF) are software implementations of network functions NFV Infrastructure (NFVI) is the totality of all hardware and software components which build up the environment in which VNFs are deployed. Network Functions Virtualization Management and Orchestration Architectural Framework (NFV-MANO Architectural Framework) these functional blocks exchange information for the purpose of managing and orchestrating NFVI and VNFs.
- #27: Mention QualiSystems
- #28: Fat – factory acceptance testing – so this is the testing I expect vendor to perform
- #29: VNF Design Patterns VNF Internal Structure VNF Instantiation VNFC States VNF Load Balancing Models VNF Scaling Models VNF Component Re-Use VNF Update and Upgrade Automatic procedure Control Update and Upgrade process Requesting virtual resources Roll-back VNF's Properties Hardware Independence Virtualization and Container Awareness Elasticity VOID VNF Policy Management Migration operations VNF State VNF Internal Structure Reliability Location Awareness Application Management Diversity and Evolution of VNF Properties VNF Topological Characteristics Deployment Behaviour Virtualisation containers NFVI Resources Components and Relationship Location VNF States and Transitions States and Transitions as Architectural Patterns The VNF Descriptor VNF Instantiation VNFC Instantiation VNFC Instance Termination VNF Instance Termination
- #31: busy hours traffic rates (BHTR)
- #35: Bryon do his questions first Then benny.. Questions Then benny wrap it up…. Thankyou for attending, we will send out a recording of the webinar. Please let us know if you have any requirements we can support.