Psi multi accessgateway_casestudy
- 1. PrimeSoft Solutions, Inc. Multi Access Gateway: Case Study August 26, 2010 This document contains information proprietary to PrimeSoft Solutions, Inc. Reproduction, release, or use of the information contained in this document is strictly prohibited without prior written permission from PrimeSoft Solutions, Inc.
- 2. Multi Access Gateway : Case Study Multi Access Gateway Client Offshore Development Center (ODC) This Santa Clara, CA, company develops and markets industry's first multi-access convergence gateway designed for stateful management of tens of thousands of secure, reliable, high-quality subscriber sessions over multiple network access types including Wi- Fi and WiMAX. To date, the company has raised over $50M and is backed by Sequoia Capital and Kleiner Perkins among other venture capital firms. Relationship Model PrimeSoft is the chosen Outsourced Product Development partner for the client. PrimeSoft developed this relationship by providing the following at our offshore development center 1. Building testing, automation and development teams 2. Identifying an Project Manager 3. Setting a Lab and setting it up with Client specific software and hardware components 4. Providing supported from HR, Admin and Facilities Management functions The offshore team consists of 18 engineering resources divided into QA, Automation and Development. The teams function as an extension to the client’s team in USA, and participate in various projects during a release. Weekly status reports are provided by the Project Manager on all active offshore projects, followed by conference calls to discuss any open issues. UMA Handset Simulator Technology Brief Unlicensed Mobile Access (UMA) technology provides access to GSM and GPRS mobile services over unlicensed spectrum technologies, including Bluetooth and 802.11. By deploying UMA technology, service providers can enable subscribers to roam and perform handover between cellular networks and unlicensed wireless networks using dual-mode mobile handsets. With UMA, subscribers receive a consistent user experience for their mobile voice and data services as they transition between networks. Project Objectives The objective of this project was to develop a UMA Handset Simulator. UMA simulator came out as a requirement from the development and automation engineers to help them perform unit testing and automation of UMA functionality. PrimeSoft Confidential Page 2 of 11
- 3. Multi Access Gateway : Case Study UMA simulator was required support functional, to help automate more than 90% of all functional tests and help development engineers run pre-commit unit tests to ensure basic functionality. PrimeSoft Confidential Page 3 of 11
- 4. Multi Access Gateway : Case Study Our Approach The Team PrimeSoft identified a technical lead, development & test engineers to staff this project. The team had prior experience in working with telecom and data communications protocols and call processing. Project Execution Requirements Analysis A team from PrimeSoft traveled to US client site for gathering requirements. The team had brainstorming sessions with client’s technical team and end users of the UMA simulator. The requirements were documented in a Functional Requirements Document Template and shared with all the stake holders, followed by a formal review. Review comments were incorporated and the updated document was checked into CVS and posted on internal Wiki. Functional Specifications & Software Design Functional Requirements Document was an input for developing a Functional Specifications Document. The HLD (High Level Design) and LLD (Low Level Design) documents were developed based on the functional specifications. The design document identified modules, interfaces, and presented pseudo code. Coding, Unit Testing & Acceptance Testing Coding of the tool involved setting up the environment and developing code based on the low level design. The code was unit tested by development engineers. System testing of the simulator was carried out by an independent team that developed system test plan document based on the requirements document and executed multiple test cycles and logged bugs using Bugzilla. User acceptance test plan was developed by PrimeSoft QA team and shared with the client. Acceptance testing was performed jointly by PrimeSoft and client teams. Salient Features Object oriented Design approach provides well defined programming interfaces to the scripting layer Provides interfaces to build and parse and UMA and GSM Layer3 messages Multithreaded implementation for RTP sends and receives for multiple handset instances Python extensions provided for abstracting the complex scripting functionality and reduce scripting effort Supports complex handover scenarios by simulating two UMA connections from a single Handset PrimeSoft Confidential Page 4 of 11
- 5. Multi Access Gateway : Case Study Project Deliverables Deliverable Details Functional Requiremnts Document Functional Specifications Document Documents Design Document Users Guide Release Notes UMA Simulator Source Code (C/C++, Makefiles, Shell Source Code Scripts) Unit Test Plan Test Documents Unit Test Matrix Bugs BUGS Logged into Bugzilla Sample Scripts Sample Automated Scripts Users Guide Automation suite and scripts users guide Traces all the requirements against Functional Requirements Traceability Specifications, Design, System Testing, Unit Testing Matrix and Users Guide Tools & Technologies Title Details Protocols UMA, GSM, TCP/IP Operating Systems Redhat Linux Programming & Scripting C, C++, Python, Shell Scripts, Makefiles Configuration Management CVS PrimeSoft Confidential Page 5 of 11
- 6. Multi Access Gateway : Case Study Multi Access Gateway Testing PrimeSoft houses the largest testing team for our client. Our testing team is involved in the entire test life cycle of the product including test strategy development, test plans development, tests cases development, manual testing, filing bugs and running regression cycles. Project Objectives The main objective of the testing group is to work as an extension to client’s onsite engineering teams, and be responsible for test and automation activities involved in the project. The offshore team is responsible of the following project activities 1. Defining QA and Automation Strategy for a identified release 2. Developing QA plan for the identified release 3. Developing Test Plans and Test Case documents based on a. Marketing and Product Requirements Documents b. Functional Specifications c. Design Documents d. RFC’s Supported 4. Performing Functional, System, Sanity, Regression, Load and Stress, Security and Interop testing 5. Identifying and logging bugs, coordinating with developers in re-producing and analyzing bugs 6. Developing test matrix and reporting on manual tests progress on a daily/weekly basis Testing Projects PrimeSoft team worked with client on multiple releases of this product and was involved in testing and owning QA and Automation for the following modules IPSec IKE v2, IKEv1 MOBIKE (Mobile IKE) RIP/OSPF VLAN (Virtual LAN) Session Initiation Protocol (SIP) QoS (Quality of Service) AAA (Authentication, Authorization and Accounting) and RADIUS Change of Authorization (CoA) Internet Protocol Slow Path PrimeSoft Confidential Page 6 of 11
- 7. Multi Access Gateway : Case Study Point to Point over Ethernet (PPPoE) Dynamic Host Control Protocol (DHCP) PDIF (Protocol Data Interworking Function) CDR (Call Detail Records) IP-IP Encapsulation DOS Attacks ACL (Access Control List) SNMP (Simple Network Management Protocol) RTC (Real Time Charging) HA (High Availability) System Testing Performance Testing Our Approach The Team We created a dedicated test team who were technically competent in networking technologies. Project Manager and a core group of engineers started the project by performing manual testing on one of the releases and logged bugs. The team developed its competency on the product by organizing Internal training seminars Technical hands on sessions Developing internal wiki pages Going through past bugs reports Communicating with developers and PLM teams Test Beds A Site-to-Site VPN was setup between PrimeSoft, and Client location in USA, allowing engineers to reserve, configure and access remote test setups. PrimeSoft also has a lab setup with client specific software and hardware available locally for engineers. Project Execution Test Projects execution followed the following phases. Analysis The aim of this phase was to technically be competent in the technology and enhance product understanding and gather all the information necessary to develop a test plan document and completely define the scope of testing activities. The team initiated the projects by going through RFC’s and product documentation including, Requirements, Functional Specifications and High Level Design. Email PrimeSoft Confidential Page 7 of 11
- 8. Multi Access Gateway : Case Study exchanges and conference calls were conducted to understand the nuances of implementation on client’s product. Test Plan Development Test plan development involved, identification and documenting 1. Test Topology 2. Software and Hardware Tools 3. Entry and Exit Criteria 4. Conformance scenarios from RFC’s 5. Test cases against Requirements 6. CLI Related test cases 7. Functional test cases 8. Negative test cases 9. End-to-End tests 10. SNMP related test cases 11. High availability test cases Internal & External Reviews Internal peer reviews (board review) is performed and review comments are updated into the document. An internal review checklist is used to ensure the quality of the test plans is consistent. The update document is sent out for an external/client review followed by a walk through of the test plan through a conference call. Review comments received from architects, designers, implementers and PLM are consolidated and updated into the document. Updated document is checked into CVS and posted on internal Wiki. Manual Test Cycle Manual test cycles involved setting up test bed and executing test cases against the SUT (System Under Test) and tracking progress. A Test Matrix is filled for each cycle of manual test cycles across multiple builds, the test matrix captures testing results documented as PASS, FAIL or BLOCKED. The matrix also provides key metrics of the testing cycle including testing effort, bugs logged and passes percentage. Bugs were logged into client maintained bugzilla, with information including 1. Bug description 2. Build version 3. Steps to reproduce PrimeSoft Confidential Page 8 of 11
- 9. Multi Access Gateway : Case Study 4. Logs and core files path 5. Initial diagnosis findings 6. Observation and Comments Tools & Technologies Title Detail Protocols IPSec, IKE v1/v2, AAA, CoA, PPPoE, DHCP, SIP, SNMP Operating Systems REDHAT Linux, QNX Scripting Python, Shell Scripts Ethereal (Network Analyzer), Scapy (Packet Generator), Tools Silvercreek, SIPp, QA Robot, Smartbits, Landslide Avalanche, IXIA, IKEv1/v2 clients, DOS Attacks tools Configuration Management CVS, Bugzilla Project Deliverables Deliverable Details Test Plan Test Plan Document Test Matrix Test Cycle Summary in a Test Matrix Document Bugs BUGS Logged into Bugzilla Checklist Test Plan Release Checklist Requirements Traceability Traces all the requirements against the test cases and Matrix bugs Multi Access Gateway Automation With a growing feature list in the product we sent out with a target of automating 50% of our test cases, and this could significantly reduce the manual test cycle timeline. PrimeSoft was involved in test automation from the very beginning and participated in developing an automation framework, developing automated test scripts and suites. Project Objectives The objective of the automation group at PrimeSoft is to Design, Develop and Enhance an Object Oriented Automated Framework Develop automated test scripts Run automated regression on daily/weekly builds PrimeSoft Confidential Page 9 of 11
- 10. Multi Access Gateway : Case Study Develop test suites for pre-commit, sanity, weekly regression and feature specific suites Identify and log bugs Fix scripts and maintain them across release and builds Our Approach The Team We created a dedicated automation group within PrimeSoft who were technically competent in scripting languages and had experience with relevant networking technologies. Framework Enhancements Our team participated in design and development activities of the automation framework. An object oriented framework was designed using Python as the scripting language. Enhancements to the framework were performed and tested (using automated unit tests); and the framework was released to carry out the development of the scripts. Automated Scripts A team of automation engineers was selected to automate the test plans using the enhanced automation framework. Automation engineers focused on developing automated scripts and scripts development involved the following scripts Manual testing Develop and automated test script Running the test on a stable build to verify the behavior of the script Fixing all issues with the scripts Adding the script in a suite and running the suite Every developed script went through a through review with at least two reviewers and focused on functionality and coding guidelines. Functional Review: Review focused on the functionality of the scripts and compared them with what is described in the test cases Coding Guidelines Review: The review focused on usability, maintainability and adherence to coding guidelines Checked into CVS Project Salient Features Python based Object Oriented Automation Framework o Can load/clear and check configuration on system under test o Can load/clear and check configuration on Juniper Netscreen hardware o Can initiate IKE v1 and v2 sessions from end clients o Can initiate PPPoE sessions from Linux based end clients o Can handle High Availability cases Exposes library functions for script developers PrimeSoft Confidential Page 10 of 11
- 11. Multi Access Gateway : Case Study Automated Tests o Topology File: Can be changed to point to a new topology o Configuration File: Captures all common configuration for a test o Suite file: Can include multiple test cases to be run in a suite Tools & Technologies Title Detail SIP, SDP, RTP Protocols IPSec IKE v1/v2 RIP, OSPF, BGP, DHCP, DNS, Radius, TCP/IP, PPPoE Clients Hardware/Software Netscreen 5GT, XpressVPN, PPPoE Linux Client, SIPp Used test tool, Cisco Terminal Server Operating Systems Redhat Linux, QNX Programming & Scripting Python, Perl, Shell Scripts Configuration Management CVS, Bugzilla Project Deliverables Deliverable Details Automated Scripts Developed as part of each Scripts functionality Test Suits Automated test suites for various types of testing Framework Libraries Framework Libraries and Documentation Checklist Scripts Development and Review Checklist Requirements Traceability Traces all the requirements against the test cases and Matrix bugs PrimeSoft Confidential Page 11 of 11