Infrastructure Testing: The Ultimate “Shift Left”
- 1. T18 Special Topics 5/11/17 13:30 Infrastructure Testing: The Ultimate “Shift Left” Presented by: Carl Delmolino, Travelers Insurance Hitesh Patel, Infosys Ltd. Travelers Insurance Brought to you by: 350 Corporate Way, Suite 400, Orange Park, FL 32073 888-‐-‐-‐268-‐-‐-‐8770 ·∙·∙ 904-‐-‐-‐278-‐-‐-‐0524 -‐ info@techwell.com -‐ http://www.starwest.techwell.com/
- 2. Carl Delmolino Carl Delmolino is the director of quality assurance within the infrastructure technology division of Travelers Insurance. Before going off the rails to build an infrastructure testing practice, Carl spent more than twenty years in application software testing, leading BA, QA, and automated test teams in the personal and commercial areas at Travelers. As the creator of a test automation community of practice and a past chairperson of Traveler's Quality Assurance Roundtable, Carl is a proponent of enterprise QA collaboration. Hitesh Patel Hitesh Patel is a group project manager at Infosys Ltd. and co-‐leads the Infrastructure quality assurance team with Carl. Hitesh has experience in various QA roles in his eighteen years in planning, executing, and delivering on-‐time, end-‐to-‐ end testing solutions. He is passionate about quality and providing innovative QA solutions for the challenges faced by testing teams.
- 3. Infrastructure Tes-ng The Ul-mate Shi5 Le5 StarEast 2017, Thursday, May 11th 1:30pm – T18 1
- 4. Carl Delmolino Travelers Insurance Hitesh Patel Infosys Presenters: 2
- 5. Infrastructure Failures DALLAS -- Southwest Airlines struggled to recover Thursday from a massive computer outage, as several hundred more flights were canceled or delayed around the country. Airline executives said a router breakdown set off a chain of failures in critical technology systems a day earlier. Backup systems didn't work as expected, they said. 3
- 6. Infrastructure Failures John HackeN, HSBC COO said: "We profoundly apologize for any inconvenience this has caused. "There is a complex technical issue with our internet banking systems, and our IT team has been working non-‐stop since yesterday morning to find a solu-on. This has involved many tests, diagnos-cs and trial runs. We are geZng closer to solving the problem, but are not there yet. We can, however, confirm this is not a cyber-‐aNack or any other malicious act." Karl Flinders, Emea Content Editor, Computer Weekly 05 Jan 2016 15:20 4
- 7. Infrastructure Failures “Airlines have spent heavily to introduce new features such as automated check-‐in kiosks, real-‐-me luggage tracking and slick mobile apps. But they have avoided the steep cost of rebuilding their reserva-ons systems from the ground up, former airline execu-ves said.” A failure to ensure backup power for some of its computer servers led to the meltdown at Delta Air Lines this week that forced more than 2,000 flight cancella-ons through Wednesday a5ernoon, the airline’s top execu-ve said. 5
- 8. Infrastructure Failures Amazon's cloud service has outage, disrup-ng sites SAN FRANCISCO — Por-ons of Amazon Web Services, the na-on's largest cloud compu-ng company, went offline Tuesday a5ernoon, affec-ng millions of companies across the United States. "This is a preNy big outage," said Dave BartoleZ, a cloud analyst with Forrester. "AWS had not had a lot of outages and when they happen, they're famous. People s-ll talk about the one in September of 2015 that lasted five hours," he said. The outage appeared to have begun around 12:35 pm ET, according to Catchpoint Systems, a digital experience monitoring company. It was centered in AWS' S3 storage system on the east coast. Elizabeth Weise , USATODAY , KARE 3:20 PM. CST February 28, 2017 6
- 9. Why is Infrastructure Tes-ng needed? Schedule slippage due to last minute issues Diverse and rapidly changing landscape Lack of tes-ng rigor and tes-ng skills Missing industry best prac-ces and standards High cost of Infrastructure projects 7
- 10. How does Infrastructure Tes-ng shi5 quality le5? Infrastructure Tes-ng achieves “Ul-mate Shi5 Le5” status when it… • Employs a methodical quality approach to the design, evalua-on, standing up, and maintenance of various infrastructure. • Takes place before any related development of applica-ons on a given infrastructure begins. • Enables subsequent so5ware development on an evaluated founda-on. • Helps segregate issues found in So5ware development to the so5ware itself, rather than Infrastructure issues being suspect. • Exists!!!! What is “Shi5 Le5”? • An approach used in so5ware tes-ng and system tes-ng in which tes-ng is performed earlier in the lifecycle (i.e., moved le5 on the project -meline). • An approach to accelerate so5ware tes-ng and facilitate development by moving the tes-ng process to an earlier point in the development cycle. 8 Prod Live UAT Perf Test Func-onal Test Unit Test Req/Design Valida-on Infrastructure Tes-ng
- 11. What is Infrastructure? • Computer Hardware -‐ IBM, HP, Linux, Apple, Switches, Hubs, Servers, Virtual, etc • End User Compu-ng -‐ Desktop and Mobile technologies • Middleware -‐ CICS, IMS, MQ, Jboss, .Net, • Internet Plauorms -‐ Apache, MS IIS, .Net, Unix, Cisco, Java • Content & Collabora-on Services -‐ Email, Messaging, Fax, KX, Audio/Visual, SP, Filenet • Opera-ng Systems -‐ MS Windows, Unix, Linux, Mac OS • Enterprise Applica-ons -‐ SAP, Oracle, MS Produc-vity, Source Code Mgmt, Customer Rela-onship Mgmt, Employee Mgmt, etc. • Database Administra-on • Networking/Telecommunica-ons -‐ Windows Server, Linux, Cisco, VOIP, VPN, Wireless • Mainframe • Voice & Service Center Technology • Data Management & Storage -‐ IBM DB2, SQL Server, Oracle • Data Centers/Facili-es -‐ Print, Disaster Recovery • IT Security 9
- 12. When to do Infrastructure Tes-ng Infrastructure Acquisi-on • New infrastructure design and build • New infrastructure func-onality and integra-on • New scalable infrastructure is planned Infrastructure Migra-on • Standardiza-on/ Consolida-on • Migra-ons – Infrastructure, applica-on • Data Center migra-ons • Database migra-on • Cloud transi-on Infrastructure BAU changes • Environment provisioning • Patch upgrades and updates • OS or Database changes • Environment standardiza-on New Laptops New phones New 3rd party tools Windows 10 W2K3 to W2K8 W2K8 to W2K12 New patches Security fixes Upgrades Examples: 10
- 13. What is Infrastructure Tes-ng? • Environment Verifica-on Tes-ng • Deployment / Installa-on / Uninstalla-on / Fallback Tes-ng • Hardware / Device Tes-ng • Connec-vity Tes-ng • Opera-onal Readiness Tes-ng • Disaster Recovery Tes-ng • Image Tes-ng (Client / Server) Core Infrastructure • Configura-on Tes-ng • System Integra-on Tes-ng • Volume & Performance Tes-ng • Security Tes-ng • MDM Tes-ng • Component Tes-ng 3rd Party App Integra-on • Smoke/Sanity Tes-ng • Func-onality / System Tes-ng • Integra-on Tes-ng • Regression Tes-ng • Performance Tes-ng • User Acceptance Tes-ng • Usability Tes-ng Home Grown App’s 11
- 14. Examples of Infrastructure Tes-ng…… Client/Server • Test and Cer-fy the quality of Applica-ons, Servers, Desktops, Patches ,OS, Image and Hardware. • Test the infrastructure components to reduce the possible down-me impact in produc-on and help to improve the performance of the infrastructure. Cloud • Test and cer-fy the Private Cloud -‐ IaaS service offerings for Release and Itera-ons. • Self service portal – Catalog service request form. • Build -‐ VM lifecycle management – VM Provisioning/ De-‐Provisioning • VM Templates– OS images and Packages • Test and Cer-fy the Cloud automa-ons and Orchestra-ons. Data Center Migra3on • Test and cer-fy the migra-on of DR capabili-es from DR datacenter to new data center. • Test the different migra-on methodologies such as New server build, V2V, P2P, Li5 and shi5. • Test the servers and ensure that there are no changes due to migra-on • Test and ensure no applica-on latency or performance are observed. Mobility • Verify corporate mobile devices are able to manage their corporate emails, calendar , contacts, personal documents, intranet access, text and video messaging. • Verify Mobile Device Mgmt (MDM) func-onality a5er changes (MDM, OS version, App version) FileNet • Ensure Filenet IBM Content Navigator(ICN) func-onali-es are not impacted a5er every bundle releases (consists of IBM updates, Security updates mandated by IS security team) • Control, Maintain and support for future Filenet upgrades. • Ensure Filenet web services are not impacted a5er every bundle release. Test Environment Mgmt • So5ware installa-on • Applica-on configura-on setup and changes • Inventory management • Ac-ve Directory administra-on • Currency maintenance, Problem resolu-on 12
- 15. How to build a Infrastructure Tes-ng prac-ce? Process Metrics & Repor/ng Opera/ng Model Infrastructure Organiza/on Methodology § Weekly / Monthly Report § Produc-vity and quality § Basic Tes-ng metrics § Advanced Tes-ng metrics § Test techniques and methods § Test Case management § Test Data Management § Defect Preven-on § Performance tes-ng § Tools & Automa-on § Test Environment § Technology stack § Sourcing Model § Effort Es-ma-on § Workload Management § Communica-on § Knowledge Management § Structure § Goals & Objec-ves § Collabora-on § People capabili-es § Strategic Alignment § R&R § Quality plan § Opera-ons guide § Test Requirement § Test Strategy & Planning § Change Request § Risk Management 13
- 16. What are the Benefits of Infrastructure Tes-ng? Improved quality of service Higher availability of systems, stable environment Reduced incidents caused by change Reduced opera-onal risks & costs Improved -me to market Improved stability of applica-ons Enhanced custom knowledgebase – faster problem resolu-on 14
- 17. Summary 1. Catastrophic Infrastructure failures are rare but, when they happen, are costly in many ways 2. Smaller infrastructure issues are more frequent and these result in a loss of produc-vity and increased opera-ng cost, which can be cumula-vely significant 3. Infrastructure change is inevitable 4. Infrastructure tes-ng helps you shi5 le5, by finding issues long before other types of tes-ng begin 5. An Infrastructure Tes-ng Team’s job is to iden-fy what, when and how to test 6. SeZng up a Infrastructure tes-ng team will help your organiza-on in the long term
- 18. THANKS 16
- 19. Infrastructure Tes-ng – Appendix 17 1. Server / Client 2. Cloud 3. Data Center Migra-on 4. Disaster Recovery 5. Test Environment Management 6. Mobility 7. Filenet 8. Sharepoint 9. Documaker
- 20. Infrastructure Tes-ng Procedure – Server/Client Test Objec3ve • Test and Cer-fy the quality of Applica-ons, Servers, Desktops, Patches ,OS, Image and Hardware. • Test the infrastructure components to reduce the possible down-me impact in produc-on and help to improve the performance of the infrastructure. • Infrastructure tes-ng helps to ensure an error free environment in produc-on and keeps the environment healthy. • Reduce the security vulnerability risks earlier by tes-ng in QA before releasing to Produc-on. • Ensure fallback plans are tested and ready before the actual implementa-on in produc-on • Provides worry free migra-on of So5ware/Hardware/OS upgrades in infrastructure. Test Procedure • Stakeholder submits the test request with all required informa-on needed for tes-ng • Tester collects the requirements from the request and dra5s a Test plan with an understanding in structure including in-‐scope/out of scope considera-ons, prerequisites, applicable OS, fresh/upgrade scenarios and func-onality tes-ng. • Tested discusses the created Test plan with the Stakeholder and obtains the signoff. • Tester executes the scenarios outline in Test plan on selec-ve target machines and validates the expected checkout results. In case of discrepancies, a defect is logged in the defect tracking tool and tracked un-l closure. • Stakeholder will respond to the defect(s). Tester will retest and verify that defect(s) are fixed. • Tester documents the results obtained during execu-on of test plan. • The procedure/package is cer-fied and approved for implementa-on in produc-on. 18
- 21. Infrastructure Tes-ng Procedure – Server/Client Test Types • Stability Test • Smoke tes-ng • Installa-on / Uninstalla-on tes-ng • Package /Bundle tes-ng • Security Patch tes-ng • Browser /extension tes-ng • Fallback Tes-ng • Integra-on Tes-ng • Func-onality Tes-ng • Hardware Cer-fica-on Benefits • Package methods used for so5ware fresh/upgrade installa-on and configura-on changes are tested on QA servers/clients. This validates the code behavior earlier and thereby reduces the possible impact in produc-on. • Microso5 security patches are released every month and are tested in QA on different opera-ng plauorms such as Windows 7, Windows 10, Windows Server 2008, Windows Server 2012, etc., before rollout to produc-on. • All types of browsers and extensions released by vendor are tested in QA before rollout in produc-on . • Monthly bundles, consis-ng of several standalone packages are tested on servers/clients in the QA environment, providing the ability to iden-fy poten-al risks earlier . • Fallback procedures/packages are tested in QA to ensure back out plans are working and in place before the actual migra-on/deployment in produc-on. 19
- 22. Infrastructure Tes-ng Procedure – Cloud Test Objec3ve • Test and cer-fy the Private Cloud -‐ IaaS service offerings for Release and Itera-ons. • Self service portal – Catalog service request form. • Build -‐ VM lifecycle management – VM Provisioning/ De-‐Provisioning • VM Templates– OS images and Packages • Test and Cer-fy the Cloud automa-ons and Orchestra-ons. Test Procedure • Migrate the templates, Code and Blue prints from DEV to QA Environment. • Configure and validate the Catalogue items, Blue prints, Build Profiles, Proper-es and workflows. • Set up the QA environment to kick off the builds based on the requirements. • Build -‐ Provision the VM’s for OS, Web and DB service offerings. • Test -‐ Perform the regression tes-ng and ensure that required so5ware's are installed. • Cer-fy – Provide the QA sign off to migrate the templates, code and Blue prints to the produc-on environment. Test types and results • Component Tes-ng • System Tes-ng • Installa-on Tes-ng • Uninstalla-on Tes-ng • End to End Tes-ng • Regression Tes-ng Benefits • Error free implementa-on in produc-on environment. • Simplified request form for beNer user experience • Automated test cases reduces the tes-ng cycle and improves the go to market -me. 20
- 23. Infrastructure Tes-ng Procedure – Data center Migra-on Test Objec3ve Ensuring the partnership with Data center team to test & cer-fy the datacenter migra-on and focusing on con-nuous Quality improvements. • Test and cer-fy the migra-on of DR capabili-es from DR datacenter to new data center. • Test the different migra-on methodologies such as New server build, V2V, P2P, Li5 and shi5. • Test the servers and ensure that there are no changes due to migra-on • Test and ensure no applica-on latency or performance are observed. Test Procedure – Pre, Post and Compare • VM / Physical -‐ Test the servers before and a5er the migra-on and ensure that there are no configura-on changes. • New Build – Test and ensure that newly build servers are built based on the exis-ng server requirements. • P2P /Li5 shi5 – Ensure that checklist is followed to physically move server from exis-ng DC to new Datacenter and no changes on configura-on during the physical move. • Produc-on checkouts are performed a5er the successful infrastructure migra-on. • Confirmed that there are no changes on applica-on performance. Test types and results • Installa-on test • Configura-on Test • Cer-fica-on Test • Stability Test • Produc-on checkout Value adds • Iden-fies the poten-al server configura-on mismatches • Iden-fies the server build issues in during the build phase • Improves the Quality of deliverable across the project life cycle 21
- 24. Infrastructure Tes-ng Procedure – Disaster Recovery Test Test Objec3ve Disaster Recovery Tes-ng is to conduct disaster recovery exercises in the DR Data Center that test end-‐to-‐end business func-ons and services • Reduce Overall Risk during disaster. • Maintain and Test Your Disaster Recovery Plan • Improve the confident of business owners • Restore cri-cal business func-ons and mission cri-cal applica-ons • Rapid Response during the failure Test Procedure • Isolated network – Designed to perform the DR test without impac-ng the produc-on. • DR tests are designed to perform the tes-ng on business func-ons or mission cri-cal applica-ons. • Core infrastructure recovery – Bring up and test the core infra services like AD, DNS and DHCP in Isolated network. • Infrastructure recovery – The required infrastructure will be brought up on the isolated network and infrastructure checkout will be performed. • Applica3on recovery – The applica-ons are brought up as required and sanity test will be performed on the recovered applica-ons. Test types and results • Component Tes-ng • Tabletop Test • Dry Run Test • Structured Walkthrough Test • Parallel / Combined Component Tes-ng • Cutover Test Value adds • Improve the recovery -me • Con-nuously improve the RPO and RTO • Automate the recovery plans 22
- 25. Tes3ng Objec3ves • So5ware installa-on • Applica-on configura-on setup and changes • Inventory management • Ac-ve Directory administra-on • Currency maintenance • Problem resolu-on TEMS Roles and Responsibili3es 1. Inventory (Monthly) a. Asset management b. Scans & reports c. Server, client, and storage costs d. LAN Center & domain breakdown of QA technology resources 2. Administra3on (Day-‐to-‐day) a. Manage system access & permissions for QA technology resources b. Domains including IP Address configura-ons, Sites c. Client lab d. Environment incidents e. Monitors & alerts f. Test bed designa-ons for servers 3. Provisioning (On demand) a. Submit server build requests b. Build client machines c. Implementa-on target machine groups 4. Compliance (Monthly) a. Schedule backfills b. Execute and review compliance jobs c. Report discrepancies 5. Environment backfills & currency (Ongoing) a. Execute IQA backfill requests b. Define targets c. Maintain machine groups 6. Schedule development (Annual) a. Environment backfills including gold server images b. Develop backfill schedules for all QA technology resources c. Schedule server build image updates for monthly infra releases – Windows & Unix 7. Repor3ng (Monthly) a. Provisioning for all QA technology resources b. Project ac-vity c. QA alerts with a breakdown by classifica-on (warning, low, med, high) d. Backfill volumes and work effort 8. Monitors & Alerts a. QA environment alert implementa-on b. Produc-on LFA/ELA/other… alerts for QA tes-ng in QA environment 9. Ac3ve Directory health checks (Annual) a. Par-cipate in Microso5 review sessions b. Implement Microso5 produc-on recommenda-ons in QA c. Resolve ac-ve directory issues 10. Manage server and client environment issues a. Resolve environmental incidents for all technology resources b. Designate environment incidents to owning poruolios as necessary Test Environment Management 23
- 26. Infrastructure Tes-ng Procedure – Mobility Test Objec3ve • Ensure employees having corporates iPhone are able to manage their corporate emails, Calendar , contacts , personal documents , Access intranet sites , manage Skype for business on mobile. • Ensure MDM solu-on is correctly managing corporate devices. • Ensure MDM func-onality works a5er any change( AirWatch update , IOS version , Applica-on version etc.) Test Procedure • Receive Test request from Project team • Create test strategy if required , Test cases etc. • Prepare devices for tes-ng • Verify func-onality in lower environment and provide sign-‐off • PROD change and checkout • Tes-ng on IOS versions , IOS beta versions , Applica-on beta version in PROD region Test types and results • Configura3on tes3ng – Test configura-on of servers , Enterprise systems integra-on , Ac-ve Directory Sync • Enrollment tes3ng -‐ Test Device enrollment in AirWatch , test user authen-ca-on and policy or terms tes-ng • Security Tes3ng – Test all restric-ons applied on managed devices as per corporate or industry standards , Compliance tes-ng • Profiles/Policies tes3ng – Test policies for app user creden-als for accessing enterprise resources , Test profiles applied to managed devices – e.g. – Email profile , VPN profile , Wi-‐Fi profile etc. • Distribu3on tes3ng – Test mul-ple devices to ensure internal/public/purchased apps installed/distributed through company app catalogue • Monitor and tes3ng – Test console/dashboard to monitor devices a5er distribu-on of apps , Test push no-fica-on , test commands for managed devices • Func3onality tes3ng -‐ Test Corporate Emails , contacts , calendar , Content management , browser to access intranet sites etc. on managed devices , Skype for Business mobile • Backup and restore tes3ng – Test iTunes to ensure device backup and restore works on latest IOS version • Regression tes3ng – Test all exis-ng test scenarios with every IOS release , Environment updates etc. , Execute few cri-cal test cases everyday in PROD region to iden-fy outage if any • Beta tes3ng – Test on IOS beta version , AirWatch beta applica-ons to ensure func-onality works on beta versions 24
- 27. Infrastructure Tes-ng Procedure – SharePoint Test Objec3ve • Ensure SharePoint func-onality is not impacted a5er Microso5 patches are applied to the environment. • Control, Maintain and support for future SharePoint upgrades(e.g SharePoint 2013 to SharePoint 2016). • Ensure SharePoint func-onali-es are not impacted a5er every enhancement. • Ensure that Company branding (look and feel of the SharePoint portal) is not affected by any of the above men-oned change. Test Procedure • Receive Test request from SharePoint Dev Ops team or engage in the project through the Demand Management process. • Schedule a mee-ng with PM, BA, Engineers to discuss the requirements. • Define Test Strategy, Design Test Condi-ons, Test Cases. • Ensure Test Environment readiness and prepare test data. • Perform Test execu-on and publish the Test Execu-on report, defect repor-ng. Test types and results • Smoke tes-ng • System Integra-on tes-ng • Func-onal tes-ng • UAT tes-ng • Regression tes-ng • Produc-on checkout 25
- 28. Infrastructure Tes-ng Procedure – Filenet Test Objec3ve • Ensure Filenet IBM Content Navigator(ICN) func-onali-es are not impacted a5er every bundle releases (consists of IBM updates, Security updates mandated by IS security team) • Control, Maintain and support for future Filenet upgrades. • Ensure Filenet web services are not impacted a5er every bundle releases. • Ensure ICC Inges-on is working as expected. Test Procedure • Receive Test request from Filenet team or engage in the project through the Demand Management process. • Schedule a mee-ng with PM, BA, Engineers to discuss the requirements. • Define Test Strategy, Design Test Condi-ons, Test Cases. • Ensure Test Environment readiness and prepare test data. • Perform Test execu-on and publish the Test Execu-on report, defect repor-ng. Test types and results • Smoke tes-ng • System Integra-on tes-ng • Func-onal tes-ng • Middleware tes-ng • UAT Tes-ng • Security Tes-ng • Performance Tes-ng • Regression tes-ng • Produc-on checkout 26
- 29. Infrastructure Tes-ng Procedure – Documaker Test Objec3ve • Ensure Documaker forms are working a5er every Documaker upgrade. • Ensure each LOB(Lines of Business) forms are working by execu-ng the XML script and verify the file generated. • Ensure Documaker forms are working a5er every patches applied to Documaker tool. Test Procedure • Receive Test request from the project team or engage in the project through the Demand Management process. • Schedule a mee-ng with PM, BA, Engineers to discuss the requirements. • Define Test Strategy, Design Test Condi-ons, Test Cases. • Ensure Test Environment readiness and prepare test data. • Perform Test execu-on and publish the Test Execu-on report, defect repor-ng. Test types and results • Smoke tes-ng • System Integra-on tes-ng • Func-onal tes-ng • UAT tes-ng • Regression tes-ng • Produc-on checkout 27