Drp Bcp Testing Alternatives
- 1. DRP/BCM - Testing Alternatives A Roundtable Discussion April 26, 2009 Gabe Gewurtz
- 2. DRP/BCM SIG - Testing Alternatives DR Program Objective: Risk management of a disastrous event Potential Cost of a disaster outage: recover cost + potential losses in Risk Analysis vs. Cost of a program to survive a disaster: mitigation, recovery plans + Testing Purpose of DR Program: Support recovery of critical business function(s) Support BCP of critical business function(s) Help the Enterprise survive a disaster event with minimum losses & cost Purpose of DR Testing: Increase Confidence that business functions Will Survive a Disaster Successful DR Testing Requires: Well understood & articulated purpose, objectives & scope
- 3. DRP/BCM SIG - Testing Alternatives Structure of a DR Plan; relevant for testing Assumptions: TECHNOLOGY High-level or Planning Assumptions; recovery site, licenses, security certificates Detailed or Working Assumptions; FW rules, SW releases, versions Are the Assumptions Complete, Valid & Correct Recovery Teams: PEOPLE Test Participants Do Recovery Teams Know their Roles & Responsibilities and can they execute plans with minimal problems Detailed Recovery Procedures: PROCESS “ The Payload” Are the Procedures correct & can be executed within RTO DR Testing: Results in Confidence that can recover Technology & Support the Business Processes, as expected within RTO at minimum cost & losses
- 4. DRP/BCM SIG - Testing Alternatives Types of DR Plans: Organization DR Plan: affecting multiple sites; eg. A virus, Y2K Site DR Plans: affecting a single site ; eg. A fire Infrastructure DR Plans: affecting several LOBs or applications Eg. Network, DNS servers, Firewall appliances, antivirus servers Shared Services DR Plans: affecting several LOBs, groups Enterprise backup/recovery, DBMS “farm”, Exchange server(s), etc. Application DR Plans: affecting individual applications ALSO: Alignment with Site & LOB BCPs Alignment with Vendor & Client BCP, where applicable
- 5. BCP silos by LOB, Biz Unit & Process DRP silos by System & Application BCP silos by LOB, Biz Unit & Process DRP silos by System & Application External Vendor & Client BCPs DRPs & other Plans ……… . Enterprise-wide, Infrastructure & Shared Services Enterprise Wide DRPs & BCPs Enterprise DR & BC Program interdependencies Site Infrastructure & Shared Services Site Infrastructure & Shared Services External Infrastructure & Shared Services Site 1 Site 2
- 6. DRP/BCM SIG - Testing Alternatives Possible Test Objectives Are the Planning Assumptions valid; sites , licenses, certificates updated Are the Working Assumptions valid ; SW at correct levels, FW rules, etc Do the Recovery Teams know what & how to recover; new staff Can the Recovery Teams execute the Procedures, skill improvement Are the Recovery Procedures correct, as documented; any errors? Are the Recovery Procedures complete, as documented; any omissions? Can the Recovery Requirements be achieved Can the Recovery be completed within the RTO Can Recovery procedures be executed by non-designated teams Are the Recovery & Results repeatable Are Recovery capabilities Robust; can they handle unexpected problems
- 7. DRP/BCM SIG - Testing Alternatives Possible Test Scenarios Site Outage, including all the hosted infrastructure, servers & software Single Application Outage, including all its servers, software & infrastructure Short Outage Long Outage Test / Don’t Test Return-Home Procedures Include / Don’t include BCP for Recovery & Support Teams Do NOT impact Production: Isolated Test configuration Run Production at Recovery site Introduce Test Faults (Murphys)
- 8. Assess Risk Assessment Business Impact Analysis Inventory and Rank Applications Define High Level Strategy & Requirements Develop Obtain Management Approval Launch DR Project Design and Specification of Recovery Solutions Identify Roles & Responsibilities Cost/Benefit Analysis Maintain and Test Awareness Training Desktop Tests, Configuration Tests Live Tests & Training Mock Disasters Use Change Management Implement Form Teams Ensure Proper/Suitable Backups Develop Recovery and Alternate Processing Plans Arrange/Build Alternate Site(s) Negotiate/Finalize 3ed-Party Contracts Develop / unit test Recovery Plans & Procedures Recovery infrastructure / Facilities Support considerations The Disaster Recovery Planning Process Typical DRP Life-Cycle phases Applies to New & Mature systems & their DR Plans For New Systems; more effort on Develop & Implement For Mature Systems; more effort on Maintain & Test
- 9. Assess Risk Assessment Business Impact Analysis Inventory and Rank Business Functions Define High Level Strategy & Requirements Develop Obtain Management Approval Launch BC Project Design and Specification of Recovery Solutions Identify Roles & Responsibilities Cost/Benefit Analysis Maintain and Test Awareness Training Desktop Tests, Configuration Tests Live Tests & Training Mock Disasters Use Change Management Implement Form Teams Ensure Proper/Suitable Backups Develop Recovery and Alternate Processing Plans Arrange/Build Alternate Site(s) Negotiate/Finalize 3ed-Party Contracts Develop/ unit test Recovery Plans & Procedures Recovery infrastructure / Facilities Support considerations The Business Continuity Planning Process BCP Life-Cycle phases; similar to DRP Life-Cycle
- 11. DRP/BCM SIG - Testing Alternatives Types Disaster Recovery Plan Tests: DRP Walkthrough: Verbal tabletop review by all stakeholders Alternate site DR Server (Planning Assumption) Validation Tests: Check assumptions that DR HW, SW & applications are correct & ready for activation Alternate site DR Infrastructure (Working Assumption) Validation Tests: Check assumptions that server intra-connections (FTP and other network connections) are correct & ready for activation DR Application Component (Unit) Test: Validate the recovery procedures of a single or logical group of applications, as documented in the DRP. Eg. Health Checks that the new recovery solutions and procedures being developed are correct Configuration changes for DR: Health Checks that distributed systems & surviving user locations can access & use the DR alternate HW & SW, as documented in the DRP Technology changes to support the BCP Changes to systems in the recovery centre to support the BCP of the impacted centre, as documented in the DRP or BCP Data Recovery Test Test data restoration; the recovery, reconciliation, synchronization of in-flight data, back-out data that is not required or is to be re-captured. Full End-to-End Test: Simulation tests 4, 5, 6, and 7 above to validate the RTO. Does not interfere with Production. Full Production Test: Same as 8 above, but run production at alternate site Surprise Test: Any of the above with no warning Return Home Test Difficult to Simulate Repeatable Tests Able to repeat test Results Tests with Planned Faults (Murphys) Somewhat Resilient to Potential Problems. Eg. I/O errors on backup media at time of data restoration
- 12. DRP/BCM SIG - Testing Alternatives Test Concerns: Many recovery teams with alternates = Many Test Participants Site hosts many applications with separate DR Plans Due Diligence & Audit typically requires regular Testing / Validation Testing could become unyielding & costly
- 13. DRP/BCM SIG - Testing Alternatives Managing DR Testing Satisfy: Enterprise Standards & Good Practice Internal & External Audits Business Requirements: support business Processes, user access BCP Technology Requirements Client Expectation & Contracts Confidence that can help the business survive a disaster at minimum cost Control: Cost Potential Losses (Risks) Time
- 14. DRP/BCM SIG - Testing Alternatives Cost of progressively more complex test types Potential Cost of an outage due to progressively more complex test types