Transitpresentationmarch2012
- 1. Business Continuity Planning Fundamentals Presented By: Jon Sherman, Vice President Continuity Solutions, Inc.
- 2. All Kinds of Disasters
- 3. What is a Disaster? Any Situation That Impedes On Day-to-Day Operations Natural Disaster • Tornadoes, severe winter storms, earthquakes, fires, dam failure, (floods and water leaks are statistically the number one threat), etc. Man-Made Disasters • Disgruntled employees, spouses, significant others • Union Strikes • Hazardous material spills • Terrorist (Foreign and Domestic) • Construction workers cutting power communication lines • Biological, chemical, nuclear devices • Civil uprisings Technical Disasters • Hackers, cyber-terrorism, power outages, voice and data communications line failure, software and hardware failures
- 4. What is Disaster Recovery? Business Continuity Program: An ongoing process supported by senior management and funded to ensure that the necessary steps are taken to identify the impact of potential losses, maintain viable recovery strategies and recovery plans, and ensure continuity of services through personnel training, plan testing, and maintenance. Business Continuity Planning (BCP): Process of developing advance arrangements and procedures that enable an organization to respond to an event in such a manner that critical business functions continue with planned levels of interruption or essential change. MANY SIMILAR TERMS: Contingency Planning, Business Resumption, Resiliency. IT Disaster Recovery Planning (DRP): Process of developing advance arrangements and procedures that enable an IT department to respond to an event in such a manner that critical business functions continue with planned levels of interruption or essential change.
- 5. Emergency Preparedness and Response WORKING WITH THE PUBLIC SECTOR WHILE RESPONDING Strategic Phase: A plan set to identify who performs what function, when and how. -Establish Relationships - Police, Fire, Medical -Develop a monitoring and reporting process – Command and Control -Identify the first response teams -Review and receive signoff, establish Emergency Operations Center Tactical Phase: Strategy for dealing effectively with the emergency. -First Responders - (Incident Command System) - Evacuation - Medical care and personnel counseling - Hazardous material response - Fire fighting - Internal and external communication - Emergency Stabilization - Site safety, security, salvage, restoration - Facility Stabilization
- 6. Emergency Response TRANSIT IN EMERGENCY RESPONSE: Public transit agencies have a history of providing assistance during crisis situations, performing vital services such as evacuation of victims and transport of emergency personnel. In the aftermath of major disasters, public transit systems have often supplemented or replaced damaged or blocked roadways, maintaining mobility for residents and for repair and recovery workers.
- 7. Emergency Preparedness and Response Disaster Response and Recovery Resource for Transit Agencies Contents – From FEMA Homeland Security Office 1. Introduction and Background How to find Information and Resources in this Document 2. Frequently Asked Questions 2.1 Info for Transit Providers in Affected Areas 2.2 Info for Transit Providers serving Displaced/Relocated Persons 2.3 Charter Service Requirements 2.4 Emergency Transportation for Persons with Special Needs (including ADA Paratransit) 2.5 Funding Eligibility and Reimbursement 2.6 How to Help Emergency Evacuees 2.7 Assisting Special Needs Populations 3. The Role of Federal Agencies and States in Disaster Response 3.1 The National Response Plan and NIMS. 3.2 State Emergency Management Plans 3.3 FEMA 3.4 FTA 4. Local Disaster Response Resources and Best Practices 4.1 Introduction and Background 4.2 Emergency Preparedness: Planning and Best Practices 4.3 Disaster Response 4.4 Disaster Recovery 4.5 Characterizing Possible MPO Roles in System Operations and Security/Disaster Planning 4.6 Providing Emergency Services to Persons with Special Needs 4.7 The Transit Industry at Large Appendix A: FTA Emergency Response Personnel Contact List Appendix B: FEMA and State Contact List
- 8. Emergency Preparedness and Response NATIONAL TRANSPORTATION RECOVERY STRATEGY (NTRS) The National Transportation Recovery Strategy (NTRS) is designed to help transportation industry stakeholders and local, tribal, and State government officials prepare for and manage the transportation recovery process following a major disaster. The overall goal of this Strategy is to promote a recovery process for transportation networks – and subsequently of communities in general – that results in a greater level of resilience.
- 9. Emergency Preparedness and Response
- 10. Transit Helps in Emergencies Headlines As Washington’s Columbia River and nearby waterways threatened to flood in February 1996, C-TRAN of Vancouver began monitoring water levels and planning with other local agencies for emergency services. When flood waters began to affect rural roads, C-TRAN detoured its routes to keep service running. As streets and bridges in Vancouver and Portland became hazardous, C-TRAN’s urban routes began early and increased commuter service to get residents home; for several days, mass transit was the primary mode of travel in downtown Portland. In addition, buses performed emergency evacuations and transported emergency and recovery personnel throughout the crisis (1). During the following year, C-TRAN evacuated and sheltered Vancouver residents during two chemical spills and a downtown fire (2). C Harrisburg, Pennsylvania’s Capitol Area Transit (CAT) responded to a variety of emergency conditions during the blizzard of 1996 and its aftermath. From a sudden increase in transportation demand when all government employees were sent home during the blizzard, to the evacuation of residents in flood zones, to the transport and shelter of firefighters during a four-alarm fire in late January, CAT vehicles and employees made significant contributions to Harrisburg’s winter storm response and recovery (3). C After the bombing of the Alfred P. Murrah Federal Building in Oklahoma City, Metro Transit began running 24-hour service to accommodate transportation needs. In addition to maintaining all regular service, Metro Transit buses transported firefighters, rescue teams, and medical personnel, and evacuated residents from a nearby housing complex. Metro Transit personnel also manned the Multi-Agency Command Center, which coordinated communications during relief efforts (4). C The 1989 San Francisco earthquake destroyed some of the area’s primary traffic arterials and damaged others to the point of impassability. The San Francisco-Oakland Bay Bridge and the I- 880 freeway, which together comprised the main connection between the cities of San Francisco and Oakland, were closed after sections of these roadways collapsed. Several other freeways and
- 11. TAKING ACTION Erie PA Metropolitan Transit Authority Erie Metropolitan Transit Authority (EMTA) has made significant investments over the past few years in its IT infrastructure to increase operational knowledge, vehicle tracking, and overall efficiency of the organization. Furthermore, EMTA is a major resource during emergency events, such as man-made and natural disasters. EMTA has the vehicles, professional drivers, and systems to assist with large-scale evacuations and to provide shelter-in- place facilities as rest and recovery stations for first responders. In order for EMTA to ensure that these critical systems are available during a disaster and in order for EMTA to assist with emergency recovery and evacuation efforts, EMTA engaged DR vendor to develop an all-hazards Disaster Recovery Plan (DRP) to ensure that critical business functions continue during an emergency
- 12. PREPAIRING Pioneer Valley Transit Authority Conduct Disaster Drill SPRINGFIELD, MA.,-The mock disaster scene will consist of an armored car and bus accident. The bus, with 22 passengers, is hit by the armored car which theoretically was used in a robbery, flips on its side, doors against the pavement, trapping the passengers inside. First responders participating in the drill are Springfield fire and police, American Medical Response, Mercy Medical Center, Pioneer Valley Red Cross. The PVTA has worked closely with Springfield’s Director of Emergency Preparedness, Robert Hassett to plan. “The goal is to build on, and maintain, good and open relationships with first responders in Springfield. It is important fire and police are familiar with our buses, and it’s important for us to test response time, communications, rescue and recovery. The Federal Transit Administration requires all Regional Transit Authorities to devote 1% of capital expenses toward safety and security.
- 13. PROTECTING DATA Utah Transit Authority Relies on Data Backup Vendor for Disaster Recovery Improves its object storage archive to assure regulatory compliance and seamlessly support its disaster recovery strategy. Using disk based data backup UTA has dramatically reduced its backup window from five days to one while improving the overall performance, integrity and availability of its data, video archives, and CAD drawing archives. As with most public transportation agencies, a large volume of the data UTA must protect comes from video surveillance of the stations and vehicles it operates. Literally mountains of surveillance video can be captured every day. This combined with heavy operational usage of unstructured data was creating volumes of data that UTA needed to efficiently and effectively archive to meet compliance and regulatory requirements.
- 14. FUNDING CINCINNATI Article from: US Fed News Service, Including US State News | August 24, 2011 WASHINGTON, Aug. 23 -- The office of Sen. Sherrod Brown, D- Ohio, has issued the following news release: New safety improvements will be made to protect southwest Ohio's public transportation system from potential disasters and other emergencies. U. S. Sen. Sherrod Brown (D-OH) today announced that new federal resources were awarded to Southwest Ohio Regional Transit Authority (SORTA) create a disaster preparedness plan to protect Ohioans from acts of terrorism, major natural disasters, and other emergencies. "Our state's public transportation systems are critical for connecting Ohioans with schools, health care facilities, and employment opportunities,"
- 15. Being Ready to Apply for Funding South Jersey Transportation Authority applies for $1 million in disaster aid following Hurricane Irene The South Jersey Transportation Authority is seeking federal disaster funding to offset an estimated $1.1 million in lost revenue from Hurricane Irene, including about $320,000 from waiving toll collections on the Atlantic City Expressway. The authority is applying to the Federal Emergency Management Agency to cover some of the losses. . The late August hurricane cost an estimated $905,000 in total toll revenues. State officials suspended tolls on the expressway and the Garden State Parkway to aid evacuations. Other hurricane costs included $130,000 in emergency staffing levels, as well as $35,000 in losses, including lost landing fees at Atlantic City International Airport, SJTA spokeswoman Sharon Gordon said. These included a three-day shutdown of Atlantic City casinos and a mandatory evacuation of Cape May County and other shore towns at the height of summer.
- 16. Why Should You Develop A Business Continuity and Disaster Recovery Plan? As a Leader in your Organization PROTECT YOUR REPUTATION
- 17. Why Should You Develop A Business Continuity Program? Protect the Organization’s Assets • People, Equipment, Information (Data) Minimize damage and loss Minimize confusion, indecision Instills confidence in staff, public and customers Ensure employee welfare and safety Disaster Plan may be used for daily activities A Business Continuity Program saves TIME and MONEY responding to disasters Deal with the media in an appropriate fashion Expedite the return to “business as usual”
- 18. Business Continuity Methodology The Path To Successful Planning Recovery Analysis Interviews Project Planning Risk Assessments Observations Schedule and Kickoff Business Impact Analysis Recovery Strategy Options Data Collection fdsfdfs fdsfdfs Polic fdsfdfs ies and Proc Gui de edures Present Recovery Solutions Analyze Data Plan Consider Viable Options Development Plan Testing Plan Enhancement Exercise Plan Maintenance Rehearsals
- 19. Business Continuity Methods Backup and Restore of Information NO DATA NO RECOVERY
- 20. Business Continuity Methods Information Media Recovery Microfiche • Are they backed up and stored off-site? • Paper Records • Use fire proof filing or fire resistant filing cabinets • Use an imaging system Critical stand alone pc’s are they backed up? • Backup nightly - critical files to network storage, tape, or thumb drive/CD/DVDs *be careful while conducting incremental backups Is the IT department effective with data backups? Are backups tested? Offsite storage, NAS (network attached storage, SAN (storage area networks) VSN (virtual storage networks) Off-Site storage facility should be used for paper documents CDs, hard drives tapes, etc. (test your storage provider ask for a backup tape periodically) Fire proof vault for cash, checks, blank checks, contracts, insurance policies, etc.
- 21. RECOVERY ANALYSIS CONDUCT A BUSINESS IMPACT ANALYSIS A management level analysis that identifies the impacts of losing the entity’s resources. The analysis measures the effect of resource loss and escalating losses over time in order to provide the entity with reliable data upon which to base decisions concerning hazard mitigation, recovery strategies, and continuity planning.
- 22. RECOVERY ANALYSIS UNDERSTANDING Business Impact Analysis(BIA) Describes the business functions at the process level Identifies critical equipment (all the equipment you need to operate in disaster mode) Frequency of operations/functions • Continuously, annually, daily, weekly, etc. Identifies periods of high volume Financial, operational and service impacts identified Considers if job descriptions and operational procedures exist Sets business process priorities Identifies single-points-of-failure Do vendors have business continuity plans?
- 23. RECOVERY ANALYSIS UNDERSTANDING Business Impact Analysis(BIA) What are Critical Business Processes to Transit Authorities? Number ONE - PUBLIC SAFTEY • Fleet, Funding, Human Resources • Customer Services, Maintenance, Line Services • Fixed Routes, Scat Services, Special Services • Passport, Title, Speakers Bureau
- 24. RECOVERY ANALYSIS UNDERSTANDING Business Impact Analysis(BIA cont.) Recovery Time Objective (RTO) - The period of time that systems, applications, or functions must be recovered after an outage (e.g. one business day). RTO’s are often used as the basis for the development of recovery strategies, and as a determinant as to whether or not to implement the recovery strategies during a disaster situation. CLASSIFY Priorities - Processes, Servers, Files Priority One, Two, Three, Four, Five Many organization use terms like Continuous Availability High Availability, Highly Recoverable, Less Critical to classify priorities business and computing priorities. Consider classifying new systems and operations as they evolve, turn BIA into part of the company lifecycle. Recovery Point Objective (RPO) - The maximum amount of data loss an organization can sustain during an event. Last backup till disaster. Recovery Time Actual (RTA) - The actual time it takes to recover a business function, consider gaps.
- 25. RECOVERY ANALYSIS QUESTION What is the best way to recover from a Disaster?
- 26. RECOVERY ANALYSIS ANSWER Never have one in the first place! CONDUCT A RISK ASSESSMENT
- 27. RECOVERY ANALYSIS How to Prevent Disasters Identify Hazards That May Cause A Disaster Mitigate The Identified Hazards
- 28. RECOVERY ANALYSIS CONDUCT A RISK ASSESSMENT Identifies vulnerabilities and ranks hazards/threats Examines all possible risk sources…physical security, systems security, facility, location, surrounding area The report will prioritize findings and recommendations for mitigation consideration Computer Based Security Assessment Tools are recommended starting points for computer security risk assessments
- 29. RECOVERY ANALYSIS CONDUCT A RISK ASSESSMENT Items To Assess Uninterrupted Power Supplies and Power Generators • In a secured location, • Is it tested regularly • Fuel contract (refill after testing) and a major supplier of fuel and an alternate Fire Suppression System Wet or dry pipes Fire extinguishers and usage training
- 30. RECOVERY ANALYSIS CONDUCT A RISK ASSESSMENT Items To Assess Physical facility security Electrical power grid feeds Telecommunication central offices used Multiple voice and data communication providers routing through same central office Evaluation of data center and network security vulnerabilities Virus protection, trojans, worms, adware/spyware detection, unnecessary open ports and services being used on servers, workstations and network equipment, identify opportunities hackers would use to attack your network Evaluate the security of vital records and one of a kind documents Business Interruption Insurance (do you have enough and the right coverage) Legal Considerations
- 31. RECOVERY ANALYSIS DETERMINE RECOVERY STRATEGIES Alternate site arrangements Communications and network equipment Unique and/or irreplaceable equipment Resources: staff, operations support, office supplies, life support (food, water, shelter) Emergency relocation costs Disaster restoration contracts Unique and/or irreplaceable equipment Environmental and off-site requirements Identification and suspension of non-critical functions or tasks Implementing manual processing functions and tasks (is this realistic in the aftermath?) Recovery facilities should be at least 30-60 miles away from the primary site Consider different power grids and telecom points of presence
- 32. RECOVERY ANALYSIS DETERMINE RECOVERY STRATEGIES Use internal methods when possible - Use your own facilities first Alternate site arrangements • Vendor Hot Site, Co-location Facilities, Company Owned Hot Site, Mobile Facilities, Managed Services • Service Bureau, Office or Warehouse Space, Reciprocal Agreement, Equipment Leasing, Drying Companies and Emergency Cleaning Companies • Cold Site, Warm Site • Work Area Recovery (Call Centers, Mail Room, Specialized Equipment) • Networking and Telephone Considerations • Continuous and High-Availability • Mirroring, Replication, Clustering, Virtualization • E-Vaulting, Disk to Disk (SAN, IP SAN, NAS, ATA) • Grid Technology - supports distributed processing connecting multiple organizational sites, devices and platforms transparently, Grid is designed to assist in recovery from system failures. Cloud Computing.
- 33. Plan for Proper Decisions “If you don’t know where you’re going, you’re liable to end up someplace else” - Yogi Berra
- 34. Business Continuity Planning Plans Must be DOCUMENTED Invisible Plans don't work
- 35. Business Continuity FAMILY FIRST PEOPLE RECOVER FROM DISASTERS NOT COMPUTERS!
- 36. Business Continuity Methods Developing the Business Continuity Plan Brings the research, analysis, strategies, procedures and recovery team assignments together Tasks managed and controlled at the Command Center location Contains recovery team(s) information Details the entire emergency response/crisis management process Contains contact information and notification procedures Details tasks and responsibilities Further identification of critical operations, functions and/or computer applications and how they will be recovered Specify business process recovery and restore requirements Specify software recovery and hardware configuration requirements Specify off-site storage location for your data and vital documents
- 37. Business Continuity Methods Developing the BCP (cont.) Detail recovery task sequence and functional interdependencies Identify everything that might be needed to perform part of the process: teams of people, equipment, transportation, support items, support providers, etc. Contain all procedures that might be used in the recovery process Contain a list of all vendors, service providers you will need to support your recovery strategies Contains a list of critical customers to contact Management Succession Contain standard forms (POs, Blank Checks, Travel Advances etc.), supplies and documents Moving from Disaster Mode to Normal
- 38. Business Continuity Methods WHAT DOES A PLAN LOOK LIKE? TABLE OF CONTENTS PAGE # Charter 1 EXECUTIVE MANAGEMENT TEAM 3 Definition of Team Members and Recovery Plan Responsibilities 4 Financial TEAM 5 BUSINESS CONTINUITY TEAM TASKS AND ASSIGNMENTS 6 TEAM TASKS 7 LAST MINUTE PREPARATION PHASE – STEP 1 7 EVALUATION PHASE – STEP 2 9 ACTIVATION PHASE – STEP 3 13 RECOVERY CENTER START-UP PHASE – STEP 4 15 RESTORATION/MOVING BACK PHASE – STEP 5 18 Disaster Recovery Contact List 19 RESOURCES (SEE ATTACHED SECTION)
- 39. Business Continuity Methods TABLE OF CONTENTS PAGE # Command Center Guidelines 1 Personnel Notification Guidelines and Location Notification Guidelines 4 Personnel Notification Control Log 5 Emergency Telephone Numbers 6 911 CALL INSTRUCTIONS 8 EMERGENCY EVENT PROCEDURES 9 EVALUATION CHECKLISTS 10 Declaration of Disaster (Press Release Sample) 19 EMPLOYEE LOCATION LOG 20 TRAVEL REQUEST FORM 21 Progress Log (Used to prepare daily status reports) 23 Purchase Order Forms 24 RECOVERY NEEDS 25 EVACUATION MAP 26 CRITICAL FUNCTION PRIORITIES RECOVERY MATRIX 27 MINIMUM RESOURCES REQUIRMENTS 28
- 40. Business Continuity Methods BUSINESS CONTINUITY TEAM TASKS AND ASSIGNMENTS Initial Notification Provide Team Member Personnel Information Team Leaders to: Call and/or Assemble and Brief Team Members Deploy Teams to Alternate Facilities or have the Team Members Stay at Home Teams working from home Teams Implement Recovery Plans Operate In Crisis Mode Coordinate Recovery Actions Status Reports and Periodic Briefings (TBD) Salvage and Restoration Return Back-to-Facility/Transition Planning Post Incident Review Develop Lessons Learned Write After Action Report Update Recovery Plans
- 41. Business Continuity Methods Develop specific tasks for your office to follow ncident Management Team Tasks If the incident calls for an evacuation, Ensure that an orderly evacuation is taking place; Evacuate to the pre-determined meeting location; Take control of the response; Activate Incident Command Center, if required; Daytime: Assemble team at pre-determined location Daytime Primary Assembly Location: Daytime Secondary Assembly Location: Nighttime: Contact team members by telephone; Nighttime Primary Assembly Location: Nighttime Secondary Assembly Location: Set up a command and control center that can establish liaison with emergency responders, customers, the media, employees and their families, suppliers, etc.; Assemble Incident Assessment Team; Determines extent of damage from Damage Assessment Team reports;
- 42. Business Continuity Methods LAST MINUTE PREPARATION PHASE – STEP 1 1.1 Person Uncovering an Incident If you become aware of a potential incident within the facility: Perform all appropriate emergency notification actions (e.g. sound fire alarm, etc.). Notify Local Emergency Responder with the following information: •Your name; •Description of incident; •Preliminary report of damages and injuries; •Information regarding any attempted or actual notification contacts; •Phone number and location where you can be reached. Department Emergency Evacuation 1.2 Team Leader Notification If in the building during the incident: Determine if equipment shutdowns are required. Contact all affected areas. Re-confirm NO ONE except Executive Management Team and/or Public Affairs is to talk with the media. Note: Team Leader will notify the Executive Management Team of function(s) shutdown.
- 43. Business Continuity Methods Procedures Lists to be Developed and Maintained for all Departmental Business Continuity Plans Offsite Storage Retrieval Procedures Department Operational Procedures Emergency Procurement Procedures Vendors Employee Names/Addresses/Phone Numbers Department Equipment List Job Descriptions Resource Requirements
- 44. Business Continuity Testing Plan Exercising - The Plan is Alive Before any recovery plan can be considered complete, it must be validated. Plan testing is a “practice recovery;” it allow you to validate the strategies, procedures and recovery team structures documented in your recovery plan. Plan testing normally consists of a mock disaster scenario or moving your critical applications to an alternate facility. We recommend that your recovery teams participate fully in the plan rehearsal, to validate team structures and responsibilities.
- 45. Business Continuity Program Lifecycle and Maintenance Plan Review Component Testing Integrated Update Plan Standards Planning and Testing Awareness Exercise Training Plan Perform Maintenance Schedule
- 46. Discussion – Thank You Thank you for attending this presentation Continuity Solutions, Inc. 5900 Roche Drive Columbus, Ohio 43229 (614)-569-3292 www.csigroup.cc