Up and Running, even during disaster
0 likes206 views
CSS Corp implemented a cost-effective disaster recovery solution for a leading consumer goods company to ensure continuity of their brand websites during disasters. Utilizing a pilot light model in AWS, the approach provided quick recovery time, minimized disruption to existing infrastructure, and achieved compliance with data protection regulations. Key outcomes included a reduced recovery time objective of 5 hours and a recovery point objective of 4 hours, facilitating automated infrastructure provisioning and data consistency.
Up and Running, even during disaster
- 1. A cost-effective, fully-automated, reduced time-to-recover disaster recovery solution for a leading consumer goods company CLIENTSUCCESSSTORY Up and running, even during disaster Brand websites are today an important medium to deliver brand experiences. And when you are one of the top three consumer brand companies in the world, ensuring continuity of your brand websites in case of a disaster is critical. CSS Corp planned and executed a cost-effective disaster recovery of the content management platform of a leading consumer brand company without disturbing their existing production infrastructure.
- 2. Being prepared for possible risks and unplanned outages is a must to minimize, if not eliminate, the damage to reputation and financial losses that disasters may cause for any business. And the cost and impact is far greater when you are one of the top three players in your industry. A carefully planned, crafted, and tested disaster recovery (DR) is essential. Our client, a leading consumer goods company, knew the importance of DR and approached us to test its business-critical content delivery infrastructure. Our pilot light method gave a quick recovery time and ensured cost-effectiveness without disturbing the existing production infrastructure. One of the top three consumer brand companies in the world, providing some of the best-known global products to people in more than 190 countries. About the Client Client Situation The company uses a well-known content management system to host their top brand commercial websites. Client's business considers this as a platinum platform. This infrastructure is hosted in the US East. The company wanted to set up disaster recovery (DR) in the US West. The company mandated CSS Corp with planning and architecting the recovery of the infrastructure that hosts customer-facing Content Delivery Application (CDA) from the US East to the US West, in case of a disaster. Disaster recovery in Amazon Web Services (AWS) cloud – the context Understanding the difference between AWS availability zones and AWS regions is very important while designing DR for AWS cloud. In AWS, Availability Zones are one or more independent data centers. Architecting and designing deployment architectures across Availability Zones is for high availability of the application. CSS Corp designs and deploys every tier of the deployment in high availability mode. Further, in AWS, all availability zones within a particular region form a region. Regional failure is considered as disaster. In this context, recovering from one region to another region is disaster recovery. The CSS Corp Solution CSS Corp adopted the approach of creating a pilot light model DR for the platinum platform to achieve operating efficiencies and cost optimization. The team created a comprehensive plan to provision / deploy the entire platinum platform's CDA infrastructure in the US West region once the recovery site was built. CSS also executed the Disaster Recovery process to test the recovery without disturbing the current production infrastructure in the US East. To this end, the CSS team worked collaboratively with the company’s digital marketing team, the service continuity team, their platform management partners, and AWS solution architects. Security Group Availability Zone 3 DNS SWITCH OVER Public Subnet Private Subnet US East US West REGIONAL FAILURE DETECTED DISASTER RECOVERY Push-button Recovery ORCHESTRATION BUILD ON REGION 2 CSS Corp - Automation and Recovery Strategy SNAPSHOT - AMI A Synchronous Mirror Witness Monitoring Active/Passive Master Slave Data transfer Log Shipping 3 4 5 2 AMI PILOTLIGHTDR PRODUCTION RECOVERY Security Group Availability Zone 2 VPC Security Group Availability Zone 1 M Security Group Availability Zone 1 MS SQL M Security Group Availability Zone 2 MS SQL 1 Private Subnet M Security Group Availability Zone 1 MS SQL Public Subnet Security Group Availability Zone 1 West VPC Minimal Running Footprint 6 P A S s M ity Group lability Zone 1 Slave ne 2 Subnet s Mirror Monitoring Active/Passive Securit Availability L Sec Availabili RECOVERY TR Y B Snapshot Pilot light DR model
- 3. As Backup is the base for any recovery process, CSS Corp worked on a custom backup strategy. – Instances are bundled and snapshots placed in S3 Database point-in-time recovery 1. Take transaction log backup 2. Take full backup 3. Restore full backup with “No Recovery” option 4. Restore transaction log backup using stop-at option in point-in-time The team began by building a lightweight replica of the production infrastructure in cloud in the US West. This was followed by building an “as-is” database server with log shipping in the US West. All other components were replicated in passive through snapshots and AMI shipping. Prerequisites AMI Snapcopy Log shipping During DR Make pilot light – production heavy Add all moving parts – ELBs, SSLs Dummy DNS to test production websites Validate production sites with Platform Partners Sign off from client Post implementation review Log shipping and DB mirroring for High availability Ongoing ELB / SSL cert mapping in DR EBS snapcopy – ongoing
- 4. Mind map of the DR flow Business Outcomes Our approach resulted in six key benefits for the client: Cost-effective: By planning and executing a pilot light disaster recovery, which can be scaled-up and made production-ready, our team ensured cost-effective deployment. Low Recovery Time Objective (RTO): The pilot light method gave a quicker recovery time (of 5 hours) than the ‘backup and restore’ scenario as the core pieces of the system are already running and are continually kept up-to-date. Target RTO from business: 24 hours. Recovery Point Objective (RPO): We currently have RPO of four hours as per the backup schedule and recovery strategy. Fully automated reduced time to recover: Our fully integrated automated platform (HAPX) and AWS API facilitates automated provisioning and configuration of the infrastructure resources. Data consistency: Log shipping ensured data consistency. PII compliance: As DR for US East was deployed in US West, data within the geography stayed within the region thus ensuring compliance to PII regulations. For more information, please mail us at info@cssscorp.com AWS Common Technology Platform Login Sync Job Snap Copy EBS Snapshot Log Shipping SSL SSL Out of 16 servers, 9 AMI’s will be bundled from US east region to US west Script to automate EBS snap copy from region 1 to region 2 every 12 hours Disaster Recovery Site Production Site US East US West Not Running Elastic Load Balancing EC2 DB on Instance EC2 DB on Instance Elastic Load Balancing