![[@IndeedEng] Redundant Array of Inexpensive Datacenters](https://image.slidesharecdn.com/slideshareversion-version2-indeedeng-operationstechtalk-june262013-130801160333-phpapp02/85/IndeedEng-Redundant-Array-of-Inexpensive-Datacenters-5-320.jpg)
![[@IndeedEng] Redundant Array of Inexpensive Datacenters](https://image.slidesharecdn.com/slideshareversion-version2-indeedeng-operationstechtalk-june262013-130801160333-phpapp02/85/IndeedEng-Redundant-Array-of-Inexpensive-Datacenters-6-320.jpg)
[@IndeedEng] Redundant Array of Inexpensive Datacenters
- 1. Redundant Array of Inexpensive Datacenters Charles Valentine and Chris Graf June 2013
- 2. Overview Charles Valentine VP, Technology Services
- 4. Indeed ● 100 million unique visitors per month ● Over 50 countries and 26 languages ● 3 Billion job searches per month
- 7. Indeed Ops ● Assist development in designing new products ● Engineer scalable systems to support applications ● Monitor applications ● Fix systems when they break
- 8. Indeed Lingo Datacenter = Point of Presence
- 9. Each Presence is Full Stack ● Applications ● Services ● Read/Write Data systems ● Communications ● Monitoring We need serious processing power in each datacenter!
- 10. Applications per Datacenter ● Over 40 Java-based web applications ● Over 90 Java-based services
- 11. Data Systems ● MySQL databases ● Mongo databases ● Memcached instances ● LSM Trees ● Search indexes ● Numerous other data stores
- 12. Goals ● Fast ● Reliable ● Inexpensive
- 16. Fast Speed is a product feature ● Server Time ● Client Time
- 18. 1 ms, 3 Billion Times/Month 1 ms = 34 job seeker days per month
- 19. 20 ms, 3 Billion Times/Month 20 ms = 22 jobseeker months
- 20. 100 ms, 3 Billion Times/Month 100 ms = 9.5 jobseeker years
- 21. Reliable Reliability is a product feature
- 22. Impact of Downtime 8,000 Disappointed Job Seekers every minute
- 23. People get hired on Indeed 7 seconds
- 24. Availability ● Jobseekers can find jobs ● Less focus on mitigating failure ● More focus on recovering quickly
- 25. Availability is Good for Job Seekers 9's
- 26. Good 99.9% availability => down for 525 minutes At peak 4,500 jobseekers don't get a job
- 27. Better 99.99% availability => down for 52 minutes At peak 450 jobseekers don't get a job
- 28. Almost Best 99.999% uptime => down for 5 minutes At peak 45 jobseekers don't get a job
- 29. Indeed is Always there for Job Seekers Availability > 99.999% Less than 5 minutes downtime per year
- 30. How It Works Chris Graf Operations Manager
- 31. Maximize Availability Beyond 99.999% No downtime, scheduled or otherwise
- 32. Maximize Performance Optimize page load times to the millisecond
- 33. Minimize Cost Minimize cost while meeting performance and availability goals
- 34. Hosting Models ● Traditional Colocation ● The Cloud ● Managed Hosting
- 35. Traditional Colocation ● You buy the servers, network gear, cables... ● You send people to set it up ● You send people to fix stuff when it breaks ● You manage your own pipes (maybe)
- 36. Traditional Colocation Expansion 1. Acquire rack space 2. Buy the hardware 3. Wait for manufacturing 4. Wait for delivery 5. Send people to the datacenter to set it all up Expansion can take weeks
- 37. Traditional Colocation Good if you have ● Fairly static environment ● Really beefy hardware ● Some centralized functionality ● Time to wait ● Lots of cap-ex budget ● Like signing long-term deals ● People to do stuff
- 38. ● You rent access to computing power ● You pay to reserve it if you aren't using it ● Usually abstracted from hardware layer The Cloud
- 39. Expanding Cloud-based systems 1. Order new instances 2. Wait a few minutes 3. Provision them Expansion takes minutes.
- 40. The Cloud is good! If you have significant, unpredictable changes in load
- 41. The Cloud is bad! Costs more if you need all your instances available all of the time
- 42. Managed Hosting ● Rent hardware from provider ● Provider buys and hosts servers, network, etc. ● Provider deals with hardware issues
- 43. Expanding Managed Hosting 1. Order new servers 2. Wait a few hours 3. Provision Expansion takes hours (depending on provider)
- 44. Indeed Uses Managed Hosting Least expensive overall Access to real bare metal hardware Agile enough
- 45. Steps for beyond 99.999% uptime 1. Find a provider 2. Sign contract for 100% uptime with 100% revenue protection 3. Profit Right?
- 46. Providers "guarantee" availability "Service Level Agreement" (SLA) guarantees some percentage of uptime
- 47. SLA: brief outages aren't outages Less than 30 minutes downtime not counted against "100% SLA" One 5-minute outage per month < 99.99% Two 25-minute outages per month < 99.9% The provider can call that 100% available
- 48. SLA: maintenance is not downtime Scheduled maintenance not counted against SLA 1 hour maintenance each month < 99.9% The provider can call that 100% available
- 49. SLA credits don't cover your business You get a refund for the services, not for lost business and lost customer confidence Providers lose your hosting fees You lose your revenue
- 50. 100% is not really 100% Hosting is complicated A single datacenter is rarely 100% available
- 51. Bug in provider hardware caused total loss of Internet access under certain load Core network problem
- 52. Power outage 1. Utility power was disrupted 2. Backup generator and UPS couldn't handle load 3. Core network went offline 4. Servers lost power 5. Upon power restoration, router did not recover
- 53. Power Outage Aftermath ● Event duration = 54 minutes ● Recovery duration = 12 hours ● 5% monthly credit for affected hardware
- 54. Backhoe Induced Fiber Failure (BIFF)
- 55. Wet servers Tornado peeled back the roof of an AT&T datacenter in 2004.
- 56. Other Disasters ● Hurricanes ● Floods ● Earthquakes ● Fires ● Etc.
- 57. Need better uptime than providers Can only get ~99.7% after asterisks We have to build something better
- 58. Save a document to a hard disk Hard Disk Doc
- 61. Disaster Recovery Restore from an external USB drive?
- 62. Redundant Storage Simple case - RAID 1 Hard Disk A Hard Disk B
- 63. RAID - Save it twice Hard Disk A Hard Disk B Doc
- 64. RAID - Two copies of everything Hard Disk A Hard Disk B Doc Doc
- 66. RAID == Redundant Array of Inexpensive Datacenters Datacenter A Datacenter B Jobseekers
- 67. RAID makes datacenters more reliable Datacenter A Datacenter B Jobseekers
- 68. Building a more reliable system Using inexpensive, less reliable components
- 69. 99.7% in, 99.999% out Now our system can get better availability as a whole than any single provider can give us.
- 70. Expect your datacenter to fail Failure is inevitable Design for it
- 71. Simpler datacenters with RAID Only need one of everything inside each datacenter: ● Firewalls ● Load balancers ● Servers provisioned primarily for capacity not redundancy
- 72. Primary and secondary datacenters 21
- 73. Datacenter level redundancy Protects against a single datacenter failure
- 74. Datacenter level redundancy Protects against a single datacenter failure ... But there are problems that can affect more than one datacenter on the same provider
- 75. Denial of service attacks Distributed denial of service attack against another customer who had servers in the same facilities took multiple facilities offline
- 76. Network configuration errors Provider pushed a bad global route which took their entire global network offline
- 78. Protect against global provider failure Use multiple providers to get provider-level redundancy
- 81. Recovering from Failure ● Offline ● Active/Passive ● Active/Active
- 82. Offline ● One active datacenter handles all traffic ● Backup systems are offline and incomplete ● Restore backups to new systems ● Downtime during switchover is ~days
- 83. Active / Passive (Dark) ● One active datacenter handles all traffic ● A second datacenter has provisioned systems and all data ● Switch from primary to secondary ● Downtime during switchover is minutes to hours
- 84. Active / Active ● Every datacenter handles traffic ● Data and systems are replicated ● Failover activated automatically ● Downtime during switchover measured in seconds ● Scales beyond two facilities
- 85. Jobseeker Impact Offline: extended downtime for all jobseekers Active/Passive: some downtime for all jobseekers Active/Active: brief downtime for some jobseekers
- 86. Which jobseekers go to which datacenter? Offline: go to single datacenter Active/Passive: go to single datacenter Active/Active: go to many datacenters?
- 87. Send jobseekers to the best datacenter Use dynamic DNS service to send job seekers to the best, healthy data center
- 88. Anycast DNS Resolving same hostname to different IP addresses ● Client A: nslookup www.indeed.com Server: dns.client-a.com Address: 1.1.1.1 ● Client B: nslookup www.indeed.com Server: dns.client-b.com Address: 2.2.2.2
- 90. Vary response from primary DNS Indeed DNS Service www.indeed.com 1.1.1.1 www.indeed.com 1.1.1.1 Indeed DNS Service www.indeed.com 2.2.2.2 www.indeed.com 2.2.2.2 Jobseeker DNS Server 5.5.5.5 Jobseeker DNS Server 8.8.8.8 Jobseeker A Jobseeker B
- 91. Similar jobseekers get similar responses Indeed DNS Service www.indeed.com 1.1.1.1 www.indeed.com 1.1.1.1 Indeed DNS Service www.indeed.com 2.2.2.2 www.indeed.com 2.2.2.2 Indeed DNS Service www.indeed.com 2.2.2.2 www.indeed.com 2.2.2.2 Jobseeker DNS Server 5.5.5.5 Jobseeker DNS Server 8.8.8.8 Jobseeker DNS Server 8.8.8.8 Jobseeker A Jobseeker B Jobseeker C
- 92. Remap jobseekers via DNS changes Indeed DNS Service www.indeed.com 1.1.1.1 www.indeed.com 1.1.1.1 R e c o n f i g Indeed DNS Service www.indeed.com www.indeed.com 2.2.2.22.2.2.2 Jobseeker DNS Server 5.5.5.5 Jobseeker DNS Server 5.5.5.5 Jobseeker A Jobseeker A
- 93. Outsource your DNS service Doing this well is an investment
- 94. Outsource your DNS service ● Robust ● Flexible ● Inexpensive Our core competency is jobs Their core competency is DNS
- 96. Degradation and Failure Manually switch datacenter on service degradation Automatically switch datacenter on failure
- 97. DNS propagation delays 1. Healthcheck cycle - up to 30 seconds 2. Healthcheck server to nearest PoP 3. Jobseeker's DNS server cache refresh 4. Jobseeker's local DNS cache refresh
- 98. DNS Time-to-live (TTL) TTL tells local name servers and clients how long to wait before looking up a domain name again TTL limits load, but also slows change propagation
- 99. Some clients and servers ignore TTL We specify a 30 second TTL, but local DNS servers and clients can ignore it
- 100. Impact of propagation delay 90 second traffic hole
- 101. 30 minute tail Well-behaved clients Ignoring our TTL
- 102. Big Picture 90 second hole Failing datacenter Total traffic
- 103. Accepting DNS limitations Complete datacenter failure is extremely rare Predictable limitation Massive costs to reduce propagation delay
- 104. Remapping Manually The same system allows us to reroute traffic whenever we want ● Datacenter maintenance ● Non-critical performance problems ● Non-critical feature loss ● Other degradation of jobseeker experience
- 105. Datacenter Redirection datacenter disabled traffic moves to others
- 106. Anycast DNS for performance This capability is also used to improve performance
- 107. Closer to the jobseekers The DNS service can give the IP address of the datacenter closest to the jobseeker.
- 108. Network hops Based on network hops between jobseeker DNS server and our DNS service POP
- 109. Network paths Estimates how many networks traffic must pass through to reach our servers
- 110. Count hops Picks estimated shortest path
- 111. Optimize for network distance We can push our data center presences closer to the jobseekers to reduce network latency
- 112. Datacenters for redundancy only
- 113. Fast for some jobseekers
- 114. Datacenters close to the jobseekers
- 115. Fast for most jobseekers
- 116. Sent to the East Coast
- 117. Sent to Central US
- 118. Sent to the West Coast
- 119. No downtime for datacenter replacement Incrementally send traffic to new datacenters Incrementally reduce traffic to old data centers
- 120. Move West Coast hosting? ?
- 121. Move West Coast hosting! -20 ms
- 123. Don't move European hosting! +50 ms!
- 124. Search Engine Performance Source GrabPerf.org
- 126. Summary and Results Charles Valentine
- 127. ● Higher-capacity network equipment ● Redundant firewalls ● Redundant load balancers ● Bigger Internet connections ● Redundant Internet connections This is "vertical scaling." Traditional Scaling Model
- 128. Horizontal Scaling with RAID Add capacity by adding datacenters Add redundancy by adding datacenters Rent "good" datacenters, not "best"
- 129. You can RAID too!
- 130. Avoid using proprietary features ● Load balancer ● Security devices ● Virtualization ● Servers
- 131. Be Hardware Agnostic
- 133. Use free software No licensing costs or recurring maintenance fees
- 134. Agile Providers ● New hardware racked and ready in a few hours ● No need to over provision
- 135. Automate configuration ● Cobbler ● Puppet
- 136. Rent instead of buying ● Obsolete hardware is not your problem ● No depreciation ● No hardware maintenance ● No need to hire people to maintain the hardware
- 137. Architect Applications for RAID Work with your development teams
- 138. Traditional Hardware Scaling ● Old hardware supports baseline traffic ● New hardware supports growth
- 139. Indeed Hardware Scaling Old hardware gets replaced by new, on demand
- 140. Moore's Law Hardware is always getting better ● Faster processors ● More memory per chassis ● Larger, faster disks
- 141. Higher capacity, lower cost ● Number of machines drives cost ● Power of machines drives cost ● More machines => more problems ● Compute power grows faster than compute cost
- 142. Replace hardware every 18 months Managed hosting + Moore's Law + RAID = new and powerful hardware every 18 months
- 143. Amazon EC2? ● Amazon is a single provider ● Costs more to run 24x7 ○ 2x without bandwidth cost ● Can't be as close to the jobseeker
- 144. What RAID gets you ● Servers closer to your customers ● Disposable datacenters ○ Datacenter-level failover ○ Get modern hardware every 18 months ● Many hosting options
- 145. Spend Time On... ● Automation ● Managed DNS ● Investigating Providers ● Monitoring
- 146. Spend Less On ● Proprietary hardware ● Network Infrastructure ● Support Contracts ● Software Licenses ● Headcount
- 147. Monthly Server Count vs Job Search
- 148. Inexpensive ● Cost as a percentage of revenue ● Cost of delivery per job search
- 149. Revenue vs Infrastructure Cost
- 151. Fast ● 100 ms average client time Reliable ● > 99.999% availability in 2012 Cost Effective ● Cost of delivery < 0.5% of revenue RAIDing FTW
- 152. Q&A