Learning to build distributed systems the hard way
0 likes413 views
This document discusses some key lessons the author has learned about building distributed systems at scale: - Start with more servers than you need and scale out capacity from the beginning to solve scaling problems proactively. - Put significant thought into how data is keyed and identified to facilitate queries and sharding. - Consistency is not always critical, and distributed systems can function with some degree of inconsistency. - Precompute frequently queried data when possible to avoid ad hoc queries. - Separate processing from storage to allow independent scaling. - Carefully plan how data will be deleted as removal is complex.
Learning to build distributed systems the hard way
- 1. LEARNING TO BUILD DISTRIBUTED SYSTEMS THE HARD WAY @iconara
- 2. LEARNING TO BUILD DISTRIBUTED SYSTEMS THE HARD WAY BIG DATA @iconara
- 5. chief architect at BURT
- 6. let’s make online advertising a great experience
- 8. MAKING THIS
- 9. INTO THIS
- 10. HOWHARDCANITBE?
- 12. 30K REQUESTS PER SECOND more than a billion requests per day, over 1 TB raw data
- 13. ONE VISIT CAN CHANGE UP TO 100K COUNTERS hundreds of millions of individual counters per day, plus counting uniques and visitor histories
- 14. IN REAL TIME or near real time, if you want to be pedantic ×
- 15. HOWHARDCANITBE?
- 16. START WITH TWO OF EVERYTHING going from one to two is the hardest, solve the scaling problem up front
- 17. START WITH TWO OF EVERYTHING you’ll solve the scaling problem, and need less overcapacity THREE
- 18. GIVE A LOT OF THOUGHT TO KEYS AND IDS and think about your queries first
- 19. MEIHO0 JME57Z monotonically increasing, sorts nicely a timestamp something random
- 20. JME57Z MEIHO0 uniformly distributed, works nicely with sharding something random a timestamp
- 21. CONSISTENCY IS OVERRATED don’t fear R + W < N
- 22. PRECOMPUTE ALL THE THINGS your users most likely don’t know what they want, so why let them do ad hoc queries?
- 23. SEPARATE PROCESSING FROM STORAGE that way you can scale each independently
- 24. PLAN HOW TO GET RID OF YOUR DATA deleting stuff is harder than you might think × × × × × × ×
- 26. DIVIDE THE LOAD big data systems are all about routing and partitioning
- 27. RANDOM when you have no interdependencies between things it’s easy to scale out
- 28. CONSISTENT when there are interdependencies you need to route using some property of the objects, but make sure you get a uniform distribution
- 29. NUMEROLOGY
- 30. 12
- 31. 2 | 12 3 | 12 4 | 12 6 | 12
- 32. 8 | 24 5 | 60
- 33. A DIVERSION ABOUT COUNTING TO 60 the reason why there’s 60 seconds to a minute, and 360 degrees to a circle ××
- 34. 3 SEGMENTS ON EACH FINGER = 12
- 35. 3 SEGMENTS ON EACH FINGER = 12 FIVE FINGERS ON OTHER HAND = 60
- 36. 12, 60, 120, 360 superior highly composite numbers
- 37. 12, 60, 120, 360 superior highly composite numbers
- 38. 12, 60, 120, 360 superior highly composite numbers
- 39. 12, 60, 120, 360 superior highly composite numbers
- 40. 12, 60, 120, 360 superior highly composite numbers
- 41. 12, 60, 120, 360 superior highly composite numbers
- 42. 12, 60, 120, 360 superior highly composite numbers
- 43. 12, 60, 120, 360 superior highly composite numbers
- 44. 12, 60, 120, 360 superior highly composite numbers
- 45. 12, 60, 120, 360 superior highly composite numbers
- 46. 12, 60, 120, 360 superior highly composite numbers
- 47. 12, 60, 120, 360 superior highly composite numbers
- 48. use multiples of 12 to scale without always having to double
- 49. BLAH BLAH BLAH use multiples of 12 to scale without always having to double
- 50. log2(366) ≈ 31
- 52. log2(366) ≈ 31
- 53. log2(366) ≈ 31 six characters 0-9, A-Z can represent 31 bits, which is kind of almost very close to four bytes
- 54. MEIHO0
- 57. YOU CAN’T SCALE TO REAL TIME and don’t trust code that doesn’t run continuously ×
- 58. DO YOU REALLY NEED A BACKUP? if you got 3x replication over multiple availability zones, is that backup really worth it?
- 59. PRODUCTION IS THE ONLY REAL TEST ENVIRONMENT when thousands of things happen every second, new, weird and unforeseen things happen all the time, your tests can only cover the foreseeable =