Never gonna give you up
1 like866 views
The document discusses 7 lessons learned from building high performance and highly available systems. The lessons are: 1) Make assumptions explicit and keep challenging them, 2) Performance and high availability are not extra features, 3) Do not reinvent the wheel but keep things simple, 4) Be wary of cargo-cult optimization, 5) High availability requires more than just redundancy, 6) Embrace diversity, and 7) Monitoring is essential but can be improved.
![@EdMcBane
LESS(1) General Commands Manual LESS(1)
NAME
less - opposite of more
SYNOPSIS
less -?
less --help
less -V
less --version
less [-[+]aABcCdeEfFgGiIJKLmMnNqQrRsSuUVwWX~]
[-b space] [-h lines] [-j line] [-k keyfile]
[-{oO} logfile] [-p pattern] [-P prompt] [-t tag]
[-T tagsfile] [-x tab,...] [-y lines] [-[z] lines]
[-# shift] [+[+]cmd] [--] [filename]...
(See the OPTIONS section for alternate option syntax with long option
names.)
DESCRIPTION
LESS IS similar to MORE (1), but has many more features.
Less does not have to read the entire input file before starting, so
with large input files it starts up faster than text editors like vi
(1). Less uses termcap (or terminfo on some systems), so it can run on
Manual page less(1) line 1 (press h for help or q to quit) .](https://image.slidesharecdn.com/nevergonnagiveyouup-150411054216-conversion-gate01/85/Never-gonna-give-you-up-16-320.jpg)
Never gonna give you up
- 1. @EdMcBane 7 lessons learned building HP/HA systems Never gonna give you up Never gonna let you down
- 2. @EdMcBane Francesco Degrassi Enthusiastic yet pragmatic Lean Software Developer. Uppish and cynical nihilist from time to time.
- 3. @EdMcBane Lean Software Development and team coaching Continuous Delivery, High availability, performance Security sensitive & high uncertainty domains
- 4. @EdMcBane The challenge ● Primary european client ● Innovative service for the consumer market ● Large userbase (200K+ users) ● Very high request rate ● Low latency requirement (<< RTT)
- 6. @EdMcBane What did we learn?
- 7. @EdMcBane Make your assumptions explicit and keep testing them Don’t eat the yellow snow
- 8. @EdMcBane Make your assumptions explicit and keep testing them #1 Make your assumptions explicit and keep challenging them
- 9. @EdMcBane Make your assumptions explicit and keep testing them #2 Performance & High Availability are not extra features
- 10. @EdMcBane
- 11. @EdMcBane Make your assumptions explicit and keep testing them #3 Do not reinvent the wheel ...but keep things simple
- 12. @EdMcBane
- 13. @EdMcBane ● Everything was good with the single core scenario In our case...
- 14. @EdMcBane SO_REUSEPORT For TCP, so_reuseport allows multiple listener sockets to be bound to the same port. Received packets are distributed to multiple sockets bound to the same port using a 4-tuple hash. With so_reuseport the distribution is uniform.
- 15. @EdMcBane Everything should be made as simple as possible, but not simpler — Albert Einstein
- 16. @EdMcBane LESS(1) General Commands Manual LESS(1) NAME less - opposite of more SYNOPSIS less -? less --help less -V less --version less [-[+]aABcCdeEfFgGiIJKLmMnNqQrRsSuUVwWX~] [-b space] [-h lines] [-j line] [-k keyfile] [-{oO} logfile] [-p pattern] [-P prompt] [-t tag] [-T tagsfile] [-x tab,...] [-y lines] [-[z] lines] [-# shift] [+[+]cmd] [--] [filename]... (See the OPTIONS section for alternate option syntax with long option names.) DESCRIPTION LESS IS similar to MORE (1), but has many more features. Less does not have to read the entire input file before starting, so with large input files it starts up faster than text editors like vi (1). Less uses termcap (or terminfo on some systems), so it can run on Manual page less(1) line 1 (press h for help or q to quit) .
- 17. @EdMcBane Make your assumptions explicit and keep testing them #4 Be wary of cargo-cult optimization
- 18. @EdMcBane
- 19. @EdMcBane TCP_TW_RECYCLE Enable fast recycling TIME-WAIT sockets. Default value is 0. It should not be changed without advice/request of technical experts. Linux will drop any segment from the remote host whose timestamp is not strictly bigger than the latest recorded timestamp TCP_TW_RECYCLE + NAT = MADNESS
- 20. @EdMcBane
- 21. @EdMcBane Make your assumptions explicit and keep testing them #5 High Availability is much more than just redundancy
- 22. @EdMcBane
- 23. @EdMcBane ● Redundant hardware ● Redundant software components But there’s more! ● Graceful degradation ● Incremental rollouts Failure impact
- 24. @EdMcBane Failure frequency But then also: ● proven technology ● high quality hardware ● automation (to avoid errors)
- 25. @EdMcBane ● Effective monitoring ○ realtime ○ reliable ○ understandable ○ thorough ○ meaningful ○ actionable ● Rollback / rollforward ● Automation (for speed) Time to recover
- 26. @EdMcBane Our response plan goes something like this... AaaaaAAaaaah
- 27. @EdMcBane ...but be prepared to improvise ● In house experience ● Developers on call ● Drills (chaos monkeys) Processes designed for ordinary times are not resilient in a crisis and need to be changed.
- 28. @EdMcBane Make your assumptions explicit and keep testing them #6 Embrace diversity
- 29. @EdMcBane
- 30. @EdMcBane
- 31. @EdMcBane Make your assumptions explicit and keep testing them #7 Monitoring is essential … and we can do way better
- 32. @EdMcBane No one size fits all ● “Monitor everything”, like “100% test coverage” is a nice slogan. ● Each environment requires a slightly different solution ● Balance between data availability, cost and ability to keep it actionable
- 33. @EdMcBane
- 34. @EdMcBane We are doing logging wrong ● Unstructured ● Inconsistent ● Poor defaults ● Complex, obscure components ● A huge waste of computing power
- 35. @EdMcBane We need a complete overview ● Logs ● Metrics ● Alerts ● Together, coherent, cross-referenced
- 36. @EdMcBane Human beings, who are almost unique in having the ability to learn from the experience of others, are also remarkable for their apparent disinclination to do so. Douglas Adams “ ”