Familiar Smells I've Detected in Your Systems Engineering Organization - SRECon APAC
0 likes171 views
This document summarizes a talk on systems engineering smells and fixes. It discusses common problems observed such as outsourced operations, lack of testing and automation, and suppressed volatility. It recommends adopting practices like infrastructure testing, automated deployments, and chaos engineering to improve reliability and prevent failures by learning from errors in a staged manner. Configuration should be coded and production readiness validated through gamedays to ensure systems can withstand failures.
![@davemangot
But, mah Load Balancer{
"Comment": "Creating Alias resource record sets in Route 53",
"Changes": [{
"Action": "CREATE",
"ResourceRecordSet": {
"Name": "elb.example.com",
"Type": "A",
"AliasTarget":{
"HostedZoneId": "Z1H1FL5HABSF5",
"DNSName": "ALB-xxxxxxxx.us-west-2.elb.amazonaws.com",
"EvaluateTargetHealth": false
}}
}]
}](https://image.slidesharecdn.com/smells-190614163515/85/Familiar-Smells-I-ve-Detected-in-Your-Systems-Engineering-Organization-SRECon-APAC-32-320.jpg)
Familiar Smells I've Detected in Your Systems Engineering Organization - SRECon APAC
- 1. @davemangot Familiar Smells I’ve Detected in Your Systems Engineering Organization… and How to Fix Them Dave Mangot SRECon APAC 2019
- 2. @davemangot Familiar Smells I’ve Detected in Your Systems Engineering Organization… and How to Fix Them Dave Mangot @davemangot
- 3. @davemangot About Me 20+ years in systems engineering, have led DevOps and Systems Engineering transformations at multiple companies Big companies: Cable & Wireless, Salesforce, SolarWinds Small Companies: Terracotta, Tagged, Librato International DevOps contributor
- 4. @davemangot
- 5. @davemangot What do you want to spend your engineering tokens on this week?
- 6. @davemangot
- 7. @davemangot I’ve got, so much trouble in my mind… Sir Joe Quarterman & Free Soul - I've Got So Much Trouble On My Mind - https://www.youtube.com/watch?v=0sZedHraIiY
- 9. @davemangot
- 10. @davemangot
- 11. @davemangot Outsourced Ops The road to hell is paved with short term projects that never died
- 12. @davemangot
- 13. @davemangot Don’t Fix Problems Solve Them
- 14. @davemangot Crawl - Walk - Run
- 16. @davemangot Conway’s Law - Eng/Ops VP of Architecture VP of Engineering VP of Operations
- 17. @davemangot Follow the Sun MTTR SLO You must be this tall…
- 18. @davemangot I’ve got, so much trouble in my mind… Sir Joe Quarterman & Free Soul - I've Got So Much Trouble On My Mind - https://www.youtube.com/watch?v=0sZedHraIiY
- 20. @davemangot Run Books (procedural) DON’T DO IT!
- 22. @davemangot Crawl - Walk - Run
- 23. @davemangot Infra Unit Tests • Rspec/Test Kitchen • ServerSpec • https://goss.rocks
- 24. @davemangot Tests in Staging • Stage is like prod • Stage is like prod • Stage is like prod
- 25. @davemangot Automated Deployment • Dark Launches / Feature Flags • Canaries • Blue Green Deploys • Cfg Management (stateful tiers)
- 26. @davemangot I’ve got, so much trouble in my mind… Sir Joe Quarterman & Free Soul - I've Got So Much Trouble On My Mind - https://www.youtube.com/watch?v=0sZedHraIiY
- 28. @davemangot
- 29. @davemangot Avoiding Failure •bonded network interfaces •RAID cards •SAN storage
- 30. @davemangot Taleb Black Swan “the problem with artificially suppressed volatility is not just that the system tends to become extremely fragile; it is that, at the same time, it exhibits no visible risks... These artificially constrained systems become prone to Black Swans. Such environments eventually experience massive blowups... catching everyone off guard and undoing years of stability or, in almost all cases, ending up far worse than they were in their initial volatile state” (p105)1. - Nassim Taleb https://continuousdelivery.com/2013/01/on-antifragility-in-systems-and-organizational-architecture/
- 31. @davemangot MTTR > MTBF 'So instead of trying to prevent errors, optimize for a relatively frequent occurrence of low-impact errors. “Failure free operations require experience with failure.” For infrequently occurring errors, this means we need to induce them.' - Aaron Blohowiak https://www.linkedin.com/pulse/reliability-works-aaron-blohowiak/?published=t (cattle) (pets)
- 32. @davemangot But, mah Load Balancer{ "Comment": "Creating Alias resource record sets in Route 53", "Changes": [{ "Action": "CREATE", "ResourceRecordSet": { "Name": "elb.example.com", "Type": "A", "AliasTarget":{ "HostedZoneId": "Z1H1FL5HABSF5", "DNSName": "ALB-xxxxxxxx.us-west-2.elb.amazonaws.com", "EvaluateTargetHealth": false }} }] }
- 33. @davemangot Remote Hands is not automation
- 34. @davemangot
- 35. @davemangot Crawl - Walk - Run
- 39. @davemangot Crawl - Walk - Run Stage is like prod (x 3) Choose Your Incentives!