Badge Poser v3.0 - A DevOps Journey
0 likes53 views
The document outlines a series of steps and troubleshooting strategies taken by Fabio Cicerchia, a software and cloud engineer, in managing and configuring a virtual machine and its software stack, including Apache, PHP, Redis, and MySQL. It discusses a range of issues encountered, fixes applied, transitions to AWS services, and emphasizes the importance of observability and efficient resource management. Key takeaways stress the significance of careful versioning, monitoring, and using specific versions of software to avoid potential complications.
Badge Poser v3.0 - A DevOps Journey
- 3. Hello! I AM FABIO CICERCHIA SW & Cloud Engineer @ You can find me at: @fabiocicerchia
- 4. Disclaimer
- 8. Let’s Start
- 13. Step #1 What I got myself into?!
- 15. Describe VM Config RAM: 2GB CPU: 2 HDD: 50GB Software: Apache 2.4.10, PHP 5.6.19, Redis 2.8.17, MySQL 5.5.47
- 16. ● Apache v2.4.10 ○ Released on 2014-07-19: Age 6 years ○ Available v2.4.43 ● PHP v5.6.19 ○ Released on 2016-03-03: Age 4 years ○ Available v7.4.5 ○ EOL: 2018-12-31 http://archive.apache.org/dist/httpd/ https://www.php.net/releases/index.php https://www.php.net/supported-versions.php https://github.com/redis/redis https://docs.redislabs.com/latest/rs/administering/product-lifecycle/ Describe VM Config - Notes ● Redis v2.8.17 ○ Released on 2014-09-19: Age 6 years ○ Available v6.0.1 ● MySQL v5.5.47 ○ Released on 2015-12-07: Age 5 years ○ Available v8.0.20
- 17. Step #2 What do I need to do?!
- 18. Define a “plan”
- 19. Step #3 Find time to do it
- 21. ● Nginx v1.18.0 ● PHP v7.4.7 ● Redis v4.0.10 Just Start! https://www.nginx.com/ https://www.php.net/ https://redis.io/
- 23. ● Ansible → Provisioning ● Ansible Galaxy → Ansible’s Recipes Repo ● AWS CloudFormation → Infrastructure as Code* ● Let’s Encrypt → SSLTLS Certificate** * Terraform is way cooler **Yes, SSL is deprecated ...Then Refine https://www.ansible.com/ https://galaxy.ansible.com/ https://aws.amazon.com/cloudformation/ https://letsencrypt.org/
- 25. https://github.com/PUGX/badge-poser/blob/master/sys/cloudformation/alpine-stack.yaml
- 27. Ansible: What’s for? - Ansible is perfect for VMs (for example EC2 in our scenario). - It is redundant for ECS with Fargate, since the underlying layer is fully managed by AWS. - It could be useful for ECS without Fargate, so it’ll provision the EC2 where the containers will run. - Useful for deploy and rollback.
- 29. Wait for it...
- 33. Start Throwing a Bunch of Things At It
- 34. ● pm.max_children = 150 pm.start_servers = 5 pm.min_spare_servers = 5 pm.max_spare_servers = 35 ● emergency_restart_threshold 10 emergency_restart_interval 1m process_control_timeout 10s ● memory_limit = 192M Workaround #1: Not Quite There Yet
- 36. Added Logz.io & Filebeat Added UptimeRobot It Keeps Crashing: Need Visibility https://logz.io/ https://www.elastic.co/beats/filebeat https://uptimerobot.com/
- 45. Redis Down: OOM Killer http://turnoff.us/geek/oom-killer/
- 48. Handle Redis Daemon via Supervisor Redis Down: OOM Killer: Workaround #2
- 49. Zero CPU Credits
- 50. Zero CPU Credits CPU capped at 20% https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/burstable-credits-baseline-concepts.html
- 53. CPU capped at 20% http://nginx.org/en/docs/http/ngx_http_fastcgi_module.html
- 54. Zero CPU Credits
- 56. Step #6 Where Are We At Now?
- 61. ● AWS ECS ● AWS ECR Container - Part 1 https://aws.amazon.com/ecs/ https://aws.amazon.com/ecr/
- 63. OOM Killer - The Revenge http://turnoff.us/geek/oom-killer/
- 65. OOM Killer - The Revenge: Workaround #3 ● Set Autoscaling fixed to min 1 running container
- 66. ● Split All-In-One Container in Multi Container ● Use Alpine Container - Part 2
- 67. OOM Killer - Highlander
- 69. One Step Back
- 75. Despite the Working Fix...
- 76. ...Alpine Wasn’t Quite Stable
- 78. Since the multi-container on Alpine was unstable just switched back to the good ol’ working one-container-has-all on Debian. Switch Back to All-in-One Debian
- 81. Alpine: Trial & Errors
- 85. Caching to the rescue
- 86. NO STALE!
- 87. MISS – The response was not found in the cache and so was fetched from an origin server. The response might then have been cached. BYPASS – The response was fetched from the origin server instead of served from the cache because the request matched a proxy_cache_bypass directive (see Can I Punch a Hole Through My Cache? below.) The response might then have been cached. EXPIRED – The entry in the cache has expired. The response contains fresh content from the origin server. Cache Statuses https://www.nginx.com/blog/nginx-caching-guide/
- 88. Cache Statuses STALE – The content is stale because the origin server is not responding correctly, and proxy_cache_use_stale was configured. UPDATING – The content is stale because the entry is currently being updated in response to a previous request, and proxy_cache_use_stale updating is configured. REVALIDATED – The proxy_cache_revalidate directive was enabled and NGINX verified that the current cached content was still valid (If-Modified-Since or If-None-Match). HIT – The response contains valid, fresh content direct from the cache. https://www.nginx.com/blog/nginx-caching-guide/
- 90. Moving away from EC2 and from Logz.io. Again?
- 91. Need to know the traffic trend CloudWatch
- 92. Get More Metrics & Desiderata
- 93. 0 0
- 108. Dark Canary 10% / 25% 100% https://landing.google.com/sre/workbook/chapters/canarying-releases/
- 112. FORKED TRAFFIC
- 116. Interlude #3 Full Page Caching w/ Redis
- 118. Step #11 Uptime
- 124. ...but at the end....
- 130. Confidence Level
- 131. Step #12 Billing
- 134. Elastic Static IP with Global Accelerator
- 139. Reduce Costs!
- 140. So what did I learn?!
- 141. Key Takeaways - Never trust code - Never trust yourself - Do small steps - It’ll help you figuring out what went wrong - Version everything - Commit as often as possible - Never use latest tag - Use specific versions - Think outside the box - Don’t stick to playing by the manual - Prefer quick and easy fixes - Reduce the odds of breaking things - Use the tools to make your life easier - So choose them carefully - Monitor & Benchmark! - Your best friends for troubleshooting * random order
- 142. Questions?
- 143. Thank You!