Logging in The World of DevOps
0 likes184 views
This document summarizes Cecep Mahbub's experience working on logging infrastructure for various companies and discusses best practices for centralized logging. It outlines Cecep's background maintaining logging systems at KMK and BBM, and their current role setting up Bukalapak's logging infrastructure on Google Cloud Platform. Key topics covered include common uses of logs, benefits of a centralized logging server, examples of logging servers and SaaS options, and integrating logs into DevOps processes.
Logging in The World of DevOps
- 1. Cecep Mahbub cecep.mahbub@bukalapak.com Logging In The World of DevOps
- 2. About Me 3.5 years with KMK (Liputan6, BBM, Vidio). In the last 2 years working on Logging Infrastructure for BBM. On the last Google Cloud Summit Jakarta (Oct, 2018), I gave a presentation entitled "Logging at Petabytes scale on GCP" Starting this month (Apr 2019), I have been joining at Bukalapak. I am working on migration Bukalapak logging infrastructure to GCP
- 3. Searching on your logs, is usually just a simple steps like this ssh admin@192.168.1.101 cd /var/log/ grep "something interesting" app.log
- 4. Then your user base is growth significantly... And now you have more servers. ssh admin@192.168.1.101 cd /var/log/ grep "something interesting" app.log ssh admin@192.168.1.102 cd /var/log/ grep "something interesting" app.log
- 6. and bigger logs size
- 9. Common Usage of Logs ● Debugging ● Source of Metrics/Monitoring ● Audit tracking
- 10. Centralized Logs Server ● Logs is indexed, searching on your logs should be fast ● Easy to aggregate search result ● Easy to create data visualization (trending over time, spot anomali traffic, comparing current data with previous time windows) ● Role based access control ● Scalable (horizontal/vertical)
- 11. Logs Servers ● ELK (Elasticsearch, Logstash, Kibana) ● Graylog ● Splunk
- 12. Logs Servers (Cloud) ● CloudWatch (AWS) ● Stackdriver (GCP) ● Azure? ● Alibaba Cloud?
- 13. Logs Servers (SaaS) ● Datadog ● Elastic Cloud ● Loggly ● Scalyr
- 14. How to integrate your logs into DevOps process
- 15. Everyone should have an access to your logging tools ● Everyone should know how to search their logs (i.e learn query DSL) ● Minimize Silo ● Team is not too dependent on other teams ● Security and personal data protection still applied
- 16. Examples Case: Monitoring Dashboard
- 17. Examples Case: Canary Release Source: https://www.infoq.com/articles/gilt-deploying-microservices-aws
- 18. Logging Infrastructure at Bukalapak
- 19. ~7 TB logs/days ● Net Appliance ● Microservices running under Docker / Kubernetes ● Application Logs ● Server App Logs (Nginx, MySQL, MongoDB etc)
- 21. Current Issues ● Too many index (1 microservices ~ 1 index) ● Field mapping conflict ● Unparsed logs
- 22. Bukalapak Logging Infrastructure (GCP)
- 23. Thank You!