predicting-m3-devopsconMunich-2023.pptx
- 1. Predicting Production Outages: Unleashing the Power of Micro-Metrics Ram Lakshmanan Architect yCrash
- 6. GC Throughput Micrometric Source: Garbage Collection Log
- 7. How does 96% GC Throughput sound? 1 day = 1440 Minutes (i.e., 24 hours x 60 minutes) 96% GC Throughput means app pausing for 57.6 minutes/day 7 What is GC Throughput? Amount of time application spends in processing customer transactions vs Amount of time application spends in processing garbage collection activity
- 8. 1. GC Log 10. netstat 12. vmstat 2. Thread Dump 9. dmesg 3. Heap Dump 6. ps 8. Disk Usage 5. top 13. iostat 11. ping 14. Kernel Params 15. App Logs 16. metadata 4. Heap Substitute 7. top -H 8 Open-source script: https://github.com/ycrash/yc-data-script 360° Data
- 10. Application Architecture JDBC SOAP MainFrame REST Server Thread Pool Application Server HTTP(S) request 10
- 11. Application Architecture JDBC SOAP MainFrame REST Server Thread Pool Application Server HTTP(S) request 11
- 12. Threads with identical Stack trace Micrometric Source: Thread Dump
- 13. 13 Case Study Backend Slowdown in a Major Financial Institution in N. America
- 15. top –H –p <PROCESS_ID>’ Secrete Option: 15 We all might have used ‘top’
- 16. Case Study Major Trading app in N. America https://blog.fastthread.io/2020/04/23/troubleshooting-cpu-spike-in-a-major-trading-application/ 16
- 18. public void synchronized getData() { doSomething(); } Thread 1 Thread 2 Thread 1 BLOCKED THREADS Concurrency Problem 18
- 19. BLOCKED state threads Micrometric Source: Thread Dump
- 20. Case Study Major Leisure Travel Service Provider https://blog.fastthread.io/2020/04/23/troubleshooting-cpu-spike-in-a-major-trading-application/ 20
- 22. My App yCrash agent yCrash Server Container/Machine 1 Every 3 minutes Micro-Metrics* are captured 2 Metrics are transmitted 4 If problem forecasted, 360 ° data capture is triggered 3 ML, Patterns applied on the Micro-Metrics Cloud/On-premise 22 Micro-Metrics * 1. Garbage Collection Log 2. Thread Dump + top –H 3. Application Log Micro-Metrics Monitoring Architecture
- 23. 1. GC Log 10. netstat 12. vmstat 2. Thread Dump 9. dmesg 3. Heap Dump 6. ps 8. Disk Usage 5. top 13. iostat 11. ping 14. Kernel Params 15. App Logs 16. metadata 4. Heap Substitute 7. top -H 23 Open-source script: https://github.com/ycrash/yc-data-script 360° Data
- 24. Ram Lakshmanan ram@tier1app.com @tier1app https://www.linkedin.com/company/ycrash This deck will be published in: https://blog.ycrash.io Learn to troubleshoot like a pro with my online training program 24 THANK YOU FRIENDS