Devoxx Belgium 2022 - Debugging distributed systems
0 likes92 views
This document outlines a structured approach for debugging distributed systems. It begins with observing and documenting what is known about the problem. The next steps are to create a minimal reproducer, debug the client side, check DNS and routing, and inspect the connection. Further debugging involves inspecting traffic and messages, debugging the server side, and wrapping up with documentation and a post-mortem. Key tools mentioned include logging, tracing, tests, SSH, network inspection tools, and remote debugging. The document emphasizes focusing on details, eliminating single points of failure, and gaining an understanding of potential failure modes through experience debugging real issues.
Devoxx Belgium 2022 - Debugging distributed systems
- 1. bertjan@openvalue.eu Debugging distributed systems Bert Jan Schrijver @bjschrijver
- 2. Debugging distributed systems: the good parts bertjan@openvalue.eu Bert Jan Schrijver @bjschrijver Networking 101 How the internet works
- 3. Why?
- 4. Bert Jan Schrijver L e t ’ s m e e t @bjschrijver
- 5. Why are distributed systems difficult? Networking 101 What? Why? ✅ Demo War stories Conclusion W h a t ‘ s n e x t ? Outline A structured approach @bjschrijver
- 6. What is a distributed system?
- 7. A distributed system is a system whose components are located on different networked computers which communicate and coordinate their actions by passing messages to one another.
- 8. • Concurrency of components • Lack of a global clock • Independent failure of components ➡ Distributed systems are harder to reason about Characteristics of distributed systems Source: http://www.nasa.gov/images/content/218652main_STOCC_FS_img_lg.jpg
- 9. Working with distributed systems is fundamentally different from writing software on a single computer - Martin Kleppmann - and the main difference is that there are lots of new and exciting ways for things to go wrong. “ ” Photo: Dave Lehl ”
- 10. Why do things go wrong? “ ” Photo: Dave Lehl
- 11. The fallacies of distributed computing are a set of assertions made by L Peter Deutsch and others at Sun Microsystems describing false assumptions that programmers new to distributed applications invariably make.
- 12. 1. The network is reliable; 2. Latency is zero; 3. Bandwidth is infinite; 4. The network is secure; 5. Topology doesn't change; 6. There is one administrator; 7. Transport cost is zero; 8. The network is homogeneous. Fallacies of distributed computing
- 13. What could possibly go wrong? “ ” Photo: Dave Lehl
- 14. OSI & TCP/IP Source: https://www.guru99.com/difference-tcp-ip-vs-osi-model.html
- 15. .. in your browser’s address bar and press Enter What happens when you type google.com… Source: https://github.com/alex/what-happens-when
- 16. 16
- 18. A structured approach to debugging distributed systems @bjschrijver Check DNS & routing Check connection Debug client side Create minimal reproducer Debug server side Observe & document Wrap up & post mortem Inspect traffic / messages
- 19. Step 1: Observe & document • What do you know about the problem? • Inspect logging, errors, metrics, tracing • Draw the path from source to target - what’s in between? Focus on details! • Document what you know • Can we reproduce in a test? • By injecting errors, for example Tools Whiteboard, documentation, logging, metrics, tracing (opentracing.io), tests, jepsen.io
- 20. Step 1: Observe & document
- 21. Step 2: Create minimal reproducer • Goal: maximise the amount of debugging cycles • Focus on short development iterations / feedback loops • Get close to the action! Tools IDE, Shell scripts, SSH tunnels, Curl
- 22. Step 3: Debug client side • Focus on eliminating anything that could be wrong on the client side • Are we connecting to the right host? • Do we send the right message? • Do we receive a response? • Not much different from local debugging Tools IDE, debugger, logging
- 23. Step 4: Check DNS & routing • DNS: • Make sure you know what IP address the hostname should resolve to • Verify that this actually happens at the client • Routing: • Verify you can reach the target machine Tools host, nslookup, dig, whois, ping, traceroute, nslookup.io, dnschecker.org
- 24. Step 5: Check connection • Can we connect to the port? • If not, do we get a REJECT or a DROP? • Does the connection open and stay open? • Are we talking TLS? • What is the connection speed between us? Tools telnet, nc, curl, iperf
- 25. Step 6: Inspect traffic / messages • Do we send the right request? • Do we receive the right response? • How do we know? • How do we handle TLS? • Are there any load balancers or proxies in between? Tools curl, wireshark, tcpdump, network tab in browser, mitm/tls proxy
- 26. Step 7: Debug server side • Inspect the remote host • Can we attach a remote debugger? • See https://youtube.com/OpenValue • Profiling • Java Flight Recorder • Strace Tools SSH tunnels, remote debugger, profiler, strace, JFR
- 27. Step 8: Wrap up & post mortem • Document the issue: • Timeline • What did we see? • Why did it happen? • What was the impact? • How did we find out? • What did we do to mitigate and fix? • What should we do to prevent repetition? Tools Whiteboard, documentation
- 28. If you really want a reliable system, you have to understand what its failure modes are. You have to actually have witnessed it misbehaving. - Jason Cahoon “ ”
- 29. Distributed systems war stories
- 30. The one where it worked half of the time…
- 31. The one at a school…
- 33. The one with breaking news…
- 34. Summary: a structured approach to debugging distributed systems @bjschrijver Check DNS & routing Check connection Debug client side Create minimal reproducer Debug server side Observe & document Wrap up & post mortem Inspect traffic / messages
- 37. Thanks for your time. Got feedback? Tweet it! All pictures belong to their respective authors @bjschrijver