Debugging your varnish instance
0 likes68 views
Varnish debugging involves gathering data from tools like varnishgather, varnishlog, and varnishtop to diagnose different types of issues like stability problems, functional bugs, operational performance issues, and capacity limitations. The document provides examples of how to use these tools to debug crashes, trace code execution, examine request headers, monitor hit rates and response times, and analyze backend communication. It emphasizes that debugging requires understanding the problem being experienced and leveraging tools like varnishlog to obtain request details, error codes, and timing information.
![Some examples
● [5-9]xx responses:
○ varnishlog -q 'RespStatus >= 500'
● Misses:
○ varnishlog -q 'VCL_call ~ MISS'
● Timing details:
○ varnishlog -i Timestamp
● Requests taking more than 5 seconds:
○ varnishlog -q 'Timestamp:Fetch[2] > 5.’
● Top requested URLs:
○ varnishtop -i ReqURL](https://image.slidesharecdn.com/debuggingyourvarnishinstance-160923051849/85/Debugging-your-varnish-instance-14-320.jpg)
Debugging your varnish instance
- 2. Debugging your Varnish Instance Varnish Summit, Los Angeles
- 3. Federico G. Schwindt Technical Presales & Engineer London, UK fgsch@varnish-software.com
- 4. Problems ● Stability ● Functional ● Operational ● Capacity ● Not my problem™
- 5. Getting the data ● https://github.com/varnish/varnishgather was born ● Gathers OS,Varnish and companion tools details ● One file per command
- 6. Stability ● Crash, panic or something else? ● Check varnishadm panic.show and syslog ● Start from command line ● Install debugging symbols, allow coredumps ● Get your hands dirty
- 7. Functional ● Code not reached, wrong logic or expectations ● printf debugging (std.log / std.syslog) ● Enable tracing (param.set vsl_mask +VCL_trace) ● Reproduce with varnishtest ● Check varnishlog
- 8. Anatomy of a .vtc varnishtest "Hello world" server s1 { rxreq expect req.http.msg == "hello" txresp -body "world" } -start varnish v1 -vcl+backend { sub vcl_recv { set req.http.msg = "hello"; } } -start client c1 { txreq rxresp expect resp.status == 200 expect resp.body == "world" } -run
- 9. varnishlog $ varnishlog -g request - Begin req 459063 rxreq - Timestamp Start: 1474493939.499338 0.000000 0.000000 - Timestamp Req: 1474493939.499338 0.000000 0.000000 - ReqStart 1.2.3.4 38780 - ReqMethod GET - ReqURL / - ReqProtocol HTTP/1.1 - ReqHeader Host: example.com - ReqHeader Accept: */* - ReqHeader Accept-Encoding: gzip - ReqHeader User-Agent: Mozilla/5.0 (X11; Linux x86_64) ... - ReqHeader Connection: close - ReqHeader X-Forwarded-For: 1.2.3.4 - VCL_call RECV - VCL_return hash - VCL_call HASH - VCL_return lookup - VCL_call MISS ...
- 10. Operational ● Low hitrate or high response times ● Inspect headers on the client side ● Use varnishtop ● Increase your EXP with varnishlog ● Understand the waitinglist
- 11. Request coalescing ● Automatically enabled for misses ● Only one request at a time goes to the backend ● Other requests for the same object will wait ● Avoids overloading the backend
- 12. Capacity ● Inadequate number of threads ● Dataset too big / cache too small ● Monitor varnishstat output ● Did I say varnishlog?
- 13. Not my problem ● “It must be the cache” axiom ● Test bypassing Varnish ● Check timeouts ● The truth is on the wire (aka look at tcpdump)
- 14. Some examples ● [5-9]xx responses: ○ varnishlog -q 'RespStatus >= 500' ● Misses: ○ varnishlog -q 'VCL_call ~ MISS' ● Timing details: ○ varnishlog -i Timestamp ● Requests taking more than 5 seconds: ○ varnishlog -q 'Timestamp:Fetch[2] > 5.’ ● Top requested URLs: ○ varnishtop -i ReqURL
- 15. The bottom line ● It is not always the cache ● Not all errors are the same ● Become familiarised with the tools ● varnishlog is the swiss army knife of varnish
- 16. Thank you!