[Test Bash Manchester] Observability and Testing
1 like309 views
The document discusses observability and testing systems. It notes that pre-production testing only covers known failure modes and observability is needed to understand unknown issues. Observability involves collecting metrics, structured logging, and distributed tracing to understand system behavior and troubleshoot problems. When combined with testing, observability can help reduce the need for extensive pre-production testing and allow issues to be discovered and debugged after deployment.
![@PierreVincent
A
F
H
D
J
B
E
C
G
a1b2c3
a1b2c3
a1b2c3
ERROR [svc=H][trace=a1b2c3] Failed to save order
Cause: Cassandra timeout exception
ERROR [svc=F][trace=a1b2c3] Failed to complete order
Cause: Shipping service responded with 500
ERROR [svc=A][trace=a1b2c3] Failed to process order
Cause: Order process manager responded with 500
a1b2c3
INFO [svc=G][trace=a1b2c3] Items verified in stock
Log Correlation](https://image.slidesharecdn.com/testbashmanchesterobservabilityandtesting-191003095825/85/Test-Bash-Manchester-Observability-and-Testing-14-320.jpg)
[Test Bash Manchester] Observability and Testing
- 1. @PierreVincent Observability & Testing Explore whatâs really happening under the hood March 8th, 2018 â QCon London @PierreVincent pvincent.io
- 3. @PierreVincent Pierre Vincent Infra. & Reliability Manager @PierreVincent pvincent.io
- 4. @PierreVincent Reaching production is only the beginning
- 5. @PierreVincent No system is immune to failure Be ready to recover
- 6. @PierreVincent When distributing a system, weâre also distributing the places where things might go wrong
- 7. @PierreVincent Pre-Prod Testing & Monitoring only cover known failure modes. What about everything else?
- 8. @PierreVincent Monitoring tells you whether the system works. Observability lets you ask why it's not working. â ââ Baron Schwarz
- 10. @PierreVincent System metrics Application metrics Business metrics CPU usage Error rates Customer conversions Metrics
- 12. @PierreVincent Watch out for over-reliance on metrics Limitations at high-cardinality Not every metric deserves an alert Real-time querying means some trade-offs on retention Limit alerting to user-impacting symptoms Poor fine-grained debugging e.g. CustomerId Not suitable for long-term trend analysis
- 13. @PierreVincent Searchable Correlated Logging Making sense of (a lot of) logs Centralised
- 14. @PierreVincent A F H D J B E C G a1b2c3 a1b2c3 a1b2c3 ERROR [svc=H][trace=a1b2c3] Failed to save order Cause: Cassandra timeout exception ERROR [svc=F][trace=a1b2c3] Failed to complete order Cause: Shipping service responded with 500 ERROR [svc=A][trace=a1b2c3] Failed to process order Cause: Order process manager responded with 500 a1b2c3 INFO [svc=G][trace=a1b2c3] Items verified in stock Log Correlation
- 15. @PierreVincent JSON 2018-02-20T16:38:23+00:00 ERROR Read timed out timestamp 2018-02-20T16:38:23+00:00 requestID ec667cb45 level ERROR team eventsservice registration-service commit 542a8b8e build 542a8b8e.7 node node_e79f3e52 log Read timed out region europe-west2 runtime java-1.8.0_161 When did it happen? Can I trace it? What is it? What is it running? Where is it? What is the message? customerID 55123Who caused it? userID 458 ... ...Any other info? Hmmm thanks⊠?
- 16. @PierreVincent Structured logs unleash high-cardinality exploration Error rate spike isolated by build version Activity spike isolated for single customer
- 17. @PierreVincent Tracing get_confirmed_attendees get_attendees get_confirm_status cassandra/select mysql/select event-mgt-api attendees-service registration-service Trace Spans 0ms 50ms 100ms 150ms 172ms 172ms 73ms 54ms 78ms 41ms
- 18. @PierreVincent
- 19. @PierreVincent Unknown Unknowns Known Unknowns Healthchecks Metrics (Alerting) Logs Metrics (Queries) Tracing Events Monitoring & Resiliency Debugging & Exploration
- 20. @PierreVincent Observability is the key to unlock new ways of testing
- 21. @PierreVincent Test (a little bit*) lessTest (a little bit*) lessDonât spend all your time testing ...keep some for instrumenting Thank you! @PierreVincent pvincent.io