Operating Consul as an Early Adopter
2 likes1,269 views
This talk discusses Stripe's experience adopting Consul early on to manage service discovery and configuration. Some key lessons learned include: - Initial rollouts had issues like memory leaks and TLS verification failures that caused outages - Watching all services at once in Consul template led to self-DDOS due to high churn - Stripe encountered goroutine leaks that consumed memory until fixes were merged upstream - Moving to generate DNS from Consul template improved stability over watching directly - Incremental rollout, monitoring, limiting scope, isolation and being willing to help upstream improved experience as an early adopter over time.
![Rolling out 0.3
• 0.3 changed TLS verification to check the
cert name
• Change our SSL issuing to add SANs
• 2014/06/16 16:52:57 [ERR] raft: Failed to make RequestVote
RPC to 10.100.29.175:8300: x509: certificate is valid for
[remote host], not [local host]](https://image.slidesharecdn.com/hashiconf-consul-2015-150929162756-lva1-app6891/85/Operating-Consul-as-an-Early-Adopter-12-320.jpg)
Operating Consul as an Early Adopter
- 1. a talk Nelson Elhage, @nelhage Operating Consul As an Early Adopter
- 2. This Talk • consul @ Stripe • War Stories • Lessons Learned
- 3. Consul at Stripe The Good, The Bad, The Outages
- 4. Why Consul? • Early 2014 • Stripe Infra gaining complexity • Nightmarish in-house service registry • Host lists distributed via puppet
- 5. Why Consul? • Wanted a better service/host store • consul had everything baked in • Decided to do some test deployments
- 6. Initial Rollout • Rolled out across all servers • (started with bake-in in QA) • No clients at all
- 7. What Could Go Wrong? • We worried about memory leaks
- 8. Our First Production Issue • Noticed one node taking >100M RAM • (others all <50M) • Reached out to armon for advice • bug in the stats framework: • https://github.com/armon/go-metrics/commit/02567bbc4f518a43853d262b651a3c8257c3f141
- 9. StartedAdding Clients • Hooked into our deploy tool • kept a manual emergency fallback • Generated LB config from consul • Noticed a surprising rate of errors
- 10. Raft Instability • Seeing >1 failover/minute • Reached out toArmon • “Try 0.3” • “consul is not optimized for spinning disk”
- 11. Rolling out 0.3 • Roll to QAfirst • Nothing works! • Check logs: TLS verification errors
- 12. Rolling out 0.3 • 0.3 changed TLS verification to check the cert name • Change our SSL issuing to add SANs • 2014/06/16 16:52:57 [ERR] raft: Failed to make RequestVote RPC to 10.100.29.175:8300: x509: certificate is valid for [remote host], not [local host]
- 13. 0.3 TLS Woes • Whoops! consul was checking the remote cert against the local node name • armon> we just use "demo.consul.io" as the CN for all of them • 0.3 essentially completely broke TLS
- 14. 0.3.1 • I wrote and got merged a patch to restore 0.2 behavior • Rolled forward to 0.3.1 • Upgraded to SSD-backed servers
- 15. Increasing Rollout • Switched various operational tools from flatfile to consul • Main app started using consul at startup
- 17. consul-template • Generating haproxy config using consul-template • https://github.com/hashicorp/consul-template/ issues/168 – `consul-template` takes O(N²) time with N services
- 18. consul-template • Got that fixed, turned it on • consul immediately fell over • multiple elections/minute • 2M allocations/minute
- 19. consul-template • Service Watches churn when any service changes health state • Watching services on a large cluster → self-DDOS
- 20. consul-template • We use `consul-template -once` in cron now • Worse latency, but it works reliably
- 21. consul for leader election • Our data team wanted a leader-election primitive • Built on top of consul, cribbing example code
- 22. Sometime Later…
- 23. goroutine leak • consul would rapidly eat all memory • larger heap -> large GC pauses -> raft instability • manually restarted cluster 1/day
- 24. goroutine leak • Reached out toArmon • Very helpful in debugging • Found several unrelated memory leaks
- 25. goroutine leak • Tried to figure out what changed • Eventually correlated to a session leak in our leader election code
- 26. goroutine leak • Fixed our leader-election code • New policy: No non-discovery uses of consul
- 27. consul DNS • Increasingly reliant on consul for internal discovery • Unhappy at exposure to periodic instability • Still have fallbacks, but outages remain painful
- 28. consul DNS • Solution: Use consul-template to compile consul DNS to a zone file • Serve that out of a normal DNS server • Refresh every 15s
- 29. Current Status • Run consul everywhere • Register all services • Request-path lookups hit cached DNS • Operational tools use HTTP interface • Also generate config from consul-template
- 30. Final Stability Note • consul 0.5.2 fixed our memory leaks • consul has been quite stable for us of late • consul-template watches still don’t scale • 0.6 should help
- 31. Lessons Learned being an early adopter without bringing down the site (too many times)
- 32. Expect It To Be Rough
- 34. (graph all the things)
- 36. Limit Scope
- 37. Isolation
- 39. Get To Know Upstream
- 40. Be Willing to Dive In
- 41. Questions?