ES Hero @ Instacart
- 1. ES Hero @ Instacart Nick Elser
- 2. Elasticsearch • Denormalized product catalog • Relatively dynamic queries + fast lookups • Multi-tenant • Hosted via Elastic Cloud • 18 clusters under management • Primary catalog is > 1.5TB (RAM), 3 primary clusters
- 3. Multitenant datastore Multiple teams leveraging Elasticsearch Each with their own access pattern
- 4. The Biggest Problem Lack of visibility
- 5. You can’t fix things if you don’t know what’s broken
- 6. Existing Monitoring • Slow Query Log • 3rd Party APM • Error reporting • Marvel • Raw logs
- 7. Problems 1. Which specific queries are causing problems? 2. Which API endpoints are causing problems? 3. How can we prioritize important calls?
- 8. These added up to huge problems Repeated outages during peak hours
- 9. ESHero to the rescue! Client-side instrumentation, aggregated
- 10. How do we anonymize and group like queries?
- 11. Remove all variables in queries InstacartSearch::QueryAnonymizer
- 13. Collected Attributes caller location cluster name user level data owner of query execution time status code query fingerprint request UUID controller#action application
- 14. Collecting all this data app Kinesis Druid HTTP middleware + query anonymizer Production ES ESHero Eventer
- 15. Back to the production problems
- 16. 1. Which specific queries are causing problems? 2. Which API endpoints are causing problems? 3. How can we prioritize important calls?
- 18. A few bad queries can really hurt the cluster
- 21. 1. Which specific queries are causing problems? 2. Which API endpoints are causing problems? 3. How can we prioritize important calls?
- 22. Group by API request UUID!
- 23. A single page load can result in hundreds of queries (N+1)
- 24. 1. Which specific queries are causing problems? 2. Which applications are causing problems? 3. How can we prioritize important work?
- 25. Arbitrary retries are extremely dangerous
- 26. Extremely fast queries that are likely to succeed are safe to retry *Manually whitelisted, for now
- 27. What’s next for ESHero? • open source • bindings for other languages • proxy support? Your feedback is really helpful!
- 28. We are hiring! 😀 Nick Elser github.com/nickelser Original slides by Jon Phillips twitter.com/elguapo1611 Thanks to John Meagher for feedback + content. Thank You! Questions?