Protecting Your Cluster from Your Humans
0 likes3,781 views
The document discusses strategies for protecting a data cluster at Kroger from human-related issues, specifically within their logging solution, Echo. It identifies various problems such as excessive data ingestion, sensitive data handling, and high query costs, along with proposed solutions like custom ingestion pipelines, data masking, and alert systems. The emphasis is on encouraging best practices among users to enhance performance and observability.
Protecting Your Cluster from Your Humans
- 1. Protecting Your Cluster From Your Humans Jaclyn Priede
- 2. Protecting Your Cluster From Your Humans Jaclyn Priede
- 3. 3 Kroger • Grocery retailer offering in store and online shopping • Founded by Barney Kroger 1883 • 2,700 stores • Ranked 20 in Fortune 500
- 4. 4 Echo • Kroger’s Enterprise Log Aggregation Solution • Based on Elasticsearch and Kibana • Custom Solutions: • Ingestion • Alerting • Data management • Rollup • Clients • Why Echo?
- 5. 5 Utilization Echo Stats Over a thousand aliases 165 teams 100K Messages/Second 71.7 TB Storage Use Cases SOC Development SRE ORE Alerting
- 6. 6 Have you ever had a query cripple your cluster? Is that acceptable?
- 7. 7 Data Ingestion • Problem: Sending too much data to Echo • Solution: • Custom Ingestion Pipeline • Default quota • Rates Page
- 8. 8 Data Ingestion • Problem: Sending too much data to Echo • Solution: • Custom Ingestion Pipeline • Default quota • Rates Page
- 9. 9 Data Ingestion • Problem: Sending too much data to Echo • Solution: • Custom Ingestion Pipeline • Default quota • Rates Page
- 10. 10 Sensitive Data • Problem: Sending sensitive data to Echo
- 11. 11 Sensitive Data • Problem: Sending sensitive data to Echo • Solution: • Terms and Conditions • Protected Spaces • Data Masking
- 12. 12 Batch Failures • Problem: • Parsing Failures • Indexing Failures • Solution: • Alerting users
- 13. 13 Data Storage • Problem: Storing too much data in Echo • Solution: • Default retention • Alerting on disk space
- 14. 14 Indexing Fields • Problem: • Fields with too many characters • Fields with high cardinality data • Solution: • Truncating fields • Kafka & Elasticsearch • Alerting users
- 15. 15 Expensive Queries • Problem: Executing expensive queries in Elastic • Solution: • Slow Logging • Slow Indexing
- 16. 16 Long Running Tasks • Problem: Tasks failing to complete in a timely fashion • Solution: • Automated job to kill tasks >5 min
- 17. 17 Refresh Rates • Problem: Excessively high refresh rates in Kibana • Solution: • Removing refresh intervals under 30 seconds from the UI
- 18. 18 High Search Volume • Problem: Excessively high search volumes against Elastic • Solution: • Audit Log • Alerting users
- 19. 19 Coaching Humans • Encourage people to be developers • Advocate best practices for efficient querying • Structure your data to make it easier to search
- 20. 20 Benefits Improved performance Better end user experience Increased observability throughout the Enterprise
- 21. Thank you