![TRIAGE
Urgent Severity] Red cluster
Vital signs
➔ Cluster in red health
➔ No ingest into any
indices
Symptoms
➔ Beats fail to ingest
➔ Cluster is responsive, search and REST API still work](https://image.slidesharecdn.com/troubleshootingyourelasticsearchclusterlikeasupportengineer-210709095514/85/Troubleshooting-your-elasticsearch-cluster-like-a-support-engineer-15-320.jpg)
![TRIAGE
Urgent Severity] Red cluster
What happened?
➔ Out of the blue, no changes
Any attempts to fix it?
➔ No
Next steps
➔ Share a support diagnostics that will provide REST API calls
https://www.elastic.co/blog/why-does-elastic-support-keep-asking-for-diagnostic-files
https://github.com/elastic/support-diagnostics/blob/main/src/main/resources/elastic-rest.yml
https://github.com/elastic/support-diagnostics
> ./diagnostics.sh --host https://localhost -u elastic -p --port 9200 --ssl --type api --noVerify](https://image.slidesharecdn.com/troubleshootingyourelasticsearchclusterlikeasupportengineer-210709095514/85/Troubleshooting-your-elasticsearch-cluster-like-a-support-engineer-16-320.jpg)
![DIAGNOSTIC
Urgent Severity] Red cluster
Why is the cluster red?
➔ REST API calls - CAT Indices API
https://www.elastic.co/guide/en/elasticsearch/reference/7.11/rest-apis.html
https://www.elastic.co/guide/en/elasticsearch/reference/7.11/cat-indices.html](https://image.slidesharecdn.com/troubleshootingyourelasticsearchclusterlikeasupportengineer-210709095514/85/Troubleshooting-your-elasticsearch-cluster-like-a-support-engineer-17-320.jpg)
![DIAGNOSTIC
Urgent Severity] Red cluster
Why is an index red?
➔ Check shards that are not started:
INITIALIZING or UNASSIGNED
https://www.elastic.co/guide/en/elasticsearch/reference/7.11/cat-shards.html](https://image.slidesharecdn.com/troubleshootingyourelasticsearchclusterlikeasupportengineer-210709095514/85/Troubleshooting-your-elasticsearch-cluster-like-a-support-engineer-18-320.jpg)
![DIAGNOSTIC
Urgent Severity] Red cluster
Why is a shard UNASSIGNED?
➔ Cluster allocation explain API
https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-allocation-explain.html](https://image.slidesharecdn.com/troubleshootingyourelasticsearchclusterlikeasupportengineer-210709095514/85/Troubleshooting-your-elasticsearch-cluster-like-a-support-engineer-19-320.jpg)
![DIAGNOSTIC
Urgent Severity] Red cluster
Why is a shard UNASSIGNED?](https://image.slidesharecdn.com/troubleshootingyourelasticsearchclusterlikeasupportengineer-210709095514/85/Troubleshooting-your-elasticsearch-cluster-like-a-support-engineer-20-320.jpg)
![DIAGNOSTIC
Urgent Severity] Red cluster
Have we used all the cluster storage?
➔ Use CAT Allocation API
https://www.elastic.co/guide/en/elasticsearch/reference/7.11/cat-allocation.html](https://image.slidesharecdn.com/troubleshootingyourelasticsearchclusterlikeasupportengineer-210709095514/85/Troubleshooting-your-elasticsearch-cluster-like-a-support-engineer-21-320.jpg)
![DIAGNOSTIC
Urgent Severity] Red cluster
Interpret data
➔ Cluster reached its disk high watermark
https://www.elastic.co/guide/en/elasticsearch/reference/7.11/modules-cluster.html#disk-based-shard-allocation](https://image.slidesharecdn.com/troubleshootingyourelasticsearchclusterlikeasupportengineer-210709095514/85/Troubleshooting-your-elasticsearch-cluster-like-a-support-engineer-22-320.jpg)
![DIAGNOSTIC
Urgent Severity] Red cluster
Interpret data
➔ Existing indices are blocked for write
https://www.elastic.co/guide/en/elasticsearch/reference/7.11/cluster-get-settings.html](https://image.slidesharecdn.com/troubleshootingyourelasticsearchclusterlikeasupportengineer-210709095514/85/Troubleshooting-your-elasticsearch-cluster-like-a-support-engineer-23-320.jpg)
![TREATMENT
Urgent Severity] Red cluster
Fixing the root cause
➔ Delete indices to increase available storage
https://www.elastic.co/guide/en/elasticsearch/reference/7.11/indices-delete-index.html
Do we have snapshots? We can restore later.
https://www.elastic.co/guide/en/elasticsearch/reference/7.11/snapshot-restore.html
➔ Add nodes or increase storage capacity (easier on cloud)](https://image.slidesharecdn.com/troubleshootingyourelasticsearchclusterlikeasupportengineer-210709095514/85/Troubleshooting-your-elasticsearch-cluster-like-a-support-engineer-24-320.jpg)
![TREATMENT
Urgent Severity] Red cluster
Temporary Hotfix
➔ Alter the cluster settings to temporarily allow a higher disk
usage
https://www.elastic.co/guide/en/elasticsearch/reference/7.11/cluster-update-settings.html](https://image.slidesharecdn.com/troubleshootingyourelasticsearchclusterlikeasupportengineer-210709095514/85/Troubleshooting-your-elasticsearch-cluster-like-a-support-engineer-25-320.jpg)
![TREATMENT
Urgent Severity] Red cluster
Remove write block on the indices
➔ Once we have enough disk, remove the index block if needed
https://www.elastic.co/guide/en/elasticsearch/reference/7.11/indices-update-settings.html](https://image.slidesharecdn.com/troubleshootingyourelasticsearchclusterlikeasupportengineer-210709095514/85/Troubleshooting-your-elasticsearch-cluster-like-a-support-engineer-26-320.jpg)
![TREATMENT
Urgent Severity] Red cluster
Bonus track
➔ If corrupted shards, and no snapshots, we can force allocation
accepting potential data loss
https://www.elastic.co/guide/en/elasticsearch/reference/7.11/cluster-reroute.html#cluster-reroute-api-request-body](https://image.slidesharecdn.com/troubleshootingyourelasticsearchclusterlikeasupportengineer-210709095514/85/Troubleshooting-your-elasticsearch-cluster-like-a-support-engineer-27-320.jpg)
![DISCHARGE
Urgent Severity] Red cluster
Takeaways
➔ Proactively monitor disk usage on each node / Alerts
Aim to 75% used storage to be on the safe side (< 85%
➔ Plan for data retention / deletion with ILM or Data Tiers
Index Lifecycle Management (ILM can help automate
https://www.elastic.co/guide/en/elasticsearch/reference/7.11/index-lifecycle-management.html
https://www.elastic.co/guide/en/elasticsearch/reference/7.11/data-tiers.html
➔ Snapshot / Snapshot Lifecycle Management (SLM) for backups
https://www.elastic.co/guide/en/elasticsearch/reference/7.11/snapshot-lifecycle-management.html](https://image.slidesharecdn.com/troubleshootingyourelasticsearchclusterlikeasupportengineer-210709095514/85/Troubleshooting-your-elasticsearch-cluster-like-a-support-engineer-28-320.jpg)
![Treatment ➔ Proactively
monitor disk
usage (alerts)
➔ Snapshots
➔ Index Lifecycle
Management
deletes old data
and manages
replicas
➔ Data Tiers with
Cold Tier
Discharge
Diagnostic
➔ Delete indices
➔ Add data node/s
➔ Update index
settings / allow
write
Reached high disk
watermark
➔ CAT APIs
➔ Allocation Explain
➔ Cluster and index
settings
Support diagnostics
Triage
➔ Cluster health is
red
➔ Stopped
ingesting
➔ Search works
SUMMARY
Urgent Severity] Red cluster](https://image.slidesharecdn.com/troubleshootingyourelasticsearchclusterlikeasupportengineer-210709095514/85/Troubleshooting-your-elasticsearch-cluster-like-a-support-engineer-29-320.jpg)
Troubleshooting your elasticsearch cluster like a support engineer
- 1. Community Conference 2021 Troubleshooting your Elasticsearch cluster like a Support Engineer Janko Strassburg, Imma Valls Sr. Support Engineers, Elastic @jankopueh, @eyeveebee
- 15. TRIAGE Urgent Severity] Red cluster Vital signs ➔ Cluster in red health ➔ No ingest into any indices Symptoms ➔ Beats fail to ingest ➔ Cluster is responsive, search and REST API still work
- 16. TRIAGE Urgent Severity] Red cluster What happened? ➔ Out of the blue, no changes Any attempts to fix it? ➔ No Next steps ➔ Share a support diagnostics that will provide REST API calls https://www.elastic.co/blog/why-does-elastic-support-keep-asking-for-diagnostic-files https://github.com/elastic/support-diagnostics/blob/main/src/main/resources/elastic-rest.yml https://github.com/elastic/support-diagnostics > ./diagnostics.sh --host https://localhost -u elastic -p --port 9200 --ssl --type api --noVerify
- 17. DIAGNOSTIC Urgent Severity] Red cluster Why is the cluster red? ➔ REST API calls - CAT Indices API https://www.elastic.co/guide/en/elasticsearch/reference/7.11/rest-apis.html https://www.elastic.co/guide/en/elasticsearch/reference/7.11/cat-indices.html
- 18. DIAGNOSTIC Urgent Severity] Red cluster Why is an index red? ➔ Check shards that are not started: INITIALIZING or UNASSIGNED https://www.elastic.co/guide/en/elasticsearch/reference/7.11/cat-shards.html
- 19. DIAGNOSTIC Urgent Severity] Red cluster Why is a shard UNASSIGNED? ➔ Cluster allocation explain API https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-allocation-explain.html
- 20. DIAGNOSTIC Urgent Severity] Red cluster Why is a shard UNASSIGNED?
- 21. DIAGNOSTIC Urgent Severity] Red cluster Have we used all the cluster storage? ➔ Use CAT Allocation API https://www.elastic.co/guide/en/elasticsearch/reference/7.11/cat-allocation.html
- 22. DIAGNOSTIC Urgent Severity] Red cluster Interpret data ➔ Cluster reached its disk high watermark https://www.elastic.co/guide/en/elasticsearch/reference/7.11/modules-cluster.html#disk-based-shard-allocation
- 23. DIAGNOSTIC Urgent Severity] Red cluster Interpret data ➔ Existing indices are blocked for write https://www.elastic.co/guide/en/elasticsearch/reference/7.11/cluster-get-settings.html
- 24. TREATMENT Urgent Severity] Red cluster Fixing the root cause ➔ Delete indices to increase available storage https://www.elastic.co/guide/en/elasticsearch/reference/7.11/indices-delete-index.html Do we have snapshots? We can restore later. https://www.elastic.co/guide/en/elasticsearch/reference/7.11/snapshot-restore.html ➔ Add nodes or increase storage capacity (easier on cloud)
- 25. TREATMENT Urgent Severity] Red cluster Temporary Hotfix ➔ Alter the cluster settings to temporarily allow a higher disk usage https://www.elastic.co/guide/en/elasticsearch/reference/7.11/cluster-update-settings.html
- 26. TREATMENT Urgent Severity] Red cluster Remove write block on the indices ➔ Once we have enough disk, remove the index block if needed https://www.elastic.co/guide/en/elasticsearch/reference/7.11/indices-update-settings.html
- 27. TREATMENT Urgent Severity] Red cluster Bonus track ➔ If corrupted shards, and no snapshots, we can force allocation accepting potential data loss https://www.elastic.co/guide/en/elasticsearch/reference/7.11/cluster-reroute.html#cluster-reroute-api-request-body
- 28. DISCHARGE Urgent Severity] Red cluster Takeaways ➔ Proactively monitor disk usage on each node / Alerts Aim to 75% used storage to be on the safe side (< 85% ➔ Plan for data retention / deletion with ILM or Data Tiers Index Lifecycle Management (ILM can help automate https://www.elastic.co/guide/en/elasticsearch/reference/7.11/index-lifecycle-management.html https://www.elastic.co/guide/en/elasticsearch/reference/7.11/data-tiers.html ➔ Snapshot / Snapshot Lifecycle Management (SLM) for backups https://www.elastic.co/guide/en/elasticsearch/reference/7.11/snapshot-lifecycle-management.html
- 29. Treatment ➔ Proactively monitor disk usage (alerts) ➔ Snapshots ➔ Index Lifecycle Management deletes old data and manages replicas ➔ Data Tiers with Cold Tier Discharge Diagnostic ➔ Delete indices ➔ Add data node/s ➔ Update index settings / allow write Reached high disk watermark ➔ CAT APIs ➔ Allocation Explain ➔ Cluster and index settings Support diagnostics Triage ➔ Cluster health is red ➔ Stopped ingesting ➔ Search works SUMMARY Urgent Severity] Red cluster
- 30. More Tools & Resources
- 31. Monitoring ➔ Monitoring in production - dedicated cluster https://www.elastic.co/guide/en/elasticsearch/reference/7.11/monitoring-production.html
- 32. Monitoring ➔ Nodes’ memory usage
- 33. Monitoring ➔ Ingest and Search queues and rejections https://www.elastic.co/guide/en/elasticsearch/reference/7.11/cat-thread-pool.html
- 34. Monitoring ➔ Example High CPU usage
- 35. Monitoring ➔ Example High CPU usage
- 36. And CAT APIs again! ➔ Example High CPU usage https://www.elastic.co/guide/en/elasticsearch/reference/7.11/cat-shards.html https://www.elastic.co/guide/en/elasticsearch/reference/7.11/cluster-nodes-hot-threads.html
- 38. Sizing - how many shards per node and what size ➔ https://www.elastic.co/guide/en/elasticsearch/reference/7.11/size-your-shards.html ➔ https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster ➔ https://www.elastic.co/guide/en/cloud/current/ec-reference-hardware.html ➔ https://benchmarks.elastic.co/ ➔ https://esrally.readthedocs.io/ Storage ➔ https://www.elastic.co/blog/how-to-design-your-elasticsearch-data-storage-architecture-for-scale ➔ https://www.elastic.co/guide/en/elasticsearch/reference/7.11/tune-for-disk-usage.html JVM Heap - do not go over ~30Gb heap ➔ https://www.elastic.co/blog/a-heap-of-trouble Hot/Warm/Cold architectures for time series data ➔ https://www.elastic.co/blog/optimizing-costs-elastic-cloud-hot-warm-index-lifecycle-management Common Resources Shared
- 39. Common Resources Shared Tuning for search - slow searches ➔ https://www.elastic.co/blog/advanced-tuning-finding-and-fixing-slow-elasticsearch-queries ➔ https://www.elastic.co/guide/en/elasticsearch/reference/7.11/tune-for-search-speed.html Tuning for ingest - use bulk! ➔ https://www.elastic.co/guide/en/elasticsearch/reference/7.11/tune-for-indexing-speed.html ➔ https://www.elastic.co/guide/en/elasticsearch/reference/7.11/docs-bulk.html Upgrading the Stack - be prepared, test and snapshots! ➔ https://www.elastic.co/webinars/expert-tips-for-upgrading-the-elk-stack ➔ https://www.elastic.co/guide/en/elastic-stack/7.11/upgrading-elastic-stack.html Secure the Stack ➔ https://www.elastic.co/blog/configuring-ssl-tls-and-https-to-secure-elasticsearch-kibana-beats-and-l ogstash Optimize Mappings ➔ https://www.elastic.co/blog/strings-are-dead-long-live-strings
- 40. Wrapping Up Triage incidents ➔ How critical is it? ➔ Do we need urgent care or is there a workaround to stabilize? Have tools ready ➔ REST APIs / Support diagnostics ➔ Monitoring & Alerts ➔ Log Analysis / Kibana Discover ➔ Search Elastic discuss, Stackoverflow, Elastic GitHub repos, etc.. Lessons learned ➔ Follow best practices ➔ Prevent future incidents - proactively investigate unexpected logs, etc.
- 41. Q & A Thank You