![TRIAGE
Urgent Severity] Red cluster
Vital signs
➔ Cluster in red health
➔ No ingest into any
indices
Symptoms
➔ Beats fail to ingest
➔ Cluster is responsive, search and REST API still work](https://image.slidesharecdn.com/troubleshooting-elasticsearch-cluster-elasticon-global-211209201002/85/Troubleshooting-your-Elasticsearch-cluster-like-an-Elastic-Support-Engineer-20-320.jpg)
![TRIAGE
Urgent Severity] Red cluster
What happened?
➔ Out of the blue, no changes
Any attempts to fix it?
➔ No
Next steps
➔ Share a support diagnostics that will provide REST API calls
https://www.elastic.co/blog/why-does-elastic-support-keep-asking-for-diagnostic-files
https://github.com/elastic/support-diagnostics/blob/main/src/main/resources/elastic-rest.yml
https://github.com/elastic/support-diagnostics
> ./diagnostics.sh --host https://localhost -u elastic -p --port 9200 --ssl --type api --noVerify](https://image.slidesharecdn.com/troubleshooting-elasticsearch-cluster-elasticon-global-211209201002/85/Troubleshooting-your-Elasticsearch-cluster-like-an-Elastic-Support-Engineer-21-320.jpg)
![DIAGNOSTIC
Urgent Severity] Red cluster
Why is the cluster red?
➔ REST API calls - CAT Indices API
https://www.elastic.co/guide/en/elasticsearch/reference/7.14/rest-apis.html
https://www.elastic.co/guide/en/elasticsearch/reference/7.14/cat-indices.html](https://image.slidesharecdn.com/troubleshooting-elasticsearch-cluster-elasticon-global-211209201002/85/Troubleshooting-your-Elasticsearch-cluster-like-an-Elastic-Support-Engineer-22-320.jpg)
![DIAGNOSTIC
Urgent Severity] Red cluster
Why is an index red?
➔ Check shards that are not started:
INITIALIZING or UNASSIGNED
https://www.elastic.co/guide/en/elasticsearch/reference/7.14/cat-shards.html](https://image.slidesharecdn.com/troubleshooting-elasticsearch-cluster-elasticon-global-211209201002/85/Troubleshooting-your-Elasticsearch-cluster-like-an-Elastic-Support-Engineer-23-320.jpg)
![DIAGNOSTIC
Urgent Severity] Red cluster
Why is a shard UNASSIGNED?
➔ Cluster allocation explain API
https://www.elastic.co/guide/en/elasticsearch/reference/7.14/cluster-allocation-explain.html](https://image.slidesharecdn.com/troubleshooting-elasticsearch-cluster-elasticon-global-211209201002/85/Troubleshooting-your-Elasticsearch-cluster-like-an-Elastic-Support-Engineer-24-320.jpg)
![DIAGNOSTIC
Urgent Severity] Red cluster
Why is a shard UNASSIGNED?](https://image.slidesharecdn.com/troubleshooting-elasticsearch-cluster-elasticon-global-211209201002/85/Troubleshooting-your-Elasticsearch-cluster-like-an-Elastic-Support-Engineer-25-320.jpg)
![DIAGNOSTIC
Urgent Severity] Red cluster
Have we used all the cluster storage?
➔ Use CAT Allocation API
https://www.elastic.co/guide/en/elasticsearch/reference/7.14/cat-allocation.html](https://image.slidesharecdn.com/troubleshooting-elasticsearch-cluster-elasticon-global-211209201002/85/Troubleshooting-your-Elasticsearch-cluster-like-an-Elastic-Support-Engineer-26-320.jpg)
![DIAGNOSTIC
Urgent Severity] Red cluster
Interpret data
➔ Cluster reached its flood stage disk watermark
https://www.elastic.co/guide/en/elasticsearch/reference/7.14/modules-cluster.html#disk-based-shard-allocation](https://image.slidesharecdn.com/troubleshooting-elasticsearch-cluster-elasticon-global-211209201002/85/Troubleshooting-your-Elasticsearch-cluster-like-an-Elastic-Support-Engineer-27-320.jpg)
![DIAGNOSTIC
Urgent Severity] Red cluster
Interpret data
➔ Existing indices are blocked for write
https://www.elastic.co/guide/en/elasticsearch/reference/7.14/cluster-get-settings.html](https://image.slidesharecdn.com/troubleshooting-elasticsearch-cluster-elasticon-global-211209201002/85/Troubleshooting-your-Elasticsearch-cluster-like-an-Elastic-Support-Engineer-28-320.jpg)
![TREATMENT
Urgent Severity] Red cluster
Fixing the root cause
➔ Delete indices to increase available storage
https://www.elastic.co/guide/en/elasticsearch/reference/7.14/indices-delete-index.html
Do we have snapshots? We can restore later.
https://www.elastic.co/guide/en/elasticsearch/reference/7.14/snapshot-restore.html
➔ Add nodes or increase storage capacity (easier on cloud)](https://image.slidesharecdn.com/troubleshooting-elasticsearch-cluster-elasticon-global-211209201002/85/Troubleshooting-your-Elasticsearch-cluster-like-an-Elastic-Support-Engineer-29-320.jpg)
![TREATMENT
Urgent Severity] Red cluster
Temporary Hotfix
➔ Alter the cluster settings to temporarily allow a higher disk usage
https://www.elastic.co/guide/en/elasticsearch/reference/7.14/cluster-update-settings.html](https://image.slidesharecdn.com/troubleshooting-elasticsearch-cluster-elasticon-global-211209201002/85/Troubleshooting-your-Elasticsearch-cluster-like-an-Elastic-Support-Engineer-30-320.jpg)
![TREATMENT
Urgent Severity] Red cluster
Remove write block on the indices
➔ Once we have enough disk, remove the index block if needed
https://www.elastic.co/guide/en/elasticsearch/reference/7.14/indices-update-settings.html](https://image.slidesharecdn.com/troubleshooting-elasticsearch-cluster-elasticon-global-211209201002/85/Troubleshooting-your-Elasticsearch-cluster-like-an-Elastic-Support-Engineer-31-320.jpg)
![TREATMENT
Urgent Severity] Red cluster
Bonus track
➔ If corrupted shards, and no snapshots, we can force allocation
accepting potential data loss
https://www.elastic.co/guide/en/elasticsearch/reference/7.14/cluster-reroute.html#cluster-reroute-api-request-body](https://image.slidesharecdn.com/troubleshooting-elasticsearch-cluster-elasticon-global-211209201002/85/Troubleshooting-your-Elasticsearch-cluster-like-an-Elastic-Support-Engineer-32-320.jpg)
![DISCHARGE
Urgent Severity] Red cluster
Takeaways
➔ Proactively monitor disk usage on each node / Alerts
Aim to 75% used storage to be on the safe side (< 85%)
➔ Plan for data retention / deletion with ILM or Data Tiers
Index Lifecycle Management (ILM) can help automate
https://www.elastic.co/guide/en/elasticsearch/reference/7.14/index-lifecycle-management.html
https://www.elastic.co/guide/en/elasticsearch/reference/7.14/data-tiers.html
➔ Snapshot / Snapshot Lifecycle Management (SLM) for backups
https://www.elastic.co/guide/en/elasticsearch/reference/7.14/snapshot-lifecycle-management.html](https://image.slidesharecdn.com/troubleshooting-elasticsearch-cluster-elasticon-global-211209201002/85/Troubleshooting-your-Elasticsearch-cluster-like-an-Elastic-Support-Engineer-33-320.jpg)
![Treatment ➔ Proactively
monitor disk
usage (alerts)
➔ Snapshots
➔ Index Lifecycle
Management
deletes old data
and manages
replicas
➔ Data Tiers with
Cold Tier or
Frozen Tiers
Discharge
Diagnostic
➔ Delete indices
➔ Add data node/s
➔ Update index
settings / allow
write
Reached flood stage
disk watermark
➔ CAT APIs
➔ Allocation Explain
➔ Cluster and index
settings
Support diagnostics
Triage
➔ Cluster health is
red
➔ Stopped
ingesting
➔ Search works
SUMMARY
Urgent Severity] Red cluster](https://image.slidesharecdn.com/troubleshooting-elasticsearch-cluster-elasticon-global-211209201002/85/Troubleshooting-your-Elasticsearch-cluster-like-an-Elastic-Support-Engineer-34-320.jpg)
![TRIAGE
High Severity] Unbalanced CPU Usage
Vital signs
➔ Green cluster
➔ Monitoring alerts
high CPU usage
Symptoms
➔ Unbalance CPU usage between different nodes
➔ CPU pressure switches to different nodes over time
➔ Some ingest delays](https://image.slidesharecdn.com/troubleshooting-elasticsearch-cluster-elasticon-global-211209201002/85/Troubleshooting-your-Elasticsearch-cluster-like-an-Elastic-Support-Engineer-36-320.jpg)
![TRIAGE
High Severity] Unbalanced CPU usage
What happened?
➔ Was there any benchmarking done
before going into production?
➔ Any changes in data ingest
volumes?
Any attempts to fix it?
➔ Tried adding nodes,
did not help
Next steps
➔ Export monitoring data and share a support diagnostics
https://github.com/elastic/support-diagnostics#extracting-time-series-diagnostics-from-monitoring
https://github.com/elastic/support-diagnostics](https://image.slidesharecdn.com/troubleshooting-elasticsearch-cluster-elasticon-global-211209201002/85/Troubleshooting-your-Elasticsearch-cluster-like-an-Elastic-Support-Engineer-37-320.jpg)
![DIAGNOSTIC
High Severity] Unbalanced CPU Usage
Why high CPU usage?
➔ Monitoring
https://www.elastic.co/guide/en/elasticsearch/reference/7.14/monitoring-production.html](https://image.slidesharecdn.com/troubleshooting-elasticsearch-cluster-elasticon-global-211209201002/85/Troubleshooting-your-Elasticsearch-cluster-like-an-Elastic-Support-Engineer-38-320.jpg)
![DIAGNOSTIC
High Severity] Unbalanced CPU Usage
Why high CPU usage?
➔ REST API calls - Cat Shards and Get Index settings API
https://www.elastic.co/guide/en/elasticsearch/reference/7.14/cat-shards.html
https://www.elastic.co/guide/en/elasticsearch/reference/7.14/indices-get-settings.html](https://image.slidesharecdn.com/troubleshooting-elasticsearch-cluster-elasticon-global-211209201002/85/Troubleshooting-your-Elasticsearch-cluster-like-an-Elastic-Support-Engineer-39-320.jpg)
![DIAGNOSTIC
High Severity] Unbalanced CPU Usage
Why high CPU usage?
➔ Hot threads API to confirm CPU usage is on write
https://www.elastic.co/guide/en/elasticsearch/reference/7.14/cluster-nodes-hot-threads.html](https://image.slidesharecdn.com/troubleshooting-elasticsearch-cluster-elasticon-global-211209201002/85/Troubleshooting-your-Elasticsearch-cluster-like-an-Elastic-Support-Engineer-40-320.jpg)
![TREATMENT
High Severity] Unbalanced CPU Usage
Fixing the root cause
➔ Increase primary shards
In the example: 3 nodes → 3 primary shards for hot index logs-201998
➔ How? Change index template and rollover index if using ILM
https://www.elastic.co/guide/en/elasticsearch/reference/7.14/index-templates.html
https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started-index-lifecycle-
management.html
https://www.elastic.co/guide/en/elasticsearch/reference/7.14/indices-rollover-index.html](https://image.slidesharecdn.com/troubleshooting-elasticsearch-cluster-elasticon-global-211209201002/85/Troubleshooting-your-Elasticsearch-cluster-like-an-Elastic-Support-Engineer-41-320.jpg)
![TREATMENT
High Severity] Unbalanced CPU Usage
Fixing the root cause](https://image.slidesharecdn.com/troubleshooting-elasticsearch-cluster-elasticon-global-211209201002/85/Troubleshooting-your-Elasticsearch-cluster-like-an-Elastic-Support-Engineer-42-320.jpg)
![TREATMENT
High Severity] Unbalanced CPU usage
Bonus track
➔ Use with caution index.routing.allocation.total_shards_per_node
https://www.elastic.co/guide/en/elasticsearch/reference/7.14/allocation-total-shards.html](https://image.slidesharecdn.com/troubleshooting-elasticsearch-cluster-elasticon-global-211209201002/85/Troubleshooting-your-Elasticsearch-cluster-like-an-Elastic-Support-Engineer-43-320.jpg)
![DISCHARGE
High Severity] Unbalanced CPU Usage
Takeaways
➔ Proactively monitor CPU usage on each node / Alerts
https://www.elastic.co/guide/en/elasticsearch/reference/7.14/monitoring-production.html
➔ Benchmark with tools like ES Rally
https://esrally.readthedocs.io/en/stable/
https://benchmarks.elastic.co/index.html
https://esrally.readthedocs.io/en/stable/adding_tracks.html](https://image.slidesharecdn.com/troubleshooting-elasticsearch-cluster-elasticon-global-211209201002/85/Troubleshooting-your-Elasticsearch-cluster-like-an-Elastic-Support-Engineer-44-320.jpg)
![Treatment ➔ Proactively
monitor cpu
usage (alerts)
➔ Benchmark (ES
rally) before
going into
production or if
there is any
changes in the
data volumes
Discharge
Diagnostic
➔ Create index
settings
➔ Index templates
to add additional
primary shards
Ingest is hot on an
index with 1 primary
shard
➔ Monitoring
➔ CAT shards API
➔ Index settings
Triage
➔ Cluster health is
green
➔ Unbalanced high
CPU usage
switching nodes
over time
SUMMARY
High Severity] Unbalanced CPU Usage](https://image.slidesharecdn.com/troubleshooting-elasticsearch-cluster-elasticon-global-211209201002/85/Troubleshooting-your-Elasticsearch-cluster-like-an-Elastic-Support-Engineer-45-320.jpg)
Troubleshooting your Elasticsearch cluster like an Elastic Support Engineer
- 1. Troubleshooting your Elasticsearch cluster like a Support Engineer Imma Valls Support Engineer, Elastic @eyeveebee http://eyeveebee.net
- 4. How can we approach troubleshooting?
- 5. The Emergency Room model
- 9. 9
- 11. 11
- 13. 13
- 15. 15
- 17. 17
- 20. TRIAGE Urgent Severity] Red cluster Vital signs ➔ Cluster in red health ➔ No ingest into any indices Symptoms ➔ Beats fail to ingest ➔ Cluster is responsive, search and REST API still work
- 21. TRIAGE Urgent Severity] Red cluster What happened? ➔ Out of the blue, no changes Any attempts to fix it? ➔ No Next steps ➔ Share a support diagnostics that will provide REST API calls https://www.elastic.co/blog/why-does-elastic-support-keep-asking-for-diagnostic-files https://github.com/elastic/support-diagnostics/blob/main/src/main/resources/elastic-rest.yml https://github.com/elastic/support-diagnostics > ./diagnostics.sh --host https://localhost -u elastic -p --port 9200 --ssl --type api --noVerify
- 22. DIAGNOSTIC Urgent Severity] Red cluster Why is the cluster red? ➔ REST API calls - CAT Indices API https://www.elastic.co/guide/en/elasticsearch/reference/7.14/rest-apis.html https://www.elastic.co/guide/en/elasticsearch/reference/7.14/cat-indices.html
- 23. DIAGNOSTIC Urgent Severity] Red cluster Why is an index red? ➔ Check shards that are not started: INITIALIZING or UNASSIGNED https://www.elastic.co/guide/en/elasticsearch/reference/7.14/cat-shards.html
- 24. DIAGNOSTIC Urgent Severity] Red cluster Why is a shard UNASSIGNED? ➔ Cluster allocation explain API https://www.elastic.co/guide/en/elasticsearch/reference/7.14/cluster-allocation-explain.html
- 25. DIAGNOSTIC Urgent Severity] Red cluster Why is a shard UNASSIGNED?
- 26. DIAGNOSTIC Urgent Severity] Red cluster Have we used all the cluster storage? ➔ Use CAT Allocation API https://www.elastic.co/guide/en/elasticsearch/reference/7.14/cat-allocation.html
- 27. DIAGNOSTIC Urgent Severity] Red cluster Interpret data ➔ Cluster reached its flood stage disk watermark https://www.elastic.co/guide/en/elasticsearch/reference/7.14/modules-cluster.html#disk-based-shard-allocation
- 28. DIAGNOSTIC Urgent Severity] Red cluster Interpret data ➔ Existing indices are blocked for write https://www.elastic.co/guide/en/elasticsearch/reference/7.14/cluster-get-settings.html
- 29. TREATMENT Urgent Severity] Red cluster Fixing the root cause ➔ Delete indices to increase available storage https://www.elastic.co/guide/en/elasticsearch/reference/7.14/indices-delete-index.html Do we have snapshots? We can restore later. https://www.elastic.co/guide/en/elasticsearch/reference/7.14/snapshot-restore.html ➔ Add nodes or increase storage capacity (easier on cloud)
- 30. TREATMENT Urgent Severity] Red cluster Temporary Hotfix ➔ Alter the cluster settings to temporarily allow a higher disk usage https://www.elastic.co/guide/en/elasticsearch/reference/7.14/cluster-update-settings.html
- 31. TREATMENT Urgent Severity] Red cluster Remove write block on the indices ➔ Once we have enough disk, remove the index block if needed https://www.elastic.co/guide/en/elasticsearch/reference/7.14/indices-update-settings.html
- 32. TREATMENT Urgent Severity] Red cluster Bonus track ➔ If corrupted shards, and no snapshots, we can force allocation accepting potential data loss https://www.elastic.co/guide/en/elasticsearch/reference/7.14/cluster-reroute.html#cluster-reroute-api-request-body
- 33. DISCHARGE Urgent Severity] Red cluster Takeaways ➔ Proactively monitor disk usage on each node / Alerts Aim to 75% used storage to be on the safe side (< 85%) ➔ Plan for data retention / deletion with ILM or Data Tiers Index Lifecycle Management (ILM) can help automate https://www.elastic.co/guide/en/elasticsearch/reference/7.14/index-lifecycle-management.html https://www.elastic.co/guide/en/elasticsearch/reference/7.14/data-tiers.html ➔ Snapshot / Snapshot Lifecycle Management (SLM) for backups https://www.elastic.co/guide/en/elasticsearch/reference/7.14/snapshot-lifecycle-management.html
- 34. Treatment ➔ Proactively monitor disk usage (alerts) ➔ Snapshots ➔ Index Lifecycle Management deletes old data and manages replicas ➔ Data Tiers with Cold Tier or Frozen Tiers Discharge Diagnostic ➔ Delete indices ➔ Add data node/s ➔ Update index settings / allow write Reached flood stage disk watermark ➔ CAT APIs ➔ Allocation Explain ➔ Cluster and index settings Support diagnostics Triage ➔ Cluster health is red ➔ Stopped ingesting ➔ Search works SUMMARY Urgent Severity] Red cluster
- 35. Troubleshooting by Example Unbalanced CPU usage 35
- 36. TRIAGE High Severity] Unbalanced CPU Usage Vital signs ➔ Green cluster ➔ Monitoring alerts high CPU usage Symptoms ➔ Unbalance CPU usage between different nodes ➔ CPU pressure switches to different nodes over time ➔ Some ingest delays
- 37. TRIAGE High Severity] Unbalanced CPU usage What happened? ➔ Was there any benchmarking done before going into production? ➔ Any changes in data ingest volumes? Any attempts to fix it? ➔ Tried adding nodes, did not help Next steps ➔ Export monitoring data and share a support diagnostics https://github.com/elastic/support-diagnostics#extracting-time-series-diagnostics-from-monitoring https://github.com/elastic/support-diagnostics
- 38. DIAGNOSTIC High Severity] Unbalanced CPU Usage Why high CPU usage? ➔ Monitoring https://www.elastic.co/guide/en/elasticsearch/reference/7.14/monitoring-production.html
- 39. DIAGNOSTIC High Severity] Unbalanced CPU Usage Why high CPU usage? ➔ REST API calls - Cat Shards and Get Index settings API https://www.elastic.co/guide/en/elasticsearch/reference/7.14/cat-shards.html https://www.elastic.co/guide/en/elasticsearch/reference/7.14/indices-get-settings.html
- 40. DIAGNOSTIC High Severity] Unbalanced CPU Usage Why high CPU usage? ➔ Hot threads API to confirm CPU usage is on write https://www.elastic.co/guide/en/elasticsearch/reference/7.14/cluster-nodes-hot-threads.html
- 41. TREATMENT High Severity] Unbalanced CPU Usage Fixing the root cause ➔ Increase primary shards In the example: 3 nodes → 3 primary shards for hot index logs-201998 ➔ How? Change index template and rollover index if using ILM https://www.elastic.co/guide/en/elasticsearch/reference/7.14/index-templates.html https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started-index-lifecycle- management.html https://www.elastic.co/guide/en/elasticsearch/reference/7.14/indices-rollover-index.html
- 42. TREATMENT High Severity] Unbalanced CPU Usage Fixing the root cause
- 43. TREATMENT High Severity] Unbalanced CPU usage Bonus track ➔ Use with caution index.routing.allocation.total_shards_per_node https://www.elastic.co/guide/en/elasticsearch/reference/7.14/allocation-total-shards.html
- 44. DISCHARGE High Severity] Unbalanced CPU Usage Takeaways ➔ Proactively monitor CPU usage on each node / Alerts https://www.elastic.co/guide/en/elasticsearch/reference/7.14/monitoring-production.html ➔ Benchmark with tools like ES Rally https://esrally.readthedocs.io/en/stable/ https://benchmarks.elastic.co/index.html https://esrally.readthedocs.io/en/stable/adding_tracks.html
- 45. Treatment ➔ Proactively monitor cpu usage (alerts) ➔ Benchmark (ES rally) before going into production or if there is any changes in the data volumes Discharge Diagnostic ➔ Create index settings ➔ Index templates to add additional primary shards Ingest is hot on an index with 1 primary shard ➔ Monitoring ➔ CAT shards API ➔ Index settings Triage ➔ Cluster health is green ➔ Unbalanced high CPU usage switching nodes over time SUMMARY High Severity] Unbalanced CPU Usage
- 46. ➔ How critical is it? ➔ Do we need urgent care or is there a workaround to stabilize? Have your tools ready ➔ REST APIs / Support diagnostics ➔ Monitoring & Alerts ➔ Log Analysis: Use Kibana! ➔ Search Elastic discuss, Stackoverflow, Elastic GitHub repos, etc. Lessons learned ➔ Follow best practices ➔ Prevent future incidents - proactively investigate unexpected logs, etc. Wrapping up Triage incidents
- 47. 47 Thank You