Stop Exhausting Yourself in Operating Multiple Elasticsearch Clusters
0 likes2,854 views
The document discusses the operational challenges of managing multiple Elasticsearch clusters, emphasizing the need for a robust management tool to prevent chaos in plugin installation and monitoring. It introduces 'Rubber Band,' a toolkit designed to enhance cluster health monitoring and alerting by aggregating information and enabling client-side metrics access. The document outlines the development phases of Rubber Band, including its eventual open-source release and highlights the team's hiring opportunity.
Stop Exhausting Yourself in Operating Multiple Elasticsearch Clusters
- 2. In-game Community / Marketing Platform WHAT'S PION?
- 3. • As a time series DB • As a search engine • As a log store WHAT’S Elasticsearch?
- 4. Logging WE HAVE MULTIPLE CLUSTERS FOR.. Event Processi ng Service Develop ment RealSandbox Purpose Environment
- 5. “Which clusters did I install which plugins on?” For example.. MULTIPLE CLUSTERS WILL CAUSE.. Basically our clusters are provisioned by Ansible BUT… Someone: “Hey, let’s try the XXX plugin on the node YYY of the cluster ZZZ in DEV environment!” They forgot to record XXX, YYY, ZZZ… Easily go down to chaos!
- 6. WE NEED A MANAGEMENT TOOL!
- 7. • ElasticHQ (OSS) • Kibana (by Elastic) • cerebro (OSS) EXISTING TOOLS For a single cluster One of its strengths is that it can support multiple clusters OK, let’s use this! However its main purpose is also deep management of a single cluster Not for browsing a cluster list
- 8. ANOTHER PROBLEM ON Elasticsearch Not too easy to: monitor an Elasticsearch cluster alert us to the abnormal status based on the result of monitoring properly Kibana or many OSS are very nice, but: Some detailed metrics (like latency 95%ile) cannot retrieved directly We cannot see them when Es is under too heavy load
- 9. COMPARISON Multiple clusters? Monitoring? Alerting? Kibana partial support (cross cluster search, dedicated separate cluster for monitoring) partial support (server side metrics) ✔ (with Watcher) ElasticHQ ✔ (not for browsing) partial support (server side metrics) ✘ What we need ✔ (w/ high browsability) ✔ ✔ OK, let’s make it by ourselves!
- 10. Screenshots RUBBER BAND - TOOLKIT FOR ES MANAGEMENT
- 11. Rubber Band UI, Health Watcher, Client - architecture
- 12. Rubber Band UI, Health Watcher, Client - architecture
- 13. Rubber Band UI, Health Watcher, Client - architecture
- 14. TWO OPTIONS FOR MONITORING Monitor clusters’ states directly /_cat/*** /_cluster/health /_nodes/*** Monitor client-side metrics can compute detailed metrics can access even when a cluster is highly loaded (via our tool)
- 15. Rubber Band UI, Health Watcher, Client - architecture
- 16. HOW TO ALERT ON A CLUSTER STATUS? The X-Pack GOLD license supports Watcher, which also can be used to check the cluster health out-of-the-box! { "trigger" : { "schedule" : { "interval" : "10s" } }, "input" : { "http" : { "request" : { "host" : "localhost", "port" : 9200, "path" : "/_cluster/health" } } Uses cluster health API! We can also utilize it by ourselves:)
- 17. EXAMPLES OF ALERT FROM HEALTH WATCHER
- 18. Rubber Band UI, Health Watcher, Client - architecture
- 19. MILESTONE PHASE 1 Rubber Band UI Rubber Band Health Watcher Rubber Band Client (Simple REST client wrapper) PHASE 2 • Rubber Band Curator (Centralized wrapper of curator) • Open to the other internal teams PHASE 3 • Publish it as a OSS
- 20. KEY TAKEAWAYS How can we manage multiple clusters without any chaos? Our toolkit: Rubber Band A simple UI with information aggregation and appropriate delegation How can we do proper monitoring and alerting? Uses both of direct server states and client metrics Implements a simple health-check server by ourselves And..
- 21. WE ARE HIRING!
- 22. THANK YOU
- 23. @Component public class ElasticsearchClientWrapper { private final RestHighLevelClient elasticsearchClient; private final MeterRegistry meterRegistry; public ElasticsearchClientWrapper(RestHighLevelClient elasticsearchClient, MeterRegistry meterRegistry) { this.elasticsearchClient = elasticsearchClient; this.meterRegistry = meterRegistry; } public void searchAndGetAggregationAsync(SearchRequest searchRequest) { Timer.Sample sample = Timer.start(meterRegistry); elasticsearchClient.searchAsync(searchRequest, new ActionListener<SearchResponse>() { @Override public void onResponse(SearchResponse searchResponse) { sample.stop(meterRegistry.timer("metrics.timer", "success")); // do stuff.. } @Override public void onFailure(Exception e) { sample.stop(meterRegistry.timer("metrics.timer", "failure")); // do fallback.. } }); } Wrap the official HighLevelRESTClient See also: Elasticsearch を検索エンジンとして利用する際のポイント https://engineering.linecorp.com/ja/blog/detail/99