DevOpsDays Warsaw 2015: Running High Performance And Fault Tolerant Elasticsearch Clusters On Docker – Rafał Kuć
Download as PPTX, PDF1 like1,813 views
The document presents a comprehensive guide on running high-performance Elasticsearch clusters using Docker, highlighting the advantages of containers over traditional virtual machines. It discusses various operational aspects such as container constraints, storage management, and configuration settings necessary for optimizing Elasticsearch deployments. Additionally, it emphasizes best practices for managing network configurations and data handling to ensure fault tolerance and efficiency in cluster operations.
DevOpsDays Warsaw 2015: Running High Performance And Fault Tolerant Elasticsearch Clusters On Docker – Rafał Kuć
- 1. Elasticsearch & Docker Rafał Kuć – Sematext Group, Inc. @kucrafal @sematext sematext.com Running High Performance Fault Tolerant Elasticsearch Clusters On Docker
- 2. About me… Sematext consultant & engineer Solr.pl co-founder Father and husband :)
- 5. You Are Probably Familiar With That Development
- 6. You Are Probably Familiar With That Development Test
- 7. You Are Probably Familiar With That Development Test QA
- 8. You Are Probably Familiar With That Development Test QA Production enviroment
- 9. And Problems That Come With It Resources not utilized
- 10. And Problems That Come With It Resources not utilized Overprovisoned Servers
- 11. And Problems That Come With It Resources not utilized Overprovisoned Servers ≠ ≠
- 12. The solution Development Test QA Production
- 14. What is Docker Light weight Based on Open Standards Secure
- 15. Containers vs Virtual Machines Hardware Traditional Virtual Machine
- 16. Containers vs Virtual Machines Hardware Host Operating System Traditional Virtual Machine
- 17. Containers vs Virtual Machines Hardware Host Operating System Hypervisor Traditional Virtual Machine
- 18. Containers vs Virtual Machines Hardware Host Operating System Hypervisor Guest OS Guest OS Traditional Virtual Machine
- 19. Containers vs Virtual Machines Hardware Host Operating System Hypervisor Guest OS Guest OS Libraries Libraries Traditional Virtual Machine
- 20. Containers vs Virtual Machines Hardware Host Operating System Hypervisor Guest OS Guest OS Libraries Libraries Application 1 Application 2 Traditional Virtual Machine
- 21. Containers vs Virtual Machines Hardware Host Operating System Hypervisor Guest OS Guest OS Libraries Libraries Application 1 Application 2 Hardware Host Operating System Traditional Virtual MachineContainer
- 22. Containers vs Virtual Machines Hardware Host Operating System Hypervisor Guest OS Guest OS Libraries Libraries Application 1 Application 2 Hardware Host Operating System Docker Engine Traditional Virtual MachineContainer
- 23. Containers vs Virtual Machines Hardware Host Operating System Hypervisor Guest OS Guest OS Libraries Libraries Application 1 Application 2 Hardware Host Operating System Docker Engine Libraries Libraries Application 1 Application 2 Traditional Virtual MachineContainer
- 24. What is Elasticsearch Reasonable defaults Distributed by design http://www.dailypets.co.uk/2007/06/17/kittens-rest-at-half-time/
- 25. Running Offical Elasticsearch Container $ docker run -d elasticsearch
- 26. Running Offical Elasticsearch Container $ docker run -d elasticsearch == docker run -d elasticsearch:latest
- 27. Running Offical Elasticsearch Container $ docker run -d elasticsearch:1.7 $ docker run -d elasticsearch == docker run -d elasticsearch:latest
- 28. Running Offical Elasticsearch Container $ docker run -d elasticsearch == docker run -d elasticsearch:latest $ docker run --name es_1 -h es_master_1 elasticsearch $ docker run -d elasticsearch:1.7
- 29. Running Offical Elasticsearch Container $ docker run -d elasticsearch == docker run -d elasticsearch:latest $ docker run --name es_1 -h es_master_1 elasticsearch $ docker run -d elasticsearch:1.7
- 30. Container Constraints $ docker run -d -m 2G elasticsearch http://docs.docker.com/engine/reference/run/
- 31. Container Constraints $ docker run -d -m 2G elasticsearch $ docker run -d -m 2G --memory-swappiness=0 elasticsearch http://docs.docker.com/engine/reference/run/
- 32. Container Constraints $ docker run -d -m 2G elasticsearch $ docker run -d -m 2G --memory-swappiness=0 elasticsearch $ docker run -d --cpuset-cpus="1,3" elasticsearch http://docs.docker.com/engine/reference/run/
- 33. Container Constraints $ docker run -d -m 2G elasticsearch $ docker run -d -m 2G --memory-swappiness=0 elasticsearch $ docker run -d --cpuset-cpus="1,3" elasticsearch $ docker run -d --cpuset-cpus="1,3" elasticsearch http://docs.docker.com/engine/reference/run/
- 34. Container Constraints $ docker run -d -m 2G elasticsearch $ docker run -d -m 2G --memory-swappiness=0 elasticsearch $ docker run -d --cpuset-cpus="1,3" elasticsearch $ docker run -d --cpuset-cpus="1,3" elasticsearch http://docs.docker.com/engine/reference/run/ $ docker run -d --cpu-period=50000 --cpu-quota=25000 elasticsearch
- 35. Creating Optimized Image Dockerfile: FROM elasticsearch ADD ./elasticsearch.yml /usr/share/elasticsearch/config/
- 36. Creating Optimized Image Dockerfile: FROM elasticsearch ADD ./elasticsearch.yml /usr/share/elasticsearch/config/ $ docker build -t devops/example .
- 37. Creating Optimized Image Dockerfile: FROM elasticsearch ADD ./elasticsearch.yml /usr/share/elasticsearch/config/ $ docker build -t devops/example . Sending build context to Docker daemon 3.072 kB Step 1 : FROM elasticsearch ---> 8112755253f1 Step 2 : ADD ./elasticsearch.yml /usr/share/elasticsearch/config/ ---> Using cache ---> c9ca48a22e58 Successfully built c9ca48a22e58
- 38. Dealing With Network $ docker run -d -p 9200:9200 -p 9300:9300 elasticsearch
- 39. Dealing With Network $ docker run -d -p 9200:9200 -p 9300:9300 elasticsearch $ docker run -d elasticsearch -Dnetwork.publish_host=192.168.1.1
- 40. Dealing With Network $ docker run -d -p 9200:9200 -p 9300:9300 elasticsearch $ docker run -d elasticsearch -Dnetwork.publish_host=192.168.1.1 $ docker run -d -p 9200:9200 -p 9300:9300 elasticsearch -Dnetwork.publish_host=192.168.1.1
- 41. Dealing With Network $ docker run -d -p 9200:9200 -p 9300:9300 elasticsearch $ docker run -d elasticsearch -Dnetwork.publish_host=192.168.1.1 $ docker run -d -p 9200:9200 -p 9300:9300 elasticsearch -Dnetwork.publish_host=192.168.1.1 $ docker run -d -p 9200:9200 -p 9300:9300 elasticsearch -Dnetwork.publish_host=0.0.0.0
- 42. Network - Good Practices Separate network for Elasticsearch cluster
- 43. Network - Good Practices Separate network for Elasticsearch cluster Common host names for containers $ docker run -d -h es_node_1 elasticsearch
- 44. Network - Good Practices Separate network for Elasticsearch cluster Common host names for containers $ docker run -d -h es_node_1 elasticsearch Expose 9200 & 9300 ports only for client nodes
- 45. Network - Good Practices Separate network for Elasticsearch cluster Common host names for containers $ docker run -d -h es_node_1 elasticsearch Expose 9200 & 9300 ports only for client nodes Elasticsearch data & client nodes point to masters only
- 46. Dealing With Storage By default in /usr/share/elasticsearch/data
- 47. Dealing With Storage By default in /usr/share/elasticsearch/data By default not persisted
- 48. Dealing With Storage By default in /usr/share/elasticsearch/data By default not persisted $ docker run -d -v /opt/elasticsearch/data:/usr/share/elasticsearch/data elasticsearch
- 49. Dealing With Storage $ docker run -d -v /opt/elasticsearch/data:/usr/share/elasticsearch/data elasticsearch By default in /usr/share/elasticsearch/data By default not persisted Use data only containers Permissions
- 50. Data Only Docker Volumes Bypasses Union File System
- 51. Data Only Docker Volumes Bypasses Union File System Can be shared between containers
- 52. Data Only Docker Volumes Bypasses Union File System Can be shared between containers Data volumes persist if the container itself is deleted
- 53. Data Only Docker Volumes Bypasses Union File System Can be shared between containers Data volumes persist if the container itself is deleted $ docker create -v /mnt/es/data:/usr/share/elasticsearch/data --name esdata elasticsearch Permissions
- 54. Data Only Docker Volumes Bypasses Union File System Can be shared between containers Data volumes persist if the container itself is deleted $ docker create -v /mnt/es/data:/usr/share/elasticsearch/data --name esdata elasticsearch $ docker run --volumes-from esdata elasticsearch
- 55. Highly Available Cluster Master only Master only Master only Data only Data only Data only Data only Data only Data only Client only Client only
- 56. Highly Available Cluster Master only Master only Master only Data only Data only Data only Data only Data only Data only Client only Client only minimum_master_nodes = N/2 + 1
- 57. Highly Available Cluster Master only Master only Master only Data only Data only Data only Data only Data only Data only Client only Client only minimum_master_nodes = N/2 + 1 recovery.after.nodes recovery.expected.nodes cluster.routing.allocation.node_concurrent_recoveries index.unassigned.node_left.delayed_timeout index.priority
- 58. Master Nodes & Docker $ docker run -d elasticsearch -Dnode.master=true -Dnode.data=false -Dnode.client=false
- 59. Client Nodes & Docker $ docker run -d elasticsearch -Dnode.master=false -Dnode.data=false -Dnode.client=true
- 60. Data Nodes & Docker $ docker run -d elasticsearch -Dnode.master=false -Dnode.data=true -Dnode.client=false
- 61. Scaling Elasticsearch Node Elasticsearch Node Elasticsearch Node Elasticsearch Node
- 62. Scaling curl -XPUT 'http://localhost:9200/devops/' -d '{ "settings" : { "index" : { "number_of_shards" : 4, "number_of_replicas" : 0 } } }'
- 63. Scaling P P P P
- 64. Scaling curl -XPUT 'http://localhost:9200/devops/_settings' -d '{ "index.number_of_replicas" : 1 }'
- 65. Scaling P P P P R R R R
- 66. Scaling curl -XPUT 'http://localhost:9200/devops/_settings' -d '{ "index.number_of_replicas" : 2 }'
- 67. Scaling P P P P R R R R R R R R
- 68. Scaling curl -XPUT 'http://localhost:9200/devops/_settings' -d '{ "index.number_of_replicas" : 1 }'
- 69. Scaling P P P P R R R R
- 70. Scaling P P P P R R R R
- 72. Use the Doc Values Luke! Default for non analyzed fields in 2.0 Used for sorting, aggregations and scripting Used instead of expensive fielddata cache
- 73. Use the Doc Values Luke! Default for non analyzed fields in 2.0 Used for sorting, aggregations and scripting Used instead of expensive fielddata cache curl -XPOST 'localhost:9200/devops' -d '{ "mappings" : { "logs" : { "properties" : { "tag" : { "type" : "string", "index" : "not_analyzed", "doc_values": true } }}}}'
- 74. Index Refresh 1s refresh -> 2K docs/sec 5s refresh -> 2.5K docs/sec 30s refresh -> 3.4K docs/sec
- 75. Index Refresh 1s refresh -> 2K docs/sec 5s refresh -> 2.5K docs/sec 30s refresh -> 3.4K docs/sec curl -XPUT 'localhost:9200/days/_settings' -d '{ "index" : { "refresh_interval" : "5s" } }'
- 76. RAM Buffer indices.memory.index_buffer_size: 10% indices.memory.min_index_buffer_size: 48mb indices.memory.max_index_buffer_size (unbounded) indices.memory.min_shard_index_buffer_size: 4mb
- 77. RAM Buffer indices.memory.index_buffer_size: 10% indices.memory.min_index_buffer_size: 48mb indices.memory.max_index_buffer_size (unbounded) indices.memory.min_shard_index_buffer_size: 4mb Higher Indexing Throughput Lower Indexing Throughput defaults ><
- 79. Time Based Data? 2015-11-23 2015-11-24 TODAY WEEK
- 80. Time Based Data? 2015-11-23 2015-11-24 2014-11-25 TODAY WEEK
- 81. Multiple Tiers node.tag=hot node.tag=cold node.tag=cold
- 82. Multiple Tiers curl -XPUT 'localhost:9200/data_2015-11-23' -d '{ "settings": { "index.routing.allocation.include.tag" : "hot" } }'
- 83. Multiple Tiers node.tag=hot node.tag=cold node.tag=cold data_2015-11-23 data_2015-11-23
- 84. Multiple Tiers curl -XPUT 'localhost:9200/data_2015-11-23/_settings' -d '{ "settings": { "index.routing.allocation.exclude.tag" : "hot", "index.routing.allocation.include.tag" : "cold", } }'
- 85. Multiple Tiers node.tag=hot node.tag=cold node.tag=cold data_2015-11-23 data_2015-11-23
- 86. Multiple Tiers curl -XPUT 'localhost:9200/data_2015-11-24' -d '{ "settings": { "index.routing.allocation.include.tag" : "hot" } }'
- 87. Multiple Tiers node.tag=hot node.tag=cold node.tag=cold data_2015-11-23 data_2015-11-23 data_2015-11-24 data_2015-11-24
- 88. Multiple Tiers node.tag=hot node.tag=cold node.tag=cold data_2015-11-23 data_2015-11-23 data_2015-11-25 data_2015-11-25 data_2015-11-24 data_2015-11-24
- 89. Multiple Tenants
- 92. Indexing Without Routing Shard 1 Shard 2 Shard 3 Shard 4 Shard 5 Shard 6 Shard 7 Shard 8 Elasticsearch Application userA userA userA userA userAuserA userA userA
- 93. Indexing With Routing Shard 1 Shard 2 Shard 3 Shard 4 Shard 5 Shard 6 Shard 7 Shard 8 Elasticsearch Application
- 94. Querying Without Routing Shard 1 Shard 2 Shard 3 Shard 4 Shard 5 Shard 6 Shard 7 Shard 8 Elasticsearch Application
- 95. Querying With Routing Shard 1 Shard 2 Shard 3 Shard 4 Shard 5 Shard 6 Shard 7 Shard 8 Elasticsearch Application
- 96. Routing vs No Routing Queries without routing (200 shards, 1 replica) #threads Avg response time Throughput 90% line Median CPU Utilization 1 3169ms 19,0/min 5214ms 2692ms 95 – 99%
- 97. Routing vs No Routing Queries without routing (200 shards, 1 replica) #threads Avg response time Throughput 90% line Median CPU Utilization 1 3169ms 19,0/min 5214ms 2692ms 95 – 99% Queries with routing (200 shards, 1 replica) #threads Avg response time Throughput 90% line Median CPU Utilization 10 196ms 50,6/sec 642ms 29ms 25 – 40% 20 218ms 91,2/sec 718ms 11ms 10 – 15%
- 100. We Are Hiring ! Dig Search ? Dig Analytics ? Dig Big Data ? Dig Performance ? Dig Logging ? Dig working with and in open – source ? We’re hiring world – wide ! http://sematext.com/about/jobs.html
Editor's Notes
- #10: Problems with standard deployment like: Resources not utilized Need to provision machines before deployment Differences between development, test, QA and production environments Hard to scale automatically
- #11: Problems with standard deployment like: Resources not utilized Need to provision machines before deployment Differences between development, test, QA and production environments Hard to scale automatically
- #12: Problems with standard deployment like: Resources not utilized Need to provision machines before deployment Differences between development, test, QA and production environments Hard to scale automatically
- #14: Amazon EC2 container service Spoonium Kubernetes RKT
- #62: Shards and Replicas
- #64: Shards and Replicas
- #66: Shards and Replicas
- #68: Shards and Replicas
- #70: Shards and Replicas
- #71: Shards and Replicas
- #72: Shards and Replicas
- #90: No time based indices No tiers Possible solution - routing
- #91: No time based indices No tiers Possible solution - routing
- #92: No time based indices No tiers Possible solution - routing