SlideShare a Scribd company logo

The elastic stack on docker

SmartWave, profile picture
SmartWave

This document discusses running the Elastic Stack (Elasticsearch, Kibana, and Logstash) using Docker. It begins with an introduction and overview of the Elastic ecosystem. It then covers installing and running Elasticsearch, Kibana, and Logstash as Docker images. It demonstrates how to create custom Docker images for each component using Dockerfiles. Finally, it shows how to tie the components together using Docker Compose to deploy the full Elastic Stack with one command.

Technology
1 of 51
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
The Elastic Stack on Docker Elastic in a Box
Agenda ● Getting acquainted ● Ecosystem ● Features overview ● What are we talking about? ● Installation ● ES on Docker ● Kibana on Docker ● Logstash on Docker ● Tying it all together ● What’s next?
Getting acquainted - Who am I? ● Valentin Crettaz (mailto:valentin.crettaz@consulthys.com) ○ https://www.linkedin.com/in/valentincrettaz/ ○ https://twitter.com/consulthys ● Developing Java software since 1996 ● Fell in love with Elasticsearch in 2010 (v0.9.0) ○ https://www.elastic.co/blog/you-know-for-search ● Running Elasticsearch meetups in Switzerland since early 2016 ○ https://www.meetup.com/fr-FR/elasticsearch-switzerland ● Active open-source contributor ○ https://github.com/consulthys ● Active Stack Overflow contributor ○ http://stackoverflow.com/users/4604579/val
Getting acquainted - Who are you? ● How many of you have already… ○ … heard of Elasticsearch? ○ … downloaded Elasticsearch? ○ … installed/run Elasticsearch? ○ … extended Elasticsearch? ● Your background? ● Define Elasticsearch with your own words
Getting acquainted - Disclaimer
Agenda ● Getting acquainted ● Ecosystem ● Features overview ● What are we talking about? ● Installation ● ES on Docker ● Kibana on Docker ● Logstash on Docker ● Tying it all together ● What’s next?
Ecosystem - The Big Picture
Ecosystem - An Even Bigger Picture
Ecosystem - Logstash 2013: https://www.elastic.co/blog/welcome-jordan-logstash
Ecosystem - Kibana 2013: https://www.elastic.co/blog/welcome-drew-rashid
Ecosystem - Beats 2015: https://www.elastic.co/blog/welcome-packetbeat-tudor-monica
Ecosystem - Monitoring
Ecosystem - Machine Learning 2016: https://www.elastic.co/blog/welcome-prelert-to-the-elastic-team
Ecosystem - Graph
Ecosystem - Elastic Cloud 2015: https://www.elastic.co/blog/welcome-found
Ecosystem - APM 2017: https://www.elastic.co/blog/welcome-opbeat-to-the-elastic-family
Ecosystem - Site + Application Search 2017: https://www.elastic.co/blog/swiftype-joins-forces-with-elastic
Ecosystem - Semantic Source Search 2018: https://www.elastic.co/blog/welcome-insight-io-to-the-elastic-team
Ecosystem - and more... ● Watcher (for alerting) ● Shield (for security) ● Report (for reporting) ● Canvas (for infographics) ● ...
Agenda ● Getting acquainted ● Ecosystem ● Features overview ● What are we talking about? ● Installation ● ES on Docker ● Kibana on Docker ● Logstash on Docker ● Tying it all together ● What’s next?
Elasticsearch Elasticsearch is a distributed, RESTful search and analytics engine capable of solving a growing number of use cases.
Use cases ● Full-text search ● Application search ● Enterprise search ● Business analytics ● Metrics analytics ● Security analytics ● Operational logs analytics ● Anomaly detection ● ...
Features overview ● Distributed and scalable ● Resilient ● Fault tolerant ● High availability ● RESTful interface ● Document-oriented ● Schema free ● Multi-tenancy ● Extensible ● Growing and active community ● Query DSL ● Aggregations ● Full-text search (Lucene) ● Structured search ● Geo-spatial search ● Suggesters ● Highlighters ● Percolation ● Profiling ● Client libraries in 10+ languages ● ...
Agenda ● Getting acquainted ● Ecosystem ● Features overview ● What are we talking about? ● Installation ● ES on Docker ● Kibana on Docker ● Logstash on Docker ● Tying it all together ● What’s next?
What are we talking about?
What are we talking about?
Agenda ● Getting acquainted ● Ecosystem ● Features overview ● What are we talking about? ● Installation ● ES on Docker ● Kibana on Docker ● Logstash on Docker ● Tying it all together ● What’s next?
Installation (Vanilla) - https://elastic.co/downloads 1. Download and unzip (or yum/dpkg/msi it) 2. ./bin/elasticsearch 3. curl http://localhost:9200/ 4. That’s all folks, now you can brag about it !
Installation (Docker) - https://www.docker.elastic.co/
Agenda ● Getting acquainted ● Ecosystem ● Features overview ● What are we talking about? ● Installation ● ES on Docker ● Kibana on Docker ● Logstash on Docker ● Tying it all together ● What’s next?
Running Official Docker Images (1/2) docker run docker.elastic.co/elasticsearch/elasticsearch:6.4.1 docker run -e "node.name=es_node1" docker.elastic.co/elasticsearch/elasticsearch:6.4.1 docker run -e "node.name=es_node1" -e ES_JAVA_OPTS="-Xmx2g -Xms2g" -e "cluster.name=my_es_cluster" docker.elastic.co/elasticsearch/elasticsearch:6.4.1 ...
Running Official Docker Images (2/2) ... docker run -e "node.name=es_node1" -e ES_JAVA_OPTS="-Xmx2g -Xms2g" -e "cluster.name=my_es_cluster" -v /elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml docker.elastic.co/elasticsearch/elasticsearch:6.4.1 docker run -e "node.name=es_node1" -e ES_JAVA_OPTS="-Xmx2g -Xms2g" -e "cluster.name=my_es_cluster" -v /elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml -v /mnt/es-data:/usr/share/elasticsearch/data -p 9201:9200 docker.elastic.co/elasticsearch/elasticsearch:6.4.1
Creating custom images using Dockerfile (1/5) FROM docker.elastic.co/elasticsearch/elasticsearch:6.4.1 ENV ES_JAVA_OPTS="-Xmx2g -Xms2g" COPY /conf/elasticsearch.yml /usr/share/elasticsearch/config/ VOLUME /mnt/es-data EXPOSE 9200 9300 HEALTHCHECK CMD curl --fail http://localhost:9200 || exit 1
$> docker build . -t custom_es641 Sending build context to Docker daemon 45.44MB Step 1/6 : FROM docker.elastic.co/elasticsearch/elasticsearch:6.4.1 ---> 96dd1575de0f Step 2/6 : ENV ES_JAVA_OPTS "-Xmx2g -Xms2g" ---> Using cache ---> d9b744d2ce1a Step 3/6 : COPY /conf/elasticsearch.yml /usr/share/elasticsearch/config/ ---> Using cache ---> 920f451da5b0 Step 4/6 : VOLUME /mnt/es-data ---> Using cache ---> 9b1e957f4820 Step 5/6 : EXPOSE 9200 9300 ---> Using cache ---> 8b62712bc1e0 Step 6/6 : HEALTHCHECK CMD curl --fail http://localhost:9200 || exit 1 ---> Using cache ---> c673dde527e4 Successfully built c673dde527e4 Successfully tagged custom_es641:latest Creating custom images using Dockerfile (2/5)
$> docker run -p 9200:9200 custom_es641:latest $> curl localhost:9200 { "name" : "Slo94R2", "cluster_name" : "elasticsearch", "cluster_uuid" : "ccJTixm4QOmtUQn8wnWa7Q", "version" : { "number" : "6.4.1", "build_flavor" : "default", "build_type" : "tar", "build_hash" : "e36acdb", "build_date" : "2018-09-13T22:18:07.696808Z", "build_snapshot" : false, "lucene_version" : "7.4.0", "minimum_wire_compatibility_version" : "5.6.0", "minimum_index_compatibility_version" : "5.0.0" }, "tagline" : "You Know, for Search" } Creating custom images using Dockerfile (3/5) elasticsearch Slo94R2
Creating custom images using Dockerfile (4/5) docker run -e "node.name=es1" --name es1 -p 9200:9200 -p 9300:9300 -e "discovery.zen.ping.unicast.hosts= es1,es2,es3" -e "cluster.name=3_node_cluster" custom_es641:latest docker run -e "node.name=es2" --name es2 -p 9201:9200 -p 9301:9300 -e "discovery.zen.ping.unicast.hosts= es1,es2,es3" -e "cluster.name=3_node_cluster" custom_es641:latest docker run -e "node.name=es3" --name es3 -p 9202:9200 -p 9302:9300 -e "discovery.zen.ping.unicast.hosts= es1,es2,es3" -e "cluster.name=3_node_cluster" custom_es641:latest
Creating custom images using Dockerfile (5/5) $ curl localhost:9200/_cat/nodes?v ip heap.% ram.% cpu load_1m load_5m load_15m node.role master name 172.17.0.4 36 58 42 1.47 0.59 0.46 mdi - es3 172.17.0.3 56 58 53 1.47 0.59 0.46 mdi - es2 172.17.0.2 42 58 30 1.47 0.59 0.46 mdi * es1 3_node_cluster es2es1* es3
Agenda ● Getting acquainted ● Ecosystem ● Features overview ● What are we talking about? ● Installation ● ES on Docker ● Kibana on Docker ● Logstash on Docker ● Tying it all together ● What’s next?
Kibana on Docker (1/2) docker run docker.elastic.co/kibana/kibana:6.4.1 docker run --rm -p 5601:5601 -e "ELASTICSEARCH_URL=http://es1:9200" docker.elastic.co/kibana/kibana:6.4.1 docker run --rm -p 5601:5601 -e "ELASTICSEARCH_URL=http://es1:9200" -e "SERVER_NAME=my-kibana" docker.elastic.co/kibana/kibana:6.4.1 ...
Kibana on Docker (2/2) FROM docker.elastic.co/kibana/kibana:6.4.1 ENV ELASTICSEARCH_URL="http://es1:9200" COPY /conf/kibana.yml /usr/share/kibana/config/ EXPOSE 5601 HEALTHCHECK CMD curl --fail http://localhost:5601 || exit 1
Agenda ● Getting acquainted ● Ecosystem ● Features overview ● What are we talking about? ● Installation ● ES on Docker ● Kibana on Docker ● Logstash on Docker ● Tying it all together ● What’s next?
Logstash on Docker (1/2) docker run docker.elastic.co/logstash/logstash:6.4.1 docker run --rm -p 9600:9600 -e "PIPELINE_WORKERS=3" docker.elastic.co/logstash/logstash:6.4.1 docker run --rm -p 9600:9600 -e "PIPELINE_WORKERS=3" -e "LOG_LEVEL=DEBUG" docker.elastic.co/logstash/logstash:6.4.1 ...
Logstash on Docker (2/2) FROM docker.elastic.co/logstash/logstash:6.4.1 RUN rm -f /usr/share/logstash/pipeline/logstash.conf ADD pipeline/ /usr/share/logstash/pipeline/ ADD config/ /usr/share/logstash/config/ EXPOSE 9600
Agenda ● Getting acquainted ● Ecosystem ● Features overview ● What are we talking about? ● Installation ● ES on Docker ● Kibana on Docker ● Logstash on Docker ● Tying it all together ● What’s next?
Tying it all together using docker-compose (1/2) version: "3.3" services: elasticsearch: image: docker.elastic.co/elasticsearch/elasticsearch:6.4.1 volumes: - ./elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml - ./jvm.options:/usr/share/elasticsearch/config/jvm.options - ./data:/usr/share/elasticsearch/data - ./snapshots:/tmp/es6_dev_repo ports: - "9200:9200" - "9300:9300" environment: ES_JAVA_OPTS: "-Xmx2g -Xms2g" networks: - elk ... ... kibana: image: docker.elastic.co/kibana/kibana:6.4.1 volumes: - ./kibana/config/:/usr/share/kibana/config - ./kibana/cache/:/usr/share/kibana/optimize ports: - "5601:5601" networks: - elk depends_on: - elasticsearch ...
Tying it all together using docker-compose (2/2) ... logstash: image: docker.elastic.co/logstash/logstash:6.4.1 volumes: - ./logstash/config/:/usr/share/logstash/config/ ports: - "9600:9600" networks: - elk depends_on: - elasticsearch networks: elk: driver: bridge $ docker-compose up
What are we talking about?
Tying it all together https://www.elastic.co/blog/a-full-stack-in-one-command https://github.com/elastic/examples/tree/master/Miscellaneous/docker/full_stack_example
Agenda ● Getting acquainted ● Ecosystem ● Features overview ● What are we talking about? ● Installation ● ES on Docker ● Kibana on Docker ● Logstash on Docker ● Tying it all together ● What’s next?
● Docs: https://www.elastic.co/guide/index.html ● Videos: https://www.elastic.co/videos ● Slides: https://speakerdeck.com/elastic/ ● Blog: https://www.elastic.co/blog ● Source: https://github.com/elastic ● Meetups: https://www.meetup.com/fr-FR/elastic-switzerland/ ● Conference: https://www.elastic.co/elasticon/conf/2018/sf ● Discuss: ■ https://stackoverflow.com/questions/tagged/elasticsearch ■ https://discuss.elastic.co/ ● Articles: ■ https://www.elastic.co/blog/a-full-stack-in-one-command ■ https://www.elastic.co/blog/docker-networking ■ https://sematext.com/blog/elasticsearch-in-docker/ ■ https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html ■ https://www.elastic.co/guide/en/kibana/current/docker.html ■ https://www.elastic.co/guide/en/logstash/current/docker.html What’s next?
Q&A

More Related Content

PDF
Deploy Python apps in 5 min with a PaaS
Appsembler
 
PDF
Zun presentation (OpenStack Barcelona summit)
hongbin034
 
PDF
Elastic 101 tutorial - Percona Europe 2018
Antonios Giannopoulos
 
KEY
Dcjq node.js presentation
async_io
 
PPT
Large Scale Log collection using LogStash & mongoDB
Gaurav Bhardwaj
 
PPTX
Attack monitoring using ElasticSearch Logstash and Kibana
Prajal Kulkarni
 
PDF
OpenStack: running manually installed components on VirtualBox
Ian Choi
 
PDF
Talk on PHP Day Uruguay about Docker
Wellington Silva
 
Deploy Python apps in 5 min with a PaaS
Appsembler
 
Zun presentation (OpenStack Barcelona summit)
hongbin034
 
Elastic 101 tutorial - Percona Europe 2018
Antonios Giannopoulos
 
Dcjq node.js presentation
async_io
 
Large Scale Log collection using LogStash & mongoDB
Gaurav Bhardwaj
 
Attack monitoring using ElasticSearch Logstash and Kibana
Prajal Kulkarni
 
OpenStack: running manually installed components on VirtualBox
Ian Choi
 
Talk on PHP Day Uruguay about Docker
Wellington Silva
 

What's hot (20)

PDF
Zun project update (boston summit)
hongbin034
 
PPTX
Easy Docker on Microsoft Azure
Docker, Inc.
 
PDF
How to master OpenStack in 2 hours
OpenCity Community
 
PDF
Wordcamp Bratislava 2017 - Docker! Why?
Adam Štipák
 
PDF
Docker 1 0 1 0 1: a Docker introduction, actualized for the stable release of...
Jérôme Petazzoni
 
PPTX
Powershell dcpp
artisriva
 
PDF
Mobile Analytics mit Elasticsearch und Kibana
inovex GmbH
 
PDF
Docker Introduction + what is new in 0.9
Jérôme Petazzoni
 
PDF
Drools Workshop @JBCNCONF 2016
Mauricio (Salaboy) Salatino
 
PDF
2013 PyCon SG - Building your cloud infrastructure with Python
George Goh
 
PDF
Nova: Openstack Compute-as-a-service
Pratik Bandarkar
 
PDF
Wso2 con 2014-us-tutorial-apache stratos-wso2 private paas with docker integr...
Lakmal Warusawithana
 
PDF
Getting Started with PureScript
John De Goes
 
PPTX
Тарас Кирилюк — Docker basics. How-to for Drupal developers
LEDC 2016
 
PDF
Node.js, toy or power tool?
Ovidiu Dimulescu
 
PPTX
State of Containers in OpenStack
openstackindia
 
PPTX
Running High Performance & Fault-tolerant Elasticsearch Clusters on Docker
Sematext Group, Inc.
 
PDF
Node.js 101 with Rami Sayar
FITC
 
PDF
Philly Tech Week Introduction to NodeJS
Ross Kukulinski
 
PDF
Intro to containerization
Balint Pato
 
Zun project update (boston summit)
hongbin034
 
Easy Docker on Microsoft Azure
Docker, Inc.
 
How to master OpenStack in 2 hours
OpenCity Community
 
Wordcamp Bratislava 2017 - Docker! Why?
Adam Štipák
 
Docker 1 0 1 0 1: a Docker introduction, actualized for the stable release of...
Jérôme Petazzoni
 
Powershell dcpp
artisriva
 
Mobile Analytics mit Elasticsearch und Kibana
inovex GmbH
 
Docker Introduction + what is new in 0.9
Jérôme Petazzoni
 
Drools Workshop @JBCNCONF 2016
Mauricio (Salaboy) Salatino
 
2013 PyCon SG - Building your cloud infrastructure with Python
George Goh
 
Nova: Openstack Compute-as-a-service
Pratik Bandarkar
 
Wso2 con 2014-us-tutorial-apache stratos-wso2 private paas with docker integr...
Lakmal Warusawithana
 
Getting Started with PureScript
John De Goes
 
Тарас Кирилюк — Docker basics. How-to for Drupal developers
LEDC 2016
 
Node.js, toy or power tool?
Ovidiu Dimulescu
 
State of Containers in OpenStack
openstackindia
 
Running High Performance & Fault-tolerant Elasticsearch Clusters on Docker
Sematext Group, Inc.
 
Node.js 101 with Rami Sayar
FITC
 
Philly Tech Week Introduction to NodeJS
Ross Kukulinski
 
Intro to containerization
Balint Pato
 
Ad

Similar to The elastic stack on docker (20)

PDF
Philipp Krenn, Elastic. From Containers to Kubernetes Operators
IT Arena
 
PPTX
DevOpsDays Warsaw 2015: Running High Performance And Fault Tolerant Elasticse...
PROIDEA
 
PDF
How LogDNA Scaled Elasticsearch on Kubernetes
LogDNA
 
PPT
Running High Performance and Fault Tolerant Elasticsearch Clusters on Docker
Sematext Group, Inc.
 
PDF
Log analysis with elastic stack
Bangladesh Network Operators Group
 
PDF
Run the elastic stack on kubernetes with eck
Daliya Spasova
 
PPTX
Deploying E.L.K stack w Puppet
Colin Brown
 
PDF
Null Bachaav - May 07 Attack Monitoring workshop.
Prajal Kulkarni
 
PDF
Ejecución del Elastic Stack en Kubernetes
Elasticsearch
 
PDF
Elasticsearch on Kubernetes
Joerg Henning
 
PDF
Deploying Elasticsearch on Docker with Weave
Ilya Dmitrichenko
 
PDF
Elastic101tutorial Percona Live Europe 2018
Alex Cercel
 
PPTX
Perl and Elasticsearch
Dean Hamstead
 
PDF
Elasticsearch, a distributed search engine with real-time analytics
Tiziano Fagni
 
PPT
Elk presentation1#3
uzzal basak
 
PDF
Elastic{ON} Seminar New York (2017)
Franklin Angulo
 
PDF
2015 03-16-elk at-bsides
Jeremy Cohoe
 
PDF
Is your Elastic Cluster Stable and Production Ready?
DoiT International
 
PPTX
Elastic Meetup Belgium - December 2018
Arthur Eyckerman
 
PDF
Elasticsearch Introduction at BigData meetup
Eric Rodriguez (Hiring in Lex)
 
Philipp Krenn, Elastic. From Containers to Kubernetes Operators
IT Arena
 
DevOpsDays Warsaw 2015: Running High Performance And Fault Tolerant Elasticse...
PROIDEA
 
How LogDNA Scaled Elasticsearch on Kubernetes
LogDNA
 
Running High Performance and Fault Tolerant Elasticsearch Clusters on Docker
Sematext Group, Inc.
 
Log analysis with elastic stack
Bangladesh Network Operators Group
 
Run the elastic stack on kubernetes with eck
Daliya Spasova
 
Deploying E.L.K stack w Puppet
Colin Brown
 
Null Bachaav - May 07 Attack Monitoring workshop.
Prajal Kulkarni
 
Ejecución del Elastic Stack en Kubernetes
Elasticsearch
 
Elasticsearch on Kubernetes
Joerg Henning
 
Deploying Elasticsearch on Docker with Weave
Ilya Dmitrichenko
 
Elastic101tutorial Percona Live Europe 2018
Alex Cercel
 
Perl and Elasticsearch
Dean Hamstead
 
Elasticsearch, a distributed search engine with real-time analytics
Tiziano Fagni
 
Elk presentation1#3
uzzal basak
 
Elastic{ON} Seminar New York (2017)
Franklin Angulo
 
2015 03-16-elk at-bsides
Jeremy Cohoe
 
Is your Elastic Cluster Stable and Production Ready?
DoiT International
 
Elastic Meetup Belgium - December 2018
Arthur Eyckerman
 
Elasticsearch Introduction at BigData meetup
Eric Rodriguez (Hiring in Lex)
 
Ad

More from SmartWave (20)

PDF
How to build an API strategy - Dorian Rougierx.
SmartWave
 
PDF
Répondre aux défis de la gestion des factures fournisseurs
SmartWave
 
PDF
SmartTechTalk : Asynchronous messaging
SmartWave
 
PPTX
Data Virtualisation and API Management United
SmartWave
 
PPTX
Data Agility and Security with Data Virtualisation
SmartWave
 
PPTX
API Program Lessons learned
SmartWave
 
PDF
Customer testimonal API Program Lessons learned
SmartWave
 
PDF
API Management Microservices beyond HIP
SmartWave
 
PDF
How does an API management strategy support your digital transformation?
SmartWave
 
PDF
Monitoring docker, k8s and your applications with the elastic stack
SmartWave
 
PDF
Gestion des logs de vos containers avec elastic !
SmartWave
 
PDF
API Trends
SmartWave
 
PDF
How api management supports the digital transformation process
SmartWave
 
PDF
Docker Geneva Meetup - Jelastic
SmartWave
 
PPTX
Docker Geneva Meetup - Swarm
SmartWave
 
PDF
Docker Geneva Meetup - Kubernetes
SmartWave
 
PPTX
Dématérialisation du traitement des factures
SmartWave
 
PDF
Axway amplify api management platform
SmartWave
 
PDF
Api gateway @ vaudoise assurances
SmartWave
 
PDF
MSC Digital transformation with Axway API Management products and SmartWave S...
SmartWave
 
How to build an API strategy - Dorian Rougierx.
SmartWave
 
Répondre aux défis de la gestion des factures fournisseurs
SmartWave
 
SmartTechTalk : Asynchronous messaging
SmartWave
 
Data Virtualisation and API Management United
SmartWave
 
Data Agility and Security with Data Virtualisation
SmartWave
 
API Program Lessons learned
SmartWave
 
Customer testimonal API Program Lessons learned
SmartWave
 
API Management Microservices beyond HIP
SmartWave
 
How does an API management strategy support your digital transformation?
SmartWave
 
Monitoring docker, k8s and your applications with the elastic stack
SmartWave
 
Gestion des logs de vos containers avec elastic !
SmartWave
 
API Trends
SmartWave
 
How api management supports the digital transformation process
SmartWave
 
Docker Geneva Meetup - Jelastic
SmartWave
 
Docker Geneva Meetup - Swarm
SmartWave
 
Docker Geneva Meetup - Kubernetes
SmartWave
 
Dématérialisation du traitement des factures
SmartWave
 
Axway amplify api management platform
SmartWave
 
Api gateway @ vaudoise assurances
SmartWave
 
MSC Digital transformation with Axway API Management products and SmartWave S...
SmartWave
 

Recently uploaded (20)

PDF
Optimiser vos workloads AI/ML sur Amazon EC2 et AWS Graviton
Julien SIMON
 
PDF
The Rise and Fall of 3GPP – Time for a Sabbatical?
3G4G
 
PDF
Profit Center Accounting in SAP S/4HANA, S4F28 Col11
Libreria ERP
 
PPTX
Detection-First SIEM: Rule Types, Dashboards, and Threat-Informed Strategy
ReZa AdineH
 
PDF
Spectral efficient network and resource selection model in 5G networks
IAESIJAI
 
PDF
Encapsulation theory and applications.pdf
gurumoop
 
PDF
TokAI - TikTok AI Agent : The First AI Application That Analyzes 10,000+ Vira...
SOFTTECHHUB
 
DOCX
The AUB Centre for AI in Media Proposal.docx
GAONE9
 
PDF
Review of recent advances in non-invasive hemoglobin estimation
IAESIJAI
 
PPTX
Spectroscopy.pptx food analysis technology
nawahassan45
 
PPTX
sap open course for s4hana steps from ECC to s4
sreeni2106invites
 
PPTX
MYSQL Presentation for SQL database connectivity
Swati270511
 
PDF
NewMind AI Weekly Chronicles - August'25 Week I
NewMind AI
 
PPTX
20250228 LYD VKU AI Blended-Learning.pptx
DatVoNgoc
 
PDF
cuic standard and advanced reporting.pdf
SimoneTitaniumTomase
 
PDF
Per capita expenditure prediction using model stacking based on satellite ima...
IAESIJAI
 
PDF
Agricultural_Statistics_at_a_Glance_2022_0.pdf
SandeepSingh286037
 
PPTX
Programs and apps: productivity, graphics, security and other tools
4mqw9zch22
 
PDF
Advanced methodologies resolving dimensionality complications for autism neur...
IAESIJAI
 
PDF
7 ChatGPT Prompts to Help You Define Your Ideal Customer Profile.pdf
SOFTTECHHUB
 
Optimiser vos workloads AI/ML sur Amazon EC2 et AWS Graviton
Julien SIMON
 
The Rise and Fall of 3GPP – Time for a Sabbatical?
3G4G
 
Profit Center Accounting in SAP S/4HANA, S4F28 Col11
Libreria ERP
 
Detection-First SIEM: Rule Types, Dashboards, and Threat-Informed Strategy
ReZa AdineH
 
Spectral efficient network and resource selection model in 5G networks
IAESIJAI
 
Encapsulation theory and applications.pdf
gurumoop
 
TokAI - TikTok AI Agent : The First AI Application That Analyzes 10,000+ Vira...
SOFTTECHHUB
 
The AUB Centre for AI in Media Proposal.docx
GAONE9
 
Review of recent advances in non-invasive hemoglobin estimation
IAESIJAI
 
Spectroscopy.pptx food analysis technology
nawahassan45
 
sap open course for s4hana steps from ECC to s4
sreeni2106invites
 
MYSQL Presentation for SQL database connectivity
Swati270511
 
NewMind AI Weekly Chronicles - August'25 Week I
NewMind AI
 
20250228 LYD VKU AI Blended-Learning.pptx
DatVoNgoc
 
cuic standard and advanced reporting.pdf
SimoneTitaniumTomase
 
Per capita expenditure prediction using model stacking based on satellite ima...
IAESIJAI
 
Agricultural_Statistics_at_a_Glance_2022_0.pdf
SandeepSingh286037
 
Programs and apps: productivity, graphics, security and other tools
4mqw9zch22
 
Advanced methodologies resolving dimensionality complications for autism neur...
IAESIJAI
 
7 ChatGPT Prompts to Help You Define Your Ideal Customer Profile.pdf
SOFTTECHHUB
 

The elastic stack on docker

  • 1. The Elastic Stack on Docker Elastic in a Box
  • 2. Agenda ● Getting acquainted ● Ecosystem ● Features overview ● What are we talking about? ● Installation ● ES on Docker ● Kibana on Docker ● Logstash on Docker ● Tying it all together ● What’s next?
  • 3. Getting acquainted - Who am I? ● Valentin Crettaz (mailto:valentin.crettaz@consulthys.com) ○ https://www.linkedin.com/in/valentincrettaz/ ○ https://twitter.com/consulthys ● Developing Java software since 1996 ● Fell in love with Elasticsearch in 2010 (v0.9.0) ○ https://www.elastic.co/blog/you-know-for-search ● Running Elasticsearch meetups in Switzerland since early 2016 ○ https://www.meetup.com/fr-FR/elasticsearch-switzerland ● Active open-source contributor ○ https://github.com/consulthys ● Active Stack Overflow contributor ○ http://stackoverflow.com/users/4604579/val
  • 4. Getting acquainted - Who are you? ● How many of you have already… ○ … heard of Elasticsearch? ○ … downloaded Elasticsearch? ○ … installed/run Elasticsearch? ○ … extended Elasticsearch? ● Your background? ● Define Elasticsearch with your own words
  • 5. Getting acquainted - Disclaimer
  • 6. Agenda ● Getting acquainted ● Ecosystem ● Features overview ● What are we talking about? ● Installation ● ES on Docker ● Kibana on Docker ● Logstash on Docker ● Tying it all together ● What’s next?
  • 7. Ecosystem - The Big Picture
  • 8. Ecosystem - An Even Bigger Picture
  • 9. Ecosystem - Logstash 2013: https://www.elastic.co/blog/welcome-jordan-logstash
  • 10. Ecosystem - Kibana 2013: https://www.elastic.co/blog/welcome-drew-rashid
  • 11. Ecosystem - Beats 2015: https://www.elastic.co/blog/welcome-packetbeat-tudor-monica
  • 13. Ecosystem - Machine Learning 2016: https://www.elastic.co/blog/welcome-prelert-to-the-elastic-team
  • 15. Ecosystem - Elastic Cloud 2015: https://www.elastic.co/blog/welcome-found
  • 16. Ecosystem - APM 2017: https://www.elastic.co/blog/welcome-opbeat-to-the-elastic-family
  • 17. Ecosystem - Site + Application Search 2017: https://www.elastic.co/blog/swiftype-joins-forces-with-elastic
  • 18. Ecosystem - Semantic Source Search 2018: https://www.elastic.co/blog/welcome-insight-io-to-the-elastic-team
  • 19. Ecosystem - and more... ● Watcher (for alerting) ● Shield (for security) ● Report (for reporting) ● Canvas (for infographics) ● ...
  • 20. Agenda ● Getting acquainted ● Ecosystem ● Features overview ● What are we talking about? ● Installation ● ES on Docker ● Kibana on Docker ● Logstash on Docker ● Tying it all together ● What’s next?
  • 21. Elasticsearch Elasticsearch is a distributed, RESTful search and analytics engine capable of solving a growing number of use cases.
  • 22. Use cases ● Full-text search ● Application search ● Enterprise search ● Business analytics ● Metrics analytics ● Security analytics ● Operational logs analytics ● Anomaly detection ● ...
  • 23. Features overview ● Distributed and scalable ● Resilient ● Fault tolerant ● High availability ● RESTful interface ● Document-oriented ● Schema free ● Multi-tenancy ● Extensible ● Growing and active community ● Query DSL ● Aggregations ● Full-text search (Lucene) ● Structured search ● Geo-spatial search ● Suggesters ● Highlighters ● Percolation ● Profiling ● Client libraries in 10+ languages ● ...
  • 24. Agenda ● Getting acquainted ● Ecosystem ● Features overview ● What are we talking about? ● Installation ● ES on Docker ● Kibana on Docker ● Logstash on Docker ● Tying it all together ● What’s next?
  • 25. What are we talking about?
  • 26. What are we talking about?
  • 27. Agenda ● Getting acquainted ● Ecosystem ● Features overview ● What are we talking about? ● Installation ● ES on Docker ● Kibana on Docker ● Logstash on Docker ● Tying it all together ● What’s next?
  • 28. Installation (Vanilla) - https://elastic.co/downloads 1. Download and unzip (or yum/dpkg/msi it) 2. ./bin/elasticsearch 3. curl http://localhost:9200/ 4. That’s all folks, now you can brag about it !
  • 29. Installation (Docker) - https://www.docker.elastic.co/
  • 30. Agenda ● Getting acquainted ● Ecosystem ● Features overview ● What are we talking about? ● Installation ● ES on Docker ● Kibana on Docker ● Logstash on Docker ● Tying it all together ● What’s next?
  • 31. Running Official Docker Images (1/2) docker run docker.elastic.co/elasticsearch/elasticsearch:6.4.1 docker run -e "node.name=es_node1" docker.elastic.co/elasticsearch/elasticsearch:6.4.1 docker run -e "node.name=es_node1" -e ES_JAVA_OPTS="-Xmx2g -Xms2g" -e "cluster.name=my_es_cluster" docker.elastic.co/elasticsearch/elasticsearch:6.4.1 ...
  • 32. Running Official Docker Images (2/2) ... docker run -e "node.name=es_node1" -e ES_JAVA_OPTS="-Xmx2g -Xms2g" -e "cluster.name=my_es_cluster" -v /elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml docker.elastic.co/elasticsearch/elasticsearch:6.4.1 docker run -e "node.name=es_node1" -e ES_JAVA_OPTS="-Xmx2g -Xms2g" -e "cluster.name=my_es_cluster" -v /elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml -v /mnt/es-data:/usr/share/elasticsearch/data -p 9201:9200 docker.elastic.co/elasticsearch/elasticsearch:6.4.1
  • 33. Creating custom images using Dockerfile (1/5) FROM docker.elastic.co/elasticsearch/elasticsearch:6.4.1 ENV ES_JAVA_OPTS="-Xmx2g -Xms2g" COPY /conf/elasticsearch.yml /usr/share/elasticsearch/config/ VOLUME /mnt/es-data EXPOSE 9200 9300 HEALTHCHECK CMD curl --fail http://localhost:9200 || exit 1
  • 34. $> docker build . -t custom_es641 Sending build context to Docker daemon 45.44MB Step 1/6 : FROM docker.elastic.co/elasticsearch/elasticsearch:6.4.1 ---> 96dd1575de0f Step 2/6 : ENV ES_JAVA_OPTS "-Xmx2g -Xms2g" ---> Using cache ---> d9b744d2ce1a Step 3/6 : COPY /conf/elasticsearch.yml /usr/share/elasticsearch/config/ ---> Using cache ---> 920f451da5b0 Step 4/6 : VOLUME /mnt/es-data ---> Using cache ---> 9b1e957f4820 Step 5/6 : EXPOSE 9200 9300 ---> Using cache ---> 8b62712bc1e0 Step 6/6 : HEALTHCHECK CMD curl --fail http://localhost:9200 || exit 1 ---> Using cache ---> c673dde527e4 Successfully built c673dde527e4 Successfully tagged custom_es641:latest Creating custom images using Dockerfile (2/5)
  • 35. $> docker run -p 9200:9200 custom_es641:latest $> curl localhost:9200 { "name" : "Slo94R2", "cluster_name" : "elasticsearch", "cluster_uuid" : "ccJTixm4QOmtUQn8wnWa7Q", "version" : { "number" : "6.4.1", "build_flavor" : "default", "build_type" : "tar", "build_hash" : "e36acdb", "build_date" : "2018-09-13T22:18:07.696808Z", "build_snapshot" : false, "lucene_version" : "7.4.0", "minimum_wire_compatibility_version" : "5.6.0", "minimum_index_compatibility_version" : "5.0.0" }, "tagline" : "You Know, for Search" } Creating custom images using Dockerfile (3/5) elasticsearch Slo94R2
  • 36. Creating custom images using Dockerfile (4/5) docker run -e "node.name=es1" --name es1 -p 9200:9200 -p 9300:9300 -e "discovery.zen.ping.unicast.hosts= es1,es2,es3" -e "cluster.name=3_node_cluster" custom_es641:latest docker run -e "node.name=es2" --name es2 -p 9201:9200 -p 9301:9300 -e "discovery.zen.ping.unicast.hosts= es1,es2,es3" -e "cluster.name=3_node_cluster" custom_es641:latest docker run -e "node.name=es3" --name es3 -p 9202:9200 -p 9302:9300 -e "discovery.zen.ping.unicast.hosts= es1,es2,es3" -e "cluster.name=3_node_cluster" custom_es641:latest
  • 37. Creating custom images using Dockerfile (5/5) $ curl localhost:9200/_cat/nodes?v ip heap.% ram.% cpu load_1m load_5m load_15m node.role master name 172.17.0.4 36 58 42 1.47 0.59 0.46 mdi - es3 172.17.0.3 56 58 53 1.47 0.59 0.46 mdi - es2 172.17.0.2 42 58 30 1.47 0.59 0.46 mdi * es1 3_node_cluster es2es1* es3
  • 38. Agenda ● Getting acquainted ● Ecosystem ● Features overview ● What are we talking about? ● Installation ● ES on Docker ● Kibana on Docker ● Logstash on Docker ● Tying it all together ● What’s next?
  • 39. Kibana on Docker (1/2) docker run docker.elastic.co/kibana/kibana:6.4.1 docker run --rm -p 5601:5601 -e "ELASTICSEARCH_URL=http://es1:9200" docker.elastic.co/kibana/kibana:6.4.1 docker run --rm -p 5601:5601 -e "ELASTICSEARCH_URL=http://es1:9200" -e "SERVER_NAME=my-kibana" docker.elastic.co/kibana/kibana:6.4.1 ...
  • 40. Kibana on Docker (2/2) FROM docker.elastic.co/kibana/kibana:6.4.1 ENV ELASTICSEARCH_URL="http://es1:9200" COPY /conf/kibana.yml /usr/share/kibana/config/ EXPOSE 5601 HEALTHCHECK CMD curl --fail http://localhost:5601 || exit 1
  • 41. Agenda ● Getting acquainted ● Ecosystem ● Features overview ● What are we talking about? ● Installation ● ES on Docker ● Kibana on Docker ● Logstash on Docker ● Tying it all together ● What’s next?
  • 42. Logstash on Docker (1/2) docker run docker.elastic.co/logstash/logstash:6.4.1 docker run --rm -p 9600:9600 -e "PIPELINE_WORKERS=3" docker.elastic.co/logstash/logstash:6.4.1 docker run --rm -p 9600:9600 -e "PIPELINE_WORKERS=3" -e "LOG_LEVEL=DEBUG" docker.elastic.co/logstash/logstash:6.4.1 ...
  • 43. Logstash on Docker (2/2) FROM docker.elastic.co/logstash/logstash:6.4.1 RUN rm -f /usr/share/logstash/pipeline/logstash.conf ADD pipeline/ /usr/share/logstash/pipeline/ ADD config/ /usr/share/logstash/config/ EXPOSE 9600
  • 44. Agenda ● Getting acquainted ● Ecosystem ● Features overview ● What are we talking about? ● Installation ● ES on Docker ● Kibana on Docker ● Logstash on Docker ● Tying it all together ● What’s next?
  • 45. Tying it all together using docker-compose (1/2) version: "3.3" services: elasticsearch: image: docker.elastic.co/elasticsearch/elasticsearch:6.4.1 volumes: - ./elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml - ./jvm.options:/usr/share/elasticsearch/config/jvm.options - ./data:/usr/share/elasticsearch/data - ./snapshots:/tmp/es6_dev_repo ports: - "9200:9200" - "9300:9300" environment: ES_JAVA_OPTS: "-Xmx2g -Xms2g" networks: - elk ... ... kibana: image: docker.elastic.co/kibana/kibana:6.4.1 volumes: - ./kibana/config/:/usr/share/kibana/config - ./kibana/cache/:/usr/share/kibana/optimize ports: - "5601:5601" networks: - elk depends_on: - elasticsearch ...
  • 46. Tying it all together using docker-compose (2/2) ... logstash: image: docker.elastic.co/logstash/logstash:6.4.1 volumes: - ./logstash/config/:/usr/share/logstash/config/ ports: - "9600:9600" networks: - elk depends_on: - elasticsearch networks: elk: driver: bridge $ docker-compose up
  • 47. What are we talking about?
  • 48. Tying it all together https://www.elastic.co/blog/a-full-stack-in-one-command https://github.com/elastic/examples/tree/master/Miscellaneous/docker/full_stack_example
  • 49. Agenda ● Getting acquainted ● Ecosystem ● Features overview ● What are we talking about? ● Installation ● ES on Docker ● Kibana on Docker ● Logstash on Docker ● Tying it all together ● What’s next?
  • 50. ● Docs: https://www.elastic.co/guide/index.html ● Videos: https://www.elastic.co/videos ● Slides: https://speakerdeck.com/elastic/ ● Blog: https://www.elastic.co/blog ● Source: https://github.com/elastic ● Meetups: https://www.meetup.com/fr-FR/elastic-switzerland/ ● Conference: https://www.elastic.co/elasticon/conf/2018/sf ● Discuss: ■ https://stackoverflow.com/questions/tagged/elasticsearch ■ https://discuss.elastic.co/ ● Articles: ■ https://www.elastic.co/blog/a-full-stack-in-one-command ■ https://www.elastic.co/blog/docker-networking ■ https://sematext.com/blog/elasticsearch-in-docker/ ■ https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html ■ https://www.elastic.co/guide/en/kibana/current/docker.html ■ https://www.elastic.co/guide/en/logstash/current/docker.html What’s next?
  • 51. Q&A