Data Con LA 2018 - Big Data as a Service: Running Elasticsearch on Pure by Brian Gold
1 like574 views
The document discusses the use of Elasticsearch, focusing on its indexing capabilities for various types of data, including emails and web pages. It emphasizes the importance of proper infrastructure and configuration for optimal performance, particularly with flash storage solutions to achieve high write throughput. Key use cases highlighted include application search, business analytics, and security analytics, showcasing Elasticsearch's ability to provide rapid insights into large datasets.
![© 2018 PURE STORAGE INC.6
Search Engines: Indexes in the Digital Age
Tokenizing
Filtering
Stemming
Normalizing
Term Document: Locations
voided {‘doc1’: [221]}
toil {‘doc3’: [12]}
house {‘doc2’: [248]}
topics {‘doc1’: [23, 206, 342]}
mandate {‘doc3’: [143]}
edition {‘doc2’: [178]}
job {‘doc1’: [282]}
week {‘doc1’: [22], ‘doc2’: [84]}
buildings {‘doc3’: [832]}
Inverted Index
THE REST IS JUST ENGINEERING ☺](https://image.slidesharecdn.com/bddla-briangold-elastic-180911044130/85/Data-Con-LA-2018-Big-Data-as-a-Service-Running-Elasticsearch-on-Pure-by-Brian-Gold-6-320.jpg)
Data Con LA 2018 - Big Data as a Service: Running Elasticsearch on Pure by Brian Gold
- 1. © 2018 PURE STORAGE INC.1 #PUREACCELERATE #NEWMEETSNOW RUNNING ELASTICSEARCH ON PURE BIG DATA AS A SERVICE Brian Gold | Pure Storage @briantgold
- 2. WHEN, WHY, AND HOW SHOULD I USE ELASTICSEARCH?
- 3. © 2018 PURE STORAGE INC.3 Searching a Large Document Corpus Find the book containing “It was the best of times, it was the worst of times”
- 6. © 2018 PURE STORAGE INC.6 Search Engines: Indexes in the Digital Age Tokenizing Filtering Stemming Normalizing Term Document: Locations voided {‘doc1’: [221]} toil {‘doc3’: [12]} house {‘doc2’: [248]} topics {‘doc1’: [23, 206, 342]} mandate {‘doc3’: [143]} edition {‘doc2’: [178]} job {‘doc1’: [282]} week {‘doc1’: [22], ‘doc2’: [84]} buildings {‘doc3’: [832]} Inverted Index THE REST IS JUST ENGINEERING ☺
- 7. © 2018 PURE STORAGE INC.7 Data shippers (e.g., crawler) Library for preprocessing and indexing text Documents AS DATA VOLUMES GROW, WE MUST SCALE OUT Basic indexing workflow with Apache Lucene
- 8. © 2018 PURE STORAGE INC.8 Data shippers (e.g., crawler) Documents Scaling out with Elasticsearch cluster frontend Elasticsearch POWERING SEARCH BACKEND FOR MAJOR WEBSITES (WIKIPEDIA)
- 9. © 2018 PURE STORAGE INC.9 “Data shippers”“Documents” What’s a document? cluster frontend Elasticsearch Emails Web pages Records in a DBMS Syslog entries Security logs … USE ELASTICSEARCH TO INDEX ANY KIND OF TEXT Logstash Fluentd Apache Spark Filebeat Web crawlers …
- 10. © 2018 PURE STORAGE INC.10 Top use cases Application search Business analytics Log analytics Security analytics
- 11. © 2018 PURE STORAGE INC.11 Common theme: rapid insights into data
- 12. © 2018 PURE STORAGE INC.12 #PUREACCELERATE #NEWMEETSNOW INFRASTRUCTURE REQUIREMENTS AND BEST PRACTICES ELASTICSEARCH IN DEPTH
- 13. © 2018 PURE STORAGE INC.13 How does Elasticsearch store my data? Elasticsearch translog segments Persistent data structures Ephemeral process Inverted indexes & associated metadata Write ahead log
- 14. © 2018 PURE STORAGE INC.14 Elasticsearch makes scale-out simple Elasticsearch translog segments Elasticsearch translog segments Elasticsearch translog segments ElasticSearch cluster - data nodes
- 15. © 2018 PURE STORAGE INC.15 Mapping concepts to hardware implementation Storage (SSDs or disks) CPU & DRAM Network Interface Elasticsearch
- 16. © 2018 PURE STORAGE INC.16 Scale out with commodity servers Elasticsearch Elasticsearch Elasticsearch Cluster Interconnect Pros Excellent scalability Cons Deployment requires specific server configuration Hard to change compute-to-storage ratios
- 17. © 2018 PURE STORAGE INC.17 What if we disaggregate? Elasticsearch Elasticsearch Elasticsearch Cluster Interconnect
- 18. © 2018 PURE STORAGE INC.18 Disaggregation enables Elasticsearch-as-a-service ES ES ES Cluster Interconnect ES ES ESES Pros Run nodes anywhere, anyhow Scale storage and compute independently Cons Network bottlenecks? Storage bottlenecks?
- 19. © 2018 PURE STORAGE INC.19 Do not use remote-mounted storage... The latency introduced here is antithetical to performance. “ ”- Elastic.co – Indexing Performance Tips
- 20. © 2018 PURE STORAGE INC.20 CHALLENGE ACCEPTED
- 21. © 2018 PURE STORAGE INC.21 FLASHBLADEPURPOSE-BUILT FOR MODERN ANALYTICS BLADE PURITY SCALE-OUT FABRIC Powerful, Elastic Data Processing & Storage Unit Massively Distributed Software for Limitless Scale Software-defined fabric that scales linearly with more data & clients
- 22. © 2018 PURE STORAGE INC.22 segment segment segment segment segment segment translog merge refresh Incoming documents ① ② ③ Why should an all-flash array perform well with Elasticsearch? Indexing needs high write throughput at consistent, low-enough latency
- 23. © 2018 PURE STORAGE INC.23 20 Elasticsearch nodes • 40 vCPUs @ 2.6GHz • 192 GB DRAM • 256GB local SSD • 2x10GbE network • Elasticsearch 6.2.1 (2 per node in containers) 5 Rally load generators • 40 vCPUs @ 2.6GHz • 192 GB DRAM • 2x10GbE network • NYC Taxis dataset 8x replicated => 595GB raw FlashBlade • 15 blades, 17TB each • 162TB usable via NFS • 8x40GbE network Indexing benchmark methodology
- 24. © 2018 PURE STORAGE INC.24 Over 3M documents per second and as fast as local SSD 0 1 2 3 0 5 10 15 20 Number of nodes (2x Elasticsearch instances per node) Direct-attach SSD FlashBlade Indexing throughput Millions of documents per sec
- 25. © 2018 PURE STORAGE INC.25 High throughput at consistently low latency Real-time monitoring aids performance tuning and diagnostics
- 26. © 2018 PURE STORAGE INC.26 ① Don’t fear “remote-mounted storage”! ② Do extensive testing at scale => Elasticsearch needs high storage throughput at consistent, low-enough latency ③ Disaggregation enables service delivery and scaling models not possible with local drives. Lessons learned
- 27. © 2018 PURE STORAGE INC.27 THANK YOU! bgold@purestorage.com @briantgold
- 28. © 2017 PURE STORAGE INC.28