(Elastic)search in big data
9 likes3,857 views
This document discusses search in big data and how Elasticsearch provides a solution. It addresses the challenges of fancy search features requiring distributed architecture to process large volumes of data across multiple servers. Elasticsearch implements a distributed search engine that allows real-time analytics on large, document-oriented data through its use of Lucene, JSON over HTTP, and sharding of data and queries across multiple nodes.
(Elastic)search in big data
- 1. (Elastic)search in Big Data Radu Gheorghe @radu0gheorghe @sematext
- 2. What is “search in Big Data”? Challenges? Some solutions? How does Elasticsearch do it? Agenda
- 4. headphones for iPhone 4, iPhone 5, iPhone 6 and iPhone 7 iPhone 5 iPhone 4 Relevancy... iphone
- 5. iphone iphone 5 Institute of Public Health ...and autocomplete... iph
- 6. No results found for “iphnoe” iPhone 5 iPhone 4 … and fuzziness... iphnoe
- 7. Did you mean “iPhone”? iPhone 5 iPhone 4 ...and corrections... iphnoe shows results anyway
- 8. iPhone 5 iPhone 4 iPhone 3 Galaxy S4 ...and similar terms... iphone
- 9. iPhone 5 iPhone 4 ...and don’t forget the statistics! iphone ☑ iOS ☐ other ☑ <100RON ☐ 100-200RON ☐ >200RON
- 10. Wait. Fancy search == Big Data?
- 11. Fancy stuff isn’t free iphone ☑ iOS ☐ other ☑ <100RON ☐ 100-200RON ☐ >200RON N requests for autocomplete Did you mean... iPhone 5 iPhone 4 iPhone 3 Galaxy S4 1 request for each of the stats 1 request for synonyms, 1 for exact matches, etc 1 request for corrections
- 12. Distributed search. When one server doesn’t cut it
- 13. Log Search web_server01 database01 backend01 search engine 10:01 - webapp - DB connect error 10:00 - DB - I/O error error
- 14. Log Analytics unique IPs: 7584 iPhone 5 iPhone 4 Galaxy S4 best sellers Romania: 200 France: 150 Hungary: 120 users per country revenue per day
- 15. Distributed search solutions Elasticsearch Solr Others: SenseiDB, Sphinx… SaaS: CloudSearch, Logsene... built on top of Lucene
- 16. Document-oriented Lucene awesome: index & store data, relevancy, fuzzy, suggesters... ...all wrapped up in JSON over HTTP Elasticsearch
- 17. Aggregations revenue per day unique IPs: 7584
- 18. Aggregations revenue per day unique IPs: 7584 Romania: 200 France: 150 Hungary: 120 unique IPs per country
- 19. Aggregations revenue per day Romania: 200 France: 150 Hungary: 120 unique IPs per country unique IPs per country per day Romania unique IPs: 7584
- 20. Node 1
- 21. Node 1
- 22. Node 1 Node 2
- 23. Node 1 Node 2
- 24. Node 1 Node 2 Node 3
- 25. Node 1 Node 2 Node 3
- 26. Node 1 Node 2 Node 3
- 27. Node 1 Node 2
- 28. Node 1 Node 2
- 29. Big Data distributed search search and real- time analytics
- 30. Big Data distributed search search and real- time analytics more search features
- 31. Big Data distributed search search and real- time analytics more search features clients usage (logs)
- 32. Thank you! radu.gheorghe@sematext.com @radu0gheorghe @sematext Big Data distributed search search and real- time analytics more search features clients usage (logs)