Elasticsearch Basics
17 likes4,733 views
This document provides an overview of Elasticsearch including: - Elasticsearch is a database server that is implemented using RESTful HTTP/JSON and is easily scalable. It is based on Lucene. - Features include being schema-free, real-time, easy to extend with plugins, automatic peer discovery in clusters, failover and replication, and community support. - Terminology includes index, type, document, and field which make up the data structure inside Elasticsearch. Searches can be performed across multiple indices. - Elasticsearch works using full-text searching via inverted indexing and analysis. Analysis extracts terms from text through techniques like removing stopwords, lowercase conversion, and stemming. - Elasticsearch can be accessed in a RESTful manner
![FULL TEXT SEARCH
stars shine
sky
stars stars
sky
cloudy
sky
[0] [1] [2]](https://image.slidesharecdn.com/elasticsearch-131005153216-phpapp01/85/Elasticsearch-Basics-15-320.jpg)
Elasticsearch Basics
- 2. Why do we need search options?
- 3. We need it to make our lives easier Find stuff that is relevant to us Find it faster
- 4. How do we add seach features to our projects? Options available: ● Standard Database Query ● Lucene libraries ● Solr ● Sphinx ● Elasticsearch
- 6. Topics for today! Full Text Searching RESTful ES Elasticsearch in Rails Advanced features of ES Note: ES = ElasticSearch
- 7. What is elasticsearch? ●Database server ●Implemented with RESTful HTTP/JSON ●Easily scalable (hence the name elasticsearch) ●Based on Lucene
- 8. Features of elasticsearch ●Schema-free ●Real-time ●Easy to extend with a plugin system for new functionality ●Automatic discovery of peers in a cluster ●Failover and replication ●Community support: Multiple clients available in various languages for easy integration Tire gem- ruby client available for ActiveModel integration
- 11. An Index can have multiple types. Each type can have multiple documents. Each document contains data in JSON format DOC-TYPE INDEX DOC-TYPE {..} {..} Document
- 12. INDEX DOC-TYPE DOC-TYPE Searches can also be performed in multiple indices
- 13. How does elasticsearch work?
- 14. Full text searching ●Inverted indexing ●Analysis
- 15. FULL TEXT SEARCH stars shine sky stars stars sky cloudy sky [0] [1] [2]
- 16. FULL TEXT SEARCH stars shine sky stars stars sky cloudy sky WORD LOCATION stars 0,1 shine 0 sky 0,1,2 cloudy 2
- 17. FULL TEXT SEARCH stars shine sky stars stars sky cloudy sky WORD LOCATION stars 0,1 shine 0 sky 0,1,2 cloudy 2
- 18. FULL TEXT SEARCH stars shine sky stars stars sky cloudy sky WORD LOCATION POSITION stars 0 0 1 0,1 shine 0 1 sky 0 2 1 2 2 1 cloudy 2 0
- 19. FULL TEXT SEARCH stars shine sky stars stars sky cloudy sky WORD LOCATION POSITION stars 0 0 1 0,1 shine 0 1 sky 0 2 1 2 2 1 cloudy 2 0 Are these positions in the document are consecutive ?
- 20. FULL TEXT SEARCH stars shine sky stars stars sky cloudy sky WORD LOCATION POSITION stars 0 0 1 0,1 shine 0 1 sky 0 2 1 2 2 1 cloudy 2 0 We find the words in consecutive positions in Document 1
- 21. ANALYSIS Analyzing is extracting “terms” from given text. Processing natural language to make it computer searchable.
- 22. ruby dynamic reflective general purpose oop language combine syntax inspire perl smalltalk feature first design develop mid 1990 yukihiro matz matsumoto japan Ruby is a dynamic, reflective, general-purpose OOP language that combines syntax inspired by Perl with Smalltalk-like features. Ruby was first designed and developed in the mid-1990s by Yukihiro "Matz" Matsumoto in Japan. Stopwords:-removal of words of less semantic significance
- 23. ruby dynamic reflective general purpose oop language combine syntax inspire perl smalltalk feature first design develop mid 1990 yukihiro matz matsumoto japan Ruby is a dynamic, reflective, general-purpose OOP language that combines syntax inspired by Perl with Smalltalk-like features. Ruby was first designed and developed in the mid-1990s by Yukihiro "Matz" Matsumoto in Japan. Lowercase and punctuation marks
- 24. ruby dynamic reflective general purpose oop language combine syntax inspire perl smalltalk feature first design develop mid 1990 yukihiro matz matsumoto japan Ruby is a dynamic, reflective, general-purpose OOP language that combines syntax inspired by Perl with Smalltalk-like features. Ruby was first designed and developed in the mid-1990s by Yukihiro "Matz" Matsumoto in Japan. Stemmer:-Deriving root of words
- 25. ES Analyzer ●Analyzer: Consists of one tokenizer and multiple token filters eg: Whitespace, Snowball,etc ●Tokenizer: It tokenizes all words. Splits sentences into individual 'terms' . Ngram and EdgeNgram highly useful for autocomplete feature. Path hierarchy tokenizers. ●Token filter: Actions on tokenized words, basic lowercase to phonetic filters and stemmers (available in many languages)
- 26. Popular Analyzers ES is easy to use. Readymade analyzers for general usage. ● Snowball – excellent for natural language Standard tokenizer, with standard filter, lowercase filter, stop filter, and snowball filter ● Some fancy analyzers: Pattern analyzers For all the Regular expressions guys out there! "type": "pattern", "pattern":"s+"
- 27. How do we access Elasticsearch?
- 28. How do we access Elasticsearch? Its RESTful
- 29. How do we access Elasticsearch? Its RESTful GET POST PUT DELETE
- 37. curl -XPUT 'localhost:9200/twitter_development/tweet/1' -d '{ "tweet" : " Elasticsearch is cool! ", "name" : "Mr Developer" }'
- 38. curl -XPUT 'localhost:9200/twitter_development/tweet/1' -d '{ "tweet" : " Elasticsearch is cool! ", "name" : "Mr Developer" }' { "_index":"twitter_development", "_type":"tweet", "_id":"1", "_version": 1, "ok":true }
- 40. Similarly, { "_index" : "twitter", "_type" : "tweet", "_id" : "1", "_source" : { "tweet" : " Elasticsearch is cool! ", "name" : "Mr Developer" } } curl -XGET 'http://localhost:9200/twitter/tweet/1'
- 41. GET /index/_search GET /index1,index2/_search GET /ind*/_search GET /index/type/_search GET /index/type1,type2/_search GET /index/type*/_search GET /_all/type*/_search Search multiple or all indices (databases) Search all types (columns) Also
- 42. Similarly, DELETE curl -XDELETE 'http://localhost:9200/twitter/tweet/1' curl -XDELETE 'http://localhost:9200/twitter/' curl -XDELETE 'http://localhost:9200/twitter/tweet/_query'-d '{ "term" : { "name" : "developer" } }' Delete document Delete index
- 44. ES in your RAILS app 1.Install ES from website. Start service. 2.In your browser check 'http://localhost:9200' for confirmation. 3.Add 'tire' to Gemfile. 4.In your model file (say 'chwink.rb' ) add- include Tire::Model::Search include Tire::Model::Callbacks 5.Run rake environment tire:import CLASS=Chwink 6.In rails console type Chwink.search(“world”) or Chwink.tire.search(“world”) 7.Results!
- 45. Request for Model.search curl -XGET 'http://localhost:9200/chwink_development/chwink/_search? -d '{ "query": { "query_string": { "query": "world" } } }'
- 46. Autocomplete feature implementation and Some examples of readymade and custom analyzers
- 47. Querying More types of queries and use cases: ●Faceting : Allowing multiple filters ●Pagination : Limiting per page results ●Sort : Sorting results by relevance and scoring (using Elasticsearch's scoring algorithm)
- 48. Advanced features of Elasticsearch
- 49. USP of Elasticsearch BIG! Data Analysis in Real Time
- 50. Automatic discovery of peers in a cluster Failover and replication
- 52. A1 B1 C1 A2 A3 B2 B2 B1 A3 A1 A2 B2 A2 A3 A2 B1 A1 B2 Automatic Discovery Module Node 1 Node 2 Node 3 Shard Replica
- 53. A1 B2 B1 A2 C1 B2 A2 A3 B1 A1 B2 A3 B1 A3 A2 B1 B2 A1 Node 1 Node 2 Node 3 Node 4 Automatic Discovery Module When a new node is added...
- 54. For each index you can specify: ● Number of shards – Each index has fixed number of shards – Shards improve indexing performance ● Number of replicas – Each shard can have 0-many replicas, can be changed dynamically – Replicas improve search performance High Availability
- 55. Tips ●Make sure you make separate indices for test and development environment Eg: Add index_name "Chwink_#{Rails.env}" to your model file ●During tests whenever you save an object make sure you add : Chwink.tire.index.refresh after each test case. ●Make sure you delete and recreate the index after tests. Eg: You can add this to your Rspec configuration file config.after(:each) do Chwink.tire.Chwink_test.delete Chwink.tire.create_elasticsearch_index end ●Debug using log files : Add to Tire config file: logger "tire_#{Rails.env}.log"
- 56. Room for exploration... Big Desk plugin! Plug-in for analysis of your search data!
- 57. Room for exploration... Big Desk plugin! Plug-in for analysis of your search data!
- 58. Room for exploration... Big Desk plugin! Plug-in for analysis of your search data!
- 59. Now you can go ahead and start exploring Elasticsearch for yourself!!
- 60. Thank you! Questions are welcome!