Finding the right stuff, an intro to Elasticsearch with Ruby/Rails

clustered and sharded
document storage with powerful
language analysing features
and a query language, 
all wrapped by a REST API

Getting Started
• install elasticsearch
• needs some JDK
• start it

Getting Started
• https://github.com/elastic/elasticsearch-rails
• gems for Rails:
• elasticsearch-model & elasticsearch-rails
• without Rails / AR:
• elasticsearch-persistence

class Event < ActiveRecord::Base
include Elasticsearch::Model

def as_indexed_json(options={})
{ title: title,
description: description,
starts_at: starts_at.iso8601 }
end

Event.import
PUT /events/event/31710
{
"title": "Finding the right stuff, ...",
"description": "Searching in data sets with ...",
"starts_at": “2015-10-08T19:00:00+09:00"
}

Event.import
{
"starts_at": “2015-10-08T19:00:00+09:00"
}
index

Event.import
{
"starts_at": “2015-10-08T19:00:00+09:00"
}
index
type

Event.import
{
"starts_at": “2015-10-08T19:00:00+09:00"
}
index
type
ID

response = Event.search 'tokyo rubyist'
response.took
# => 28
response.results.total
# => 2075
response.results.first._score
# => 0.921177
response.results.first._source.title
# => "Drop in Ruby"

response.took
# => 28
response.results.total
# => 2075
response.results.first._score
# => 0.921177
response.results.first._source.title
# => "Drop in Ruby"
GET /events/event/_search?q=tokyo%20rubyist

response.records.to_a
# => [#<Event id: 12409, ...>, ...]
response.page(2).results
response.page(2).records

response.records.to_a
# => [#<Event id: 12409, ...>, ...]
response.page(2).results
response.page(2).records
supports kaminari /
will_paginate

response.records.each_with_hit do |rec,hit|
puts "* #{rec.title}: #{hit._score}"
end
# * Drop in Ruby: 0.9205564
# * Javascript meets Ruby in Kamakura: 0.8947
# * Meetup at EC Navi: 0.8766844
# * Pair Programming Session #3: 0.8603562
# * Kickoff Party: 0.8265461
# * Tales of a Ruby Committer: 0.74487066
# * One Year Anniversary Party: 0.7298212

Event.search 'tokyo rubyist'
only upcoming events?

only upcoming events?
sorted by start date?

Event.search query: {
filtered: {
query: {
simple_query_string: {
query: "tokyo rubyist",
default_operator: "and"
}
},
filter: {
and: [
{ range: { starts_at: { gte: Time.now } } },
{ term: { featured: true } }
]
}
}
}, sort: { starts_at: { order: "asc" } }

filtered: {
query: {
}
},
filter: {
and: [
]
}
}
basically same as
before

filtered: {
query: {
}
},
filter: {
and: [
]
}
}
basically same as
before
ﬁltered by conditions

filtered: {
query: {
}
},
filter: {
and: [
]
}
}
basically same as
before
ﬁltered by conditions
sorted by start time

Query DSL
• query: { <query_type>: <arguments> }
• valid arguments depend on query type
• "Filtered Query" takes a query and a ﬁlter
• "Simple Query String Query" does not allow
nested queries

Query DSL
• filter: { <filter_type>: <arguments> }
• valid arguments depend on filter type
• "And filter" takes an array of filters
• "Range filter" takes a property and lt(e), gt(e)
• "Term filter" takes a property and a value

Match Query
Multi Match Query
Bool Query
Boosting Query
Common Terms Query
Constant Score Query
Dis Max Query
Filtered Query
Fuzzy Like This Query
Fuzzy Like This Field Query
Function Score Query
Fuzzy Query
GeoShape Query
Has Child Query
Has Parent Query
Ids Query
Indices Query
Match All Query
More Like This Query
Nested Query
Preﬁx Query
Query String Query
Simple Query String Query
Range Query
Regexp Query
Span First Query
Span Multi Term Query
Span Near Query
Span Not Query
Span Or Query
Span Term Query
Term Query
Terms Query
Top Children Query
Wildcard Query
Minimum Should Match
Multi Term Query Rewrite
Template Query

And Filter
Bool Filter
Exists Filter
Geo Bounding Box Filter
Geo Distance Filter
Geo Distance Range Filter
Geo Polygon Filter
GeoShape Filter
Geohash Cell Filter
Has Child Filter
Has Parent Filter
Ids Filter
Indices Filter
Limit Filter
Match All Filter
Missing Filter
Nested Filter
Not Filter
Or Filter
Preﬁx Filter
Query Filter
Range Filter
Regexp Filter
Script Filter
Term Filter
Terms Filter
Type Filter

{ title: title,
description: description,
starts_at: starts_at.iso8601,
featured: group.featured? }
end
settings do
mapping dynamic: 'false' do
indexes :title, type: 'string'
indexes :description, type: 'string'
indexes :starts_at, type: 'date'
indexes :featured, type: 'boolean'
end
end

Event.import force: true
deletes existing index,
creates new index with settings,
imports documents

bool: {
should: [
{
}
}, {
function_score: {
filter: {
and: [
{ range: { starts_at: { lte: 'now' } } },
]
},
gauss: {
starts_at: {
origin: 'now',
scale: '10d',
decay: 0.5
},
},
boost_mode: "sum"
}
}
],
minimum_should_match: 2
}
}

Dealing with different
languages
built in analysers for arabic, armenian, basque,
brazilian, bulgarian, catalan, cjk, czech, danish,
dutch, english, ﬁnnish, french, galician, german,
greek, hindi, hungarian, indonesian, irish, italian,
latvian, norwegian, persian, portuguese, romanian,
russian, sorani, spanish, swedish, turkish, thai.

Japanese?
• install kuromoji plugin
• https://github.com/elastic/elasticsearch-
analysis-kuromoji
• plugin install elasticsearch/elasticsearch-analysis-kuromoji/2.7.0

{ title: { en: title_en, ja: title_ja },
description: { en: description_en, ja: description_ja },
end
settings do
indexes :title do
indexes :en, type: 'string', analyzer: 'english'
indexes :ja, type: 'string', analyzer: 'kuromoji'
end
indexes :description do
end
end
end

with data from other
models?

{ title: { en: title_en, ja: title_ja },
description: { en: description_en, ja: description_ja },
group_name: { en: group.name_en, ja: group.name_ja },
end
settings do
indexes :title do
end
indexes :description do
end
indexes :group_name do
end
end
end

index_name "drkpr_#{Rails.env}_events"
Index names with
environment

Test Helpers
• https://gist.github.com/mreinsch/094dc9cf63362314cef4
• Helpers:  
wait_for_elasticsearch 
wait_for_elasticsearch_removal 
clear_elasticsearch!
• specs: Tag tests which require elasticsearch

Production Ready?
• use elastic.co/found or AWS ES
• use two instances for redundancy
• elasticsearch could go away
• usually only impacts search
• keep impact at a minimum

after_save do
IndexerJob.perform_later(
'update', self.class.name, self.id)
end
after_destroy do
IndexerJob.perform_later(
'delete', self.class.name, self.id)
end
...

class IndexerJob < ActiveJob::Base
queue_as :default
def perform(action, record_type, record_id)
record_class = record_type.constantize
record_data = {
index: record_class.index_name,
type: record_class.document_type,
id: record_id
}
client = record_class.__elasticsearch__.client
case action.to_s
when 'update'
record = record_class.find(record_id)
client.index record_data.merge(body: record.as_indexed_json)
when 'delete'
client.delete record_data.merge(ignore: 404)
end
end
end
https://gist.github.com/mreinsch/acb2f6c58891e5cd4f13

Questions?
Elastic Docs 
https://www.elastic.co/guide/index.html
Ruby Gem Docs 
https://github.com/elastic/elasticsearch-rails
Resources
or ask me later:
michael@doorkeeper.jp
@mreinsch

Finding the right stuff, an intro to Elasticsearch with Ruby/Rails

More Related Content

What's hot (20)

Viewers also liked (18)

Similar to Finding the right stuff, an intro to Elasticsearch with Ruby/Rails (20)

Recently uploaded (20)

Finding the right stuff, an intro to Elasticsearch with Ruby/Rails