Scaling massive elastic search clusters - Rafał Kuć - Sematext

Scaling Massive ElasticSearch
Clusters

Rafał Kuć – Sematext International
@kucrafal @sematext sematext.com

Who Am I
• „Solr 3.1 Cookbook” author
• Sematext software engineer
• Solr.pl co-founder
• Father and husband :-)

Copyright 2012 Sematext Int’l. All rights reserved

What Will I Talk About ?
• ElasticSearch scaling
• Indexing thousands of documents per second
• Performing queries in tens of milliseconds
• Controling shard and replica placement
• Handling multilingual content
• Performance testing
• Cluster monitoring


The Challenge
• More than 50 millions of documents a day
• Real time search
• Less than 200ms average query latency
• Throughput of at least 1000 QPS
• Multilingual indexing
• Multilingual querying


Why ElasticSearch ?
• Written with NRT and cloud support in mind
• Uses Lucene and all its goodness
• Distributed indexing with document
distribution control out of the box
• Easy index, shard and replicas creation on live
cluster


Index Design
• Several indices (at least one index for each day
of data)
• Indices divided into multiple shards
• Multiple replicas of a single shard
• Real-time, synchronous replication
• Near-real-time index refresh (1 to 30 seconds)


Shard Deployment Problems
• Multiple shards per node
• Replicas on the same nodes as shards
• Not evenly distributed shards and replicas
• Some nodes being hot, while others are cold


Default Shard Deployment

Shard 1 Shard 2 Shard 3 Replica 1

Replica 2
Node 1 Node 2

Replica 3

Node 3
ElasticSearch Cluster


What Can We Do With Shards Then ?
• Contol shard placement with node tags:
– index.routing.allocation.include.tag
– index.routing.allocation.exclude.tag
• Control shard placement with nodes IP
addresses:
– cluster.routing.allocation.include._ip
– cluster.routing.allocation.exclude._ip
• Specified on index or cluster level
• Can be changed on live cluster !

Shard Allocation Examples
• Cluster level:
curl -XPUT localhost:9200/_cluster/settings -d '{
"persistent" : {
"cluster.routing.allocation.exclude._ip" : "192.168.2.1"
}
}'
• Index level:
curl -XPUT localhost:9200/sematext/ -d '{
"index.routing.allocation.include.tag" : "nodeOne,nodeTwo"
}'


Number of Shards Per Node
• Allows one to specify number of shards per
node
• Specified on index level
• Can be changed on live indices
• Example:
curl -XPUT localhost:9200/sematext -d '{
"index.routing.allocation.total_shards_per_node" : 2
}'


Controlled Shard Deployment

Shard 1 Replica 2 Shard 3 Replica 1

Node 1 Node 2

Shard 2 Replica 3

Node 3


Does Routing Matters ?
• Controls target shard for each document
• Defaults to hash of a document identifier
• Can be specified explicitly (routing parameter) or
as a field value (a bit less performant)
• Can take any value
• Example:
curl -XPUT localhost:9200/sematext/test/1?routing=1234 -d '{
"title" : "Test routing document"
}'


Indexing the Data

Shard Replica Shard Replica
1 2 3 1

Node 1 Node 2

Shard Replica
2 3

Node 3

Indexing application

How We Indexed Data

Shard 1 Shard 2

Node 1 Node 2

Node 3


Indexing application


Nodes Without Data
• Nodes used only to route data and queries to
other nodes in the cluster
• Such nodes don’t suffer from I/O waits (of
course Data Nodes don’t suffer from I/O waits
all the time)
• Not default ElasticSearch behavior
• Setup by setting node.data to false


Multilingual Indexing
• Detection of document's language before
sending it for indexing
• With, e.g. Sematext LangID or Apache Tika
• Set known language analyzers in configuration
or mappings
• Set analyzer during indexing (_analyzer field)


Multilingual Indexing Example
{
"test" : {
"_analyzer" : { "path" : "langId" },
"properties" : {
"id" : { "type" : "long", "store" : "yes", "precision_step" : "0" },
"title" : { "type" : "string", "store" : "yes", "index" : "analyzed" },
"langId" : { "type" : "string", "store" : "yes", "index" : "not_analyzed" }
}
}
}

curl -XPUT localhost:9200/sematext/test/10 -d '{
"title" : "Test document",
"langId" : "english"
}'


Multilingual Queries
• Identify language of query before its execution
(can be problematic)
• Query analyzer can be specified per query
(analyzer parameter):
curl -XGET
localhost:9200/sematext/_search?q=let+AND+me&analyzer=english


Query Performance Factors – Lucene
level
• Refresh interval
– Defaults to 1 second
– Can be specified on cluster or index level
– curl -XPUT localhost:9200/_settings -d '{ "index" : {
"refresh_interval" : "600s" } }'
• Merge factor
– Defaults to 10
– Can be specified on cluster or index level
– curl -XPUT localhost:9200/_settings -d '{ "index" : {
"merge.policy.merge_factor" : 30 } }'


Let’s Talk About Routing Once Again
• Routes a query to a particular shard
• Speeds up queries depending on number of
shards for a given index
• Have to be specified manualy with routing
parameter during query
• routing parameter can take any value:

curl -XGET
'localhost:9200/sematext/_search?q=test&routing=2012-02-16'


Querying ElasticSearch – No Routing

Shard 1 Shard 2 Shard 3 Shard 4


ElasticSearch Index

Application


Querying ElasticSearch – With Routing



ElasticSearch Index

Application


Performance Numbers
Queries without routing (200 shards, 1 replica)
#threads Avg response time Throughput 90% line Median CPU Utilization

1 3169ms 19,0/min 5214ms 2692ms 95 – 99%

Queries with routing (200 shards, 1 replica)
#threads Avg response time Throughput 90% line Median CPU Utilization

10 196ms 50,6/sec 642ms 29ms 25 – 40%
20 218ms 91,2/sec 718ms 11ms 10 – 15%


Scaling Query Throughput – What Else ?

• Increasing the number of shards for data
distribution
• Increasing the number of replicas
• Using routing
• Avoid always hitting the same node and
hotspotting it


FieldCache and OutOfMemory
• ElasticSearch default setup doesn’t limit field
data cache size


FieldCache – What We Can do With It ?
• Keep its default type and set:
– Maximum size (index.cache.field.max_size)
– Expiration time (index.cache.field.expire)
• Change its type:
– soft (index.cache.field.type)
• Change your data:
– Make your fields less precise (ie: dates)
– If you sort or facet on fields think if you can reduce
fields granularity
• Buy more servers :-)


FieldCache After Changes


Additional Problems We Encountered
• Rebalancing after full cluster restarts
– cluster.routing.allocation.disable_allocation
– cluster.routing.allocation.disable_replica_allocation
• Long startup and initialization
• Faceting with strings vs faceting on numbers on
high cardinality fields


JVM Optimization
• Remember to leave enough memory to OS for
cache
• Make GC frequent ans short vs. rare and long
– -XX:+UseParNewGC
– -XX:+UseConcMarkSweepGC
– -XX:+CMSParallelRemarkEnabled
• -XX:+AlwaysPreTouch (for short performance
tests)


Performance Testing
• Data
– How much data do I need ?
– Choosing the right queries
• Make changes
– One change at a time
– Understand the impact of the change
• Monitor your cluster (jstat, dstat/vmstat,
SPM)
• Analyze your results

ElasticSearch Cluster Monitoring
• Cluster health
• Indexing statistics
• Query rate
• JVM memory and garbage collector work
• Cache usage
• Node memory and CPU usage


Cluster Health

Node restart


Indexing Statistics


Query Rate


JVM Memory and GC


Cache Usage


CPU and Memory


Summary
• Controlling shard and replica placement
• Indexing and querying multilingual data
• How to use sharding and routing and not to
tear your hair out
• How to test your cluster performance to find
bottle-necks
• How to monitor your cluster and find
problems right away

We Are Hiring !
• Dig Search ?
• Dig Analytics ?
• Dig Big Data ?
• Dig Performance ?
• Dig working with and in open – source ?
• We’re hiring world – wide !
http://sematext.com/about/jobs.html


How to Reach Us
• Rafał Kuć
– Twitter: @kucrafal
– E-mail: rafal.kuc@sematext.com
• Sematext
– Twitter: @sematext
– Website: http://sematext.com
• Graphs used in the presentation are from:
– SPM for ElasticSearch (http://sematext.com/spm)


Scaling massive elastic search clusters - Rafał Kuć - Sematext

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Scaling massive elastic search clusters - Rafał Kuć - Sematext (20)

Recently uploaded (20)

Scaling massive elastic search clusters - Rafał Kuć - Sematext