Scylla Summit 2017: From Elasticsearch to Scylla at Zenly

PRESENTATION TITLE ON ONE LINE
AND ON TWO LINES
First and last name
Position, company
From Elasticsearch To Scylla
at Zenly
10 Month in production
Head of infrastructure, Zenly - https://zen.ly
Jean-Baptiste Dalido

AND ON TWO LINES
First and last name
Position, company
Jean-Baptiste Dalido
2
● Zenly’s Head of infrastructure, now a Snap Inc.
Company
● Experience with Mobile Startups, previously
Appgratis and Batch.com
● Worked with Cassandra in production for quite
some time

AND ON TWO LINES
First and last name
Position, company

AND ON TWO LINES
First and last name
Position, company
FROM ELASTICSEARCH

AND ON TWO LINES
First and last name
Position, company
X Use cases for X Databases
5
▪ Elasticsearch, as a main database
▪ Redis, for cache
▪ Cassandra, for gigantic datasets (chat messages)
▪ Postgres, for results of heavy computations
▪ Name it, it was in production ...

AND ON TWO LINES
First and last name
Position, company
ElasticSearch as a main database ?
6
▪ Easy data-modelling
▪ Search as a query language
o offsets data-model changes and deficiencies
o comes at a cost, search is not cheap
▪ KV Semantics
▪ Sharding/Replication
▪ Document versioning

AND ON TWO LINES
First and last name
Position, company
Why leaving ?
7
▪ Data-model became more and more stable
▪ Particular Workload, Heavy updates
▪ Unification of databases
o Less operational cost (onboarding, maintenance…)
o Simpler monitoring
▪ No real need for ElasticSearch features anymore

AND ON TWO LINES
First and last name
Position, company
WAIT, WHY NOT
STICK WITH
CASSANDRA ?

AND ON TWO LINES
First and last name
Position, company
Total time for which application threads were stopped:
(too many) seconds
AKA: Stop the world
9

AND ON TWO LINES
First and last name
Position, company
Cassandra ?
10
▪ Garbage Collection is a deal breaker for small clusters
▪ Overcome by a high number of nodes
▪ Proficiency requires knowledge and experience
▪ Cassandra without GC might be a close-to-perfect database

AND ON TWO LINES
First and last name
Position, company
MOVING TO SCYLLA
(after days of debate, and no GC)

AND ON TWO LINES
First and last name
Position, company
WE WERE TOLD NOT TO
USE DOCKER FOR
PRODUCTION
(but we did it anyway)

AND ON TWO LINES
First and last name
Position, company
First migration, cluster in Kubernetes
13
▪ Easily deploy/update within our infrastructure
o 12 Kubernetes clusters in production
▪ Except Kafka, we have everything in Docker
▪ Scale within minutes
▪ Networking, logs, everything is included

AND ON TWO LINES
First and last name
Position, company
First migration, cluster in Kubernetes
14
▪ Unexpected High latencies and load was over the roof
▪ ScyllaDB’s team to the rescue
▪ It’s all about Seastar internals

AND ON TWO LINES
First and last name
Position, company
Seastar and Kubernetes
15
▪ Seastar is sharding, one shard, one core
▪ Networking and data are sharded per core
▪ Pinned CPU in Kubernetes was not available
o keep an eye on Cpu-Manager

AND ON TWO LINES
First and last name
Position, company
Moving, part 2

AND ON TWO LINES
First and last name
Position, company
New infrastructure
17
▪ 3 clusters
▪ 2 single region clusters
▪ 1 double region cluster
o Redundancy
o Read-Heavy workload without impact on production
▪ 7 machines per cluster/region
o 10 cpus
o 36Gb of RAM
o x2 - 375GB local SSD
▪ No Docker

AND ON TWO LINES
First and last name
Position, company
GCP Deployment
18
▪ Ubuntu 16.04 LTS
▪ NVMe
o Cascade failure on GCP machines with kernel 4.8.0
o Unexpected Chaos Monkey
▪ Resiliency theory proved to be right
▪ Failure did not provoke any downtime

AND ON TWO LINES
First and last name
Position, company
Clusters as feature isolation
19
▪ More nodes
▪ Better overview of workload
o Feature teams can scale their needs
o Developers can improve performances and see impact
o Offer data teams a way to query production clusters
▪ Resiliency if implemented the right way

AND ON TWO LINES
First and last name
Position, company
Uptime
20
▪ 100% uptime for our 3 clusters
o 0 cases of major failure due to ScyllaDB
▪ Few crashes that were always machine related
▪ No impact on production

AND ON TWO LINES
First and last name
Position, company
Everything is about
performance

AND ON TWO LINES
First and last name
Position, company
300k requests per second on 7 nodes
22

AND ON TWO LINES
First and last name
Position, company
Latencies distribution over the same traffic
23
▪ p95 in under 1ms, provided by Google Cloud Platform

AND ON TWO LINES
First and last name
Position, company
Scylla is production ready
24
▪ It’s fast, so fast
o Make mistakes
o Experiments
▪ You don’t need to be a Cassandra ninja
▪ It’s a supported product (Scylla 2 is out)
▪ Keep an eye on Kubernetes

AND ON TWO LINES
First and last name
Position, company
Go behind usual use-cases
25
▪ Using Scylla as a cache instead of Redis ?
▪ I/O scheduler
▪ Memstore
▪ Cache
▪ Persistence beyond the Memory
▪ Amazing performances
▪ Do you still need a cache ?

AND ON TWO LINES
First and last name
Position, company
We don’t :)

AND ON TWO LINES
First and last name
Position, company
THANK YOU
jb@zen.ly
@jbaptistedalido
Please stay in touch

Scylla Summit 2017: From Elasticsearch to Scylla at Zenly

More Related Content

What's hot (20)

Similar to Scylla Summit 2017: From Elasticsearch to Scylla at Zenly (16)

More from ScyllaDB (20)

Recently uploaded (20)

Scylla Summit 2017: From Elasticsearch to Scylla at Zenly