Amir Salihefendic: Redis - the hacker's database

The
Hacker’s
Database

Amir
Salihefendic
(amix)

About
Me

•  Co-‐founder
and
former
CTO
of
Plurk.com

•  Helped
Plurk
scale
to
millions
of
users,

billions
of
pages
views
and
8+
billion
unique

data
items.
With
minimal
hardware!

•  Founder
of
Doist.io

creators
of
Todoist
and
Wedoist

Outline
of
the
talk

•  Plurk
Timelines
opKmizaKon:
How
we
saved

hundreds
of
thousands
of
dollars

•  What’s
great
about
Redis?

•  Diﬀerent
sample
implementaKons:

–  redis_wrap

–  redis_graph

–  redis_queue

•  Advanced
analyKcs
using
Redis

–  bitmapist
and
bitmapist.cohort

Problem

ExponenKal
data
growth
in
Social
Networks

data size

number of users

The
Easy
Solu=on

Throw
money
at
the
problem

The
Smarter
Solu=on

Reduce
to
linear
data
growth

data size

number of users

Example:
Timelines

timeline
data size

number of users

Example:
Timelines

SoluKon:
Chea=ng!

Make
Kmelines
a
ﬁxed
size
-‐
500
messages

timeline •  O(1)
inserKon

data size •  O(1)
update

•  Cache
able

number of users

Plurk’s
=melines
migra=on
path

Tokyo
Tyrant

•  Problem
with
MySQL
and
Tokyo
Tyrant?

Death
by
IO

What’s
great
about
Redis?

• Everything
is
in
memory,

but
the
data
is
persistent.

• Amazing
performance:

100.000+
SETs
pr.
sec

80.000+
GETs
pr.
sec

Redis
Rich
Datatypes

•  Rela=onal
databases

Schemas,
tables,
columns,
rows,
indexes
etc.

•  Column
databases
(BigTable,
hBase
etc.)

Schemas,
columns,
column
families,
rows
etc.

•  Redis

key-‐value,
sets,
lists,
hashes,
bitmaps,
etc.

Redis
datatypes
resemble
datatypes

in
programming
languages.

They
are
natural
to
us!

redis_wrap

•  Implements
a
wrapper
for
Redis
datatypes
so

they
mimic
the
datatypes
found
in
Python

•  100
lines
of
code

•  h_ps://github.com/Doist/redis_wrap

redis_wrap

# Mimic of Python lists # Mimic of Python sets
bears = get_list('bears') fishes = get_set('fishes')
bears.append('grizzly') assert 'nemo' not in fishes

assert len(bears) == 1 fishes.add('nemo')
assert 'grizzly' in bears
assert 'nemo' in fishes

for item in fishes:
assert item == 'nemo'

# Mimic of hashes
villains = get_hash('villains')
assert 'riddler' not in villains

villains['riddler'] = 'Edward Nigma'
assert 'riddler' in villains
assert len(villains.keys()) == 1

del villains['riddler']
assert len(villains) == 0

redis_graph

•  Implements
a
simple
graph
database
in
Python

•  Can
scale
to
a
few
million
nodes
easily

•  You
could
use
something
similar
to
implement

LinkedIn’s
“who
is
connected
to
who”
feature

•  Under
40
lines
of
code

•  h_ps://github.com/Doist/redis_graph

redis_graph

# Adding an edge between nodes
add_edge(from_node='frodo', to_node='gandalf')
assert has_edge(from_node='frodo',
to_node='gandalf') == True

# Getting neighbors of a node
assert list(neighbors('frodo')) == ['gandalf']

# Deleting edges
delete_edge(from_node='frodo', to_node='gandalf')

# Setting node values
set_node_value('frodo', '1')
assert get_node_value('frodo') == '1'

# Setting edge values
set_edge_value('frodo_baggins', '2')
assert get_edge_value('frodo_baggins') == '2'

redis_graph:
The
implementaKon

from redis_wrap import *

#--- Edges ----------------------------------------------
def add_edge(from_node, to_node, system='default'):
edges = get_set( from_node, system=system )
edges.add( to_node )

def delete_edge(from_node, to_node, system='default'):
edges = get_set( from_node, system=system )

key_node_y = to_node
if key_node_y in edges:
edges.remove( key_node_y ) #--- Node values ----------------------------
def get_node_value(node_x, system='default'):
def has_edge(from_node, to_node, system='default'): node_key = 'nv:%s' % node_x
edges = get_set( from_node, system=system ) return get_redis(system).get( node_key )
return to_node in edges
def set_node_value(node_x, value, system='default'):
def neighbors(node_x, system='default'): node_key = 'nv:%s' % node_x
return get_set( node_x, system=system ) return get_redis(system).set( node_key, value )

#--- Edge values -----------------------------
def get_edge_value(edge_x, system='default'):
edge_key = 'ev:%s' % edge_x
return get_redis(system).get( edge_key )

def set_edge_value(edge_x, value, system='default'):
edge_key = 'ev:%s' % edge_x
return get_redis(system).set( edge_key, value )

redis_queue

•  Implements
a
queue
in
Python
using
Redis

•  Used
to
process
millions
of
background
tasks
on

Plurk
/
Todoist
/
Wedoist
daily
(billions
in
total)

•  Implementa=on:
18
lines

“real”
implementaKon
a
bit
bigger

•  h_ps://github.com/Doist/redis_simple_queue

redis_queue

from redis_simple_queue import *

delete_jobs('tasks')

put_job('tasks', '42')

assert 'tasks' in get_all_queues()
assert queue_stats('tasks')['queue_size'] == 1

assert reserve_job('tasks') == '42'
assert queue_stats('tasks')['queue_size'] == 0

redis_queue:
Implementa=on

from redis_wrap import *

def put(queue, job_data, system='default'):
get_list(queue, system=system).append(job_data)

def reserve(queue, system='default'):
return get_list(queue, system=system).pop()

def delete_jobs(queue, system='default'):
get_redis(system).delete(queue)

def get_all_queues(system='default'):
return get_redis(system).keys('*').split(' ')

def queue_stats(queue, system='default'):
return {
'queue_size': len(get_list(queue))
}

bitmapist
and
bitmapist.cohort

•  Implements
an
advanced
analyKcs
library
on
top

of
Redis
bitmaps.
Saved
us
$2000
USD/month

(Mixpanel)!

•  bitmapist

h_ps://github.com/Doist/bitmapist

•  bitmapist.cohort

Cohort
analyKcs
(retenKon)

bitmapist:
What
does
it
help
with?

•  Has
user
123
been
online
today?
This
week?

•  Has
user
123
performed
acKon
"X"?

•  How
many
users
have
been
acKve
have
this
month?

•  How
many
unique
users
have
performed
acKon
"X"

this
week?

•  How
many
%
of
users
that
were
acKve
last
week
are

sKll
acKve?

•  How
many
%
of
users
that
were
acKve
last
month
are

sKll
acKve
this
month?

•  Bitmapist
can
answer
thisfor
millions
of
users
and

most
operaKons
are
O(1)!
Using
very
small
amounts

of
memory.

What
are
bitmaps?

•  Opera=ons:
SETBIT,
GETBIT,
BITCOUNT,
BITOP

•  SETBIT
somekey
8
1

•  GETBIT
somekey
8

•  BITOP
AND
destkey
somekey1
somekey2

•  h_p://en.wikipedia.org/wiki/Bit_array

bitmapist:
Using
it

# Mark user 123 as active and has played a song
mark_event('active', 123)
mark_event('song:played', 123)

# Answer if user 123 has been active this month
assert 123 in MonthEvents('active', now.year, now.month)
assert 123 in MonthEvents('song:played', now.year, now.month)

# How many users have been active this week?
print len(WeekEvents('active', now.year, now.isocalendar()[1]))

# Perform bit operations. How many users that
# have been active last month are still active this month?
active_2_months = BitOpAnd(
MonthEvents('active', last_month.year, last_month.month),
MonthEvents('active', now.year, now.month)
)
print len(active_2_months)

bitmapist.cohort:

Manage
retenKon!

h_p://amix.dk/blog/post/19718

•  Goal:
InvenKng
a
modern
way
to
work
together

•  Join
an
amazing
team
of
13
people
from
all
around

the
world.
A
proﬁtable
business.
500.000+
users.

•  Work
from
anywhere.
Hacker
friendly
culture.

Python.
CompeKKve
salaries.

•  We
are
hiring:

jobs@doist.io

www.doist.io

Ques=ons
and
Answers

•  Slides
will
be
posted
to

h_p://amix.dk/

•  For
“oﬄine”
quesKons
contact:

amix@doist.io

Amir Salihefendic: Redis - the hacker's database

More Related Content

What's hot (20)

Similar to Amir Salihefendic: Redis - the hacker's database (20)

More from it-people (20)

Recently uploaded (20)

Amir Salihefendic: Redis - the hacker's database