ZeroMQ as scriptable sockets

Bear Metal OÜ| Õismäe tee 78-25, 13513 Tallinn, Harjumaa, Estonia | info@bearmetal.eu
ZeroMQ as scriptable sockets
Realtimeconf EU 2013
Lyon - http://realtimeconf.eu/
Lourens Naudé
Monday, April 22, 13

Why this talk ?

Operations / Platform

Credentials
ZeroMQ monitoring support
czmq co-maintainer
Ruby binding
Experimental TLS support

99% of deployments
don’t need enterprise
reliability
Of reliability

Then ...

... now

Zero broker, admin, cost,
waste, price tag and
(almost) zero latency
What is ZeroMQ ?

What it isn’t

IM for apps
Not a server or message queue
A library
Communication / concurrency
Messaging semantics for sockets

BSD API
sock.bind(“inproc://workers”)
sock.connect (“tcp://0.0.0.0:30”)
sock.send (“message”)
msg = sock.recv ()
sock.close ()

Community
30+ language bindings ...

Super (scriptable) sockets
What is a ZeroMQ socket ?

bind () or connect () to
multiple endpoints,
simultaneously
Multiple endpoints

bind () or connect ()
in any order
Async connect

Socket types
PUB SUB
REQ REP
PUSH PULL

Request / reply
REQ
REP
REQ
REP REP
REP
REQ REQ

Request / Reply usage
Lock step
Bi-directional
Routable
Flow control, orchestration etc.

Publish / subscribe
PUB
SUB SUB SUB

Inversion - logging
SUB
PUB PUB PUB

Publish / subscribe usage
Market data, game state etc.
PUB socket distributes
SUB socket filters
Subscription forwarding (3.x)

Pipeline
PUSH
PULL PULL PULL

Pipeline usage
Extendable - UNIX pipes
Round-robin distribution
Tasks with variable processing

Transport agnostic
inproc:// - threads in a process
ipc:// - processes on a box
tcp:// - nodes in a network
Uniform API

Lightweight switches
Devices

pub/sub persistence
PERSISTENCE
DEVICE

Device main loop
loop do
m = sub_socket.recv ()
persist (m)
pub_socket.send (m)
end

Use cases
Proxy
Routing (LRU)
Filtering / de-dup
Translation

Work distribution
PUSH
PULL
PUSH
PULL
PULL
PUSH
PULL
PUSH
VENTILATOR
SINK

PUSH
PULL
PUSH
PULL
PULL
PUSH
PULL
PUSH
VENTILATOR
SINK
PUSH
PULL
Scalable work distribution

Interjection Principle
Scalable topologies

Saving bandwidth
SUB
PUB
PUB
SUB SUB
SUB SUB

Asynchronous external
dependencies.
Resiliency

Atomicity
Atomic message delivery
Unbound message size
Send 5GB, recv all or nothing

Availability
Delayed retry on failure
Multi bind() and connect()
Unordered bind() or connect()

Reliability as patterns
Lazy Pirate (REQ / REP)
Majordomo (queue)
Titanic (persistence)
Binary Star (High Availability)
See http://zguide.zeromq.org

ZeroMQ concurrency
PUSH
PULL PULL PULL
MAIN THREAD
THREAD 1 THREAD 2 THREAD 3
I/O
THREAD

s = ctx.bind(:PUSH, "inproc://w")
3.times{ spawn_worker(ctx) }
loop do
msg = work.shift
break if msg == “exit”
s.send(msg)
end
Master (work distributor)

Thread.new do
s = ctx.connect(:PULL, “inproc://w”)
loop do
msg = s.recv
break if msg == "quit"
process(msg)
end
end
Workers (consumer threads)

Mongrel2
Case study

How it works
Multi-protocol
ZeroMQ for backends
Async backend requests
Language agnostic (bindings)

Mongrel2 topology
PULL
PUB
PULL
PUB
PUSH
SUB
BACKEND A BACKEND B
MONGREL2
CLIENT

Mongrel2 protocol
UUID ID PATH SIZE:HEADERS,SIZE:BODY

Mongrel2 HTTP request
PUSH PULL
AB23 4 /uri SIZE:HEADERS,SIZE:BODY

Mongrel2 HTTP response
SUB PUB
AB23 4 /uri SIZE:HEADERS,SIZE:BODY

Questions ?
(... or sync up after)
O fim
fork @methodmissing
follow @methodmissing

ZeroMQ as scriptable sockets

More Related Content

Similar to ZeroMQ as scriptable sockets (17)

More from Lourens Naudé (9)

Recently uploaded (20)

ZeroMQ as scriptable sockets