Agenda
System calls
File descriptor semantics
Blocking, nonblocking and async I/O
The Reactor Pattern
Patterns and gotchas



Caveats
Can’t run before you walk
Patterns are framework agnostic
KISS
read, write and connect only



Head count ?
Eventmachine (or any client libs)
Redis
Nginx
node.js
ZeroMQ



Reactor Pattern sweet spot ?



Soft realtime systems where
throughput is more
important than processing



What does that mean ?
Lots of in flight I/O
Little CPU per request - proxies
Improve ON:OFF cpu time
NEVER about speed for a single client
or request



"Upgraded to Thin, my
app's flying!"



EDOINGITWRONG



To maintain acceptable
response times for more clients,
with the same or less
infrastructure



Example app

Client 1

Client 2
Broker
Client 3

Client 4



FIX
FIX client
Financial Information eXchange
Compact protocol - encodes a lot of
domain info



FIX message

8=FIX.4.49=4535=049=AB56=CD34=352=20000426-12:05:0610=22

8=FIX.4.4 # FIX version
9=45 # body length
35=0 # Heartbeat msg
49=AB # sender
56=CD # receiver
34=3 # sequence number
52=20000426-12:05:06 # sent at
10=220 # checksum



The kernel

Mediates access to system
resources :
I/O, memory and CPU



User mode

Sandboxed execution for
security and stability
requirements



How do we access resources ?

System calls ( syscalls )

read(5, &buf, 4000)



Syscalls

Ruby
process

Kernel disk

Ruby
process
read(5, &buf, 4000)


Definition

A protected function call
from user to kernel space
with the intent to interact
with a system resource



Characteristics
Uniform API on POSIX systems
Ditto for behavior
LIES !
Slow emulations



Syscall performance
MUCH slower than function calls
20x and more
Definite context switch
Calls can block - slow upstream peers



Who doesn't know what file
descriptors or file handles
are ?



open syscall

fd = open(“/path/file”, O_READ)
read(fd, &buf, 4000)



Examples
local file
socket
directory
pipe
Consistent API, semantics differ



Definition
Numeric handle
References a kernel allocated resource
Kernel buffer
User space buffer



Blocking I/O

A request to read a chunk of
data from a descriptor may
block depending on
buffer state



Blocked buffer state

read(5, &b, 4000)

Kernel
User buffer
buffer
2000
1000



Nonblocking I/O

fd = socket (PF_INET, SOCK_STREAM, 0);
fcntl (fd, F_SETFL, O_NONBLOCK)



Nonblocking I/O

A read request would only
be initiated if the system
call won’t block



EAGAIN or
EWOULDBLOCK



Not a silver bullet
Guideline only
Not free - invokes a syscall
Not supported by all devices



Async I/O myth

Nonblocking I/O IS NOT
asynchronous I/O



Async I/O
Windows I/O Completion Ports
AIO: POSIX Realtime Extension
Async I/O often skip double buffering
Supports file I/O only



Recap
O_NONBLOCK is a guideline only
Invoked in user space
Data transfer in user space
Blocking and nonblocking I/O terms



Reactor Pattern

Remember, the primary use
case is increased throughput



Challenges
Large number of file descriptors
Respond to descriptor state changes
Execute domain logic



WHILE maintaining acceptable
response times



The Reactor Loop

while true do
# shortened for brevity
add_new_descriptors # attach new client descriptors
modify_descriptors # update descriptor state
break if terminate? # conditional loop break
end



Registered descriptors
W
5 R
10

R
6 Reactor 9
R
7
W 8
W


Multiplexed I/O

Who's familiar with select,
poll, epoll or kqueue ?



Multiplexed I/O
Nonblocking I/O is inefficient
Retry on EAGAIN is polling
Multiplexed I/O: concurrent blocking
Notified of state changes on any
registered fd



Multiplexed I/O
fd 5
fd 6
fd 7
fd 8
I/O bound
App Multiplexer

fd 9
fd 10
fd 11
fd 12

User space Kernel space



Multiplexed I/O

1. Tell me when fds 1..200
are ready for a read or a
write



Multiplexed I/O

2. Do other work until one
of them's ready



Multiplexed I/O

3. Get woken up by the
multiplexer on state
changes



Multiplexed I/O

4. Which of fd set 1..200
are in a ready state ?



Multiplexed I/O

5. Handle all fds ready for
I/O



Event driven I/O

These notifications, or
rather state changes in
readiness for reading or
writing, that's the Events in
Event Driven I/O



Event handlers / callbacks

module MyFixApp
def connection_completed
end

def receive_data(buffer)
end
end



Callbacks
Reactor provides I/O concurrency
NO concurrent handling of user
code
Fire on the reactor thread
Runs within the reactor loop



Ticks
A turn through the event loop
The reactor can do several thousand
ticks per second
Time slice for doing work
Aim to use minimal CPU per tick



Best practices, patterns and
gotchas



#1: Coupling Layers

App handler

Dispatch handler

I/O Multiplexer

Kernel



Dispatch Handler
Handles low level events from the
multiplexed I/O framework
eg. readiness for read or write
Hide complexities from apps
Buffering
Transfer
Conversion to and from Ruby strings



Dispatch Handler

class EventableDescriptor
def read; end # Tell don't ask interface
def write; end
def heartbeat; end
def select_for_read; end # I/O multiplexer
def select_for_write; end # agnostic
end



Application Handlers
Domain layer
Higher level events
Connection accepted, read,
disconnected
Encode and decode protocols
Perform business logic



Application Handlers

class FixConnection
def receive_data(data); end # callbacks
def connection_completed; end
def unbind; end
end



Glue

class FixConnection
# Encapsulate excess business logic in callbacks
@queue.push *FIX::Parser.parse(buffer)
end
end



#2: Testing

You don’t need a transport
for testing



Testing

class FixConnection
def receive_data(data)
@queue << data
end
end

def test_enqueue_on_receive
conn.receive_data "stub data"
assert_equal 1, conn.queue.size
end



#3: Confusing sync and async

Events occur
asynchronously, but they're
handled synchronously



App handler

Dispatch handler

I/O Multiplexer

Kernel



Defer work to a thread pool

class FixConnection
# enqueue to a thread pool
EM.defer{ FIX::Parser.parse(buffer) }
end
end



Schedule on the reactor thread

class FixConnection
# allow the reactor to service other clients also
EM.schedule{ FIX::Parser.parse(buffer) }
end
end



#4: Too much ON cpu time
Reactor is single threaded
Single core on SMP systems
Pegging the core blocks the event loop
Time sharing



Tight loops

[1..n].each do |i|
process(i) # expensive work
end



Prefer a tick loop

work = [1..n]
EM.tick_loop do
if work.empty?
:stop
else
process(work.shift) # expensive work
end
end



Slow Protocol parsers

Protocol parsers incur a cost
for both encoding and
decoding from buffers



Negate encoding / decoding costs

Prefer a fast C extension /
parser to native Ruby code



#5: Name resolution and connects

EM.connect("slow-broker.net", 2000, FixConnection)



Async resolver

d = EM::DNS::Resolver.resolve "slow-broker.net"
d.callback do |addrs|
# connect to broker API when resolved
EM.connect(addrs.first, 2000, FixConnection)
end



Async HTTP APIs

conn = EM::HttpRequest.new('http://api.a-broker.net')

req1 = conn.get :path => “x”, :keepalive => true
req1.callback {
req2 = conn.get :path => “y”
req2.callback {
# same connection as req1
}
}



#6: Blocking I/O

Not all file descriptors
support O_NONBLOCK



Ruby wrapper for libeio

$ gem install eio

EIO.open(__FILE__) do |fd|
EIO.read(fd, 1000) do |buf|
p buf
EIO.close(fd)
end
end

http://github.com/methodmissing/eio



#7: All libs have to be evented

redis = EM::Protocols::Redis.connect
redis.errback do |code|
puts "Error code: #{code}"
end
redis.set "a", "foo" do |response|
redis.get "a" do |response|
puts response
end
end



Evented MySQL

client = Mysql2::EM::Client.new
defer = client.query "SELECT sleep(3) as query"
defer.callback do |result|
puts "Result: #{result.to_a.inspect}"
end



#8: Memory

The Garbage Collector
could be invoked at any
time during request
processing



Minimize GC pressure

Be careful with sloppy
allocation patterns



Connection level proxy
module ProxyConnection
def initialize(client, request)
@client, @request = client, request
end

def post_init
EM::enable_proxy(self, @client)
end

def connection_completed
send_data @request
end

def proxy_target_unbound
close_connection
# end


#9: Buffer sizes

class FixConnection
def receive_data(inbound)
# inbound.size can be a single protocol
# message or a batch - stream of data
end

def send_data(outbound)
# enqueued on a write queue
end
end



Scheduled parsing

class FixConnection
# buffer packs multiple FIX messages
Parser.parse(buffer).each do |msg|
EM.next_tick { process(msg) }
end
end
end



Takeaways
Time sharing for I/O bound services
System call overheads and behavior
Blocking, nonblocking and async I/O
Schedule through ticks
OS, dispatch and app layers
Transport last, interfaces first



Questions ?



Wildfire Interactive, Inc. is hiring



Thanks !

http://wildfireapp.com/buzz/jobs
follow @methodmissing
fork github.com/methodmissing



In the Loop - Lone Star Ruby Conference

More Related Content

Viewers also liked (8)

Similar to In the Loop - Lone Star Ruby Conference (20)

More from Lourens Naudé (6)

Recently uploaded (20)

In the Loop - Lone Star Ruby Conference