When rails hits the fan

Proprietary and
Confidential
When Rails hits the fan
Eric Saxby
@sax @ecdysone @sax
Thursday, June 6, 13

This talk is about:
Proprietary and
Confidential
User trafﬁc increasing week
over week

Proprietary and
Confidential
Databases outgrowing RAM

Proprietary and
Confidential
Avg site response time over same period

Who am I? Why should you care?
Proprietary and
Confidential
■ Application developer
Many and various technologies.
Worked with Rails for ~5 years.
■ Recent focus has been
operational
Chef, PostgreSQL, SmartOS,
monitoring
■ TDD, BDD, Agile, DevOps, SOA, etc
I care about how code is organized, and always want to
learn how to do that better.
■ Now I’m at Wanelo

What is Wanelo?
Proprietary and
Confidential
■ Wanelo (“Wah-nee-lo” from Want, Need
Love) is a global platform for shopping.

Proprietary and
Confidential
Marketing-free shopping across 100s of
thousands of unique stores

Proprietary and
Confidential
Personal feed of products from any store on the
internet
No Il8n or l10n... yet!

Technology overview
Proprietary and
Confidential
■ MRI Ruby 1.9.3 & Rails 3.2
■ PostgreSQL 9.2.4, Solr 3.6
■ Joyent Cloud, SmartOS
ZFS, ARC, raw IO performance, SmartOS, dTrace
■ Circonus, NewRelic, Boundary
Monitoring, graphing, alerting
■ Chef + Opscode
■ Amazon S3 + Fastly CDN
■ statsd, Graphite, nagios

How do you know when
you are entering the
bad place?
Proprietary and
ConfidentialThursday, June 6, 13

Let's go back to this for a sec
Proprietary and

Proprietary and

What is the bad place?
Proprietary and
Confidential
green: disk reads, red: disk writes on DB server
Is this actually a problem?

What is the bad place?
Proprietary and
Confidential
Can be difﬁcult to predict, using most of the
default metrics we track
green: disk reads, red: disk writes on DB server

Utilization
Proprietary and
Confidential
> iostat -xM 3
extended device statistics
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
sd0 11.3 48.8 0.2 0.5 0.0 0.5 8.2 0 11
sd1 13.9 42.2 0.2 0.6 0.0 0.3 5.8 0 11
sd2 12.0 24.9 0.2 0.5 0.0 0.2 5.5 0 10
sd3 12.3 27.2 0.2 0.5 0.0 0.4 9.6 0 10
sd4 7.3 15.3 0.1 0.3 0.0 0.1 6.2 0 6
sd5 13.0 14.6 0.2 0.3 0.0 0.1 4.8 0 7
sd6 9.6 50.5 0.2 0.5 0.0 0.4 5.9 0 10
sd7 7.0 46.5 0.1 0.6 0.0 0.4 7.5 0 11
sd8 9.3 33.5 0.1 0.4 0.0 0.3 6.3 0 9
sd9 7.6 32.5 0.1 0.4 0.0 0.2 6.1 0 7
sd10 6.3 52.1 0.1 0.6 0.0 0.3 6.0 0 10
sd11 7.6 50.2 0.1 0.6 0.0 0.5 8.5 0 12

Utilization
Proprietary and
Confidential
> iostat -xM 3
sd0 11.3 48.8 0.2 0.5 0.0 0.5 8.2 0 11
sd1 13.9 42.2 0.2 0.6 0.0 0.3 5.8 0 11
sd2 12.0 24.9 0.2 0.5 0.0 0.2 5.5 0 10
sd3 12.3 27.2 0.2 0.5 0.0 0.4 9.6 0 10
sd4 7.3 15.3 0.1 0.3 0.0 0.1 6.2 0 6
sd5 13.0 14.6 0.2 0.3 0.0 0.1 4.8 0 7
sd6 9.6 50.5 0.2 0.5 0.0 0.4 5.9 0 10
sd7 7.0 46.5 0.1 0.6 0.0 0.4 7.5 0 11
sd8 9.3 33.5 0.1 0.4 0.0 0.3 6.3 0 9
sd9 7.6 32.5 0.1 0.4 0.0 0.2 6.1 0 7
sd10 6.3 52.1 0.1 0.6 0.0 0.3 6.0 0 10
sd11 7.6 50.2 0.1 0.6 0.0 0.5 8.5 0 12
Develop spidey senses over time, but problems
are highly anecdotal

Saturation
Proprietary and
Confidential
> iostat -xM 3
sd0 11.3 48.8 0.2 0.5 0.0 0.5 8.2 0 11
sd1 13.9 42.2 0.2 0.6 0.0 0.3 5.8 0 11
sd2 12.0 24.9 0.2 0.5 0.0 0.2 5.5 0 10
sd3 12.3 27.2 0.2 0.5 0.0 0.4 9.6 0 10
sd4 7.3 15.3 0.1 0.3 0.0 0.1 6.2 0 6
sd5 13.0 14.6 0.2 0.3 0.0 0.1 4.8 0 7
sd6 9.6 50.5 0.2 0.5 0.0 0.4 5.9 0 10
sd7 7.0 46.5 0.1 0.6 0.0 0.4 7.5 0 11
sd8 9.3 33.5 0.1 0.4 0.0 0.3 6.3 0 9
sd9 7.6 32.5 0.1 0.4 0.0 0.2 6.1 0 7
sd10 6.3 52.1 0.1 0.6 0.0 0.3 6.0 0 10
sd11 7.6 50.2 0.1 0.6 0.0 0.5 8.5 0 12

Both are important!
Proprietary and
Confidential
■ Tracking saturation helps you predict
problems
■ Tracking utilization can tell you how to
solve that problem
■ Basically, watch every video by
Brendan Gregg on YouTube
http://www.youtube.com/results?search_query=brendan+gregg

Know the limits of your data
Proprietary and
Confidential
Averages can be extremely useful, but
do not give a complete picture

Know the limits of your data
Proprietary and
Confidential
Outliers can cause severe problems
even when average is great

This is why we ❤ PostgreSQL
Proprietary and
Confidential
■ pg_stat_activity
■ pg_stat_user_indexes
■ pg_stat_user_tables
■ pg_stat_statements
■ PostgreSQL gives you tools to monitor
it and operate at scale

Proprietary and
Confidential
pg_stat_statements

Ok, so back to the
bad place
Proprietary and

Proprietary and
Confidential
Hit ratio of File System Cache on
database server degrading over time

Proprietary and
Confidential
Average response latency
increasing over time, ties up CPU resources

Step 1: Cache all the things!
Proprietary and
Confidential
■ Reduce load on DB(s)
■ Reduce rendering time

Cache invalidation?
Proprietary and
Confidential
■ Use model updated_at in the cache
key
■ Changing DB record invalidates all
caches for that record

What were the goals of caching?
Proprietary and
Confidential
■ Using model attributes as cache keys
means you need to fetch DB records
■ At high scale, fetching DB records for
every page becomes problematic
■ Cache sweepers are painful, but
they're the only thing we've found to
be scalable and reliable

Action cache all the things!
Proprietary and
Confidential
■ Cache hits skip all rendering
■ Still able to run before ﬁlters, for
instance when doing A/B testing
■ Some cached pages can be put
behind a CDN
■ Requires page personalization to be
added via Ajax

Fragment cache the rest!
Proprietary and
Confidential
■ Fragments can be shared between pages
■ Can reduce rendering time
■ Can remove queries for related records

Proprietary and
Confidential
■ Difﬁcult to remove top level query
■ Joins/eager loading means some related
records are still queried
■ How many trips to memcached?

Proprietary and
Confidential
def multi_get_on_collection(objects,
cache_options = {})
cache_keys = objects.
inject(ActiveSupport::OrderedHash.new) do |key_map, obj|
key_map[obj] = cache_options[:cache_key_proc].call obj
key_map
end
pre_rendered_objects = Rails.cache.read_multi *cache_keys.values
cache_keys.map do |object, cache_key|
cached_html = pre_rendered_objects[cache_key]
if cached_html.present? && caching_enabled?
cached_html
else
cache_options[:render_proc].call(object).tap do |fragment|
Rails.cache.write cache_key, fragment, cache_options
end
end
end
end

MultiGet is your friend
Proprietary and
Confidential
■ Turns 100+ memcached calls into a
single request
cache_key = ->(product) do
"products_thumb_#{product.id}"
end
renderer = ->(product) do
render "products/thumb", model: product
end
multi_get_on_collection(@products,
cache_key_proc: cache_key,
render_proc: renderer,
expires_in: 6.hours).join("n").html_safe

Proprietary and
Confidential
Increasing your cache hit ratio means
less queries against your database

Proprietary and
Confidential
sd1 384.0 1157.5 48.0 116.8 0.0 8.8 5.7 2 100
sd1 368.0 1117.9 45.7 106.3 0.0 8.0 5.4 2 100
sd1 330.3 1357.5 41.3 139.1 0.0 9.5 5.6 2 100
DB latency increases
■ Even with highly efﬁcient caches, iostat
shows 100% disk busy

count(*)
Proprietary and

TOO MANY COUNTS
Proprietary and

Rails has the answer!
Proprietary and
Confidential
■ counter_cache column on table
■ Adding records executes INCR
■ Removing records executes DECR
class Product < ActiveRecord::Base
belongs_to :store, counter_cache: true
end

But...
Proprietary and
Confidential
■ On very write heavy applications,
multiple requests will update the same
record
■ DEADLOCK errors

Use background jobs
Proprietary and
Confidential
■ Stop updating counter caches on save
■ Queue a delayed job for the near future
■ Job performs a complete recalculation of
the counter cache and is idempotent
■ The higher the count, the further we
delay the job (less likely users will notice)

Deduplicate delayed jobs
Proprietary and
Confidential
■ Sidekiq with UniqueJob plugin
■ Updates are serialized via a ﬁxed
number of workers
■ Workers can be stopped to alleviate DB
load in an emergency
■ Same pattern can be applied elsewhere,
like updating Solr indexes

Deduplicate delayed jobs
Proprietary and
Confidential
class UpdateProductCountWorker < WaneloWorker
sidekiq_options queue: :product_counts,
unique: true
wait 10.minutes
def perform!(params)
id = params[:id].to_i
ActiveRecord::Base.connection.execute %Q{
update stores set product_count =
(select count(*)
from products
where store_id = stores.id)
where stores.id = #{id}
}
end
end

STILL TOO MANY COUNTS
Proprietary and

Proprietary and
Confidential
Pagination gems run counts

Proprietary and
Confidential
■ Kaminari executes a count(*) to determine
total page count

Proprietary and
Confidential
total page count
SELECT "stores".* FROM "stores"
WHERE (state = 'approved')
LIMIT 20 OFFSET 0
SELECT COUNT(*) FROM "stores" WHERE (state = 'approved')

Proprietary and
Confidential
total page count
SELECT "stores".* FROM "stores"
WHERE (state = 'approved')
LIMIT 20 OFFSET 0
SELECT COUNT(*) FROM "stores" WHERE (state = 'approved')
■ We paginate EVERYTHING
■ We often already know total count from
counter cache (or can hard code 10,000)

Proprietary and
Confidential
module Kaminari
module ActiveRecordRelationMethods
# a workaround for AR 3.0.x that returns 0 for #count when page > 1
# if +limit_value+ is specified, load all the records and count them
if ActiveRecord::VERSION::STRING < '3.1'
def count(column_name = nil, options = {}) #:nodoc:
limit_value ? length : super(column_name, options)
end
end
def total_count(column_name = nil, options = {}) #:nodoc:
@total_count ||= begin
c = except(:offset, :limit, :order)
# Remove includes only if they are irrelevant
c = c.except(:includes) unless references_eager_loaded_tables?
# .group returns an OrderdHash that responds to #count
c = c.count(column_name, options)
if c.is_a?(ActiveSupport::OrderedHash)
c.count
else
c.respond_to?(:count) ? c.count(column_name, options) : c
end
end
end
end
end

Proprietary and
Confidential
Time for some monkey patches!
module ActiveRecord
class Relation
def custom_counter(count)
@total_count ||= count
self
end
end
end
module Sunspot
module Search
class AbstractSearch
def custom_counter(count)
@total ||= count
self
end
end
end
end
@products = @store.products.
custom_counter(@store.products_count).
page(params[:page])

How do we know what
Rails is really doing?
Proprietary and

ruby-prof and pilfer
Proprietary and
Confidential
■ Proﬁle the entire stack trace of an action
or a set of classes
https://github.com/eric/pilfer
https://github.com/ruby-prof/ruby-prof

Proprietary and
Confidential
■ Can work as Rack middleware
■ Outputs HTML that will tell you...
if Rails.env.profile?
use Rack::RubyProf, :path => '/tmp/profile'
end

Proprietary and
Confidential
■ Can work as Rack middleware
■ Outputs HTML that will tell you...
if Rails.env.profile?
use Rack::RubyProf, :path => '/tmp/profile'
end
■ How long are we spending generating
URLs???

Proprietary and
Confidential
module UrlHelpers
def product_path(product)
"/p/#{product.id}/#{product.slug}"
end
end

The importance of
fast JSON rendering
Proprietary and

HTML vs JSON
Proprietary and
Confidential
■ Since launching native iOS and Android
apps, the majority of Wanelo trafﬁc is
served via a JSON API
■ The longer we spend rendering JSON,
the more CPUs are tied up, the more
servers we need

Proprietary and
Confidential
Removing RABL

Things we did not expect
Proprietary and
Confidential
■ RABL serializes, then deserializes JSON
to do merges!
■ ActiveSupport deﬁnes inefﬁcient :to_json
on every object

Proprietary and
Confidential
■ Rendering JSON partials should
hash.merge!
■ Fragment caching should marshal hashes,
not JSON
■ Caching should allow for MultiGet
■ Should allow for arbitrary composition
■ JSON conversion should use OJ to
call :to_json ONCE
https://github.com/wanelo/compositor

Proprietary and
Confidential
require 'active_support/json'
require 'active_support/core_ext/object/to_json'
[Object, Array, FalseClass, Float,
Hash, Integer, NilClass, String,
TrueClass].each do |klass|
klass.class_eval do
def to_json(options = nil)
Oj.dump(self, options)
end
end
end

Asynchronous
Commits
Proprietary and

Do you read, or do you write?
Proprietary and
Confidential
■ Tools like iostat, vmstat, kstat, collectd
pg_stat_user_tables can show you
utilization
■ Reads much easier to scale than writes.
Read/write splitting, caching.
■ What do you do when writes become the
bottleneck?

Writes committed to disk?
Proprietary and
Confidential
■ Some workloads are more lenient for
delay/possible data loss
■ Applicable to many technologies
■ Even microsecond delays can reduce
load
■ Waiting for Solr to respond can keep DB
transactions open longer

i.e. PostgreSQL
Proprietary and

i.e. Solr
Proprietary and
Confidential

<autoCommit>

<maxDocs>1000</maxDocs>

<maxTime>30000</maxTime>
</autoCommit>

What happens when data
grows larger than a
single database?
Proprietary and

Proprietary and
Confidential
Growth of one key DB table over 3 months

Service Oriented
Architecture
Proprietary and

The only real way to scale
Proprietary and
Confidential
■ Tune code, infrastructure for a very
particular workload
■ Hide sharding from other codebases
■ Allow small teams to manage small(er)
codebases

The only real way to scale
Proprietary and
Confidential
■ Tune code, infrastructure for a very
particular workload
■ Hide sharding from other codebases
■ Allow small teams to manage small(er)
codebases
■ Everyone talks about why, no-one talks
about how in the Rails world

Services are hard (the first time)
Proprietary and
Confidential
■ Synchonous vs asynchronous data
persistence
■ Message passing
■ Testing
■ Iterative development

Iteration is the key
Proprietary and
Confidential
■ Isolate data
■ Isolate interface
■ Extract interface with an database
adapter
■ Launch service layer
■ Switch interface to user service adapter

Proprietary and
Confidential
has_many :saves
end

Proprietary and
Confidential
has_many :saves
end X

Proprietary and
Confidential
has_many :saves
end X
def saves
Save.where(product_id: self.id)
end
end

Proprietary and
Confidential
has_many :saves
end X
def saves
Save.where(product_id: self.id)
end
end
Do this everywhere (you have tests, right?)

Proprietary and
Confidential
class Save < ActiveRecord::Base
establish_connection "saves_#{Rails.env}"
end
■ Set up a read replica
■ Take down site
■ Promote replica to be a master
■ Restart unicorns with new conﬁg
■ Bring up site
■ Clean up unnecessary tables on each DB

Proprietary and
Confidential
establish_connection "saves_#{Rails.env}"
def self.by_product(product)
where(product_id: product.id)
end
end
def saves
Save.by_product(self)
end
end
■ Reduce Ruby interface to minimum possible
■ Easy to deploy this

Proprietary and
Confidential
class Save
include SavesClient
end
module SavesClient
def self.included(other)
other.send(:attr_accessor, :id, :product_id)
other.extend ClientClassMethods
end
module ClientClassMethods
def by_product(*args)
adapter.new(self).by_product(*args)
end
def adapter
@adapter ||= SavesClient::DbAdapter
end
end
end

Proprietary and
Confidential
module SavesClient
class DbAdapter
def self.close_connections
# Check in the database connection,
# since we're shutting down this thread
SavesService::Save.clear_active_connections!
end
relation :by_product, *args
end
def relation(method, *args)
SavesClient::AdapterRelation.new(self,
SavesService::Save.send(method, *args))
end
def all(scope)
# Scope is an AR Relation instance returned from
# SavesService::Save.by_product(product)
scope.all.map { |m| client_class.new save_attrs_from(m) }
end
end
end

Proprietary and
Confidential
module SavesClient
class DbAdapter
def self.close_connections
# Check in the database connection,
# since we're shutting down this thread
SavesService::Save.clear_active_connections!
end
relation :by_product, *args
end
def relation(method, *args)
SavesClient::AdapterRelation.new(self,
SavesService::Save.send(method, *args))
end
def all(scope)
# Scope is an AR Relation instance returned from
# SavesService::Save.by_product(product)
scope.all.map { |m| client_class.new save_attrs_from(m) }
end
end
end
Thread safety is EXTREMELY important

Proprietary and
Confidential
module SavesClient
class AdapterRelation
attr_reader :adapter, :scope
def initialize(adapter, scope)
@adapter, @scope = adapter, scope
end
def limit(num); end
def order(order); end
def page(num); end
def first; end
def all
adapter.all(scope)
end
end
end
■ Calling :by_product instantiates a Relation
■ Calling :all executes the query

What executes the query?
Proprietary and
Confidential
■ The ActiveRecord model moves into the gem
■ The adapter translates the Save methods into AR calls,
maps columns into attributes on our class
■ Important to deploy this at this stage, as there many
problems to solve, like:
■ Getting your tests green, ﬁxtures consistent
■ Figuring out whether you really covered all access
patterns
module SavesService
end
end

Minimize the Ruby access
Proprietary and
Confidential
■ We were able to reduce everything to 7 scopes,
i.e. :by_product, :by_user, etc
■ Reduced Relation methods to these:
■ limit
■ page
■ order
■ count
■ all
■ ﬁrst
■ last
■ pluck
■ ﬁnd_in_batches

Launch the service layer
Proprietary and
Confidential
■ Now that we have a small public Ruby
interface, we can pair that to a Sinatra app
■ Sinatra can serve as a long-term fake for
Selenium, even after service is re-written

module SavesService
class Web < Sinatra::Base
set :environment, ENV['RACK_ENV'] || "development"
PAGE_SIZE = 50
ActiveRecord::Base.include_root_in_json = false
register Sinatra::ActiveRecordExtension
configure do
set :database_file, SavesService.config.db_config
set :root, File.expand_path("../../../", __FILE__)
disable :raise_errors
disable :show_exceptions
set :logger, nil
end
error do
e = env['sinatra.error']
ActiveRecord::Base.logger.error ["#{e.class}: #{e.message}",
*e.backtrace].join("n ")
status 500
body '{"errors":":("}'
end
before { content_type 'application/json' }
end
end
Proprietary and

Proprietary and
Confidential
get '/products/:pid/saves' do
paginate SavesService::Save.by_product(params[:pid])
end
private
def paginate(scope)
return count(scope) if params[:count].present?
order = params[:order] if params[:order] =~ /Adesc|ascZ/i
scope = scope.order("created_at #{order || 'desc'}")
limit = params[:limit] ? params[:limit].to_i : PAGE_SIZE
page_number = [params[:page].to_i - 1, 0].max * limit
scope = scope.offset(page_number).limit(limit)
scope = scope.pluck(params[:pluck]) if params[:pluck]
body Oj.dump(saves: scope)
end

Proprietary and
Confidential
#!/usr/bin/env ruby
require 'optparse'
options = {
:environment => 'development',
:port => 3000
}
OptionParser.new do |opts|
opts.banner = "Usage: saves_service [options]"
opts.on("-d", "--dbconfig OPT", "path to database.yml") do |opt|
options[:db_config] = File.expand_path(opt, Dir.pwd)
end
opts.on("-E", "--environment OPT", "RACK_ENV to use") do |opt|
options[:environment] = opt
end
opts.on("-p", "--port OPT", "port to use") do |opt|
options[:port] = opt
end
end.parse!
cmd_env = { 'RACK_ENV' => options[:environment],
'DB_CONFIG' => options[:db_config],
}.delete_if{|k,v| v.nil? }
rackup_file = File.expand_path('../../config.ru', __FILE__)
exec cmd_env, "unicorn -p #{options[:port]} #{rackup_file}"

DbAdapter vs HTTPAdapter
Proprietary and
Confidential
■ Maps Ruby interface to Net::HTTP::Persistent
■ Deserializes JSON into values on class
■ Scope becomes a map
by_product(1) => "/products/1/saves"
■ Finder methods map params
limit(10) => "?limit=10"
■ AdapterRelation uses Adapter to fetch records,
does not change at all

Proprietary and
Confidential
module SavesClient
class AdapterRelation
attr_reader :adapter, :scope
def initialize(adapter, scope)
@adapter, @scope = adapter, scope
end
def limit(num); end
def order(order); end
def page(num); end
def first; end
def all
adapter.all(scope)
end
end
end

Deploy as a setting change
Proprietary and
Confidential
Rails.application.config.after_initialize do |app|
if Settings.saves_service.enabled
Save.saves_base_url = Settings.saves_service.url
else
require 'saves_client/db_adapter'
Save.adapter = SavesClient::DbAdapter
require 'saves_service/save'
saves_db = "saves_#{Rails.env}"
SavesService::Save.establish_connection saves_db
end
end

Takeaways
Proprietary and

Proprietary and
Confidential
■ Choose technologies that are easy to
operate and monitor
■ Don't immediately break when they hit
resource thresholds
■ Sound replication strategies
■ Assume that data should be tracked, even if
you don't yet understand the relevance
■ Small iterative performance improvements
can have massive payoff over time

Thanks!
Proprietary and
Confidential
@sax
@ecdysone
@sax
https://github.com/wanelo
https://github.com/wanelo-chef

When rails hits the fan

More Related Content

Similar to When rails hits the fan (20)

Recently uploaded (20)

When rails hits the fan