Debugging Memory Problems in Rails

Nasos Psarrakos
psarrakos@gmail.com
github.com/nasospsa
linkedin.com/in/nasos-psarrakos
Debugging Memory
Problems in Rails

The Problem
> A migration needs more than 1GB memory
> Memory @ Heroku: 512MB

def start_script
puts "starting script..."
# Remove the X to enable the parameters for tuning.
# These are the default values as of Ruby 2.2.0.
@child = spawn(<<-EOC.split.join(" "))
XRUBY_GC_HEAP_FREE_SLOTS=4096
XRUBY_GC_HEAP_INIT_SLOTS=10000
XRUBY_GC_HEAP_GROWTH_FACTOR=1.8
XRUBY_GC_HEAP_GROWTH_MAX_SLOTS=0
XRUBY_GC_HEAP_OLDOBJECT_LIMIT_FACTOR=2.0
XRUBY_GC_MALLOC_LIMIT=16777216
XRUBY_GC_MALLOC_LIMIT_MAX=33554432
XRUBY_GC_MALLOC_LIMIT_GROWTH_FACTOR=1.4
XRUBY_GC_OLDMALLOC_LIMIT=16777216
XRUBY_GC_OLDMALLOC_LIMIT_MAX=134217728
XRUBY_GC_OLDMALLOC_LIMIT_GROWTH_FACTOR=1.2
rails runner scripts/rbup.rb & echo $! >> tmp/pids/rbup.pid
EOC
sleep 0.1 until alive?
end
The problem 1/6
Script to check memory usage
def alive?
lines = `ps -p #{server_pid} | wc -l`.to_i
lines == 2
end
def stop_script
puts "stopping script..."
Process.kill :KILL, server_pid if server_pid && alive?
Process.wait @child
delete_pid
end
def delete_pid
`if test -f tmp/pids/rbup.pid; then rm tmp/pids/rbup.pid; fi`
end
def server_pid
`cat tmp/pids/rbup.pid`.to_i
end
def memory_size_mb
(`ps -o rss= -p #{server_pid}`.to_i * 1024).to_f / 2**20
end
https://gist.github.com/nasospsa/f7fadd6c2f3510b420525f960dc12ddd

The problem 2/6
Script to check memory usage
start_script
seconds = 0
used_mb = 0
max_mem = 0
while alive?
used_mb = memory_size_mb
max_mem = [max_mem, used_mb].max
puts "#{seconds}sec - #{'%.2f' % used_mb.round} MB"
seconds += 1
sleep 1
end
stop_script
puts "Total Time: #{seconds}sec"
puts "Max Memory: #{'%.2f' % max_mem} MB"
https://gist.github.com/nasospsa/f7fadd6c2f3510b420525f960dc12ddd

The problem 3/6
mapped = {}
fields = [:weights].concat(FoodsController::EAGER_LOADED_FIELDS)
Food.includes(*fields).find_each do |f|
mapped[f.id] = { json_data: f.as_json(methods: [:nutrient_data_basic]) }
end
Food.update(mapped.keys, mapped.values)
First iteration of the migration:
Result: Max Memory 1,1GB | 300 seconds

Food.includes(*fields).find_each do |f|
f.update json_data: f.as_json(methods: [:nutrient_data_basic])
end
The problem 4/6
Second iteration of the migration:
Result: Max Memory 1GB | 273 seconds

Food.includes(*fields).find_in_batches(batch_size: 200) do |group|
ActiveRecord::Base.transaction do
group.each do |f|
f.update json_data: f.as_json(methods: [:nutrient_data_basic])
end
end
end
The problem 5/6
Third iteration of the migration:
Result: Max Memory 380MB | 160 seconds

The problem 6/6
Fourth successful iteration of the migration:
Result: Max Memory 265MB | 82 seconds
Food.includes(*fields).find_in_batches(batch_size: 50) do |group|
group.each do |f|
f.update_columns json_data: f.as_json(methods: [:nutrient_data_basic])
end
end

update_columns(attributes)
Updates the attributes directly in the database issuing an
UPDATE SQL statement and sets them in the receiver.
This is the fastest way to update attributes because it goes
straight to the database, but take into account that in
consequence the regular update procedures are totally
bypassed.

3 Aha! Moments
> a moment of sudden insight or discovery.

psarrakos@gmail.com
github.com/nasospsa
linkedin.com/in/nasos-psarrakos
Thank
You

Debugging Memory Problems in Rails

More Related Content

What's hot (20)

Similar to Debugging Memory Problems in Rails (20)

Recently uploaded (20)

Debugging Memory Problems in Rails