Background processing with Resque
0 likes10,391 views
Resque is a background processing library for Ruby that allows processing of jobs asynchronously via a queue using Redis. It uses workers that listen for jobs on queues and process them, and includes plugins like resque-lock to prevent duplicate jobs and resque-loner to relaunch failed jobs. The document discusses setting up Resque with God for monitoring workers, handling signals, and restarting workers gracefully on the bookandgolf.com project which synchronizes golf course data from multiple APIs into Redis.
![Workers graceful restart
namespace :resque do
task :restart_workers, :roles => [:app, :db] do
rake "resque:restart_workers"
end
end](https://image.slidesharecdn.com/resquebackgroundprocessing-110409132427-phpapp02/85/Background-processing-with-Resque-15-320.jpg)
Background processing with Resque
- 1. Resque Background processing System administration on a multi- instances setup Experience feedback Plugins Nicolas Blanco Novapost
- 2. My project : bookandgolf.com
- 3. bookandgolf.com At launch : • more than 60 golfs to synchronise • 4 different APIs • More than 350 000 slots (starts) to synchronize every 2 hours At first we tried to do the synchronization with cron tasks... but...
- 5. nginx resque-web :web redis-server :db worker 1 Current worker 2 :app worker 3 architecture worker 4
- 6. Resque job class class MyKikoololSleepJob @queue = :medium def self.perform(sleep_duration) sleep sleep_duration end end Resque.enqueue(MyKikoololSleepJob, 10)
- 7. Resque workers rake environment resque:work QUEUE=medium rails@www:~$ ps aux | grep resque rails 8950 sh -c cd /home/rails/www/socianalytics/current && rake environment resque:work QUEUE=high,medium,low ... rails 8951 resque-1.9.10: Forked 9105 at 1302095560 rails 9105 resque-1.9.10: Processing medium since 1302095560
- 8. Resque workers Unix signals • QUIT - Wait for child to finish processing then exit • TERM / INT - Immediately kill child then exit • USR1 - Immediately kill child but don't exit • USR2 - Don't start to process any new jobs • CONT - Start to process new jobs again after a USR2
- 12. God setup - my way! Versioned god config files by instance name, ie : • god / app.god • god / web.god • god / db.god • ... God init.d file found on gist.github.com :) god.conf : GOD_CONFIG=/home/rails/www/bookgolf/current/god/db.god GOD_COMMAND="sudo -u rails /usr/local/bin/god" GOD_LOG=/home/rails/log/god.log
- 13. God config # Resque God.watch do |w| w.env = { 'RAILS_ROOT' => rails_root, 'RAILS_ENV' => rails_env } w.name = "resque-worker" w.interval = 30.seconds w.start = "cd #{rails_root} && rake environment resque:work QUEUE=high,medium,low" w.start_grace = 10.seconds ...
- 14. Workers graceful restart Rake task to restart workers, the "graceful" way namespace :resque do task :restart_workers => :environment do pids = Array.new Resque.workers.each do |worker| pids << worker.to_s.split(/:/).second if worker.to_s.include?(Settings.resque.localhost_name) end if pids.size > 0 system("kill -QUIT #{pids.join(' ')}") end system("rm /home/rails/.god/pids/resque-worker*.pid") end end
- 15. Workers graceful restart namespace :resque do task :restart_workers, :roles => [:app, :db] do rake "resque:restart_workers" end end
- 16. Resque-web - Final setup • Launched externally with nginx reverse proxy (easy to add http basic auth) • External Ruby config file to change Redis setup, load plugins, etc. resque-web config/resque-web.rb
- 17. Resque-web God monitoring %w{5678}.each do |port| God.watch do |w| w.env = { 'RAILS_ROOT' => rails_root, 'RAILS_ENV' => rails_env } w.uid = "rails" w.name = "resque-web" w.interval = 30.seconds w.start = "cd #{rails_root} && resque-web config/resque- web.rb" w.start_grace = 15.seconds w.start_if do |start| start.condition(:process_running) do |c| c.interval = 5.seconds c.running = false end end end end
- 18. ActiveRecord - timeout / stale :( Fast workaround... ... ActiveRecord::Base.connection.reconnect! ...
- 19. Resque plugins • defunkt / resque-lock Prevent two workers from working on the same Job class with same arguments
- 20. Resque plugins • jayniz / resque-loner TIP : Do not forget to : require "resque-loner" in your Resque-web config.rb file! or... FAIL (cleared jobs won't be relaunched again!) BAD : Need inheritance in job classes :'(... But a fork exists that doesn't need it.
- 21. Other plugins... Look @ github.com/defunkt/resque Wiki! And remember that with a good system architecture, Resque is just...