Background Jobs with Resque

Background Jobs with
Resque
Jon Homan
Apprentice @ Obtiva

http://jonhoman.com
@jonhoman

Agenda
• Resque
• Examples
• Real world use
• Plugins
• Alternate Implementations
• When to use Resque

Background Jobs

• Remote API calls
• Uploading images
• Number crunching
• Updating RSS feeds

Resque
https://github.com/defunkt/resque

• Open-source library for creating background
jobs
• Redis backend
• Support for multiple queues
• Queues are processed by workers

More Resque Features
• Monitor with god or monit
• Distributed processing
• Priorities
• Persistent queues
• Memory leak resistant
• Web interface

Redis
http://redis.io/

• Key-value datastore
• data structure server
• constant time pushes and pops
• atomic operations

All “Foreground”
class FeedsController
def create
feed = Feed.create!
feed.fetch
redirect_to feed_path(feed)
end
end

class Feed
def fetch
rss = RSS::Parse.parse(url, false)
feed.title = rss.title
end
end

Queue up creation
def create
feed = Feed.create!
Queue.push(feed) # won’t block UI
end
end

class Queue
def push(feed)
queue << feed
end
end

Now with Resque
def create
feed = Feed.create!
Resque.enqueue(FeedCreator, feed.id)
end
end

How to process jobs
class FeedCreator
@queue = :feed_creation

def perform(feed_id)
feed = Feed.find(feed_id)
rss = RSS::Parser.parse(feed.url,false)
feed.title = rss.title
# create all items in the rss
end
end

Workers?

• Long-running Rake tasks
• Look for jobs every n seconds

How to start workers
$ rake resque:work

$ QUEUE=feed rake resque:work

$ QUEUE=feed, twitter rake resque:work

$ QUEUE=* rake resque:work

$ COUNT=3 QUEUE=* rake resque:work

Testing w/ RSpec

it "stores the name of the feed" do
FeedCreator.perform(feed.id)
feed.reload.title.should == "Feed
end

Testing w/ RSpec

it "creates Items for every item in the feed" do
feed.reload.items.count.should == 5
end

Testing w/ RSpec

it “parses the feed” do
RSS::Parser.should_receive(:parse)

end

Real world uses

• GitHub
• Groupon
• lifeStreams

GitHub
35 different types of jobs:
• Warming caches
• Counting disk usage
• Building tarballs
• Building Rubygems
• Building graphs
• Updating search index

Groupon

• Resque for background jobs
• Notiﬁcations
• Lockable worker
• Redis for semaphores

lifeStreams
• Process updates to blog feeds
• Publish updates to Facebook, Twitter, Email

Plugins
• resque_spec
• Resque Heroku Autoscaler
• resque multi step
• ResqueMailer
• resque-retry
• and at least 25 others

ResqueSpec
https://github.com/leshill/resque_spec

• Resque plugin that adds resque-speciﬁc
matchers
• have_queued
• have_scheduled
• have_queue_size_of

Examples
it “adds a feed to the creation queue” do
post :create, :feed
FeedCreator.should have_queued(feed.id)
end

it “schedules a feed update” do
feed.update
FeedUpdater.should have_scheduled(feed.id)
end

Resque Heroku
Autoscaler

• Scales Heroku workers
• Start worker if none running
• Stops worker after jobs are processed

RHA
require 'resque/plugins/resque_heroku_autoscaler'

class TestJob
extend Resque::Plugins::HerokuAutoscaler

@queue = :test

def perform
...awesome stuff...
end
end

Resque Mailer

• Asynchronously deliver email in Rails

Resque Mailer

class MyMailer < ActionMailer::Base
include Resque::Mailer
end

MyMailer.subject_email(params).deliver

QUEUE=mailer rake environment resque:work

resque-retry

• Retry, delay and exponential backoff support

resque-retry
require 'resque-retry'

class ExampleRetryJob
extend Resque::Plugins::Retry
@queue = :example_queue

@retry_limit = 3
@retry_delay = 60

def self.perform(*args)
# your magic/heavy lifting goes here.
end
end

resque-retry

no delay, 1m, 10m, 1h, 3h, 6h
@backoff_strategy = [0, 60, 600, 3600, 10800, 21600]

Alternate
Implementations
• C
• Java
• Scala
• .NET
• Python
• PHP
• node.js

jesque
// Add a job to the queue
final Job job = new Job("TestAction",
new Object[]{ 1, 2, true, "test", Arrays.asList("inner",
4ß)});
final Client client = new ClientImpl(config);
client.enqueue("foo", job);
client.end();

// Start a worker to run jobs from the queue
final Worker worker = new WorkerImpl(config,
Arrays.asList("foo"), Arrays.asList(TestAction.class));
final Thread workerThread = new Thread(worker);
workerThread.start();

Resque?

• Multiple queues
• Potentially HUGE queues
• Admin UI
• Expecting chaos/failures

DelayedJob?

• Numeric priorities
• Queue is small
• Add whole objects to queue
• No need to setup Redis

Resources
github.com/defunkt/resque

redis.io

redistogo.com

Background Jobs with Resque

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Background Jobs with Resque (20)

Recently uploaded (20)

Background Jobs with Resque

Editor's Notes