![Using a Different Queue
As easy as:
●
Global:
$conf['queue_default_class'] = 'MarconiQueue';
●
Specific queue
$conf['queue_class_{queue_name}'] = 'MarconiQueue';](https://image.slidesharecdn.com/message-queues-131121143913-phpapp01/85/Message-Queues-and-Drupal-9-320.jpg)
![Queues and Drush
●
drush queue-list
●
drush queue-run [queue_name]](https://image.slidesharecdn.com/message-queues-131121143913-phpapp01/85/Message-Queues-and-Drupal-10-320.jpg)
![Code.
function my_module_process_queue_item() {
$queue = DrupalQueue::get('my_queue');
$item = $queue>claimItem(30);
if ($item && $item['my_key'] == 'my_value') {
$queue>deleteItem($item);
}
else {
$queue>releaseItem($item);
}
}](https://image.slidesharecdn.com/message-queues-131121143913-phpapp01/85/Message-Queues-and-Drupal-15-320.jpg)
![Code.
function my_module_cron_queue_info() {
$queues['my_queue'] = array(
'worker callback' =>
'my_module_process_queue_item',
'time' => 60,
// Time for cron to spend on this queue, not
// the lease time.
);
return $queues;
}](https://image.slidesharecdn.com/message-queues-131121143913-phpapp01/85/Message-Queues-and-Drupal-16-320.jpg)
Message Queues and Drupal
- 1. Message Queues and Drupal
- 2. About Me ● ● ● ● Name: Brian Altenhofel IRC: VeggieMeat Twitter: @BrianAltenhofel Drupal: #550616 What I Do: Commerce, Third-party APIs, Migrations, Devops, anything with data... Favorite Tools: Vim, Scotch, Padron Serie 1926 No. 1 Maduro
- 4. What are message queues?
- 5. Consumer Producer Producer Queue Consumer Consumer Producer What are message queues? Publisher Publisher Publisher Subscriber Queue Subscriber Subscriber
- 6. You already use queues.
- 7. In Drupal...
- 8. Queues are pluggable Backends available on Drupal.org: AWS SQS – http://drupal.org/project/aws_sqs Beanstalkd – http://drupal.org/project/beanstalkd Redis – http://drupal.org/project/redis STOMP – http://drupal.org/project/stomp (And.... still in sandbox but ready to go...) Openstack Marconi http://drupal.org/sandbox/BrianAltenhofel/2107851
- 9. Using a Different Queue As easy as: ● Global: $conf['queue_default_class'] = 'MarconiQueue'; ● Specific queue $conf['queue_class_{queue_name}'] = 'MarconiQueue';
- 10. Queues and Drush ● drush queue-list ● drush queue-run [queue_name]
- 11. Code.
- 17. It's really that simple.
- 18. Debugging ● watchdog() is a Drupaler's best friend ● Log everything (for both your sanity and insanity) ● (probably want to ship these logs if you're processing a lot of items) (if you want to get started on that): http://www.youtube.com/watch?v=p0Av29yaBEI
- 19. Monitoring ● Monitor queue depths with your favorite tool (Nagios, Zabbix, Zenoss, etc.) ● In a pinch, you can get an idea with $queue->numberOfItems(). ● Take action if queues get too deep
- 20. Any questions? @BrianAltenhofel IRC: VeggieMeat (MANY channels) brian.altenhofel@vmdoh.com
Editor's Notes
- #4: Commonly used message queues include Redis – RabbitMQ – IronMQ (service) – Amazon SQS/SNS
- #5: - a datastore for messages to be consumed – like a mailbox - used for parallel processing, ensuring that a task is performed without blocking the user experience - scalability – typically think vertical (server size) or horizontal (more servers) – queues allow you to scale time and smooth spikes... great for rate-limited APIs - decoupling
- #6: Many configurations possible... two most common Producer/Consumer – producers create messages, consumers claim them when they begin working on them, release them on failure, delete on completion – i.e. Parallel processing Publisher/Subscriber – publishers create messages, subscribers consume them as long as they are in the queue, message eventually expires and disappears from queue – i.e. Data feeds
- #7: - uploading videos that need to be encoded - GitHub uses queues extensively... most of the work their service does is in the background - Drupal. Several modules implement queues (i.e. Notifications, Batch API) – every time you enable a module, it gets added to a queue to check for updates
- #11: Queue-list will list all available queues to run Queue-run will run them (but will always delete items) – advantage over cron is that cron processes can only be run 1 at a time – queue-run can be invoked as much as you want
- #17: Note: this is only necessary if Drupal is also acting as the consumer/subscriber. This is not necessary is Drupal is just producing or publishing messages to a queue to be consumed by something else.
- #20: Actions: Send alert with PagerDuty... spin up new workers...