Delayed operations with Queues
1 like745 views
This document discusses using queues to process time-consuming jobs asynchronously in Drupal to improve website performance and scalability. It provides examples of use cases for queues like preparing content for other platforms, generating content in advance, handling comment submissions, processing images, and integrating with external data sources. It also describes some implementations of queue APIs in Drupal and common queue runners as well as limitations of the current queue APIs.
![Queue API methods: Queue
QueueInterface
• Q::createItem(mixed $data: void
• Q::claimItem($lease_time = 3600: mixed $item
FALSE | stdClass + [item_id => int, data => mixed, created => timestamp]
$lease_time → Assumptions for runner, currently not used
• Q::deleteItem($item): void -> work done
• Q::releaseItem($item): bool
• Q::numberOfItems(): int → best guess, unreliable
• Q::createQueue() / Q::deleteQueue()
ReliableQueueInterface: ordering, single execution](https://image.slidesharecdn.com/queue-lviv-camp-151019072937-lva1-app6891/85/Delayed-operations-with-Queues-14-320.jpg)
Delayed operations with Queues
- 1. Delayed operations with queues Yuriy Gerasimov Frédéric G. Marand
- 2. Yuriy Gerasimov ygerasimov • FFW • Drupal architect & developer • Contrib 7 modules: services, draggableviews • Founder at Backtrac.io
- 3. Frédéric G. Marand fgm • OSInet: performance/architecture consulting • for internal teams at larger accounts • Core contributor 4.7 to 8.0.x, MongoDB + XMLRPC maintainer + others • Already 4 D8 customer projects before 8.0.0 • Customer D8 in production since 07/2015 • Frequently adds queueing to larger Drupal projects
- 4. Why use queues ? To have websites which are : • Faster for visitors • Snappier for editors • More scaleable To process time-consuming jobs : • Video encoding • High-resolution gallery uploads and processing
- 5. Concrete use cases • Prepare content for non-Drupal front-ends • Anticipate content generation • Deferred submits, e.g. comments handling • Slow operations: node saves, previews, image processing • External data sources: pull, push • Multi-step operations: batch
- 11. Image processing
- 12. Job servers • How to get results • Rerun failed jobs • Separate queue for failed jobs • Monitoring queues, workers • Supervisor
- 14. Queue API methods: Queue QueueInterface • Q::createItem(mixed $data: void • Q::claimItem($lease_time = 3600: mixed $item FALSE | stdClass + [item_id => int, data => mixed, created => timestamp] $lease_time → Assumptions for runner, currently not used • Q::deleteItem($item): void -> work done • Q::releaseItem($item): bool • Q::numberOfItems(): int → best guess, unreliable • Q::createQueue() / Q::deleteQueue() ReliableQueueInterface: ordering, single execution
- 15. Queue Runners Core / Contrib • Core Cron / Elysia Cron / Queue_Runner • Drush: queue-list / queue-run Similar limitations: • Default on in D6 / D7, default off in D8 • Single threaded, single process across queues Custom runners • Provided by queue modules or per-project one-offs • Preemption, parallel execution...
- 16. Queue API limitations • Exception handling • Priority management • Tagging • Delay, burying ... • No monitoring, peek(), deduplication
- 17. Questions ?