Background Processing - PyCon MY 2015
Download as PPTX, PDF2 likes942 views
This document discusses using background processing to improve response times for slow webpages and views. It provides an overview of common issues like slow database writes or email sending causing timeouts. Short term solutions like "spinning" are discussed, as well as breaking tasks into preparation and processing stages to run slow tasks asynchronously. Django-background-tasks and Django-RQ are presented as options for implementing background tasks in Django, along with pros and cons. Code samples and instructions are provided for setting up and running background tasks with each.
![Why timeout?
In Linux, we use NGINX – GUNICORN - DJANGO
There is a limit of exact concurrent request that a
server can handle.
General formula for no. of workers is:
(2 x [number of cores]) + 1
Therefore, 8 cores = (2 x 8) + 1 = 17
Slow response = handle less requests per second](https://image.slidesharecdn.com/pyconmy-2015-backgroundprocessing-150823175528-lva1-app6892/85/Background-Processing-PyCon-MY-2015-7-320.jpg)
![Django Background Tasks
- background processing code
import time
from background_task import background
@background(schedule=0)
def dbt_task(data):
print "[START] django-background-tasks for: " + data.get("name","")
time.sleep(10)
print "[DONE] django-background-tasks for: " + data.get("name","")](https://image.slidesharecdn.com/pyconmy-2015-backgroundprocessing-150823175528-lva1-app6892/85/Background-Processing-PyCon-MY-2015-14-320.jpg)
![django-rq
- background processing code
import time
def rq_task(data):
print "[START] rq for: " + data.get("name","")
time.sleep(10)
print "[DONE] rq for: " + data.get("name","")](https://image.slidesharecdn.com/pyconmy-2015-backgroundprocessing-150823175528-lva1-app6892/85/Background-Processing-PyCon-MY-2015-18-320.jpg)
Background Processing - PyCon MY 2015
- 1. BACKGROUND PROCESSING TO IMPROVE RESPONSE TIME by Chew Kok Hoor kokhoor@solutionx.com.
- 2. Python @ SolutionX Django Web Applications Django Mobile Apps backend Django-based Point-of-Sales system (POS) Device Connectors
- 3. Issue Issue: Webpages / Views that takes time to response Contact Us Page Registration Page Possible causes: Sending email Slow writes to database, business rules etc.
- 4. What happens next Users / Visitors / Customer complains Boss ask us to resolve it, ASAP.
- 5. Short Term Resolution We spin while we can
- 6. What happens next after what happened next earlier More Users / Visitors / Customers Timeout errors
- 7. Why timeout? In Linux, we use NGINX – GUNICORN - DJANGO There is a limit of exact concurrent request that a server can handle. General formula for no. of workers is: (2 x [number of cores]) + 1 Therefore, 8 cores = (2 x 8) + 1 = 17 Slow response = handle less requests per second
- 9. Pros n Cons Disadvantages New process required On failure unable to notify users thru response to request. Worse case, lost contact to user who submitted request. Need to monitor tasks not done / failed Advantages Timeout issue resolved Users get quicker response Can choose to process N tasks at a time Can retry tasks etc., background process can afford slower response
- 10. What should be done? 1. Break down tasks of slow responding requests 2. Process tasks in background Preparation Stage Perform slow running task in Background Submit task for processing
- 11. Tasks Breakdown Preparation Stage record necessary information for task to execute properly or for user to track status. activities to prevent inconsistency or breaking of business rules preferably a means of contacting user when it is done (email, mobile notification, etc.) Slow Running tasks Email sending Create and setup new database Setup new user login roles, etc. Complex calculations or setup
- 12. Choosing Task Queue software Software Pro Cons django-background- tasks (https://pypi.python.org/ pypi/django- background-tasks) - Very easy setup - Easy to use - Good for small volume - Uses Django ORM for backend - Absolutely terrible for medium-to-high volume RQ (http://python-rq.org/) (https://github.com/ui/dj ango-rq) - Flexible, powered by Redis - Lower memory footprint than Celery - Great for high volume - Not as many features as Celery - Medium difficulty setup - Redis only option for storage Celery (http://www.celeryprojec t.org/) - De facto Django standard - Many different storage types - Flexible, - Full-featured - Great for high volume - Challenging setup - overkill for slower traffic sites From Two Scoops of Django - Best Practices For Django 1.8 by Daniel Roy Greenfeld & Audrey Roy Greenfeld (http://twoscoopspress.org/) Choosing Task Queue Software
- 13. Django Background Tasks - background processing code INSTALLATION pip install django-background-tasks settings.py INSTALLED_APPS += ( ‘background_task’, )
- 14. Django Background Tasks - background processing code import time from background_task import background @background(schedule=0) def dbt_task(data): print "[START] django-background-tasks for: " + data.get("name","") time.sleep(10) print "[DONE] django-background-tasks for: " + data.get("name","")
- 15. Django Background Tasks - initiate background task from . import tasks tasks.dbt_task(request.POST.copy())
- 16. Django Background Tasks - start worker (only 1 allowed) To start background processing process: python manage.py process_tasks
- 17. django-rq - background processing code INSTALLATION pip install django-rq settings.py INSTALLED_APPS += ( 'django_rq', ) RQ_QUEUES = { 'default': { 'HOST': 'localhost', 'PORT': 6379, 'DB': 0, 'DEFAULT_TIMEOUT': 360, } }
- 18. django-rq - background processing code import time def rq_task(data): print "[START] rq for: " + data.get("name","") time.sleep(10) print "[DONE] rq for: " + data.get("name","")
- 19. django-rq - initiate background task from . import rq_tasks import django_rq django_rq.enqueue(rq_tasks.rq_task, request.POST.copy())
- 20. django-rq - start worker(s) First, start redis (e.g. /usr/local/bin/redis-server) To start background processing process (supports more than 1 worker(s)) python manage.py rqworker default
- 21. Demo
- 22. Other Practical Usage of Background Processing Not just for response time Scheduling (require additional components, e.g. RQ Scheduler) Enforce a separation between front-end logic and back-end logic Batch processing (daily import / export of data) Reports generation Maintenance - expiring old sessions, sweeping caches
- 23. FAQ Chew Kok Hoor Director SolutionX Software Sdn Bhd kokhoor@solutionx.com.my