Solving performance issues in Django ORM
1 like170 views
This document summarizes techniques for optimizing performance when working with large datasets in Django. It discusses using select_related, prefetch_related, values, and values_list to retrieve data in a lean way without entire model objects. It also covers string aggregation, setdefault, and get for serializing to-many relationships efficiently. The goal is to retrieve and serialize data from the database with as few queries as possible for better performance with large amounts of data.
![skip model object
slim data retrieval
.values()
# List of 10k dictionaries
form_qs = Form.objects.values(
'application_id',
'author__name',
'categories__name',
)
for form in form_qs:
print(
f"{form['application_id']}, "
f"{form['author__name']}, "
f"{form['categories__name']}"
)
slim + quick](https://image.slidesharecdn.com/solvingperformanceissuesindjangoorm-sianlerk-190826144803/85/Solving-performance-issues-in-Django-ORM-41-320.jpg)
![print(connection.queries[-1])
# actual query performed, with time taken
{'sql': "QUERY = 'SELECT ...', 'time': '0.009'}](https://image.slidesharecdn.com/solvingperformanceissuesindjangoorm-sianlerk-190826144803/85/Solving-performance-issues-in-Django-ORM-46-320.jpg)
![form_qs = Form.objects.values('id', 'categories__name')
serialized_forms = {}
for id, category in form_qs:
serialized_form = serialized_forms.setdefault(
id,
{},
)
categories = serialized_form.get('categories', [])
categories.append(category)
setdefault x get](https://image.slidesharecdn.com/solvingperformanceissuesindjangoorm-sianlerk-190826144803/85/Solving-performance-issues-in-Django-ORM-62-320.jpg)
Solving performance issues in Django ORM
- 1. SOLVING PERF ISSUES IN . ORMDJANGO
- 3. A little bit about Jewel… We’re a Singapore-based company with offices in KL and HK too. We’re hiring! We’re always looking for talented developers, data scientists, data engineers, devops engineers and more! Check out our Careers page at https://www.jewelpaymentech.com/careers.html Image Jewel Paymentech © Copyright 2018 Jewel Paymentech Pte .Ltd. – Reproduction and distribution of this presentation without written permission is prohibited. VISION
- 4. JOIN THE TEAM Python Developers React Developers Data Scientists Data Engineers and many more… Scan the QR code to visit our Careers page:
- 5. (UN)ORTHODOX THE JOURNEY OF AN OPTIMIZATION
- 6. Jafnee Jasmee Willy CahyadiSwee Meng
- 8. M V P
- 9. M V P
- 10. TOOLS (TO GET US STARTED)
- 21. just works
- 22. just works
- 25. #the10kreality
- 26. REALITY HURTS (if not your eyes)
- 27. setting baseline
- 28. #the10kdream
- 30. lazy load 1
- 31. # 10k Records found Form.objects.count() # How many queries made? for form in Form.objects.all(): print(form.author.name)
- 32. # 10k Records found Form.objects.count() # How many queries made? for form in Form.objects.all(): print(form.author.name) # 1+10k queries made! Big-O Notation
- 33. # 10k Records found Form.objects.count() # How many queries made? for form in Form.objects.all(): print(form.author.name) # 1+10k queries made! Big-O Notation O(n+1)
- 35. one-to-one one-to-many .select_related() # One query made for form in Form.objects.select_related('author'): print(form.author.name) O(1)
- 36. many-to-many .prefetch_related() # Two queries made, join done at python level for form in Form.objects.prefetch_related('categories'): for category in form.categories.all(): print(category.name) O(2)
- 37. model object 2
- 38. #expensive
- 39. skip model object slim data retrieval
- 40. # 10k Records found Form.objects.count() # List of 10k model objects for form in Form.objects.select_related('author'): print(book.author.name) fat + slow
- 41. skip model object slim data retrieval .values() # List of 10k dictionaries form_qs = Form.objects.values( 'application_id', 'author__name', 'categories__name', ) for form in form_qs: print( f"{form['application_id']}, " f"{form['author__name']}, " f"{form['categories__name']}" ) slim + quick
- 42. skip model object slim data retrieval .values_list() # List of 10k tuples form_qs = Form.objects.values_list( 'application_id', 'author__name', 'categories__name', ) for application_id, author, categories in form_qs: print(f'{application_id}, {author}, {categories}') slim + quick
- 43. skip model object slim data retrieval .values_list() # List of 10k tuples form_qs = Form.objects.values_list( 'application_id', flat=True, ) for application_id in form_qs: print(f'{application_id}') slim + quick
- 44. prove to me
- 45. from django.db import connection # prints 0 print(len(connection.queries)) # How many queries made? forms = list(Form.objects.values_list('id', 'categories__name')) # prints x? print(len(connection.queries))
- 46. print(connection.queries[-1]) # actual query performed, with time taken {'sql': "QUERY = 'SELECT ...', 'time': '0.009'}
- 48. #slow
- 50. return a dict
- 51. #eww
- 53. #okay...
- 57. SELECT form.id AS form, ... STRING_AGG(category.name, ',') AS categories, ... GROUP BY form ... string aggregation
- 58. SELECT form.id AS form, ... STRING_AGG(category.name, ',') AS categories, ... GROUP BY form ... string aggregation
- 60. #...
- 61. setdefault get
- 62. form_qs = Form.objects.values('id', 'categories__name') serialized_forms = {} for id, category in form_qs: serialized_form = serialized_forms.setdefault( id, {}, ) categories = serialized_form.get('categories', []) categories.append(category) setdefault x get
- 63. #jobdone
- 64. disclaimer