![n_documents = 80000!
batch_size = 5000!
batch = []!
!
import time!
start = time.time()
The Python Code](https://image.slidesharecdn.com/pythonprofilinggutsandglorytaipei-140529130936-phpapp01/85/Python-Performance-Profiling-The-Guts-And-The-Glory-6-320.jpg)
![What?!
The Python Code
for i in range(n_documents):!
date = datetime.fromtimestamp(!
time.mktime(min_date.timetuple())!
+ int(round(random.random() * delta)))!
!
value = random.random()!
document = {!
'created_on': date,!
'value': value}!
!
batch.append(document)!
if len(batch) == batch_size:!
collection.insert(batch)!
batch = []!](https://image.slidesharecdn.com/pythonprofilinggutsandglorytaipei-140529130936-phpapp01/85/Python-Performance-Profiling-The-Guts-And-The-Glory-8-320.jpg)
![for index in range(n_documents):!
date = datetime.fromtimestamp(!
time.mktime(min_date.timetuple())!
+ int(round(random.random() * delta)))!
!
value = random.random()!
document = {!
'created_on': date,!
'value': value}!
!
batch.append(document)!
if len(batch) == batch_size:!
collection.insert(batch)!
batch = []!
The Python Code
one third
of the time](https://image.slidesharecdn.com/pythonprofilinggutsandglorytaipei-140529130936-phpapp01/85/Python-Performance-Profiling-The-Guts-And-The-Glory-19-320.jpg)
![for index in range(n_documents):!
date = datetime.now()!
!
!
!
value = random.random()!
document = {!
'created_on': date,!
'value': value}!
!
batch.append(document)!
if len(batch) == batch_size:!
collection.insert(batch)!
batch = []!
The Python Code](https://image.slidesharecdn.com/pythonprofilinggutsandglorytaipei-140529130936-phpapp01/85/Python-Performance-Profiling-The-Guts-And-The-Glory-20-320.jpg)
Python Performance Profiling: The Guts And The Glory
- 1. Python Profiling: A. Jesse Jiryu Davis @jessejiryudavis MongoDB The Glory & The Guts
- 3. “PyMongo is slower! compared to the JavaScript version” MongoDB Node.js driver:!88,000 per second PyMongo: ! ! ! ! ! ! ! ! ! 29,000 per second
- 4. “Why Is PyMongo Slower?” From:!steve@mongodb.com! To:!! jesse@mongodb.com! CC:!! eliot@mongodb.com Hi Jesse,! ! Why is the Node MongoDB driver 3 times! faster than PyMongo?! http://dzone.com/articles/mongodb-facts-over-80000
- 5. The Python Code # Obtain a MongoDB collection.! import pymongo! ! client = pymongo.MongoClient('localhost')! db = client.random! collection = db.randomData! collection.remove()!
- 6. n_documents = 80000! batch_size = 5000! batch = []! ! import time! start = time.time() The Python Code
- 7. import random! from datetime import datetime! ! min_date = datetime(2012, 1, 1)! max_date = datetime(2013, 1, 1)! delta = (max_date - min_date).total_seconds()! The Python Code
- 8. What?! The Python Code for i in range(n_documents):! date = datetime.fromtimestamp(! time.mktime(min_date.timetuple())! + int(round(random.random() * delta)))! ! value = random.random()! document = {! 'created_on': date,! 'value': value}! ! batch.append(document)! if len(batch) == batch_size:! collection.insert(batch)! batch = []!
- 9. duration = time.time() - start! ! print 'inserted %d documents per second' % (! n_documents / duration)! The Python Code inserted 30,000 documents per second
- 10. The Node.js Code (not shown)
- 11. The Question Why is the Python script 3 times slower than the equivalent Node script?
- 12. Why Profile? • Optimization is like debugging • Hypothesis: “The following change will yield a worthwhile improvement.” • Experiment • Repeat until fast enough
- 13. Why Profile? Profiling is a way to generate hypotheses.
- 14. Which Profiler? • cProfile • GreenletProfiler • Yappi
- 16. Yappi Compared to cProfile, it is: ! • As fast • Also measures functions • Can measure CPU time, not just wall • Can measure all threads • Can export to callgrind
- 17. Yappi import yappi! ! yappi.set_clock_type('cpu')! yappi.start(builtins=True)! ! start = time.time()! ! for i in range(n_documents):! # ... same code ... ! ! duration = time.time() - start! stats = yappi.get_func_stats()! stats.save('callgrind.out', type='callgrind')! Same code as before
- 18. KCacheGrind
- 19. for index in range(n_documents):! date = datetime.fromtimestamp(! time.mktime(min_date.timetuple())! + int(round(random.random() * delta)))! ! value = random.random()! document = {! 'created_on': date,! 'value': value}! ! batch.append(document)! if len(batch) == batch_size:! collection.insert(batch)! batch = []! The Python Code one third of the time
- 20. for index in range(n_documents):! date = datetime.now()! ! ! ! value = random.random()! document = {! 'created_on': date,! 'value': value}! ! batch.append(document)! if len(batch) == batch_size:! collection.insert(batch)! batch = []! The Python Code
- 21. The Python Code • Before: 30,000 inserts per second • After: 50,000 inserts per second
- 22. Why Profile? • Generate hypotheses • Estimate possible improvement
- 23. How Does Profiling Work? int callback(PyFrameObject *frame,! int what,! PyObject *arg);! int start(void)! {! PyEval_SetProfile(callback);! }!
- 24. PyObject *! PyEval_EvalFrameEx(PyFrameObject *frame)! {! if (tstate->c_profilefunc != NULL) {! tstate->c_profilefunc(frame,! PyTrace_CALL,! Py_None);! }! ! /* ... execute bytecode in the frame! * until return or exception... */! ! if (tstate->c_profilefunc != NULL) {! tstate->c_profilefunc(frame,! PyTrace_RETURN,! retval);! }! }!
- 25. int callback(PyFrameObject *frame,! int what,! PyObject *arg)! {! switch (what) {! case PyTrace_CALL:! {! PyCodeObject *cobj = frame->f_code;! PyObject *filename = cobj->co_filename;! PyObject *funcname = cobj->co_name;! ! /* ... record the function call ... */! }! break;! ! /* ... other cases ... */! ! }! }!
- 26. A. Jesse Jiryu Davis @jessejiryudavis MongoDB