Concurrency models in python

z
Concurrency
Models in Python
And how to choose between them
Isac Casapu
isac@pricesearcher.com

z
Why concurrency?
 Multiple, distinct activities that are time consuming
 Processing intensive (CPU bound)
 Waiting on I/O (storage, network)

z
Concurrency Strategies
 Multiple Processes
 Share nothing by default
 Multiple Threads
 Potentially run on multiple CPUs
 All memory and context by default shared
 Asynchronous, Cooperative Multitasking
 Submit time-consuming requests and get called when done
 Typically runs in a single OS thread

z
Async vs Multi-Threading
ExecutionTime
Source: Python Async - Leonardo Rossetti

z
What Async code looks like
 Explicitly indicate functions called asynchronously
 Explicitly relinquish control
 Change any code that blocks to Async equivalent
 E.g. aiohttp, aiobotocore, aiomysql…
async def crawl(self):
await self.request_queue.load_robots_file(self.base_url)
self.robots_file = self.request_queue.robots_file
self.request_queue.enqueue(self.base_url)
await self.request_queue.run()
return
Excerpt from a PoC crawler written using AioHTTP.

z
Multithreading in Python
 Python interpreter is single-threaded
 1 thread at a time can execute pure
Python code
 Native code can work concurrently
 CPU bound: Numeric, XML parsing
 Making system calls
 Anything else in C, Fortran etc.
 Alternative VMs are thread safe:
 .Net (IronPython), JVM (Jython)

z
Recommendations
Your
Workload
I/O
Bound
?
Mostly
Native
?
No
Multithreading
Library
Use?
Yes
Yes
AsyncIO
No
Yes
Multiprocessing
No

z
More Info
Presentations going into greater depth:
 Introduction to Python Asyncio
 Enhance Your Python Code Beyond GIL
Code examples for asynchronous programming:
 A simple web crawler using aiohttp
 Capturing streaming output from a subprocess using AsyncIO
Isac Casapu
isac@pricesearcher.com

Concurrency models in python

More Related Content

What's hot (20)

Similar to Concurrency models in python (20)

Recently uploaded (20)

Concurrency models in python