Generators: The Final Frontier

Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Generators:
The Final Frontier
David Beazley (@dabeaz)
http://www.dabeaz.com
Presented at PyCon'2014, Montreal
1

Previously on Generators
2
• GeneratorTricks for Systems Programmers (2008)
http://www.dabeaz.com/generators/
• A ﬂying leap into generator awesomeness

Previously on Generators
3
• A Curious Course on Coroutines and Concurrency (2009)
http://www.dabeaz.com/coroutines/
• Wait, wait? There's more than iteration?

Today's Installment
4
• Everything else you ever wanted to know
about generators, but were afraid to try
• Part 3 of a trilogy

Requirements
5
• You need Python 3.4 or newer
• No third party extensions
• Code samples and notes
http://www.dabeaz.com/ﬁnalgenerator/
• Follow along if you dare!

Disclaimer
6
• This is an advanced tutorial
• Assumes general awareness of
• Core Python language features
• Iterators/generators
• Decorators
• Common programming patterns
• I learned a LOT preparing this

Will I Be Lost?
7
• Although this is the third part of a series, it's
mostly a stand-alone tutorial
• If you've seen prior tutorials, that's great
• If not, don't sweat it
• Be aware that we're focused on a speciﬁc use of
generators (you just won't get complete picture)

Focus
8
• Material in this tutorial is probably not
immediately applicable to your day job
• More thought provoking and mind expanding
• from __future__ import future
practical
utility
bleeding
edge

Part I
9
Preliminaries - Generators and Coroutines
(rock)

Generators 101
• yield statement deﬁnes a generator function
10
def countdown(n):
while n > 0:
yield n
n -= 1
• You typically use it to feed iteration
for x in countdown(10):
print('T-minus', x)
• A simple, yet elegant idea

Under the Covers
11
• Generator object runs in response to next()
>>> c = countdown(3)
>>> c
<generator object countdown at 0x10064f900>
>>> next(c)
3
>>> next(c)
2
>>> next(c)
1
>>> next(c)
Traceback (most recent call last):
File "<stdin>", line 1, in ?
StopIteration
>>>
• StopIteration raised when function returns

Interlude
12
• Generators as "iterators" misses the big picture
• There is so much more to yield

Generators as Pipelines
• Stacked generators result in processing pipelines
• Similar to shell pipes in Unix
13
generator
input
sequence
for x in s:generator generator
def process(sequence):
for s in sequence:
... do something ...
yield item
• Incredibly useful (see prior tutorial)

Coroutines 101
• yield can receive a value instead
14
def receiver():
while True:
item = yield
print('Got', item)
• It deﬁnes a generator that you send things to
recv = receiver()
next(recv) # Advance to first yield
recv.send('Hello')
recv.send('World')

Coroutines and Dataﬂow
15
• Coroutines enable dataﬂow style processing
source coroutine
coroutine
send() send()
• Publish/subscribe, event simulation, etc.
coroutine
coroutine
send()
send()
coroutine
send()

Fundamentals
• The yield statement deﬁnes a generator function
16
def generator():
...
... yield ...
...
• The mere presence of yield anywhere is enough
• Calling the function creates a generator instance
>>> g = generator()
>>> g
<generator object generator at 0x10064f120>
>>>

Advancing a Generator
• next(gen) - Advances to the next yield
17
def generator():
...
...
yield item
...
• Returns the yielded item (if any)
• It's the only allowed operation on a newly
created generator
• Note: Same as gen.__next__()

Sending to a Generator
• gen.send(item) - Send an item to a generator
18
def generator():
...
item = yield
...
...
yield value
• Wakes at last yield, returns sent value
• Runs to the next yield and emits the value
g = generator()
next(g) # Advance to yield
value = g.send(item)

Closing a Generator
• gen.close() - Terminate a generator
19
def generator():
...
try:
yield
except GeneratorExit:
# Shutting down
...
• Raises GeneratorExit at the yield
• Only allowed action is to return
• If uncaught, generator silently terminates
g = generator()
g.close() # Terminate

Raising Exceptions
• gen.throw(typ [, val [,tb]]) - Throw exception
20
def generator():
...
try:
yield
except RuntimeError as e:
...
...
yield val
• Raises exception at yield
• Returns the next yielded value (if any)
g = generator()
val = g.throw(RuntimeError,
'Broken')

Generator ReturnValues
• StopIteration raised on generator exit
21
def generator():
...
yield
...
return result
• Return value (if any) passed with exception
• Note: Python 3 only behavior (in Python 2,
generators can't return values)
g = generator()
try:
next(g)
except StopIteration as e:
result = e.value

Generator Delegation
• yield from gen - Delegate to a subgenerator
22
def generator():
...
yield value
...
return result
def func():
result = yield from generator()
• Allows generators to call other generators
• Operations take place at the current yield
• Return value (if any) is returned

Delegation Example
• Chain iterables together
23
def chain(x, y):
yield from x
yield from y
• Example:
>>> a = [1, 2, 3]
>>> b = [4, 5, 6]
>>> for x in chain(a, b):
... print(x,end=' ')
...
1 2 3 4 5 6
>>> c = [7,8,9]
>>> for x in chain(a, chain(b, c)):
... print(x, end=' ')
...
1 2 3 4 5 6 7 8 9
>>>

Mini-Reference
• Generator instance operations
24
gen = generator()
next(gen) # Advance to next yield
gen.send(item) # Send an item
gen.close() # Terminate
gen.throw(exc, val, tb) # Raise exception
result = yield from gen # Delegate
• Using these, you can do a lot of neat stuff
• Generator deﬁnition
def generator():
...
yield
...
return result

Part 2
25
and now for something completely different

A Common Motif
• Consider the following
26
f = open()
...
f.close()
lock.acquire()
...
lock.release()
db.start_transaction()
...
db.commit()
start = time.time()
...
end = time.time()
• It's so common, you'll see it everywhere!

Context Managers
• The 'with' statement
27
with open(filename) as f:
statement
statement
...
with lock:
statement
statement
...
• Allows control over entry/exit of a code block
• Typical use: everything on the previous slide

Context Management
• It's easy to make your own (@contextmanager)
28
import time
from contextlib import contextmanager
@contextmanager
def timethis(label):
start = time.time()
try:
yield
finally:
end = time.time()
print('%s: %0.3f' % (label, end-start)
• This times a block of statements

Context Management
29
• Usage
with timethis('counting'):
n = 1000000
while n > 0:
n -= 1
• Output
counting: 0.023

Context Management
• Another example: temporary directories
30
import tempfile, shutil
@contextmanager
def tempdir():
outdir = tempfile.mkdtemp()
try:
yield outdir
finally:
shutil.rmtree(outdir)
• Example
with tempdir() as dirname:
...

Whoa,Whoa, Stop!
• Another example: temporary directories
31
@contextmanager
def tempdir():
outdir = tempfile.mkdtemp()
try:
yield outdir
finally:
shutil.rmtree(outdir)
• Example
...
What is this?
• Not iteration
• Not dataﬂow
• Not concurrency
• ????

Context Management
• Under the covers
32
with obj:
statements
statements
statements
...
statements
• If an object implements these methods it can
monitor entry/exit to the code block
obj.__enter__()
obj.__exit__()

Context Manager
• Implementation template
33
class Manager(object):
def __enter__(self):
return value
def __exit__(self, exc_type, val, tb):
if exc_type is None:
return
else:
# Handle an exception (if you want)
return True if handled else False
• Use:
with Manager() as value:
statements
statements

Context Manager Example
• Automatically deleted temp directories
34
import tempfile
import shutil
class tempdir(object):
self.dirname = tempfile.mkdtemp()
return self.dirname
def __exit__(self, exc, val, tb):
shutil.rmtree(self.dirname)
• Use:
...

Alternate Formulation
• @contextmanager is just a reformulation
35
@contextmanager
def tempdir():
dirname = tempfile.mkdtemp()
try:
yield dirname
finally:
shutil.rmtree(dirname)
• It's the same code, glued together differently

Deconstruction
• How does it work?
36
@contextmanager
def tempdir():
try:
yield dirname
finally:
• Think of "yield" as scissors
• Cuts the function in half

Deconstruction
• Each half maps to context manager methods
37
@contextmanager
def tempdir():
try:
yield dirname
statements
statements
statements
...
finally:
__enter__
__exit__
user statements ('with' block)
• yield is the magic that makes it possible

Deconstruction
• There is a wrapper class (Context Manager)
38
class GeneratorCM(object):
def __init__(self, gen):
self.gen = gen
...
...
• And a decorator
def contextmanager(func):
def run(*args, **kwargs):
return GeneratorCM(func(*args, **kwargs))
return run

Deconstruction
• enter - Run the generator to the yield
39
self.gen = gen
return next(self.gen)
...
• It runs a single "iteration" step
• Returns the yielded value (if any)

Deconstruction
• exit - Resumes the generator
40
...
def __exit__(self, etype, val, tb):
try:
if etype is None:
next(self.gen)
else:
self.gen.throw(etype, val, tb)
raise RuntimeError("Generator didn't stop")
except StopIteration:
return True
except:
if sys.exc_info()[1] is not val: raise
• Either resumes it normally or raises exception

Full Disclosure
• Actual implementation is more complicated
• There are some nasty corner cases
• Exceptions with no associated value
• StopIteration raised inside a with-block
• Exceptions raised in context manager
• Read source and see PEP-343
41

Discussion
• Why start with this example?
• A completely different use of yield
• Being used to reformulate control-flow
• It simplifies programming for others (easy
definition of context managers)
• Maybe there's more... (of course there is)
42

Part 3
43
Call me, maybe

Part 3
44
Call me, maybe

Async Processing
• Consider the following execution model
45
def func(args):
...
...
...
return result
run_async(func, args)
main thread
• Examples: Run in separate process or thread,
time delay, in response to event, etc.

Example:Thread Pool
46
from concurrent.futures import ThreadPoolExecutor
def func(x, y):
'Some function. Nothing too interesting'
import time
time.sleep(5)
return x + y
pool = ThreadPoolExecutor(max_workers=8)
fut = pool.submit(func, 2, 3)
r = fut.result()
print('Got:', r)
• Runs the function in a separate thread
• Waits for a result

Futures
47
>>> fut = pool.submit(func, 2, 3)
>>> fut
<Future at 0x1011e6cf8 state=running>
>>>
• Future - A result to be computed later
• You can wait for the result to return
>>> fut.result()
5
>>>
• However, this blocks the caller

Futures
48
def run():
fut.add_done_callback(result_handler)
def result_handler(fut):
result = fut.result()
print('Got:', result)
• Alternatively, you can register a callback
• Triggered upon completion

Exceptions
49
>>> fut = pool.submit(func, 2, 'Hello')
>>> fut.result()
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.4/concurrent/futures/_base.py",
line 395, in result
return self.__get_result()
File "/usr/local/lib/python3.4/concurrent/futures/_base.py",
line 354, in __get_result
raise self._exception
File "/usr/local/lib/python3.4/concurrent/futures/thread.py",
line 54, in run
result = self.fn(*self.args, **self.kwargs)
File "future2.py", line 6, in func
return x + y
TypeError: unsupported operand type(s) for +: 'int' and 'str'
>>>

Futures w/Errors
50
def run():
try:
except Exception as e:
print('Failed: %s: %s' % (type(e).__name__, e))
• Error handling with callbacks
• Exception propagates out of fut.result() method

Interlude
51
def run():
try:
• Consider the structure of code using futures
• Meditate on it... focus on the code.
• This seems sort of familiar

Callback Hell?
52
• No, no, no.... keep focusing.

Interlude
53
def entry():
fut.add_done_callback(exit)
def exit(fut):
try:
• What if the function names are changed?
• Wait! This is almost a context manager (yes)

Inlined Futures
54
@inlined_future
def do_func(x, y):
result = yield pool.submit(func, x, y)
run_inline_future(do_func)
• Thought: Maybe you could do that yield trick
• The extra callback function is eliminated
• Now, just one "simple" function
• Inspired by @contextmanager

DéjàVu
55

DéjàVu
56
• This twisted idea has been used before...

Preview
57
t = Task(gen)
• There are two separate parts
• Part 1:Wrapping generators with a "task"
• Part 2: Implementing some runtime code
run_inline_future(gen)
• Forewarning: It will bend your mind a bit

Commentary
58
• Will continue to use threads for examples
• Mainly because they're easy to work with
• And I don't want to get sucked into an event loop
• Don't dwell on it too much
• Key thing: There is some background processing

Running the Generator
59
def do_func(x, y):
• Problem: Stepping through a generator
• Involves gluing callbacks and yields together
def do_func(x, y):
yield pool.submit(func, x, y)
result =
add_done_callback()
cut
enter
exit

60
class Task:
self._gen = gen
def step(self, value=None):
try:
fut = self._gen.send(value)
fut.add_done_callback(self._wakeup)
except StopIteration as exc:
pass
def _wakeup(self, fut):
! self.step(result)

61
class Task:
self._gen = gen
try:
pass
! self.step(result)
Task class wraps
around and represents
a running generator.

62
class Task:
self._gen = gen
try:
pass
! self.step(result)
Advance the
generator to the next
yield, sending in a
value (if any)

63
class Task:
self._gen = gen
try:
pass
! self.step(result)
Attach a callback to
the produced Future

64
class Task:
self._gen = gen
try:
pass
! self.step(result)
Collect result and
send back into the
generator

Does it Work?
65
def func(x, y):
time.sleep(1)
return x + y
def do_func(x, y):
t = Task(do_func(2, 3))
t.step()
• Try it:
• Output:
Got: 5
• Yes, it works
Note: must initiate
ﬁrst step of the task
to get it to run

Does it Work?
66
def func(x, y):
time.sleep(1)
return x + y
def do_many(n):
while n > 0:
result = yield pool.submit(func, n, n)
n -= 1
t = Task(do_many(10))
t.step()
• More advanced: multiple yields/looping
• Yes, this works too.

Exception Handling
67
class Task:
self._gen = gen
def step(self, value=None, exc=None):
try:
if exc:
fut = self._gen.throw(exc)
else:
pass
try:
self.step(result, None)
except Exception as exc:
self.step(None, exc)

Exception Handling
68
class Task:
self._gen = gen
try:
if exc:
else:
pass
try:
send() or throw()
depending on
success

Exception Handling
69
class Task:
self._gen = gen
try:
if exc:
else:
pass
try:
Catch exceptions
and pass to next
step as appropriate

Error Example
70
def do_func(x, y):
try:
print('Failed:', repr(e))
t = Task(do_func(2, 'Hello'))
t.step()
• Try it:
• Output:
Failed: TypeError("unsupported operand type(s) for +:
'int' and 'str'",)
• Yep, that works too.

Commentary
71
• This whole thing is rather bizarre
• Execution of the inlined future takes place all on
its own (concurrently with other code)
• The normal rules don't apply

Consider
72
def recursive(n):
yield pool.submit(time.sleep, 0.001)
print('Tick:', n)
Task(recursive(n+1)).step()
Task(recursive(0)).step()
• Inﬁnite recursion?
• Output:
Tick: 0
Tick: 1
Tick: 2
...
Tick: 1662773
Tick: 1662774
...

Part 4
73
yield from yield from yield from yield from future
(maybe)
source: @UrsulaWJ

A Singular Focus
74
• Focus on the future
• Not the past
• Not now
• Yes, the future.
• No, really, the future.
(but not the singularity)

A Singular Focus
75
class Task:
self._gen = gen
try:
if exc:
else:
pass
...
generator must only
produce Futures

Puzzler
76
def after(delay, gen):
'''
Run an inlined future after a time delay
'''
yield pool.submit(time.sleep, delay)
yield gen
Task(after(10, do_func(2, 3))).step()
• Can you make library functions?
• It's trying to delay the execution of a user-
supplied inlined future until later.

Puzzler
77
'''
'''
yield gen
• No
...
AttributeError: 'generator' object has no attribute
'add_done_callback'

Puzzler
78
'''
'''
yield gen
• This is busted
• gen is a generator, not a Future

Puzzler (2nd Attempt)
79
'''
'''
for f in gen:
yield f
• What about this?
• Idea: Just iterate the generator manually
• Make it produce the required Futures

Puzzler (2nd Attempt)
80
'''
'''
for f in gen:
yield f
• What about this?
• No luck.The result gets lost somewhere
Got: None
• Hmmm.

Puzzler (3rd Attempt)
81
result = None
try:
while True:
f = gen.send(result)
result = yield f
pass
• Obvious solution (duh!)
• Hey, it works!
Got: 5

Puzzler (3rd Attempt)
82
result = None
try:
while True:
f = gen.send(result)
result = yield f
pass
• Obvious solution (duh!)
• Hey, it works!
Got: 5
manual running of
generator with
results (ugh!)

Puzzler (4th Attempt)
83
yield from gen
• A better solution: yield from
• 'yield from' - Runs the generator for you
Got: 5
• And it works! (yay!)
• Awesome

PEP 380
• yield from gen - Delegate to a subgenerator
84
def generator():
...
yield value
...
return result
def func():
result = yield from generator()
• Transfer control to other generators
• Operations take place at the current yield
• Far more powerful than you might think

Conundrum
85
yield from gen
• "yield" and "yield from"?
• Two different yields in the same function
• Nobody will ﬁnd that confusing (NOT!)

86
yield from pool.submit(time.sleep, delay)
yield from gen
• Maybe this will work?
• Just use 'yield from'- always!

87
yield from gen
• Maybe this will work?
• Just use 'yield from'- always!
• No. 'yield' and 'yield from' not interchangeable:
...
TypeError: 'Future' object is not iterable
>>>

??????
88
(Can it be made to work?)

Iterable Futures
89
def patch_future(cls):
def __iter__(self):
! if not self.done():
yield self
return self.result()
cls.__iter__ = __iter__
from concurrent.futures import Future
patch_future(Future)
• A simple ingenious patch
• It makes all Future instances iterable
• They simply produce themselves and the result
• It magically makes 'yield from' work!

All Roads Lead to Future
90
yield self
• Future is the only thing
that actually yields
• Everything else delegates
using 'yield from'
• Future terminates the
chain
generator
generator
generator
Future
yield from
yield from
yield from
run()
next(gen)

The Decorator
91
import inspect
def inlined_future(func):
assert inspect.isgeneratorfunction(func)
return func
• Generators yielding futures is its own world
• Probably a good idea to have some demarcation
• Does nothing much at all, but serves as syntax
@inlined_future
yield from gen
• Alerts others about what you're doing

Task Wrangling
92
t = Task(gen)
t.step()
• The "Task" object is just weird
• No way to obtain a result
• No way to join with it
• Or do much of anything useful at all
Task
runs
????
t.step()

Tasks as Futures
93
class Task(Future):
!super().__init__()
! self._gen = gen
try:
if exc:
else:
self.set_result(exc.value)
• This tiny tweak makes it much more interesting

Tasks as Futures
94
class Task(Future):
!super().__init__()
! self._gen = gen
try:
if exc:
else:
• This tiny tweak makes it much more interesting
A Task is a Future
Set its result upon
completion

Example
95
@inlined_future
def do_func(x, y):
return result
t = Task(do_func(2,3))
t.step()
...
print("Got:", t.result())
• Obtaining the result of task
• So, you create a task that runs a generator
producing Futures
• The task is also a Future
• Right. Got it.

Example
96
@inlined_future
def do_func(x, y):
return result
t = Task(do_func(2,3))
t.step()
...
print("Got:", t.result())
class Task(Future):
...
try:
...
...

Task Runners
97
def start_inline_future(fut):
t = Task(fut)
t.step()
return t
def run_inline_future(fut):
t = start_inline_future(fut)
return t.result()
• You can make utility functions to hide details
result = run_inline_future(do_func(2,3))
• Example: Run an inline future to completion
• Example: Run inline futures in parallel
t1 = start_inline_future(do_func(2, 3))
t2 = start_inline_future(do_func(4, 5))
result1 = t1.result()

Step Back Slowly
98
• Built a generator-based task system for threads
Tasks
@inline_future
run_inline_future()
Threads
pools
concurrent.future
Futures
submit
result
• Execution of the future hidden in background
• Note: that was on purpose (for now)

asyncio
99
• Ideas are the foundation asyncio coroutines
Tasks
@coroutine
run_until_complete()
Event Loop
asyncio
Futures
result
• In fact, it's almost exactly the same
• Naturally, there are some details with event loop

Simple Example
100
import asyncio
def func(x, y):
return x + y
@asyncio.coroutine
def do_func(x, y):
yield from asyncio.sleep(1)
return func(x, y)
loop = asyncio.get_event_loop()
result = loop.run_until_complete(do_func(2,3))
print("Got:", result)
• asyncio "hello world"

Advanced Example
101
import asyncio
@asyncio.coroutine
def echo_client(reader, writer):
while True:
line = yield from reader.readline()
if not line:
break
resp = b'Got:' + line
writer.write(resp)
writer.close()
loop = asyncio.get_event_loop()
loop.run_until_complete(
asyncio.start_server(echo_client, host='', port=25000)
)
loop.run_forever()
• asyncio - Echo Server

Be on the Lookout!
102

Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com 103
(source: globalpost.com)

Part 5
104
"Gil"

Python Threads
105
• Threads, what are they good for?
• Answer: Nothing, that's what!
• Damn you GIL!!

Actually...
106
• Threads are great at doing nothing!
time.sleep(2) # Do nothing for awhile
data = sock.recv(nbytes) # Wait around for data
data = f.read(nbytes)
• In fact, great for I/O!
• Mostly just sitting around

CPU-Bound Work
107
• Threads are weak for computation
• Global interpreter lock only allows 1 CPU
• Multiple CPU-bound threads ﬁght each other
• Could be better
http://www.dabeaz.com/GIL

A Solution
108
• Naturally, we must reinvent the one thing that
threads are good at
• Namely, waiting around.
• Event-loops, async, coroutines, green threads.
• Think about it: These are focused on I/O
(yes, I know there are other potential issues with threads, but work with me here)

CPU-Bound Work
109
• Event-loops have their own issues
• Don't bug me, I'm blocking right now
(source: chicagotribune.com)

Standard Solution
110
• Delegate the work out to a process pool
python
python
python
python
python
main program
CPU
CPU
CPU
CPU
• multiprocessing, concurrent.futures, etc.

Thought Experiment
111
• Didn't we just do this with inlined futures?
def fib(n):
return 1 if n <= 2 else (fib(n-1) + fib(n-2))
@inlined_future
def compute_fibs(n):
result = []
for i in range(n):
val = yield from pool.submit(fib, i)
result.append(val)
return result
pool = ProcessPoolExecutor(4)
result = run_inline_future(compute_fibs(35))
• It runs without crashing (let's ship it!)

Thought Experiment
112
• Can you launch tasks in parallel?
t1 = start_inline_future(compute_fibs(34))
• Sequential execution
run_inline_future(compute_fibs(34))
• Recall (from earlier)
t = Task(fut)
t.step()
return t

Thought Experiment
113
t = Task(fut)
t.step()
return t
9.56s

Thought Experiment
114
run_inline_future(compute_fibs(34)) 9.56s
t = Task(fut)
t.step()
return t
4.78s
Inlined tasks running
outside conﬁnes of
the GIL?

Execution Model
115
• The way in which it works is a little odd
@inlined_future
result = []
for i in range(n):
print(threading.current_thread())
result.append(val)
return result
add
this
• Output: (2 Tasks)
<_MainThread(MainThread, started 140735086636224)>
<_MainThread(MainThread, started 140735086636224)>
<Thread(Thread-1, started daemon 4320137216)>
...
????

Process Pools
116
• Process pools involve a hidden result thread
main thread python
python
CPU
CPU
submit
result_thread
• result thread reads returned values
• Sets the result on the associated Future
• Triggers the callback function (if any)
results

The Issue
117
• Our inlined future switches execution threads
@inlined_future
result = []
for i in range(n):
result.append(val)
return result
main thread
result thread
• Switch occurs at the ﬁrst yield
• All future execution occurs in result thread
• That could be a little weird (especially if it blocked)

Important Lesson
118
• If you're going to play with control ﬂow, you
must absolutely understand possible implications
under the covers (i.e., switching threads across
the yield statement).

Insight
119
• The yield is not implementation
@inlined_future
result = []
for i in range(n):
result.append(val)
return result
• You can implement different execution models
• You don't have to follow a formulaic rule

Inlined Thread Execution
120
• Variant: Run generator entirely in a single thread
def run_inline_thread(gen):
value = None
exc = None
while True:
try:
if exc:
fut = gen.throw(exc)
else:
fut = gen.send(value)
try:
value = fut.result()
exc = None
exc = e
return exc.value
• It just steps through... no callback function

New Execution
121
• Try it again with a thread pool (because why not?)
@inlined_future
result = []
for i in range(n):
print(threading.current_thread())
result.append(val)
return result
tpool = ThreadPoolExecutor(8)
t1 = tpool.submit(run_inline_thread(compute_fibs(34)))
t2 = tpool.submit(run_inline_thread(compute_fibs(34)))

New Execution
122
• Output: (2 Threads)
<Thread(Thread-1, started 4319916032)>
...
(works perfectly)
4.60s
(a bit faster)
• Processes, threads, and futures in perfect harmony
• Uh... let's move along. Faster. Must go faster.

Big Idea
123
• You can mold and adapt generator execution
• That yield statement: magic!

Part 6
124
Fake it until you make it

Actors
125
• There is a striking similarity between coroutines
and actors (i.e., the "actor" model)
• Features of Actors
• Receive messages
• Send messages to other actors
• Create new actors
• No shared state (messages only)
• Can coroutines serve as actors?

Example
126
• A very simple example
@actor
def printer():
while True:
msg = yield
print('printer:', msg)
printer()
n = 10
while n > 0:
send('printer', n)
n -=1
idea: use generators
to deﬁne a kind of
"named" actor task

Attempt 1
127
_registry = { }
def send(name, msg):
_registry[name].send(msg)
def actor(func):
def wrapper(*args, **kwargs):
gen = func(*args, **kwargs)
next(gen)
_registry[func.__name__] = gen
return wrapper
• Make a central coroutine registry and a decorator
• Let's see if it works...

Example
128
• A very simple example
@actor
def printer():
while True:
msg = yield
print('printer:', msg)
printer()
n = 10
while n > 0:
send('printer', n)
n -=1
• It seems to work (maybe)
printer: 10
printer: 9
printer: 8
...
printer: 1

Advanced Example
129
• Recursive ping-pong (inspired by Stackless)
@actor
def ping():
while True:
n = yield
print('ping %d' % n)
send('pong', n + 1)
@actor
def pong():
while True:
n = yield
print('pong %d' % n)
send('ping', n + 1)
ping()
pong()
send('ping', 0)
ping
pong

Advanced Example
130
ping 0
pong 1
File "actor.py", line 36, in <module>
send('ping', 0)
File "actor.py", line 8, in send
File "actor.py", line 24, in ping
send('pong', n + 1)
File "actor.py", line 31, in pong
send('ping', n + 1)
ValueError: generator already executing
• Alas, it does not work

Problems
131
• Important differences between actors/coroutines
• Concurrent execution
• Asynchronous message delivery
• Although coroutines have a "send()", it's a normal
method call
• Synchronous
• Involves the call-stack
• Does not allow recursion/reentrancy

Solution 1
132
class Actor(threading.Thread):
super().__init__()
self.daemon = True
self.gen = gen
self.mailbox = Queue()
self.start()
def send(self, msg):
self.mailbox.put(msg)
def run(self):
while True:
msg = self.mailbox.get()
self.gen.send(msg)
• Wrap the generator with a thread
• Err...... no.

Solution 2
133
_registry = { }
_msg_queue = deque()
def send(name, msg):
_msg_queue.append((name, msg))
def run():
while _msg_queue:
name, msg = _msg_queue.popleft()
• Write a tiny message scheduler
• send() simply drops messages on a queue
• run() executes as long as there are messages

Advanced Example
134
• Recursive ping-pong (reprise)
@actor
def ping():
while True:
n = yield
print('ping %d' % n)
send('pong', n + 1)
@actor
def pong():
while True:
n = yield
print('pong %d' % n)
send('ping', n + 1)
ping()
pong()
send('ping', 0)
run()
ping
pong

Advanced Example
135
ping 0
pong 1
ping 2
pong 3
ping 4
ping 5
ping 6
pong 7
...
... forever
• It works!
• That's kind of amazing

Comments
136
• It's still kind of a fake actor
• Lacking in true concurrency
• Easily blocked
• Maybe it's good enough?
• I don't know
• Key idea: you can bend space-time with yield

Part 7
137
A TerrifyingVisitor

Let's Write a Compiler
138
• Well, an extremely simple one anyways...
• Evaluating mathematical expressions
2 + 3 * 4 - 5
• Why?
• Because eval() is for the weak, that's why

Compilers 101
139
• Lexing : Make tokens
2 + 3 * 4 - 5 [NUM,PLUS,NUM,TIMES,NUM,MINUS,NUM]
• Parsing : Make a parse tree
[NUM,PLUS,NUM,TIMES,NUM,MINUS,NUM]
PLUS
NUM
(2)
TIMES
NUM
(3)
NUM
(4)
MINUS
NUM
(5)

Compilers 101
140
• Evaluation :Walk the parse tree
PLUS
NUM
(2)
TIMES
NUM
(3)
NUM
(4)
MINUS
NUM
(5)
9
• It's almost too simple

Tokenizing
141
import re
from collections import namedtuple
tokens = [
r'(?P<NUM>d+)',
r'(?P<PLUS>+)',
r'(?P<MINUS>-)',
r'(?P<TIMES>*)',
r'(?P<DIVIDE>/)',
r'(?P<WS>s+)',
]
master_re = re.compile('|'.join(tokens))
Token = namedtuple('Token', ['type','value'])
def tokenize(text):
scan = master_re.scanner(text)
return (Token(m.lastgroup, m.group())
for m in iter(scan.match, None)
if m.lastgroup != 'WS')

Tokenizing
142
text = '2 + 3 * 4 - 5'
for tok in tokenize(text):
print(tok)
Token(type='NUM', value='2')
Token(type='PLUS', value='+')
Token(type='TIMES', value='*')
Token(type='MINUS', value='-')
• Example:

Parsing
143
• Must match the token stream against a grammar
expr ::= term { +|- term }*
term ::= factor { *|/ factor}*
factor ::= NUM
• An expression is just a bunch of terms
2 + 3 * 4 - 5
term + term - term
• A term is just one or more factors
term term
factor factor * factor
(2) (3) (4)

Recursive Descent Parse
144
expr ::= term { +|- term }*
term ::= factor { *|/ factor}*
factor ::= NUM
def expr():
! term()
! while accept('PLUS','MINUS'):
term()
!print('Matched expr')
def term():
! factor()
while accept('TIMES','DIVIDE'):
factor()
print('Matched term')
def factor():
if accept('NUM'):
print('Matched factor')
else:
raise SyntaxError()
Encode the grammar
as a collection of
functions
Each function steps
through the rule

Recursive Descent Parse
145
def parse(toks):
lookahead, current = next(toks, None), None
def accept(*toktypes):
nonlocal lookahead, current
if lookahead and lookahead.type in toktypes:
current, lookahead = lookahead, next(toks, None)
return True
def expr():
term()
while accept('PLUS','MINUS'):
term()
print('Matched expr')
...
expr()

Tree Building
146
class Node:
_fields = []
def __init__(self, *args):
for name, value in zip(self._fields, args):
setattr(self, name, value)
class BinOp(Node):
_fields = ['op', 'left', 'right']
class Number(Node):
_fields = ['value']
• Need some tree nodes for different things
• Example:
n1 = Number(3)
n2 = Number(4)
n3 = BinOp('*', n1, n2)

Tree Building
147
def parse(toks):
...
def expr():
left = term()
while accept('PLUS','MINUS'):
left = BinOp(current.value, left)
left.right = term()
return left
def term():
left = factor()
while accept('TIMES','DIVIDE'):
left = BinOp(current.value, left)
left.right = factor()
return left
def factor():
if accept('NUM'):
return Number(int(current.value))
else:
raise SyntaxError()
return expr()
Building nodes
and hooking
them together

Our Little Parser
148
text = '2 + 3*4 - 5'
toks = tokenize(text)
tree = parse(toks)
• Story so far...
BinOp('-',
BinOp('+',
Number(2),
BinOp('*',
Number(3),
Number(4)
)
),
Number(5)
)

Evaluation
149
class NodeVisitor:
def visit(self, node):
return getattr(self,
'visit_' + type(node).__name__)(node)
• The "Visitor" pattern
• Example:
class MyVisitor(NodeVisitor):
def visit_Number(self, node):
print(node.value)
def visit_BinOp(self, node):
self.visit(node.left)
self.visit(node.right)
print(node.op)
MyVisitor().visit(tree)
2
3
4
*
+
5
-
output

Evaluation
150
class Evaluator(NodeVisitor):
! return node.value
leftval = self.visit(node.left)
rightval = self.visit(node.right)
if node.op == '+':
return leftval + rightval
elif node.op == '-':
return leftval - rightval
elif node.op == '*':
return leftval * rightval
elif node.op == '/':
return leftval / rightval
print(Evaluator().visit(tree))
• An Expression Evaluator
9

Digression
151
• Last 12 slides a whole graduate CS course
• Plus at least one additional Python tutorial
• Don't worry about it
• Left as an exercise...

Death Spiral
152
# Make '0+1+2+3+4+...+999'
text = '+'.join(str(x) for x in range(1000))
tree = parse(toks)
val = Evaluate().visit(tree)
• And it almost works...
File "compiler.py", line 100, in <module>
val = Evaluator().visit(tree)
File "compiler.py", line 63, in visit
return getattr(self, 'visit_' + type(node).__name__)(node)
File "compiler.py", line 80, in visit_BinOp
...
RuntimeError: maximum recursion depth exceeded while calling a
Python object

Evaluation
153
! return node.value
rightval = self.visit(node.right)
if node.op == '+':
!%*@*^#^#
Recursion
(damn you to hell)

Evaluation
154
+
0 + 1 + 2 + 3 + 4 ... + 999
0
+
+
+
+
2
3
998
999
1
...
A deeply
nested tree
structure
Blows up!

• The visitor pattern is bad idea
• Better: Functional language with pattern
matching and tail-call optimization
I ToldYou So

QUESTION
156
How do you NOT do something?

QUESTION
157
How do you NOT do something?
(yield?)

Evaluation
158
! return node.value
leftval = yield node.left
rightval = yield node.right
if node.op == '+':
Nope. Not doing
that recursion.

Generator Wrapping
159
class NodeVisitor:
def genvisit(self, node):
result = getattr(self,
'visit_' + type(node).__name__)(node)
if isinstance(result, types.GeneratorType):
result = yield from result
return result
• Step 1:Wrap "visiting" with a generator
• Thinking: No matter what the visit_() method
produces, the result will be a generator
• If already a generator, then just delegate to it

Generator Wrapping
160
>>> v = Evaluator()
>>> n = Number(2)
>>> gen = v.genvisit(n)
>>> gen
<generator object genvisit at 0x10070ab88>
>>> gen.send(None)
StopIteration: 2
>>>
• Example:A method that simply returns a value
• Result: Carried as value in StopIteration

Generator Wrapping
161
>>> v = Evaluator()
>>> n = BinOp('*', Number(3), Number(4))
>>> gen
>>> gen.send(None)
<__main__.Number object at 0x1058525c0>
>>> gen.send(_.value)
<__main__.Number object at 0x105852630>
StopIteration: 12
>>>
• A method that yields nodes (iteration)
Again, note the return
mechanism

Generator Wrapping
162
>>> v = Evaluator()
>>> gen
>>> gen.send(None)
StopIteration: 12
>>>
• A method that yields nodes
return node.value
Manually carrying out this
method in the example

Running Recursion
163
class NodeVisitor:
def visit(self, node):
stack = [ self.genvisit(node) ]
! result = None
while stack:
try:
node = stack[-1].send(result)
stack.append(self.genvisit(node))
result = None
! stack.pop()
result = exc.value
! return result
• Step 2: Run depth-ﬁrst traversal with a stack
• Basically, a stack of running generators

Transcendence
164
# Make '0+1+2+3+4+...+999'
text = '+'.join(str(x) for x in range(1000))
tree = parse(toks)
val = Evaluate().visit(tree)
print(val)
• Does it work?
• Yep
499500
• Yow!

Running Recursion
165
leftval = yield node.left
rightval = yield node.right
if node.op == '+':
result = leftval + rightval
...
return result
• Each yield creates a new stack entry
• Returned values (via StopIteration) get
propagated as results
generator
generator
generator
stack
yield
yield return
return

Running Recursion
166
>>> v = Evaluator()
>>> stack = [ v.genvisit(n) ]
>>> stack[-1].send(None)
>>> stack.append(v.genvisit(_))
StopIteration: 3
>>> stack.pop()
>>> stack[-1].send(3)
StopIteration: 4
>>> stack.pop()
StopIteration: 12
>>>
Nodes are visited and
generators pushed onto
a stack

Running Recursion
167
>>> v = Evaluator()
>>> stack = [ v.genvisit(n) ]
StopIteration: 3
>>> stack.pop()
StopIteration: 4
>>> stack.pop()
StopIteration: 12
>>>
Results propagate via
StopIteration
12 (Final Result)

Final Words

Historical Perspective
169
• Generators seem to have started as a simple
way to implement iteration (Python 2.3)
• Took an interesting turn with support for
coroutines (Python 2.5)
• Taken to a whole new level with delegation
support in PEP-380 (Python 3.3).

Control Flow Bending
170
• yield statement allows you to bend control-ﬂow
to adapt it to certain kinds of problems
• Wrappers (context managers)
• Futures/concurrency
• Messaging
• Recursion
• Frankly, it blows my mind.

asyncio
171
• Inclusion of asyncio in standard library may be a
game changer
• To my knowledge, it's the only standard library
module that uses coroutines/generator
delegation in a signiﬁcant manner
• To really understand how it works, need to have
your head wrapped around generators
• Read the source for deep insight

Is it Proper?
172
• Are coroutines/generators a good idea or not?
• Answer: I still don't know
• Issue: Coroutines seem like they're "all in"
• Fraught with potential mind-bending issues
• Example:Will there be two standard libraries?

Two Libraries?
173
Python
Standard Library
Standard
coroutine library
(asyncio and friends)
?????
• If two different worlds, do they interact?
• If so, by what rules?

Personal Use
174
• My own code is dreadfully boring
• Generators for iteration: Yes.
• Everything else: Threads, recursion, etc. (sorry)
• Nevertheless:There may be something to all of
this advanced coroutine/generator business

A Bit More Information
175

Thanks!
176
• I hope you got some new ideas
• Please feel free to contact me
http://www.dabeaz.com
• Also, I teach Python classes (shameless plug)
@dabeaz (Twitter)
• Special Thanks:
Brian Curtin, Ken Izzo, George Kappel, Christian Long,
Michael Prentiss,Vladimir Urazov, Guido van Rossum

Generators: The Final Frontier

More Related Content

What's hot (20)

Similar to Generators: The Final Frontier (20)

More from David Beazley (Dabeaz LLC) (7)

Recently uploaded (20)

Generators: The Final Frontier