\[ [Index](index.md) | [Exercise 8.4](ex8_4.md) | [Exercise 8.6](ex8_6.md) \]

# Exercise 8.5

*Objectives:*

- Learn about managed generators

*Files Created:* `multitask.py`, `server.py`

A generator or coroutine function can never execute without being
driven by some other code.  For example, a generator used for
iteration doesn't do anything unless iteration is actually carried out
using a for-loop.  Similarly, a collection of coroutines won't run
unless their `send()` method is invoked somehow.

In advanced applications of generators, it is possible to drive 
generators in various unusual ways.  In this exercise, we look at a
few examples.

## (a) Generators as tasks

If a file `multitask.py`, define the following code:

```python
# multitask.py

from collections import deque

tasks = deque()
def run():
    while tasks:
        task = tasks.popleft()
        try:
            task.send(None)
            tasks.append(task)
        except StopIteration:
            print('Task done')
```

This code implements a tiny task scheduler that runs generator functions.
Try it by running it on the following functions.

```python
# multitask.py
...

def countdown(n):
    while n > 0:
        print('T-minus', n)
        yield
        n -= 1

def countup(n):
    x = 0
    while x < n:
        print('Up we go', x)
        yield
        x += 1

if __name__ == '__main__':
    tasks.append(countdown(10))
    tasks.append(countdown(5))
    tasks.append(countup(20))
    run()
```

When you run this, you should see output from all of the generators
interleaved together.  For example:

```python
T-minus 10
T-minus 5
Up we go 0
T-minus 9
T-minus 4
Up we go 1
T-minus 8
T-minus 3
Up we go 2
T-minus 7
T-minus 2
Up we go 3
T-minus 6
T-minus 1
Up we go 4
T-minus 5
Task done
Up we go 5
T-minus 4
Up we go 6
T-minus 3
Up we go 7
T-minus 2
Up we go 8
T-minus 1
Up we go 9
Task done
Up we go 10
Up we go 11
Up we go 12
Up we go 13
Up we go 14
Up we go 15
Up we go 16
Up we go 17
Up we go 18
Up we go 19
Task done
```

That's interesting, but not especially compelling.  Move on to the next example.

## (b) Generators as Tasks Serving Network Connections

Create a file `server.py` and put the following code into it:

```python
# server.py

from socket import *
from select import select
from collections import deque

tasks = deque()
recv_wait = {}   #  sock -> task
send_wait = {}   #  sock -> task

def run():
    while any([tasks, recv_wait, send_wait]):
        while not tasks:
            can_recv, can_send, _ = select(recv_wait, send_wait, [])
            for s in can_recv:
                tasks.append(recv_wait.pop(s))
            for s in can_send:
                tasks.append(send_wait.pop(s))
        task = tasks.popleft()
        try:
            reason, resource = task.send(None)
            if reason == 'recv':
                recv_wait[resource] = task
            elif reason == 'send':
                send_wait[resource] = task
            else:
                raise RuntimeError('Unknown reason %r' % reason)
        except StopIteration:
            print('Task done')
```

This code is a slightly more complicated version of the task scheduler in
part (a).  It will require a bit of study, but the idea is that not only
will each task yield, it will indicate a reason for doing so (receiving or
sending).  Depending on the reason, the task will move over to a waiting
area.  The scheduler then runs any available tasks or waits for I/O
events to occur when nothing is left to do.

It's all a bit tricky perhaps, but add the following code which implements
a simple echo server:

```python
# server.py
...

def tcp_server(address, handler):
    sock = socket(AF_INET, SOCK_STREAM)
    sock.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1)
    sock.bind(address)
    sock.listen(5)
    while True:
        yield 'recv', sock
        client, addr = sock.accept()
        tasks.append(handler(client, addr))
        
def echo_handler(client, address):
    print('Connection from', address)
    while True:
        yield 'recv', client
        data = client.recv(1000)
        if not data:
            break
        yield 'send', client
        client.send(b'GOT:' + data)
    print('Connection closed')

if __name__ == '__main__':
    tasks.append(tcp_server(('',25000), echo_handler))
    run()
```

Run this server in its own terminal window.   In another terminal, connect to it using a command such as `telnet` or `nc`.   For example:

```
bash % nc localhost 25000
Hello
Got: Hello
World
Got: World
```

If you don't have access to `nc` or `telnet` you can also use Python itself:

```
bash % python3 -m telnetlib localhost 25000
Hello
Got: Hello
World
Got: World
```

If it's working, you should see output being echoed back to you. Not only that,
if you connect multiple clients, they'll all operate concurrently.

This tricky use of generators is not something that you would
likely have to code directly.  However, they are used in certain advanced
packages such as `asyncio` that was added to the standard
library in Python 3.4.

\[ [Solution](soln8_5.md) | [Index](index.md) | [Exercise 8.4](ex8_4.md) | [Exercise 8.6](ex8_6.md) \]

----
`>>>` Advanced Python Mastery  
`...` A course by [dabeaz](https://www.dabeaz.com)  
`...` Copyright 2007-2023  

![](https://i.creativecommons.org/l/by-sa/4.0/88x31.png). This work is licensed under a [Creative Commons Attribution-ShareAlike 4.0 International License](http://creativecommons.org/licenses/by-sa/4.0/)