python-mastery/Exercises/ex8_1.md

\[ [Index](index.md) | [Exercise 7.6](ex7_6.md) | [Exercise 8.2](ex8_2.md) \]

# Exercise 8.1

*Objectives:*

- Learn how to customize iteration using generators

*Files Modified:* `structure.py`

*Files Created:* `follow.py`

## (a) A Simple Generator

If you ever find yourself wanting to customize iteration, you should
always think generator functions.  They're easy to write---simply make
a function that carries out the desired iteration logic and uses `yield`
to emit values.

For example, try this generator that allows you to iterate over a
range of numbers with fractional steps (something not supported by
the `range()` builtin):

```python
>>> def frange(start,stop,step):
        while start < stop:
            yield start
            start += step

>>> for x in frange(0, 2, 0.25):
        print(x, end=' ')

0 0.25 0.5 0.75 1.0 1.25 1.5 1.75
>>>
```

Iterating on a generator is a one-time operation. For example, here's
what happen if you try to iterate twice:

```python
>>> f = frange(0, 2, 0.25)
>>> for x in f:
        print(x, end=' ')

0 0.25 0.5 0.75 1.0 1.25 1.5 1.75
>>> for x in f:
        print(x, end=' ')

>>>
```

If you want to iterate over the same sequence, you need to recreate the generator
by calling `frange()` again.    Alternative, you could package everything into a class:

```python
>>> class FRange:
        def __init__(self, start, stop, step):
            self.start = start
            self.stop = stop
            self.step = step
        def __iter__(self):
            n = self.start
            while n < self.stop:
                yield n
                n += self.step

>>> f = FRange(0, 2, 0.25)
>>> for x in f:
        print(x, end=' ')

0 0.25 0.5 0.75 1.0 1.25 1.5 1.75
>>> for x in f:
        print(x, end=' ')

0 0.25 0.5 0.75 1.0 1.25 1.5 1.75
>>>
```

## (b) Adding Iteration to Objects

If you've created a custom class, you can make it support iteration by
defining an `__iter__()` special method.  `__iter__()` returns an
iterator as a result.  As shown in the previous example, an easy way
to do it is to define `__iter__()` as a generator.

In earlier exercises, you defined a `Structure` base class.
Add an `__iter__()` method to this class that produces the attribute values
in order. For example:

```python
class Structure(metaclass=StructureMeta):
    ...
    def __iter__(self):
        for name in self._fields:
            yield getattr(self, name)
    ...
```

Once you've done this, you should be able to iterate over the instance
attributes like this:

```python
>>> from stock import Stock
>>> s = Stock('GOOG', 100, 490.1)
>>> for val in s:
        print(val)
GOOG
100
490.1
>>>
```

## (c) The Surprising Power of Iteration

Python uses iteration in ways you might not expect.   Once you've added `__iter__()`
to the `Structure` class, you'll find that it is easy to do all sorts of new 
operations. For example, conversions to sequences and unpacking:

```python
>>> s = Stock('GOOG', 100, 490.1)
>>> list(s)
['GOOG', 100, 490.1]
>>> tuple(s)
('GOOG', 100, 490.1)
>>> name, shares, price = s
>>> name
'GOOG'
>>> shares
100
>>> price
490.1
>>> 
```

While we're at it, we can now add a comparison operator to our `Structure`
class:

```python
# structure.py
class Structure(metaclass=StructureMeta):
    ...
    def __eq__(self, other):
        return isinstance(other, type(self)) and tuple(self) == tuple(other)
    ...
```

You should now be able to compare objects:

```python
>>> a = Stock('GOOG', 100, 490.1)
>>> b = Stock('GOOG', 100, 490.1)
>>> a == b
True
>>>
```

Try running your `teststock.py` unit tests again.  Everything should be passing now.
Excellent.

## (d) Monitoring a streaming data source

Generators can also be a useful way to simply produce a stream of
data.  In this part, we'll explore this idea by writing a generator to
watch a log file.  To start, follow the next instructions carefully.

The program `Data/stocksim.py` is a program that
simulates stock market data.  As output, the program constantly writes
real-time data to a file `stocklog.csv`.  In a
command window (not IDLE) go into the `Data/` directory and run this program:

```
% python3 stocksim.py
```

If you are on Windows, just locate the `stocksim.py` program and
double-click on it to run it.  Now, forget about this program (just
let it run).  Again, just let this program run in the background---it
will run for several hours (you shouldn't need to worry about it).

Once the above program is running, let's write a little program to
open the file, seek to the end, and watch for new output.  Create a
file `follow.py` and put this code in it:

```python
# follow.py
import os
import time
f = open('Data/stocklog.csv')
f.seek(0, os.SEEK_END)   # Move file pointer 0 bytes from end of file

while True:
    line = f.readline()
    if line == '':
        time.sleep(0.1)   # Sleep briefly and retry
        continue
    fields = line.split(',')
    name = fields[0].strip('"')
    price = float(fields[1])
    change = float(fields[4])
    if change < 0:
        print('%10s %10.2f %10.2f' % (name, price, change))
```

If you run the program, you'll see a real-time stock ticker.  Under the covers,
this code is kind of like the Unix `tail -f` command that's used to watch a log file.

**Note:** The use of the `readline()` method in this example is
somewhat unusual in that it is not the usual way of reading lines from
a file (normally you would just use a `for`-loop).  However, in
this case, we are using it to repeatedly probe the end of the file to
see if more data has been added (`readline()` will either
return new data or an empty string).

If you look at the code carefully, the first part of the code is
producing lines of data whereas the statements at the end of the
`while` loop are consuming the data.  A major feature of generator
functions is that you can move all of the data production code into a
reusable function.

Modify the code so that the file-reading is performed by 
a generator function `follow(filename)`.   Make it so the following code
works:

```python
>>> for line in follow('Data/stocklog.csv'):
          print(line, end='')
   
... Should see lines of output produced here ...
```

Modify the stock ticker code so that it looks like this:

```python
for line in follow('Data/stocklog.csv'):
    fields = line.split(',')
    name = fields[0].strip('"')
    price = float(fields[1])
    change = float(fields[4])
    if change < 0:
        print('%10s %10.2f %10.2f' % (name, price, change))
```

**Discussion**

Something very powerful just happened here.  You moved an interesting iteration pattern
(reading lines at the end of a file) into its own little function.   The `follow()` function
is now this completely general purpose utility that you can use in any program.  For
example, you could use it to watch server logs, debugging logs, and other similar data sources.
That's kind of cool.

\[ [Solution](soln8_1.md) | [Index](index.md) | [Exercise 7.6](ex7_6.md) | [Exercise 8.2](ex8_2.md) \]

----
`>>>` Advanced Python Mastery  
`...` A course by [dabeaz](https://www.dabeaz.com)  
`...` Copyright 2007-2023  

![](https://i.creativecommons.org/l/by-sa/4.0/88x31.png). This work is licensed under a [Creative Commons Attribution-ShareAlike 4.0 International License](http://creativecommons.org/licenses/by-sa/4.0/)