pytudes/ipynb/Snobol.ipynb

Peter Norvig, Nov. 2017

Bad Grade, Good Experience¶

Recently I was asked a question I hadn't thought about in decades:

As a student, did you ever get a bad grade on a programming assignment?

I've forgotten most of my assignments, but there is one I do remember. It was something like this:

The Concordance Assignment¶

Using the Snobol language, read lines of text from the standard input and print a concordance, which is an alphabetized list of words in the text, with the line number(s) where each word appears. Words with different capitalization (like "A" and "a") should be merged into one entry.

After studying Snobol a bit, I realized that the expected solution was along these lines:

1. Create an empty hash table whose keys will be words and values will be lists of line numbers.
2. Read the lines of text (tracking the line numbers), split them into words, and build up the list of line numbers for each word.
3. Convert the hash table into a two-dimensional array where each row has the two columns [word, line_numbers].
4. Write a function to sort the array alphabetically (sort is not built-in to Snobol).
5. Write a function to print the array.

That would be around 40 to 60 lines of code; an easy task. But I noticed three interesting things about Snobol:

• '$' is an indirection operator, so if the variable 'word' has the value "A", then '$word = x' is the same as 'A = x'.
• Uninitialized variables are treated as the empty string, so 'A = A + "text"' works even if we haven't seen 'A' before.
• When the program ends, the Snobol interpreter prints out each variable (in sorted order), with its value, as a debugging aid.

That means I could use $ to do away with the hash table and array data structures, eliminating steps 1, 3, 4, and 5, and just do step 2!

The Concordance Solution¶

I ended up with a program similar to the following (translated from Snobol to Python, but with '$word' indirection):

In [1]:

program = """
for i, line in enumerate(input):
    for word in re.findall("[A-Z]+", line.upper()):
        $word = $word + i + ", "
"""

That's just 3 lines, not 40 to 60!

To test the program, I'll write a mock Snobol/Python interpreter, which at heart is just a call to the Python interpreter, exec(program), except that it handles the three things I mentioned about the Snobol interpreter, plus one more:

1. $word gets translated as _globals[word].
2. The interpreter calls exec(program, _globals), where _globals is a defaultdict that makes variables default to the empty string.
3. After the exec completes, the user-defined variables (but not the built-in ones) are printed.
4. Concatenating a string with an integer coerces the int to str automatically. I'll handle that with a Str class.

In [2]:

from collections import defaultdict
import re

def snobol(program, data=''):
    """A Python interpreter with four Snobol-ish features:
    1. $word indirection; 2. variables default to empty string; 
    3. post-mortem dump;  4. automatic coercing to string"""
    program = re.sub(r'\$(\w+)', r'_globals[\1]', program) # 1. 
    _globals = defaultdict(Str, vars(__builtins__))        # 4., 2.
    _globals.update(re=re, input=data.splitlines(), _globals=_globals)
    builtins = set(_globals) | {'__builtins__'}
    try:
        exec(program, _globals)
    finally:
        print('-' * 79)                                    # 3.     
        for name in sorted(_globals):
            if name not in builtins:
                print('{:10} = {}'.format(name, _globals[name]))
                
class Str(str):
    "String class with automatic coercion for +"
    def __add__(self, other):  return Str(str(self) + str(other))
    def __radd__(self, other): return Str(str(other) + str(self))

Now we can run the program on some data:

In [3]:

data = """
There she was just a-walkin' down the street, 
Singin' "Do wah diddy diddy dum diddy do"
Snappin' her fingers and shufflin' her feet, 
Singin' "Do wah diddy diddy dum diddy do"
She looked good (looked good), 
She looked fine (looked fine)
She looked good, she looked fine 
And I nearly lost my mind
"""

In [4]:

snobol(program, data)

-------------------------------------------------------------------------------
A          = 1, 
AND        = 3, 8, 
DIDDY      = 2, 2, 2, 4, 4, 4, 
DO         = 2, 2, 4, 4, 
DOWN       = 1, 
DUM        = 2, 4, 
FEET       = 3, 
FINE       = 6, 6, 7, 
FINGERS    = 3, 
GOOD       = 5, 5, 7, 
HER        = 3, 3, 
I          = 8, 
JUST       = 1, 
LOOKED     = 5, 5, 6, 6, 7, 7, 
LOST       = 8, 
MIND       = 8, 
MY         = 8, 
NEARLY     = 8, 
SHE        = 1, 5, 6, 7, 7, 
SHUFFLIN   = 3, 
SINGIN     = 2, 4, 
SNAPPIN    = 3, 
STREET     = 1, 
THE        = 1, 
THERE      = 1, 
WAH        = 2, 4, 
WALKIN     = 1, 
WAS        = 1, 
i          = 8
line       = And I nearly lost my mind
word       = MIND

Oops! The post-mortem printout includes the variables i, line, and word. Reluctantly, I'll increase the program's line count by 33%:

In [5]:

program = """
for i, line in enumerate(input):
    for word in re.findall("[A-Z]+", line.upper()):
        $word = $word + i + ", "
del i, line, word
"""

snobol(program, data)

-------------------------------------------------------------------------------
A          = 1, 
AND        = 3, 8, 
DIDDY      = 2, 2, 2, 4, 4, 4, 
DO         = 2, 2, 4, 4, 
DOWN       = 1, 
DUM        = 2, 4, 
FEET       = 3, 
FINE       = 6, 6, 7, 
FINGERS    = 3, 
GOOD       = 5, 5, 7, 
HER        = 3, 3, 
I          = 8, 
JUST       = 1, 
LOOKED     = 5, 5, 6, 6, 7, 7, 
LOST       = 8, 
MIND       = 8, 
MY         = 8, 
NEARLY     = 8, 
SHE        = 1, 5, 6, 7, 7, 
SHUFFLIN   = 3, 
SINGIN     = 2, 4, 
SNAPPIN    = 3, 
STREET     = 1, 
THE        = 1, 
THERE      = 1, 
WAH        = 2, 4, 
WALKIN     = 1, 
WAS        = 1, 

Looks good to me!¶

But sadly, the grader for the course did not agree, complaining that my program was not extensible: what if I wanted to cover two or more files in one run? What if I wanted the output to have a slightly different format? I argued that YAGNI, and if the requirements changed, then I would write the necessary 40 or 60 lines, but there's no sense doing that until then. The grader was not impressed with my arguments and I got points taken off.

Still, I was happy with my program. I felt like the purpose of the assignment was to get familiar with a new programming language with some different idioms/paradigms. By using the indirection operator I learned more about "thinking different" than if I had written the expected program.

TFW you flunk AI¶

Here's another example that I had completely forgotten about until 2016, when I was cleaning out a filing cabinet and came across my old college transcript. It turns out that I flunked an AI course! (Or at least, didn't complete it.) This course was offered by Prof. Richard Millward in the Cognitive Science program. I certainly remember a lot of influential material from this class: we read David Marr, we read Winston's just-published Psychology of Computer Vision, we read a chapter from Duda and Hart which was then only a few years old. The things I learned in that course have stuck with me for decades, but one thing that didn't stick is that, according to my transcript, I never completed the course! I'm not sure what happened. I did an independent study with Ulf Grenander that semester; my best guess is that when I started doing the independent study that would have put me over some limit, and so I had to drop the AI course.

So in both the concordance program and the Cognitive Science AI class, I had a great experience and I learned a lot, even if it wasn't well-reflected in official credit. The moral is: look for the good experiences, and don't worry about the official credit.