{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div align=\"right\" style=\"text-align: right;\"><i>Peter Norvig<br>April 2020</i></div>\n",
    "\n",
    "# Jotto and Wordle: Word Guessing Games\n",
    "\n",
    "[Jotto](https://en.wikipedia.org/wiki/Jotto) is a word game in which a **guesser** tries to guess a secret **target** word, which is chosen from a **word list** of allowable words, in as few guesses as possible. Each guess must be one of the allowable words, and the **reply** to each guess is the number of letters in common between the guess word and the target word, regardless of the positions of the letters.   \n",
    "\n",
    "Here is an example Jotto game, where I show the guesses, the replies, the number of remaining targets that are **consistent** with all the replies seen so far, and finally the letters that matched (the matches are an aid to you, the reader; they are not known to the guesser). In this game, the guesser gets to the target word, \"wonky\", in 7 guesses. \n",
    "\n",
    "    Guess 1: \"stoma\" Reply: 1 right; Consistent targets: 1118    (Matched: \"o\")\n",
    "    Guess 2: \"bairn\" Reply: 1 right; Consistent targets: 441     (Matched: \"n\")\n",
    "    Guess 3: \"swipe\" Reply: 1 right; Consistent targets: 197     (Matched: \"w\")\n",
    "    Guess 4: \"lurks\" Reply: 1 right; Consistent targets: 87      (Matched: \"k\")\n",
    "    Guess 5: \"rowdy\" Reply: 3 right; Consistent targets: 14      (Matched: \"owy\")\n",
    "    Guess 6: \"roved\" Reply: 1 right; Consistent targets: 2       (Matched: \"o\")\n",
    "    Guess 7: \"wonky\" Reply: 5 right; Consistent targets: 1       (Matched: \"wonky\")\n",
    "\n",
    "\n",
    "There are several variants of the game; here are five key questions and my answers:\n",
    "\n",
    "- Q: How many letters can each word be? <br>A: **Only five-letter words are allowed**.\n",
    "- Q: Does a guess have to be a word in the word list? <br>A: **Yes.**\n",
    "- Q: What is the reply for a word that has the same letter twice, like the \"s\" in \"stars\"? <br>A: **Only words with no repeated letters are allowed in the word list**.\n",
    "- Q: What if the reply is \"5\", but the guess is not the target?<br>A: **No two words in the word list are allowed to have the same set of five letters**.<br>For example, only one of the anagrams apers/pares/parse/pears/reaps/spare/spear is allowed.\n",
    "- Q: Who chooses the target word? <br>A: **Random chance**. Jotto is sometimes a two-person game where the chooser is an adversary, but not here.\n",
    "\n",
    "# Preliminaries\n",
    "\n",
    "First off, some Python basics: Import some modules and define the basic types `Word`, `Score`, and `Reply`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from typing      import List, Tuple, Dict, Union, Counter, Callable\n",
    "from dataclasses import dataclass\n",
    "from statistics  import mean, stdev\n",
    "from collections import defaultdict\n",
    "from math        import log\n",
    "import random\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "Word  = str # A word is a lower-case string of five distinct characters\n",
    "Score = int # A score is the number of guesses it took to get the target\n",
    "Reply = int # A reply is the number of letters in common between guess and target words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can make a Jotto word list by starting with a list of words, discarding the ones that don't have 5 distinct letters, putting the rest into a dict keyed by the set of letters, and then keeping only one word for each set of letters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def allowable(words) -> List[Word]:\n",
    "    \"\"\"Build a list of allowable Jotto words from an iterable of words.\"\"\"\n",
    "    dic = {frozenset(w): w for w in words if len(w) == 5 == len(set(w))}\n",
    "    return list(dic.values())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The file [sgb-words.txt](sgb-words.txt) (from the [Stanford GraphBase project](https://www-cs-faculty.stanford.edu/~knuth/sgb.html)) has a nice list of five-letter words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(5756, 2845)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "! [ -e sgb-words.txt ] || curl -O https://norvig.com/ngrams/sgb-words.txt\n",
    "    \n",
    "sgb_words = open('sgb-words.txt').read().split()\n",
    "wordlist  = allowable(sgb_words)\n",
    "\n",
    "len(sgb_words), len(wordlist)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see there are 2,845 allowable target words out of the 5,756 words in [sgb-words.txt](sgb-words.txt)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Playing a Game\n",
    "\n",
    "The function `play` will play a game of Jotto and return the score (the number of guesses). The function's arguments are:\n",
    "- `guesser`: a `callable` (e.g., a function) that should return the guess to make. It is passed two arguments: the reply to the previous guess, and the words that are consistent with all the guesses made so far. (If the guesser wants to keep track of all the guesses made so far, or all the words in the word list, it is welcome to do so itself. A guesser need not be a pure function; it can remember things from one call to the next.)\n",
    "- `target`: The target word. If none is given, the target word is chosen at random from the wordlist.\n",
    "- `wordlist`: The list of allowable words.\n",
    "- `verbose`: If true, print a message for each guess.\n",
    "\n",
    "Two corner cases: (1) If the guesser improperly guesses a non-word, the reply is -1. (2) If a poor guesser makes as many guesses as there are words in the list the game stops and the guesser gets as a score the number of words in the list (thus avoiding an infinite loop)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "Guesser = Callable[[Reply, List[Word]], Word]\n",
    "\n",
    "def play(guesser: Guesser, target=None, wordlist=wordlist, verbose=False) -> Score:\n",
    "    \"\"\"How many guesses does it take for `guesser` to guess the Jotto target word,\n",
    "    which is selected by `chooser` from the words in `wordlist`?\"\"\"\n",
    "    target  = target or random.choice(wordlist) # Choose a random target if none was given\n",
    "    targets = wordlist # The targets that are consistent with all replies\n",
    "    reply   = None     # For the first guess, there is no previous reply\n",
    "    for i in range(1, len(wordlist) + 1):\n",
    "        guess = guesser(reply, targets)\n",
    "        reply = reply_for(guess, target) if guess in wordlist else -1\n",
    "        targets = [w for w in targets if reply_for(guess, w) == reply]\n",
    "        if verbose: \n",
    "            print(f'Guess {i}: \"{guess}\" Reply: {reply} right; Consistent targets: {len(targets)}')\n",
    "        if guess == target:  \n",
    "            return i\n",
    "    return i\n",
    "                         \n",
    "def reply_for(guess, target) -> Reply: \n",
    "    \"The number of letters in common between the target and guess\"\n",
    "    return len(set(target).intersection(guess))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Simple Guessers\n",
    "\n",
    "Here we define two simple guessers:\n",
    "- `human_guesser` asks a human for `input`.\n",
    "- `random_guesser` guesses one of the remaining consistent  targets, picked at random. We show it in action:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Guess 1: \"henry\" Reply: 1 right; Consistent targets: 1254\n",
      "Guess 2: \"prods\" Reply: 1 right; Consistent targets: 535\n",
      "Guess 3: \"fovea\" Reply: 1 right; Consistent targets: 229\n",
      "Guess 4: \"daunt\" Reply: 3 right; Consistent targets: 25\n",
      "Guess 5: \"unmap\" Reply: 3 right; Consistent targets: 4\n",
      "Guess 6: \"mount\" Reply: 2 right; Consistent targets: 3\n",
      "Guess 7: \"ulnas\" Reply: 2 right; Consistent targets: 1\n",
      "Guess 8: \"inapt\" Reply: 5 right; Consistent targets: 1\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "8"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def human_guesser(reply, targets) -> Word: return input(f'Reply was {reply}. Guess:')\n",
    "\n",
    "def random_guesser(reply, targets) -> Word: return random.choice(targets)\n",
    "\n",
    "play(random_guesser, verbose=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluating Guessers\n",
    "\n",
    "That was just one sample game; maybe there was some luck involved. To properly evaluate a guesser we can try it with every possible target word. (This is straightforward when the guesser is a program; tedious when the guesser is a human.) The function `evaluate_guesser` plays a game with every target in `wordlist` and collects the scores into a Counter. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate_guesser(guesser, wordlist=wordlist) -> Counter[Score]:\n",
    "    \"\"\"Counter of scores for this guesser on all the targets in `wordlist`.\"\"\"\n",
    "    return Counter(play(guesser, target, wordlist, verbose=False)\n",
    "                   for target in wordlist)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Counter({9: 346,\n",
       "         6: 520,\n",
       "         8: 629,\n",
       "         7: 756,\n",
       "         10: 143,\n",
       "         5: 228,\n",
       "         11: 71,\n",
       "         2: 5,\n",
       "         12: 28,\n",
       "         4: 67,\n",
       "         13: 12,\n",
       "         14: 7,\n",
       "         15: 5,\n",
       "         3: 27,\n",
       "         16: 1})"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scores = evaluate_guesser(random_guesser)\n",
    "scores"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get a better feel for these scores, the function `show` prints the mean, standard deviation, worst case number of guesses, and total number of words; followed by a display of the cumulative percentage guessed correctly for different numbers of guesses (e.g., the notation `\"≤5:11%\"` means that 11% of the targets were guessed in 5 or fewer guesses); followed by a histogram of scores:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def show(scores: Counter[int]):\n",
    "    \"\"\"Show a histogram and statistics for these scores.\"\"\"\n",
    "    bins = range(min(scores), max(scores) + 2)\n",
    "    s = list(scores.elements()) # The individual scores\n",
    "    scale = 100 / len(s)\n",
    "    plt.hist(list(scores), weights=[scale * scores[s] for s in scores],\n",
    "             align='left', rwidth=0.9, bins=bins)\n",
    "    plt.xticks(bins[:-1])\n",
    "    plt.xlabel('Number of guesses'); plt.ylabel('% of targets')\n",
    "    print(f'mean: {mean(s):.2f} ± {stdev(s):.2f} guesses, worst: {max(s)}, N: {len(s):,d}')\n",
    "    def pct(g): return round(scale * sum(scores[i] for i in range(g + 1)))\n",
    "    print('cumulative:', ', '.join(f'≤{g}:{pct(g)}%' for g in range(3, 11)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mean: 7.36 ± 1.73 guesses, worst: 16, N: 2,845\n",
      "cumulative: ≤3:1%, ≤4:3%, ≤5:11%, ≤6:30%, ≤7:56%, ≤8:78%, ≤9:91%, ≤10:96%\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEGCAYAAABiq/5QAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAXAklEQVR4nO3de7RkZXnn8e9PQOWigtIQREgjQUdDImJLdEgMijoqDoiXicQLiRiMEe/OpIMZozGuNDF4XY4GBTQRcaGAIhgRETUYAzaES5OOIdFW0YZuNQKOCdr0M3/sfYbDoc/p6lO1z+nu9/tZq1ZV7ap69lPd5/xqn117v2+qCklSO+612A1IkhaWwS9JjTH4JakxBr8kNcbgl6TG7LjYDYxizz33rKVLly52G5K0Tbnqqqt+UFVLZi7fJoJ/6dKlrFy5crHbkKRtSpJvb2q5u3okqTEGvyQ1xuCXpMYY/JLUGINfkhpj8EtSYwx+SWqMwS9JjTH4Jakx28SZu9r+LF1+0cRqrVlx1MRqSS1wi1+SGmPwS1JjDH5JaozBL0mNMfglqTEGvyQ1xuCXpMYY/JLUmMGCP8l+SS5LsjrJDUle3S9/c5LvJbmmvzxjqB4kSfc05Jm7G4DXV9XVSe4HXJXkkv6xd1bVXw64bknSLAYL/qpaC6ztb9+eZDWw71DrkySNZkH28SdZCjwauKJfdFKS65KckWSPWV5zYpKVSVauX79+IdqUpCYMHvxJdgPOBV5TVbcB7wcOBA6h+4vg1E29rqpOq6plVbVsyZIlQ7cpSc0YNPiT7EQX+mdV1XkAVXVLVd1ZVRuBDwKHDdmDJOnuhjyqJ8DpwOqqese05ftMe9qxwKqhepAk3dOQR/UcDrwIuD7JNf2yk4HjkhwCFLAGeNmAPUiSZhjyqJ7LgWzioc8OtU5J0uZ55q4kNcbgl6TGGPyS1BiDX5IaY/BLUmMMfklqjMEvSY0x+CWpMQa/JDXG4Jekxhj8ktQYg1+SGmPwS1JjDH5JaozBL0mNMfglqTEGvyQ1ZsipF6UFtXT5RROps2bFUROpI22t3OKXpMYY/JLUGINfkhpj8EtSYwx+SWqMwS9JjTH4JakxBr8kNcbgl6TGGPyS1BiDX5IaY/BLUmMGC/4k+yW5LMnqJDckeXW//IFJLklyY3+9x1A9SJLuacgt/g3A66vqEcDjgFckeSSwHLi0qg4CLu3vS5IWyGDBX1Vrq+rq/vbtwGpgX+AY4CP90z4CPGuoHiRJ97Qg+/iTLAUeDVwB7F1Va6H7cAD2muU1JyZZmWTl+vXrF6JNSWrC4MGfZDfgXOA1VXXbqK+rqtOqallVLVuyZMlwDUpSYwYN/iQ70YX+WVV1Xr/4liT79I/vA6wbsgdJ0t0NeVRPgNOB1VX1jmkPXQAc398+Hvj0UD1Iku5pyDl3DwdeBFyf5Jp+2cnACuCcJCcA3wGeN2APkqQZBgv+qrocyCwPHznUeiVJc/PMXUlqjMEvSY0x+CWpMQa/JDXG4Jekxhj8ktQYg1+SGjPkCVzajixdftFE6qxZcdRE6kiaP7f4JakxBr8kNcbgl6TGGPyS1JjNBn+SA5Pcp799RJJXJdl9+NYkSUMYZYv/XODOJL9EN77+AcDHBu1KkjSYUYJ/Y1VtAI4F3lVVrwX2GbYtSdJQRgn+nyc5jm62rAv7ZTsN15IkaUijBP/vAo8H3lZV30pyAPDRYduSJA1llDN3n1JVr5q604f/fwzYkyRpQKNs8R+/iWW/M+E+JEkLZNYt/n6//m8DByS5YNpD9wN+OHRjkqRhzLWr5++BtcCewKnTlt8OXDdkU5Kk4cwa/FX1beDbwOOT/CJwUFV9IcnOwM50HwCSpG3MKGfu/h7wSeCv+kUPAT41ZFOSpOGM8uXuK4DDgdsAqupGYK8hm5IkDWeU4L+jqn42dSfJjkAN15IkaUijBP+Xk5wM7JzkKcAngM8M25YkaSijBP9yYD1wPfAy4LPAHw/ZlCRpOJs9c7eqNgIf7C+SpG3cZoM/yfXcc5/+rcBK4M+qypO5JGkbMspYPX8L3MldY/A/v7++Dfgw8N8n35YkaSijBP/hVXX4tPvXJ/lqVR2e5IWzvSjJGcAzgXVVdXC/7M3A79F9ZwBwclV9dn6tS5LmY5Qvd3dL8mtTd5IcBuzW390wx+s+DDxtE8vfWVWH9BdDX5IW2Chb/CcAZyaZCvvbgROS7Ar8+WwvqqqvJFk6doeSpImaM/iT3At4aFX9SpIHAKmqH097yjnzWOdJSV5M9+Xw66vq32dZ94nAiQD777//PFYjSdqUOXf19IdyntTfvnVG6M/H+4EDgUPoRv48dbYnVtVpVbWsqpYtWbJkzNVKkqaMso//kiRvSLJfkgdOXeazsqq6parunHZuwGHzqSNJmr9R9vG/pL9+xbRlBTx0S1eWZJ+qWtvfPRZYtaU1JEnjGeXM3QPmUzjJ2cARwJ5JbgL+BDgiySF0Hxxr6IaAkCQtoFG2+ElyMPBI4L5Ty6rqr+d6TVUdt4nFp29Rd5KkiRtlyIY/odtyfyTdAG1PBy4H5gx+SdLWaZQvd58LHAncXFW/CzwKuM+gXUmSBjNK8P9HfxTOhiT3B9Yxjy92JUlbh1H28a9Msjvd4ZdXAT8Brhy0K0nSYEY5qucP+psfSPI54P5Vdd2wbUmShrLZXT1JLp26XVVrquq66cskSduWWbf4k9wX2IXuOPw9gPQP3R948AL0JkkawFy7el4GvIYu5K/iruC/DXjfwH1JkgYya/BX1buBdyd5ZVW9dwF7kiQNaLP7+A19Sdq+jHIcvyRpOzJr8Cc5vL/2LF1J2o7MtcX/nv76awvRiCRpYcx1VM/Pk5wJ7JvkPTMfrKpXDdeWJGkocwX/M4EnA0+iO5xTkrQdmOtwzh8AH0+yuqquXcCeJEkDGmWQth8mOR84nG7mrMuBV1fVTYN2Ji2ypcsvmlitNSuOmlgtaVyjHM55JnAB3Rm8+wKf6ZdJkrZBowT/XlV1ZlVt6C8fBpYM3JckaSCjBP/6JC9MskN/eSHww6EbkyQNY5TgfwnwP4CbgbV0UzG+ZMimJEnDGWUilu8ARy9AL5KkBeBYPZLUGINfkhpj8EtSY0YO/iSPS/LFJF9N8qwhm5IkDWeuOXd/oapunrbodXRf8gb4e+BTA/cmSRrAXEf1fCDJVcDbq+o/gR8Dvw1spJt3V5K0DZp1V09VPQu4BrgwyYvoJl7fCOwCuKtHkrZRc+7jr6rPAP8N2B04D/hGVb2nqtYvRHOSpMmba+rFo5NcDnwRWAU8Hzg2ydlJDtxc4SRnJFmXZNW0ZQ9MckmSG/vrPSbxJiRJo5tri//P6Lb2nwOcUlU/rqrXAW8C3jZC7Q8DT5uxbDlwaVUdBFza35ckLaC5gv9Wuq385wPrphZW1Y1V9fzNFa6qrwA/mrH4GOAj/e2P4HcFkrTg5gr+Y+m+yN1AdzTPJOxdVWsB+uu9ZntikhOTrEyycv16v1KQpEnZ3NSL713AXmau/zTgNIBly5bVYvUhSdubhR6y4ZYk+wD01+s283xJ0oQtdPBfABzf3z4e+PQCr1+SmjdY8Cc5G/ga8PAkNyU5AVgBPCXJjcBT+vuSpAW02YlY5quqjpvloSOHWqckafMcllmSGmPwS1JjDH5JaozBL0mNMfglqTEGvyQ1xuCXpMYY/JLUGINfkhoz2Jm7WjxLl180sVprVhw1sVqStg5u8UtSYwx+SWqMwS9JjTH4JakxBr8kNcbgl6TGGPyS1BiDX5IaY/BLUmMMfklqjMEvSY0x+CWpMQa/JDXG4Jekxhj8ktQYg1+SGmPwS1JjDH5JaozBL0mNMfglqTGLMtl6kjXA7cCdwIaqWrYYfUhSixYl+HtPrKofLOL6JalJ7uqRpMYs1hZ/AZ9PUsBfVdVpM5+Q5ETgRID9999/gduThrF0+UUTqbNmxVETqaM2LdYW/+FVdSjwdOAVSZ4w8wlVdVpVLauqZUuWLFn4DiVpO7UowV9V3++v1wHnA4ctRh+S1KIFD/4kuya539Rt4KnAqoXuQ5JatRj7+PcGzk8ytf6PVdXnFqEPSWrSggd/VX0TeNRCr1eS1PFwTklqjMEvSY0x+CWpMQa/JDXG4Jekxhj8ktQYg1+SGmPwS1JjDH5JaozBL0mNMfglqTEGvyQ1xuCXpMYs5mTrksbkVI6aD4N/K+Avr6SF5K4eSWqMwS9JjTH4JakxBr8kNcbgl6TGGPyS1BiDX5IaY/BLUmMMfklqjMEvSY1xyIZ5cIgFba8m9bMN/nxvzdzil6TGGPyS1BiDX5IaY/BLUmMW5cvdJE8D3g3sAHyoqlYsRh+ShufBEFufBQ/+JDsA7wOeAtwEfD3JBVX1T0Osz6MUpO3LEL/TrX04LcaunsOAf62qb1bVz4CPA8csQh+S1KRU1cKuMHku8LSqeml//0XAr1XVSTOedyJwYn/34cA3BmxrT+AHW3nNFnts8T0PUbPFHlt8z5vyi1W1ZObCxdjHn00su8enT1WdBpw2fDuQZGVVLduaa7bYY4vveYiaLfbY4nveEouxq+cmYL9p9x8CfH8R+pCkJi1G8H8dOCjJAUnuDTwfuGAR+pCkJi34rp6q2pDkJOBiusM5z6iqGxa6jxmG2KU06Zot9tjiex6iZos9tvieR7bgX+5KkhaXZ+5KUmMMfklqTNPBn2S/JJclWZ3khiSvHrPefZNcmeTavt5bJtjrDkn+McmFE6i1Jsn1Sa5JsnJC/e2e5JNJ/rn/93z8GLUe3vc2dbktyWvG7O+1/f/JqiRnJ7nvOPX6mq/u690w3/6SnJFkXZJV05Y9MMklSW7sr/cYs97z+h43Jtmiwwdnqff2/v/5uiTnJ9l9AjXf2te7Jsnnkzx4nHrTHntDkkqy5wR6fHOS7037uXzGuD0meWWSb/T/P3+xJT2OpaqavQD7AIf2t+8H/AvwyDHqBditv70TcAXwuAn1+jrgY8CFE6i1Bthzwv+WHwFe2t++N7D7hOruANxMdyLKfGvsC3wL2Lm/fw7wO2P2dTCwCtiF7iCJLwAHzaPOE4BDgVXTlv0FsLy/vRw4Zcx6j6A7CfJLwLIJ9PdUYMf+9ilb0t8cNe8/7fargA+MU69fvh/dQSTf3tKf91l6fDPwhnn+vGyq3hP7n5v79Pf3GudncksuTW/xV9Xaqrq6v307sJouJOZbr6rqJ/3dnfrL2N+eJ3kIcBTwoXFrDSHJ/el+sE8HqKqfVdWPJ1T+SODfqurbY9bZEdg5yY50YT3uuSOPAP6hqn5aVRuALwPHbmmRqvoK8KMZi4+h+yClv37WOPWqanVVzevM91nqfb5/zwD/QHcuzrg1b5t2d1e24Pdmln9DgHcC/2tLao1Qc15mqfdyYEVV3dE/Z92k1rc5TQf/dEmWAo+m20ofp84OSa4B1gGXVNVY9XrvovsB3jiBWtD9Inw+yVX90BjjeiiwHjiz3x31oSS7TqAudOd5nD1Ogar6HvCXwHeAtcCtVfX5MftaBTwhyYOS7AI8g7ufmDiOvatqLXQbJ8BeE6o7hJcAfzuJQkneluS7wAuAN41Z62jge1V17SR6m+akfpfUGVuyC24WDwN+I8kVSb6c5LGTaHAUBj+QZDfgXOA1M7Y8tlhV3VlVh9BtBR2W5OAxe3smsK6qrhqnzgyHV9WhwNOBVyR5wpj1dqT7M/b9VfVo4P/S7aIYS3+C39HAJ8asswfdVvQBwIOBXZO8cJyaVbWabjfHJcDngGuBDXO+aDuT5I107/msSdSrqjdW1X59vZM29/w5+toFeCNjfnhswvuBA4FD6DYgTh2z3o7AHsDjgP8JnJNkU0PaTFzzwZ9kJ7rQP6uqzptU3X5Xx5eAp41Z6nDg6CRr6EYyfVKSj47Z2/f763XA+XQjpo7jJuCmaX/dfJLug2BcTweurqpbxqzzZOBbVbW+qn4OnAf813Gbq6rTq+rQqnoC3Z/xN45bs3dLkn0A+usF2wUwqiTHA88EXlD9DuoJ+hjwnDFefyDdh/y1/e/NQ4Crk/zCOE1V1S39ht1G4INM5vfmvH4X8ZV0f9Fv0ZfQ89V08PefrqcDq6vqHROot2TqCIckO9MFzj+PU7Oq/qiqHlJVS+l2e3yxqua9tZpk1yT3m7pN90XdPY6G2MIebwa+m+Th/aIjgUnMr3AcY+7m6X0HeFySXfr/8yPpvs8ZS5K9+uv9gWczmV6hG8Lk+P728cCnJ1R3ItJNpPSHwNFV9dMJ1Txo2t2jGeP3pqqur6q9qmpp/3tzE91BHDeP2eM+0+4ey5i/N8CngCf1tR9Gd1DE0KN1dhbqW+St8QL8Ot3+7uuAa/rLM8ao96vAP/b1VgFvmnC/RzDmUT10++Ov7S83AG+cUG+HACv79/4pYI8x6+0C/BB4wIT6ewtdmKwC/ob+SIoxa/4d3QfctcCR86xxNt1ug5/TBdQJwIOAS+n+grgUeOCY9Y7tb98B3AJcPGa9fwW+O+13ZuQjcOaoeW7/f3Md8Blg33HqzXh8DVt+VM+mevwb4Pq+xwuAfcasd2/go/37vhp40iR+1ke5OGSDJDWm6V09ktQig1+SGmPwS1JjDH5JaozBL0mNMfi16PrRE0+ddv8NSd48odofTvLcSdTazHqel25U0suGXpc0LoNfW4M7gGdv6dC5Q0uywxY8/QTgD6rqiUP1I02Kwa+twQa6+UdfO/OBmVvsSX7SXx/RD2x1TpJ/SbIiyQvSzYdwfZIDp5V5cpK/65/3zP71O/Tjyn+9H3TrZdPqXpbkY3Qn68zs57i+/qokp/TL3kR3MuAHkrx9xvPvleT/9OOtX5jks1PvJ928CHv2t5cl+VJ/e9d+ELCv94PeHdMv/+X+/V3T93xQ/9yL0s0BsSrJb/XPfUz/73NVkounDQHxqiT/1L/+4/P4v9J2YMEnW5dm8T7gumzZZBSPohse+UfAN4EPVdVh6SbUeSUwNTnKUuA36cZwuSzJLwEvphul87FJ7gN8NcnUiJ2HAQdX1bemryzd5CCnAI8B/p1uhNNnVdWfJnkS3VjtMye2eXa//l+hG2VzNXDGZt7XG+mG5nhJPwTIlUm+APw+8O6qOqsfwG4HulFBv19VR/U9PqAff+q9wDFVtb7/MHgb3Uiay4EDquqObOEEKtp+uMWvrUJ1o6L+Nd0kHKP6enVzKtwB/BswFdzX04XtlHOqamNV3Uj3AfFf6MYoenG6IbSvoBsmYWq8mCtnhn7vscCXqhvsbWpUys2NbPrrwCf69d8MjPIdwFOB5X1vXwLuC+wPfA04Ockf0k1M8x/9e31yklOS/EZV3Uo36crBwCV9jT/mrjHzrwPO6kcnbWo0Ud3FLX5tTd5FN2bJmdOWbaDfQOkHWLv3tMfumHZ747T7G7n7z/bMcUmKbra0V1bVxdMfSHIE3bDSmzKfIXPnes3/f2904T79Nc+pe06esjrJFXST8lyc5KVV9cUkj6Hb8v/z/q+W84EbqmpT018eRfdhdTTwv5P8ct01qYoa4Ra/thpV9SO6aRFPmLZ4Dd2uFejG1N9pHqWf1+9rP5BukLpv0E3J9/J+twhJHpbNTx5zBfCbSfbsv/g9jm7mrblcDjynX//edAPtTVnDXe9t+jDEFwOv7D/oSPLo/vqhwDer6j10g4T9ar/76adV9VG6yWYO7d/fkvTzHifZqf9+4F7AflV1Gd3EPrsDu22mf22H3OLX1uZU7j4JxweBTye5km6kytm2xufyDbqA3hv4/ar6zyQfotsddHUfsOvZzBSHVbU2yR/R7a4J8Nmq2tyQyefSDQO9im5O5yuAW/vH3gKcnuRk7j7z21vp/vq5ru9tDd3Y978FvDDJz+nmIf5Tut1Pb0+ykW7kx5dX1c/6L5Dfk+QBdL/n7+rX/9F+WYB31uSmyNQ2xNE5pYEl2a2qfpLkQcCVdDOgjTU2vDQOt/il4V3YH0Fzb+Cthr4Wm1v8ktQYv9yVpMYY/JLUGINfkhpj8EtSYwx+SWrM/wOOnnRhu8fG+AAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "show(scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The random guesser strategy might have seemed hopelessly naive, but it is actually a pretty decent strategy. The average is a bit more than 7 guesses and about 80% of the time it will take 8 guesses or less. Can we improve on that?\n",
    "\n",
    "#  Guessers that Partition Targets\n",
    "\n",
    "A key idea in guessing is to reduce the number of consistent targets. We can think of a guess as **partitioning** the consistent targets into different **branches** of a tree, each branch corresponding to a different reply:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def partition(guess, targets) ->  Dict[Reply, List[str]]:\n",
    "    \"\"\"A partition of targets by the possible replies to guess: {reply: [word, ...]}.\"\"\"\n",
    "    branches = defaultdict(list)\n",
    "    for target in targets:\n",
    "        branches[reply_for(guess, target)].append(target)\n",
    "    return branches"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To make the output easier to see, here's a partition of just 15 targets (every 200th word in `wordlist`):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "defaultdict(list,\n",
       "            {2: ['their', 'logic', 'grime', 'crone', 'fovea'],\n",
       "             3: ['pairs', 'recta', 'divas'],\n",
       "             1: ['flock', 'comfy', 'thrum', 'inode'],\n",
       "             0: ['sloth', 'judos'],\n",
       "             5: ['vicar']})"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "partition('vicar', wordlist[::200])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that after guesssing `'vicar'`, no matter what the reply is, we will be left with no more than five consistent targets. That seems like a good thing, and suggests a strategy: **guess a word that partitions the possible targets into small branches.**\n",
    "\n",
    "If we onjly want to know the *size* of each branch, not the list of words therein, we can use `partition_counts` rather than `partition`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def partition_counts(guess, targets) -> Counter[Reply]: \n",
    "    \"A Counter of the sizes of each branch of the partition of targets by guess.\"\n",
    "    return Counter(reply_for(guess, target) for target in targets)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Counter({2: 5, 3: 3, 1: 4, 0: 2, 5: 1})"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "partition_counts('vicar', wordlist[::200])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below are the partition counts for two possible first guesses, `ouija` and `coder`, on the complete wordlist:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Counter({1: 1848, 3: 65, 2: 755, 0: 175, 4: 1, 5: 1})"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "partition_counts('ouija', wordlist)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Counter({2: 1014, 1: 1030, 3: 327, 0: 433, 4: 40, 5: 1})"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "partition_counts('coder', wordlist)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It looks like `'coder'` is a better first guess than `'ouija'`, because `'ouija'` has a good chance of leaving you with a huge 1848-word branch, while `'coder'` is guaranteed to leave no more than 1030 in a branch, no matter what the reply.\n",
    "\n",
    "What exactly is the metric for deciding which partition is better? What we really want to know is how many additional guesses it will take to handle each branch, but since we don't know that, we can use one of the following proxy metrics and find the best partion by:\n",
    "\n",
    "- **Minimizing the maximum of the partition counts**. That is, we should deduce that `coder`, with its `max` partition count of 1030, is a better guess than `ouija`, with its `max` of 1848. \n",
    "\n",
    "- **Minimizing the expected value of the partition counts**. In probability theory the expected value or **expectation** is the weighted average of a random variable. Here it means the sum, over all branches, of the size of the branch multiplied by the probability of ending up in the branch. We are assuming that every target is equally likely, so the probability of a branch is proportional to the number of targets in it.\n",
    "\n",
    "- **Maximizing the entropy in the partition counts**. Entropy is an information-theoretic measure that is similar to expectation, except that it weights each branch size by its base 2 logarithm (whereas expectation weights it by its actual size). Since the other two metrics are minimizing, we will minimize *negative* entropy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "def max_count(scores: Counter[Reply]) -> int:\n",
    "    \"The maximum count in a Counter of scores.\"\n",
    "    return max(scores.values())\n",
    "    \n",
    "def expectation(scores: Counter[Reply]) -> float:\n",
    "    \"The expected value of a Counter of scores.\"\n",
    "    scale = 1/sum(scores.values())\n",
    "    def P(x): return scale * x\n",
    "    return sum(P(x) * x for x in scores.values())\n",
    "\n",
    "def neg_entropy(scores: Counter[Reply]) -> float: \n",
    "    \"\"\"The negation of the entropy of a Counter of scores.\"\"\"\n",
    "    scale = 1/sum(scores.values())\n",
    "    def P(x): return scale * x\n",
    "    return sum(P(x) * log(P(x), 2) for x in scores.values())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are the top and bottom 4 guess words according to each of the three metrics:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  max_count: wader:1012 cadre:1026 armed:1028 diner:1029 | quota:1548 azoic:1555 axiom:1615 ouija:1848\n",
      "expectation: raved: 813 debar: 818 roved: 827 orbed: 827 | azoic:1120 axiom:1157 audio:1184 ouija:1413\n",
      "neg_entropy: debar:1.95 alder:1.95 raved:1.94 dater:1.94 | juicy:1.57 axiom:1.56 audio:1.49 ouija:1.29\n"
     ]
    }
   ],
   "source": [
    "def extreme(n, metrics=(max_count, expectation, neg_entropy), wordlist=wordlist) -> List[Word]: \n",
    "    \"\"\"The top and bottom n words to guess, according to each metric.\"\"\"\n",
    "    for metric in metrics:\n",
    "        pairs = sorted([metric(partition_counts(g, wordlist)), g] for g in wordlist)\n",
    "        def num(m): return f'{round(m):4d}' if m > 0 else f'{abs(m):4.2f}'\n",
    "        def fmt(pairs): return \" \".join(f'{w}:{num(m)}' for m, w in pairs)\n",
    "        print(f'{metric.__name__:>11}: {fmt(pairs[:n])} | {fmt(pairs[-n:])}')\n",
    "        \n",
    "extreme(4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Which metric is best? We have the tools to answer that: we could use `evaluate_guesser` to get scores for guessers that use each of the three metrics. But that would take a long time. It takes about 5 seconds to compute the best first guess; I don't want to repeat that for every target word."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#  Guesser Trees \n",
    "\n",
    "I can speed up computation by precomputing and caching a **guesser tree**: a tree that has branches for every possible path the game might take, with the best guess for each situation already computed. A guesser tree is either:\n",
    "- An **interior node**, which has a guess and a dict of branches, `Node(guess, {reply: subtree, ...})`, where  each subtree covers all the target words that are consistent with the corresponding reply.\n",
    "- A **leaf word**, such as `'coder'`, indicating that this is the sole remaining consistent target word. Every word in the word list should appear as a leaf in exactly one place in a guesser tree."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "@dataclass \n",
    "class Node:\n",
    "    guess: Word\n",
    "    branches: Dict[Reply, 'Tree']\n",
    "\n",
    "Tree = Union[Word, Node] # A Tree is a leaf Word or an interior Node"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The function `make_tree(metric, targets)` builds a tree that covers all the targets and that, at every node, guesses a word that  minimizes the `metric` applied to the `partition_counts` of the guess."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "def make_tree(metric, targets) -> Tree:\n",
    "    \"\"\"Make a tree that guesses to minimize metric(partition_counts(guess, targets)).\"\"\"\n",
    "    if len(targets) == 1:\n",
    "        return targets[0]\n",
    "    else:\n",
    "        guess = min(targets, key=lambda guess: metric(partition_counts(guess, targets))) \n",
    "        branches = partition(guess, targets)\n",
    "        return Node(guess, {reply: make_tree(metric, branches[reply]) \n",
    "                        for reply in sorted(branches)})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is a tree that covers five words by always making a guess that minimizes the maximum number in the partition counts:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "words5 = ['purge', 'bites', 'sulky', 'patsy', 'hayed']\n",
    "\n",
    "tree5  = make_tree(max_count, words5)\n",
    "\n",
    "assert tree5 == Node(guess='bites', \n",
    "                     branches={1: Node(guess='purge', \n",
    "                                       branches={1: Node(guess='sulky', \n",
    "                                                         branches={1: 'hayed', 5: 'sulky'}), \n",
    "                                                 5: 'purge'}), \n",
    "                               2: 'patsy', \n",
    "                               5: 'bites'})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The tree says that the first guess is `'bites'`, and if the reply is `1` there is a complex subtree to consider (starting with the guess `'purge'`), but if the reply is `2` the target can only be `'patsy'` and of course if the reply is `5` then `'bites'` was the target."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Turning a Tree into a Callable Guesser\n",
    "\n",
    "Now that we've made a tree, we need to use it as a component of a guesser that `play` can call upon. A `TreeGuesser` is an object that stores a tree in the `self.root` and `self.tree` attributes, and while the game is being played, it follows branches in the tree, choosing the guess indicated at each node. This is done within the `__call__` method, because `play` expects a callable.  Note that on the first turn of a new game there is no previous turn, and hence the variable `reply` is `None`, in which case we reset `self.tree` to be the root tree. For subsequent turns, we follow the branch indicated by the reply. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "class TreeGuesser:\n",
    "    \"\"\"Given a guesser tree, use it to create a callable Guesser that can play Jotto.\"\"\"\n",
    "    def __init__(self, tree): self.root = self.tree = tree\n",
    "        \n",
    "    def __call__(self, reply, targets) -> str:\n",
    "        \"\"\"If reply is None, start a new game; otherwise follow the branch for the reply.\n",
    "        Then return the current leaf or interior node guess.\"\"\"\n",
    "        self.tree = self.root if reply is None else self.tree.branches[reply]\n",
    "        return self.tree if isinstance(self.tree, str) else self.tree.guess"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we build a tree and use it to play a game:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Guess 1: \"wader\" Reply: 2 right; Consistent targets: 1004\n",
      "Guess 2: \"lawns\" Reply: 1 right; Consistent targets: 339\n",
      "Guess 3: \"douse\" Reply: 2 right; Consistent targets: 113\n",
      "Guess 4: \"bergs\" Reply: 3 right; Consistent targets: 31\n",
      "Guess 5: \"treks\" Reply: 4 right; Consistent targets: 11\n",
      "Guess 6: \"terms\" Reply: 4 right; Consistent targets: 6\n",
      "Guess 7: \"tiers\" Reply: 4 right; Consistent targets: 5\n",
      "Guess 8: \"crest\" Reply: 4 right; Consistent targets: 4\n",
      "Guess 9: \"frets\" Reply: 4 right; Consistent targets: 3\n",
      "Guess 10: \"prest\" Reply: 4 right; Consistent targets: 2\n",
      "Guess 11: \"treys\" Reply: 4 right; Consistent targets: 1\n",
      "Guess 12: \"verst\" Reply: 5 right; Consistent targets: 1\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "12"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tree = make_tree(max_count, wordlist)\n",
    "\n",
    "play(TreeGuesser(tree), verbose=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Making Inconsistent Guesses\n",
    "\n",
    "So far,  we have always guessed one of the consistent targets. That seems reasonable; why waste a guess on a word that could not possibly be the target? But it turns out that in some cases it *is* a good strategy to guess such a word.\n",
    "\n",
    "I will redefine `make_tree` so that it is passed both the list of consistent target words and the complete word list. It also takes a flag, `inconsistent`. When this flag is true, any word in the wordlist can be considered as a guess; when false, only consistent targets are considered, as before. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "def make_tree(metric, targets, wordlist=wordlist, inconsistent=False) -> Tree:\n",
    "    \"\"\"Make a tree that guesses to minimize metric(partition_counts(guess, targets)).\"\"\"\n",
    "    if len(targets) == 1:\n",
    "        return targets[0]\n",
    "    else:\n",
    "        words = wordlist if (inconsistent and len(targets) > 3) else targets\n",
    "        guess = min(words, key=lambda guess: metric(partition_counts(guess, targets))) \n",
    "        branches = partition(guess, targets)\n",
    "        return Node(guess, {reply: make_tree(metric, branches[reply], wordlist, inconsistent) \n",
    "                            for reply in sorted(branches)})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we see that by default, `make_tree` behaves just as it did before:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "assert make_tree(max_count, words5) == (\n",
    "    Node(guess='bites', \n",
    "         branches={1: Node(guess='purge', \n",
    "                           branches={1: Node(guess='sulky', \n",
    "                                             branches={1: 'hayed', 5: 'sulky'}), \n",
    "                                     5: 'purge'}), \n",
    "                                     2: 'patsy', 5: 'bites'}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But with `inconsistent=True`, we get a better tree:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Node(guess='dashy', branches={0: 'purge', 1: 'bites', 2: 'sulky', 3: 'patsy', 4: 'hayed'})"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tree5b = make_tree(max_count, words5, inconsistent=True)\n",
    "tree5b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This tree guesses an inconsistent word (not one of the words in `words5`) with the first guess. There is no chance that this is the target, but it sets us up so that we will always be able to guess correctly on the second guess. So the minimum, mean, and maximum number of guesses is 2.\n",
    "\n",
    "In contrast, making only consistent guesses can take up to four guesses in the worst case (`'hayed'`), and has a mean number of guesses of 2.4:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mean: 2.40 ± 1.14 guesses, worst: 4, N: 5\n",
      "cumulative: ≤3:80%, ≤4:100%, ≤5:100%, ≤6:100%, ≤7:100%, ≤8:100%, ≤9:100%, ≤10:100%\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEGCAYAAABiq/5QAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAVOUlEQVR4nO3dfbRddX3n8ffHEISKisqFIiENUMZC6RD0msJKp6URHATKQ6WttCBT6UQriFSdik5nqq2uwlgEsU5tlCeXiDKiBZEWGSA6WBeYaAyhKcViOovykIvyIDMWDfnOH2dnuNzc3HtyyT4nN/v9Wuuss/fv7IfvOYTP2fe39/ntVBWSpO543rALkCQNlsEvSR1j8EtSxxj8ktQxBr8kdcxOwy6gH3vssUctWLBg2GVI0qyycuXKR6pqZGL7rAj+BQsWsGLFimGXIUmzSpJ/nqzdrh5J6hiDX5I6xuCXpI4x+CWpYwx+SeoYg1+SOqb14E8yJ8m3k9zQzO+X5I4k9yb5XJKd265BkvSMQRzxvx1YO27+AuCiqjoQeBQ4cwA1SJIarQZ/knnAccAnm/kAS4DPN4tcCZzUZg2SpGdr+5e7FwN/CLywmX8Z8FhVbWjm7wf2mWzFJEuBpQDz589vuUxNtOC8Lw+7hO3SuvOPG3YJ0nPW2hF/kuOB9VW1cnzzJItOeguwqlpWVaNVNToystlQE5KkGWrziH8xcEKSY4FdgBfR+wtg9yQ7NUf984AHWqxBkjRBa0f8VfWeqppXVQuANwC3VtXvALcBpzSLnQFc11YNkqTNDeM6/ncD70jyXXp9/pcOoQZJ6qyBDMtcVcuB5c30fcCiQexXkrQ5f7krSR1j8EtSxxj8ktQxBr8kdYzBL0kdY/BLUscY/JLUMQa/JHWMwS9JHWPwS1LHGPyS1DEGvyR1jMEvSR1j8EtSxxj8ktQxBr8kdUybN1vfJcmdSb6T5O4k72/ar0jyvSSrmsfCtmqQJG2uzTtwPQUsqaonk8wFbk/yN81r/6mqPt/iviVJW9Ba8FdVAU82s3ObR7W1P0lSf1rt408yJ8kqYD1wc1Xd0bz0wSSrk1yU5Plt1iBJerZWg7+qnq6qhcA8YFGSQ4D3AD8HvBp4KfDuydZNsjTJiiQrxsbG2ixTkjplIFf1VNVjwHLgmKp6sHqeAi4HFm1hnWVVNVpVoyMjI4MoU5I6oc2rekaS7N5M7wocBfxDkr2btgAnAWvaqkGStLk2r+rZG7gyyRx6XzDXVNUNSW5NMgIEWAW8pcUaJEkTtHlVz2rgsEnal7S1T0nS9PzlriR1jMEvSR1j8EtSxxj8ktQxBr8kdYzBL0kdY/BLUscY/JLUMQa/JHWMwS9JHWPwS1LHGPyS1DEGvyR1jMEvSR1j8EtSxxj8ktQxBr8kdUyb99zdJcmdSb6T5O4k72/a90tyR5J7k3wuyc5t1SBJ2lybR/xPAUuq6lBgIXBMksOBC4CLqupA4FHgzBZrkCRN0FrwV8+Tzezc5lHAEuDzTfuVwElt1SBJ2lyrffxJ5iRZBawHbgb+CXisqjY0i9wP7LOFdZcmWZFkxdjYWJtlSlKntBr8VfV0VS0E5gGLgIMmW2wL6y6rqtGqGh0ZGWmzTEnqlIFc1VNVjwHLgcOB3ZPs1Lw0D3hgEDVIknravKpnJMnuzfSuwFHAWuA24JRmsTOA69qqQZK0uZ2mX2TG9gauTDKH3hfMNVV1Q5K/Bz6b5APAt4FLW6xBkjRBa8FfVauBwyZpv49ef78kaQj85a4kdYzBL0kdY/BLUscY/JLUMQa/JHWMwS9JHWPwS1LHGPyS1DEGvyR1jMEvSR1j8EtSxxj8ktQx0wZ/kgOSPL+ZPjLJOZuGW5YkzT79HPFfCzyd5GfpDaG8H/CZVquSJLWmn+Df2Nwj92Tg4qr6A3pj7UuSZqF+gv8nSU6ld7esG5q2ue2VJElqUz/B/7vAEcAHq+p7SfYDPt1uWZKktvQT/EdX1TlVdTVAVX0P+NF0KyXZN8ltSdYmuTvJ25v29yX5lySrmsexz+0tSJK2Rj/Bf8Ykbf+hj/U2AO+sqoOAw4GzkhzcvHZRVS1sHjf2V6okaVvY4j13m3793wb2S3L9uJdeCHx/ug1X1YPAg830D5OsBfZ5buVKkp6rqW62/nf0gnsP4MJx7T8EVm/NTpIsoHfj9TuAxcDZSd4IrKD3V8Gjk6yzFFgKMH/+/K3ZnSRpClvs6qmqf66q5VV1BLAOmFtVXwXWArv2u4Mku9H7LcC5VfUE8JfAAcBCel8sF062XlUtq6rRqhodGRnpd3eSpGn088vd/wh8Hvirpmke8Nf9bDzJXHqhf1VVfQGgqh6uqqeraiPwCWDRTAqXJM1MPyd3z6LXPfMEQFXdC+w53UpJQu+Xvmur6sPj2sf/+OtkYM3WFCxJem6m6uPf5Kmq+nEvxyHJTkD1sd5i4HTgriSrmrb3AqcmWdhsYx3w5q0tWpI0c/0E/1eTvBfYNcnRwFuBL023UlXdDmSSl7x8U5KGqJ+unvOAMeAuekfnNwJ/1GZRkqT2THvEP+4k7CfaL0eS1LZpgz/JXWzep/84vWvwP1BV0/6YS5K0/einj/9vgKd5Zgz+NzTPTwBXAL+27cuSJLWln+BfXFWLx83fleTrVbU4yWltFSZJakc/J3d3S/KLm2aSLAJ2a2Y3tFKVJKk1/Rzxnwlc3gy9AL2xes5M8gLgz1qrTJLUiimDP8nzgP2r6heSvBhIVT02bpFrWq1OkrTNTdnV01zKeXYz/fiE0JckzUL99PHfnORdzR21Xrrp0XplkqRW9NPH/6bm+axxbQXsv+3LkSS1rZ9f7u43iEIkSYPRzxE/SQ4BDgZ22dRWVZ9qqyhJUnv6GbLhj4Ej6QX/jcDrgNsBg1+SZqF+Tu6eArwGeKiqfhc4FHh+q1VJklrTT/D/qLmsc0OSFwHr8cSuJM1a/fTxr0iyO71hmVcCTwJ3tlqVJKk1/VzV89Zm8uNJ/hZ4UVWtnm69JPvSOw/w08BGYFlVfaT5DcDngAX0br34m1X16MzKlyRtrWm7epLcsmm6qtZV1erxbVPYALyzqg4CDgfOSnIwvTt63VJVBwK3NPOSpAHZ4hF/kl2AnwL2SPISnrl/7ouAl0+34ap6EHiwmf5hkrXAPsCJ9K4SArgSWA68e2blS5K21lRdPW8GzqUX8it5JvifAD62NTtJsgA4DLgD2Kv5UqCqHkyy5xbWWQosBZg/f/7W7O5ZFpz35RmvuyNbd/5xwy5hh+S/t8lN9+/Nz23L2vh/dYtdPVX1keZXu++qqv2rar/mcWhV/UW/O2iGc74WOLeqnuh3vapaVlWjVTU6MjLS72qSpGlM28dfVR+d6caTzKUX+ldV1Rea5oeT7N28vje9y0MlSQPSz3X8M5IkwKXA2qr68LiXrgfOaKbPAK5rqwZJ0ua2GPxJFjfPM/2V7mLgdGBJklXN41jgfODoJPcCRzfzkqQBmerk7iXAq4BvAK/c2g1X1e08c0J4otds7fYkSdvGVMH/kySXA/skuWTii1V1TntlSZLaMlXwHw8cBSyhdzmnJGkHsMXgr6pHgM8mWVtV3xlgTZKkFvVzVc/3k3wxyfokDye5Nsm81iuTJLWin+C/nN4lmC+nN+TCl5o2SdIs1E/w71lVl1fVhuZxBeBPaSVpluon+MeSnJZkTvM4Dfh+24VJktrRT/C/CfhN4CF6o22e0rRJkmahfm7E8r+BEwZQiyRpAFobq0eStH0y+CWpYwx+SeqYvoM/yeFJbk3y9SQntVmUJKk9U91z96er6qFxTe+gd5I3wN8Bf91ybZKkFkx1Vc/Hk6wEPlRV/wo8Bvw2sJHefXclSbPQVPfcPQlYBdyQ5HR6N17fCPwUYFePJM1SU/bxV9WXgH8P7A58Abinqi6pqrFBFCdJ2vamuvXiCUluB24F1gBvAE5OcnWSA6bbcJLLmhE914xre1+Sf5lwK0ZJ0gBN1cf/AeAIYFfgxqpaBLwjyYHAB+l9EUzlCuAvgE9NaL+oqv58ZuVKkp6rqYL/cXrhviuwflNjVd3L9KFPVX0tyYLnWJ8kaRubqo//ZHoncjfQu5pnWzk7yeqmK+glW1ooydIkK5KsGBvzlIIkbStTXdXzSFV9tKo+XlXb6vLNvwQOABbSG+nzwin2v6yqRqtqdGTE4f8laVsZ6JANVfVwVT1dVRuBTwCLBrl/SdKAgz/J3uNmT6Z3tZAkaYCmHY9/ppJcDRwJ7JHkfuCPgSOTLAQKWAe8ua39S5Im11rwV9WpkzRf2tb+JEn9cVhmSeoYg1+SOsbgl6SOMfglqWMMfknqGINfkjrG4JekjjH4JaljDH5J6hiDX5I6xuCXpI4x+CWpYwx+SeoYg1+SOsbgl6SOMfglqWMMfknqmNaCP8llSdYnWTOu7aVJbk5yb/P8krb2L0maXJtH/FcAx0xoOw+4paoOBG5p5iVJA9Ra8FfV14AfTGg+Ebiymb4SOKmt/UuSJjfoPv69qupBgOZ5zy0tmGRpkhVJVoyNjQ2sQEna0W23J3erallVjVbV6MjIyLDLkaQdxqCD/+EkewM0z+sHvH9J6rxBB//1wBnN9BnAdQPevyR1XpuXc14NfAN4RZL7k5wJnA8cneRe4OhmXpI0QDu1teGqOnULL72mrX1Kkqa33Z7clSS1w+CXpI4x+CWpYwx+SeoYg1+SOsbgl6SOMfglqWMMfknqGINfkjrG4JekjjH4JaljDH5J6hiDX5I6xuCXpI4x+CWpYwx+SeoYg1+SOqa1O3BNJck64IfA08CGqhodRh2S1EVDCf7Gr1bVI0PcvyR1kl09ktQxwwr+Ar6SZGWSpZMtkGRpkhVJVoyNjQ24PEnacQ0r+BdX1SuB1wFnJfnliQtU1bKqGq2q0ZGRkcFXKEk7qKEEf1U90DyvB74ILBpGHZLURQMP/iQvSPLCTdPAa4E1g65DkrpqGFf17AV8Mcmm/X+mqv52CHVIUicNPPir6j7g0EHvV5LU4+WcktQxBr8kdYzBL0kdY/BLUscY/JLUMQa/JHWMwS9JHWPwS1LHGPyS1DEGvyR1jMEvSR1j8EtSxxj8ktQxBr8kdYzBL0kdY/BLUscY/JLUMUMJ/iTHJLknyXeTnDeMGiSpq4Zxs/U5wMeA1wEHA6cmOXjQdUhSVw3jiH8R8N2quq+qfgx8FjhxCHVIUielqga7w+QU4Jiq+r1m/nTgF6vq7AnLLQWWNrOvAO4ZaKHt2AN4ZNhFzEJ+bjPj5zYzO9Ln9jNVNTKxcachFJJJ2jb79qmqZcCy9ssZnCQrqmp02HXMNn5uM+PnNjNd+NyG0dVzP7DvuPl5wANDqEOSOmkYwf9N4MAk+yXZGXgDcP0Q6pCkThp4V09VbUhyNnATMAe4rKruHnQdQ7JDdV0NkJ/bzPi5zcwO/7kN/OSuJGm4/OWuJHWMwS9JHWPwD0CSy5KsT7Jm2LXMJkn2TXJbkrVJ7k7y9mHXNBsk2SXJnUm+03xu7x92TbNJkjlJvp3khmHX0haDfzCuAI4ZdhGz0AbgnVV1EHA4cJbDe/TlKWBJVR0KLASOSXL4kGuaTd4OrB12EW0y+Aegqr4G/GDYdcw2VfVgVX2rmf4hvf8Z9xluVdu/6nmymZ3bPLyKow9J5gHHAZ8cdi1tMvg1KyRZABwG3DHcSmaHprtiFbAeuLmq/Nz6czHwh8DGYRfSJoNf270kuwHXAudW1RPDrmc2qKqnq2ohvV/GL0pyyLBr2t4lOR5YX1Urh11L2wx+bdeSzKUX+ldV1ReGXc9sU1WPAcvxHFM/FgMnJFlHb9TgJUk+PdyS2mHwa7uVJMClwNqq+vCw65ktkowk2b2Z3hU4CviH4Va1/auq91TVvKpaQG8omVur6rQhl9UKg38AklwNfAN4RZL7k5w57JpmicXA6fSOvFY1j2OHXdQssDdwW5LV9MbGurmqdthLE7X1HLJBkjrGI35J6hiDX5I6xuCXpI4x+CWpYwx+SeoYg19Dl6SSXDhu/l1J3reNtn1FklO2xbam2c9vNKOI3tb2vqTnyuDX9uAp4NeT7DHsQsZLMmcrFj8TeGtV/Wpb9UjbisGv7cEGevc5/YOJL0w8Yk/yZPN8ZJKvJrkmyT8mOT/J7zTj0N+V5IBxmzkqyf9qlju+WX9Okg8l+WaS1UnePG67tyX5DHDXJPWc2mx/TZILmrb/CvwS8PEkH5qw/POS/PdmXPwbkty46f0kWbfpyy7JaJLlzfQLmns4fLMZF/7Epv3nm/e3qqn5wGbZLzdj769J8lvNsq9qPp+VSW5KsnfTfk6Sv2/W/+wM/ltpBzDwm61LW/AxYHWS/7YV6xwKHERvyOv7gE9W1aLmhi1vA85tllsA/ApwAL1ftP4s8Ebg8ap6dZLnA19P8pVm+UXAIVX1vfE7S/Jy4ALgVcCjwFeSnFRVf5JkCfCuqloxocZfb/b/C8Ce9IaWvmya9/Wf6Q0X8KZm6IU7k/xP4C3AR6rqqiQ7A3OAY4EHquq4psYXN+MbfRQ4sarGmi+DDwJvAs4D9quqpzYN66Du8Yhf24Vm1M1PAedsxWrfbMbsfwr4J2BTcN9FL2w3uaaqNlbVvfS+IH4OeC3wxmbo4juAlwEHNsvfOTH0G68GllfVWFVtAK4CfnmaGn8J+B/N/h8C+jkH8FrgvKa25cAuwHx6w368N8m7gZ+pqh817/WoJBck+XdV9TjwCuAQ4OZmG39Eb5ROgNXAVUlOo/eXljrII35tTy4GvgVcPq5tA80BSjNo287jXntq3PTGcfMbefa/7YnjkhQQ4G1VddP4F5IcCfyfLdSXad/B1q3z/98bvXAfv87rq+qeCcuvTXIHvRuF3JTk96rq1iSvonfk/2fNXy1fBO6uqiMm2edx9L6sTgD+S5Kfb77E1CEe8Wu7UVU/AK6hd6J0k3X0ulYATqR3N6mt9RtNX/sBwP7APcBNwO833SIk+TdJXjDNdu4AfiXJHs2J31OBr06zzu3A65v97wUcOe61dTzz3l4/rv0m4G3NFx1JDmue9wfuq6pLgOuBf9t0P/3fqvo08OfAK5v3N5LkiGa9uc35gecB+1bVbfRuNrI7sNs09WsH5BG/tjcXAmePm/8EcF2SO4Fb2PLR+FTuoRfQewFvqap/TfJJet1B32oCdgw4aaqNVNWDSd5Dr7smwI1Vdd00+74WeA2wBvhHel8ejzevvR+4NMl7efadxf6U3l8/q5va1gHHA78FnJbkJ8BDwJ/Q6376UJKNwE+A36+qHzcnkC9J8mJ6/59f3Oz/001bgIua8frVMY7OKbUsyW5V9WSSlwF3Aoub/n5pKDzil9p3Q3MFzc7Anxr6GjaP+CWpYzy5K0kdY/BLUscY/JLUMQa/JHWMwS9JHfP/AL3Jv99s+GYiAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "show(evaluate_guesser(TreeGuesser(tree5), words5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Comparing Metrics\n",
    "\n",
    "Now we will compare the three metrics, with and without inconsistent targets.\n",
    "The function `show_metric` makes the appropriate `TreeGuesser` and calls `evaluate_guesser` and `show` to display results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "def show_metric(metric, inconsistent=False, wordlist=wordlist) -> None:\n",
    "    \"\"\"Show statistics and histogram for a guesser that minimizes `metric` over partition counts.\"\"\"\n",
    "    tree = make_tree(metric, wordlist, wordlist, inconsistent)\n",
    "    show(evaluate_guesser(TreeGuesser(tree), wordlist))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Comparing Metrics with Consistent Guesses Only"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mean: 7.15 ± 1.81 guesses, worst: 18, N: 2,845\n",
      "cumulative: ≤3:1%, ≤4:4%, ≤5:13%, ≤6:35%, ≤7:67%, ≤8:86%, ≤9:92%, ≤10:95%\n",
      "CPU times: user 12.5 s, sys: 12.1 ms, total: 12.5 s\n",
      "Wall time: 12.5 s\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEGCAYAAABiq/5QAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAYWklEQVR4nO3de5gkdX3v8fdHLnITQRkQgZxFgh4NiYDrBg9qENCj4uHiJZGoIRGDMaKi8Zwg5hiN+gSiiJcYPSg3I15QQBGJQBA0GgV3EZbF1RBxVXRhxxvo0aDLfvNH1YZx2Jnp7pma2d16v56nn66u7vrVt2d6Pl3z6+rfL1WFJKk/7rfQBUiS5pfBL0k9Y/BLUs8Y/JLUMwa/JPXMlgtdwCB22WWXWrRo0UKXIUmblGXLlv2gqsYmr98kgn/RokUsXbp0ocuQpE1Kkm9vaL1dPZLUMwa/JPWMwS9JPWPwS1LPGPyS1DMGvyT1jMEvST1j8EtSzxj8ktQzm8Q3d7X5WXTyp0fedtWpR8xhJVL/eMQvST1j8EtSzxj8ktQzBr8k9YzBL0k9Y/BLUs8Y/JLUMwa/JPWMwS9JPWPwS1LPGPyS1DMGvyT1jMEvST3TWfAn2SbJdUluTHJzkje06/dOcm2SW5J8NMnWXdUgSbqvLo/47wYOrapHA/sDT01yEHAacEZV7Qv8GDi+wxokSZN0FvzV+Fl7c6v2UsChwMfb9ecBR3dVgyTpvjrt40+yRZIbgDXAlcA3gZ9U1dr2IbcBe0yx7QlJliZZOj4+3mWZktQrnQZ/Vd1TVfsDewJLgEdu6GFTbHtmVS2uqsVjY2NdlilJvTIvZ/VU1U+Aa4CDgJ2SrJ/ycU/g+/NRgySp0eVZPWNJdmqXtwUOB1YCVwPPbh92HPDJrmqQJN1Xl5Ot7w6cl2QLmjeYC6rq0iRfAz6S5E3AV4GzOqxBkjRJZ8FfVcuBAzaw/laa/n5J0gLwm7uS1DMGvyT1jMEvST1j8EtSzxj8ktQzBr8k9YzBL0k9Y/BLUs8Y/JLUMwa/JPWMwS9JPWPwS1LPGPyS1DMGvyT1jMEvST1j8EtSzxj8ktQzBr8k9YzBL0k9Y/BLUs8Y/JLUMwa/JPVMZ8GfZK8kVydZmeTmJK9o178+yfeS3NBent5VDZKk+9qyw7bXAn9RVdcneQCwLMmV7X1nVNVbO9y3JGkKnQV/Va0GVrfLP02yEtijq/1JkgYzL338SRYBBwDXtqtOTLI8ydlJdp5imxOSLE2ydHx8fD7KlKRe6Dz4k+wAXAicVFV3Ae8B9gH2p/mP4PQNbVdVZ1bV4qpaPDY21nWZktQbnQZ/kq1oQv/8qroIoKruqKp7qmod8D5gSZc1SJJ+XZdn9QQ4C1hZVW+bsH73CQ87BljRVQ2SpPvq8qyeg4EXADcluaFddwpwbJL9gQJWAS/usAZJ0iRdntXzBSAbuOuyrvYpSZqZ39yVpJ4x+CWpZwx+SeoZg1+Sesbgl6SeMfglqWcMfknqGYNfknrG4JeknjH4JalnDH5J6hmDX5J6xuCXpJ4x+CWpZ7ocj1+bqUUnf3rkbVedesQcViJpFDMe8SfZJ8n92+VDkrw8yU7dlyZJ6sIgXT0XAvck+U2aqRT3Bj7UaVWSpM4MEvzrqmotzfy4b6+qVwK7z7CNJGkjNUjw/yrJscBxwKXtuq26K0mS1KVBgv9PgMcBb66qbyXZG/hgt2VJkroyyFk9T66ql6+/0Yb/LzqsSZLUoUGO+I/bwLo/nuM6JEnzZMoj/rZf/w+BvZNcMuGuBwA/7LowSVI3puvq+VdgNbALcPqE9T8Fls/UcJK9gA8ADwHWAWdW1TuSPAj4KLAIWAX8flX9eJTiJUnDm7Krp6q+XVXXVNXjaAJ6q6r6HLAS2HaAttcCf1FVjwQOAl6a5FHAycBVVbUvcFV7W5I0Twb55u6fAh8H/l+7ak/gEzNtV1Wrq+r6dvmnNG8YewBHAee1DzsPOHr4siVJoxrkw92XAgcDdwFU1S3ArsPsJMki4ADgWmC3qlrdtrV6qraSnJBkaZKl4+Pjw+xOkjSNQYL/7qr65fobSbYEatAdJNmBZtiHk6rqrkG3q6ozq2pxVS0eGxsbdDNJ0gwGCf7PJTkF2DbJk4GPAZ8apPEkW9GE/vlVdVG7+o4ku7f37w6sGb5sSdKoBgn+k4Fx4CbgxcBlwF/NtFGS0AzqtrKq3jbhrku497sBxwGfHKZgSdLszPjN3apaB7yvvQzjYOAFwE1JbmjXnQKcClyQ5HjgO8BzhmxXkjQLMwZ/kpu4b5/+ncBS4E1VtcEvc1XVF4BM0exhwxQpSZo7g4zV80/APdw7Bv9z2+u7gHOB/zX3ZUmSujJI8B9cVQdPuH1Tki9W1cFJnt9VYZKkbgwS/Dsk+d2quhYgyRJgh/a+tZ1VJs3AuX+l0QwS/McD57Tn40MzVs/xSbYH/razyiRJnZg2+JPcD3hYVf12kgcCqaqfTHjIBZ1WJ0mac9Oex9+eynliu3znpNCXJG2CBvkC15VJXp1kryQPWn/pvDJJUicG6eN/YXv90gnrCnjY3JcjSeraIN/c3Xs+CpEkzY9BjvhJsh/wKGCb9euq6gNdFSVJ6s4gQzb8NXAITfBfBjwN+ALNtIqSpE3MIB/uPptmbJ3bq+pPgEcD9++0KklSZwYJ/l+0p3WuTbIjzfj5frArSZuoQfr4lybZiWZY5mXAz4DrOq1KktSZQc7q+fN28b1JPgPsWFXLuy1LktSVGbt6kly1frmqVlXV8onrJEmblimP+JNsA2wH7JJkZ+6dVGVH4KHzUJskqQPTdfW8GDiJJuSXcW/w3wW8u+O6JEkdmTL4q+odwDuSvKyq3jWPNUmSOjRjH7+hL0mbl0HO45ckbUamDP4kB7fXfktXkjYj0x3xv7O9/tJ8FCJJmh/TndXzqyTnAHskeefkO6vq5dM1nORs4BnAmqrar133euBPgfH2YadU1WWjFC5JGs10wf8M4HDgUJrTOYd1LvD33HcUzzOq6q0jtCdJmgPTnc75A+AjSVZW1Y3DNlxVn0+yaBa1SZI6MMhZPT9McnGSNUnuSHJhkj1nsc8TkyxPcnb7jeANSnJCkqVJlo6Pj0/1MEnSkAYJ/nOAS2i+wbsH8Kl23SjeA+wD7A+sBk6f6oFVdWZVLa6qxWNjYyPuTpI02SDBv2tVnVNVa9vLucBISVxVd1TVPe34/u8DlozSjiRpdIME/3iS5yfZor08H/jhKDtLsvuEm8cAK0ZpR5I0ukEmYnkhzdk5ZwAF/Gu7blpJPkwzV+8uSW4D/ho4JMn+bTuraAaCkyTNo0EmYvkOcOSwDVfVsRtYfdaw7UiS5pZj9UhSzxj8ktQzBr8k9czAwZ/koCSfTfLFJEd3WZQkqTvTzbn7kKq6fcKqV9F8yBuaM3s+0XFtkqQOTHdWz3uTLAPeUlX/AfwE+ENgHc28u5KkTdCUXT1VdTRwA3BpkhfQTLy+DtgOsKtHkjZR0/bxV9WngP8J7ARcBHyjqt5ZVY6aJkmbqOmmXjwyyReAz9IMrfBc4JgkH06yz3wVKEmaW9P18b8JeBywLXBZVS0BXpVkX+DNNG8EkqRNzHTBfydNuG8LrFm/sqpuwdCXpE3WdH38x9B8kLuW5mweSdJmYKapF981j7VIkuaBQzZIUs8Y/JLUMwa/JPWMwS9JPWPwS1LPGPyS1DMGvyT1jMEvST1j8EtSz3QW/EnOTrImyYoJ6x6U5Mokt7TXO3e1f0nShnV5xH8u8NRJ604GrqqqfYGr2tuSpHnUWfBX1eeBH01afRRwXrt8Hs7kJUnzbr77+HerqtUA7fWuUz0wyQlJliZZOj7uhF+SNFc22g93q+rMqlpcVYvHxsYWuhxJ2mzMd/DfkWR3gPZ6zQyPlyTNsfkO/kuA49rl44BPzvP+Jan3ujyd88PAl4BHJLktyfHAqcCTk9wCPLm9LUmaR9PNuTsrVXXsFHcd1tU+JUkz22g/3JUkdaOzI35tnBad/OmRt1116hFzWImkheIRvyT1jMEvST1j8EtSzxj8ktQzBr8k9YzBL0k9Y/BLUs8Y/JLUMwa/JPWMwS9JPWPwS1LPGPyS1DMGvyT1jMEvST1j8EtSzxj8ktQzBr8k9YzBL0k949SL6jWnolQfecQvST1j8EtSzyxIV0+SVcBPgXuAtVW1eCHqkKQ+Wsg+/idV1Q8WcP+S1Et29UhSzyxU8BdwRZJlSU7Y0AOSnJBkaZKl4+Pj81yeJG2+Fir4D66qA4GnAS9N8sTJD6iqM6tqcVUtHhsbm/8KJWkztSDBX1Xfb6/XABcDSxaiDknqo3kP/iTbJ3nA+mXgKcCK+a5DkvpqIc7q2Q24OMn6/X+oqj6zAHVIUi/Ne/BX1a3Ao+d7v5KkhqdzSlLPGPyS1DMGvyT1jMEvST1j8EtSzzgRizRLs5nMBZzQRfPPI35J6hmDX5J6xq6eTYjzw0qaCx7xS1LPGPyS1DMGvyT1jMEvST1j8EtSz3hWj7QR8Etgmk8e8UtSzxj8ktQzBr8k9Yx9/NJmws8JNCiP+CWpZwx+SeoZu3rmiQOsSdpYGPyS/oufE/TDggR/kqcC7wC2AN5fVacuRB2S5p5vHhu/eQ/+JFsA7waeDNwGfCXJJVX1tfmuRdLGyTePbi3EEf8S4N+r6laAJB8BjgI22uC3f17a9MzFm8fG0sZcS1XNeaPT7jB5NvDUqnpRe/sFwO9W1YmTHncCcEJ78xHANzoqaRfgB7ZhG7ZhG5twG1P5b1U1NnnlQhzxZwPr7vPuU1VnAmd2XkyytKoW24Zt2IZtbKptDGshzuO/Ddhrwu09ge8vQB2S1EsLEfxfAfZNsneSrYHnApcsQB2S1Evz3tVTVWuTnAhcTnM659lVdfN81zHBXHQn2YZt2IZtLGQbQ5n3D3clSQvLsXokqWcMfknqmd4Gf5Kzk6xJsmLE7fdKcnWSlUluTvKKEdrYJsl1SW5s23jDKLW0bW2R5KtJLh1x+1VJbkpyQ5KlI7axU5KPJ/l6+3N53JDbP6Ld//rLXUlOGqGOV7Y/zxVJPpxkmxHaeEW7/c3D1LCh11WSByW5Mskt7fXOI7TxnLaWdUlmPPVvijbe0v5ulie5OMlOI7Txxnb7G5JckeShw7Yx4b5XJ6kku4xQx+uTfG/Ca+Xpo9SR5GVJvtH+bP9uhDo+OqGGVUluGKGN/ZN8ef3fXpIl07UxJ6qqlxfgicCBwIoRt98dOLBdfgDwb8CjhmwjwA7t8lbAtcBBI9bzKuBDwKUjbr8K2GWWP9PzgBe1y1sDO82irS2A22m+gDLMdnsA3wK2bW9fAPzxkG3sB6wAtqM5AeKfgX1HfV0Bfwec3C6fDJw2QhuPpPki4zXA4hHreAqwZbt82oh17Dhh+eXAe4dto12/F80JHt+e6XU3RR2vB149xO90Q208qf3d3r+9vesoz2XC/acDrxuhjiuAp7XLTweuGeb1Osqlt0f8VfV54Eez2H51VV3fLv8UWEkTOsO0UVX1s/bmVu1l6E/bk+wJHAG8f9ht50qSHWle1GcBVNUvq+ons2jyMOCbVfXtEbbdEtg2yZY04T3s90QeCXy5qn5eVWuBzwHHDLLhFK+ro2jeFGmvjx62japaWVUDf3t9ijauaJ8PwJdpvkMzbBt3Tbi5PTO8Xqf5OzsD+D8zbT9DGwOboo2XAKdW1d3tY9aMWkeSAL8PfHiENgrYsV1+IPPwvabeBv9cSrIIOIDmiH3Ybbdo/z1cA1xZVUO3Abyd5o9o3QjbrlfAFUmWpRkuY1gPA8aBc9oup/cn2X4W9TyXGf6INqSqvge8FfgOsBq4s6quGLKZFcATkzw4yXY0R2F7zbDNdHarqtVtfauBXWfR1lx5IfBPo2yY5M1Jvgs8D3jdCNsfCXyvqm4cZf8TnNh2O509U/fZFB4OPCHJtUk+l+Sxs6jlCcAdVXXLCNueBLyl/Zm+FXjNLOoYiME/S0l2AC4ETpp0NDSQqrqnqvanOfpakmS/Iff/DGBNVS0bdt+THFxVBwJPA16a5IlDbr8lzb+w76mqA4D/T9OtMbQ0X+w7EvjYCNvuTHOEvTfwUGD7JM8fpo2qWknTFXIl8BngRmDttBttQpK8lub5nD/K9lX12qraq93+xJkeP2nf2wGvZYQ3jEneA+wD7E/zBn/6CG1sCewMHAT8b+CC9sh9FMcywoFK6yXAK9uf6Stp/2vuksE/C0m2ogn986vqotm01XaLXAM8dchNDwaOTLIK+AhwaJIPjrD/77fXa4CLaUZRHcZtwG0T/mP5OM0bwSieBlxfVXeMsO3hwLeqaryqfgVcBPyPYRupqrOq6sCqeiLNv+ajHMmtd0eS3QHa62m7FLqU5DjgGcDzqu1UnoUPAc8acpt9aN6Ub2xfs3sC1yd5yDCNVNUd7UHTOuB9DP96heY1e1Hb5XodzX/M037QvCFtl+IzgY+OUAPAcTSvU2gOdjr/cNfgH1F7ZHAWsLKq3jZiG2Prz6xIsi1NaH19mDaq6jVVtWdVLaLpHvlsVQ11hJtk+yQPWL9M8yHgUGc7VdXtwHeTPKJddRijD7U9m6On7wAHJdmu/R0dRvP5y1CS7Npe/wbNH/Wo9UAzJMlx7fJxwCdn0dbI0kyA9JfAkVX18xHb2HfCzSMZ/vV6U1XtWlWL2tfsbTQnSdw+ZB27T7h5DEO+XlufAA5t23s4zQkJo4ySeTjw9aq6bYRtoenT/712+VBmd5AxmK4/Pd5YLzR/yKuBX9G8+I4fcvvH0/SLLwduaC9PH7KN3wG+2raxghnOCBigvUMY4awemv75G9vLzcBrR9z//sDS9vl8Ath5hDa2A34IPHAWP4c30ATSCuAfac/aGLKNf6F547oROGw2ryvgwcBVNH/QVwEPGqGNY9rlu4E7gMtHaOPfge9OeL3OdEbOhtq4sP25Lgc+BewxbBuT7l/FzGf1bKiOfwRuauu4BNh9hDa2Bj7YPp/rgUNHeS7AucCfzeL18XhgWftauxZ4zKiv/UEvDtkgST1jV48k9YzBL0k9Y/BLUs8Y/JLUMwa/JPWMwa8F147QePqE269O8vo5avvcJM+ei7Zm2M9z0oxIenXX+5Jmy+DXxuBu4JkzDc8735JsMcTDjwf+vKqe1FU90lwx+LUxWEsz7+grJ98x+Yg9yc/a60PagbUuSPJvSU5N8rw08xvclGSfCc0cnuRf2sc9o91+izTj03+lHejrxRPavTrJh2i+IDS5nmPb9lckOa1d9zqaL+G8N8lbJj3+fkn+oR3v/dIkl61/Pu347bu0y4uTXNMub98OPPaVdsC7o9r1v9U+vxvamvdtH/vpNHM6rEjyB+1jH9P+fJYluXzCkBEvT/K1dvuPjPC70mZg3idbl6bwbmB5ZpgMY5JH0wyh/CPgVuD9VbUkzaQ4L6MZ9RBgEc1X4vcBrk7ym8Af0Yzc+dgk9we+mGT9KJ5LgP2q6lsTd5Zm0pHTgMcAP6YZzfToqvqbJIfSjA8/eRKbZ7b7/22aUTlXAmfP8LxeSzP0xgvbIT2uS/LPwJ8B76iq89uB7LagGTn0+1V1RFvjA9sxpN4FHFVV4+2bwZtpRuQ8Gdi7qu7ODBOxaPPlEb82CtWMbPoBmsk9BvWVauZFuBv4Js2EFtAcqS+a8LgLqmpdNUPm3gr8d5rxiP4ozZDY19IMq7B+HJrrJod+67E0k2SMVzOu/fk0cxBM5/HAx9r93w4M8hnAU4CT29quAbYBfgP4EnBKkr+kmaDmF+1zPTzJaUmeUFV30kzYsh9wZdvGX3Hv2PvLgfPTjFi62Yw4quF4xK+Nydtpxkw5Z8K6tbQHKO2ga1tPuO/uCcvrJtxex6+/tiePS1I0s5+9rKoun3hHkkNohpTekFGG7J1um/96bjThPnGbZ9V9J15ZmeRamkl3Lk/yoqr6bJLH0Bz5/237X8vFwM1VtaGpL4+gebM6Evi/SX6r7p2cRT3hEb82GlX1I5qpEo+fsHoVTdcKNOPsbzVC089p+9r3oRmQ7hs00/69pO0WIcnDM/PEMdcCv5dkl/aD32NpZueazheAZ7X7341mIL31VnHvc5s4vPHlwMvaNzqSHNBePwy4tareSTMw2e+03U8/r6oP0kzicWD7/MbSznmcZKv284H7AXtV1dU0E/fsBOwwQ/3aDHnEr43N6fz65B7vAz6Z5DqakS2nOhqfzjdoAno3mlEU/yPJ+2m6g65vA3acmadEXJ3kNTTdNQEuq6qZhli+kGZo6BU08zJfC9zZ3vcG4Kwkp/Drs7e9kea/n+VtbatoxtD/A+D5SX5FMx/x39B0P70lyTqaER9fUlW/bD9AfmeSB9L8nb+93f8H23UBzqjZTY+pTZSjc0odS7JDVf0syYOB62hmOxtq/HlpLnnEL3Xv0vYMmq2BNxr6Wmge8UtSz/jhriT1jMEvST1j8EtSzxj8ktQzBr8k9cx/ApVQwE2Dcoy8AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%time show_metric(max_count, False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mean: 7.14 ± 1.82 guesses, worst: 17, N: 2,845\n",
      "cumulative: ≤3:1%, ≤4:4%, ≤5:13%, ≤6:36%, ≤7:68%, ≤8:85%, ≤9:91%, ≤10:95%\n",
      "CPU times: user 12.3 s, sys: 10.7 ms, total: 12.3 s\n",
      "Wall time: 12.3 s\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEGCAYAAABiq/5QAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAYBklEQVR4nO3de5hkdX3n8fdHLnJV0GlwBNxBgq6JiYAjwSUa5OIquFy8JBIvJOJijHh314lmjUZ9AlHEy7q6yNWIKAooIhEJgkZjwBkchsHREHFUdGTaG+hq0GG++eOcXtpmuru6qqt6Zs779Tz11Dmn6nzPt3pqvvWr3zn1+6WqkCR1x/0WOgFJ0mhZ+CWpYyz8ktQxFn5J6hgLvyR1zLYLnUAvFi1aVEuWLFnoNCRpi7JixYofVtXY1O1bROFfsmQJy5cvX+g0JGmLkuTbm9puV48kdYyFX5I6xsIvSR1j4ZekjrHwS1LHWPglqWMs/JLUMRZ+SeoYC78kdcwW8ctddceSZZ8eaP+1px0zT5lIWy9b/JLUMRZ+SeoYC78kdYyFX5I6xsIvSR1j4ZekjrHwS1LHWPglqWMs/JLUMRZ+SeoYC78kdczQCn+SHZLckOSmJLckeVO7fd8k1ye5NclHk2w/rBwkSfc1zBb/3cDhVfUY4ADgKUkOAU4Hzqyq/YGfACcPMQdJ0hRDK/zV+Hm7ul17K+Bw4OPt9guA44eVgyTpvobax59kmyQrgfXA1cA3gZ9W1Yb2KbcDe02z7ylJlidZPj4+Psw0JalThlr4q+qeqjoA2Bs4GHjUpp42zb5nVdXSqlo6NjY2zDQlqVNGclVPVf0UuA44BNgtycQEMHsD3x9FDpKkxjCv6hlLslu7vCNwJLAGuBZ4Zvu0k4BPDisHSdJ9DXPqxcXABUm2ofmAubiqrkjyNeAjSd4CfBU4Z4g5SJKmGFrhr6pVwIGb2H4bTX+/JGkB+MtdSeoYC78kdYyFX5I6xsIvSR1j4ZekjrHwS1LHWPglqWMs/JLUMRZ+SeoYC78kdYyFX5I6xsIvSR1j4ZekjrHwS1LHWPglqWMs/JLUMRZ+SeoYC78kdYyFX5I6xsIvSR1j4ZekjrHwS1LHWPglqWOGVviT7JPk2iRrktyS5OXt9jcm+V6Sle3t6GHlIEm6r22HGHsD8OqqujHJrsCKJFe3j51ZVW8f4rElSdMYWuGvqnXAunb5Z0nWAHsN63iSpN6MpI8/yRLgQOD6dtOpSVYlOTfJ7tPsc0qS5UmWj4+PjyJNSeqEoRf+JLsAlwCvqKq7gPcB+wEH0HwjOGNT+1XVWVW1tKqWjo2NDTtNSeqMoRb+JNvRFP0Lq+pSgKq6o6ruqaqNwAeAg4eZgyTpNw3zqp4A5wBrquodk7YvnvS0E4DVw8pBknRfw7yq51DgecDNSVa2214HnJjkAKCAtcCLhpiDJGmKYV7V80Ugm3joymEdU5I0O3+5K0kdY+GXpI6x8EtSx1j4JaljLPyS1DEWfknqGAu/JHWMhV+SOmaYv9xVhyxZ9umB9l972jHzlImk2djil6SOsfBLUsdY+CWpYyz8ktQxFn5J6hgLvyR1jIVfkjpm1sKfZL8k92+XD0vysiS7DT81SdIw9NLivwS4J8lv0cyhuy/w4aFmJUkaml4K/8aq2kAzMfo7q+qVwOJZ9pEkbaZ6Kfy/TnIicBJwRbttu+GlJEkapl4K/58BjwfeWlXfSrIv8KHhpiVJGpZeBmk7qqpeNrHSFv9fDjEnSdIQ9dLiP2kT2/50nvOQJI3ItC3+tl//T4B9k1w+6aFdgR/NFjjJPsAHgYcAG4GzqupdSR4EfBRYAqwF/qiqftLvC5Akzc1MXT3/DKwDFgFnTNr+M2BVD7E3AK+uqhuT7AqsSHI1zbeFa6rqtCTLgGXAa/tJXpI0d9N29VTVt6vquqp6PE3LfLuq+jywBthxtsBVta6qbmyXf9butxdwHHBB+7QLgOMHegWSpDnp5Ze7/x34OPB/2017A5+Yy0GSLAEOBK4H9qyqddB8OAB7TLPPKUmWJ1k+Pj4+l8NJkmbQy8ndlwCHAncBVNWtTFOsNyXJLjS//n1FVd3V635VdVZVLa2qpWNjY73uJkmaRS+F/+6q+tXESpJtgeoleJLtaIr+hVV1abv5jiSL28cXA+vnlrIkaRC9FP7PJ3kdsGOSo4CPAZ+abackoRnbZ01VvWPSQ5dz7yWiJwGfnFvKkqRB9FL4lwHjwM3Ai4Argb/qYb9DgecBhydZ2d6OBk4DjkpyK3BUuy5JGpFZf7lbVRuBD7S3nlXVF4FM8/ARc4klSZo/sxb+JDdz3z79O4HlwFuqatYfc0mSNh+9jNXzD8A93DsG/7Pb+7uA84H/Nv9pSZKGpZfCf2hVHTpp/eYkX6qqQ5M8d1iJSZKGo5eTu7sk+f2JlSQHA7u0qxuGkpUkaWh6afGfDJzX/hALmrF6Tk6yM/C3Q8tMkjQUMxb+JPcDHl5Vv5vkgUCq6qeTnnLxULOTJM27Gbt62ks5T22X75xS9CVJW6Be+vivTvKaJPskedDEbeiZSZKGopc+/he09y+ZtK2Ah89/OpKkYevll7v7jiIRSdJo9NLiJ8mjgd8GdpjYVlUfHFZSkqTh6WXIhr8GDqMp/FcCTwW+SDOfriRpC9NLi/+ZwGOAr1bVnyXZEzh7uGlJg1my7NMD7b/2tGPmKRNp89PLVT2/bC/r3JDkATQTp3hiV5K2UL20+Jcn2Y1mWOYVwM+BG4aalSRpaHq5qucv2sX3J/kM8ICqWjXctCRJwzJrV0+SayaWq2ptVa2avE2StGWZtsWfZAdgJ2BRkt25dzatBwAPHUFukqQhmKmr50XAK2iK/AruLfx3Ae8dcl6SpCGZtvBX1buAdyV5aVW9Z4Q5SZKGaNY+fou+JG1dermOX5K0FZm28Cc5tL2//+jSkSQN20wt/ne391/uJ3CSc5OsT7J60rY3JvlekpXt7eh+YkuS+jfTVT2/TnIesFeSd099sKpeNkvs84H/zX0Hczuzqt4+pywlSfNmpsL/NOBI4HCayznnpKq+kGRJf2lJkoZlpss5fwh8JMmaqrppHo95apLnA8uBV1fVTzb1pCSnAKcAPOxhD5vHw0tSt/VyVc+PklzW9tffkeSSJHv3ebz3AfsBBwDrgDOme2JVnVVVS6tq6djYWJ+HkyRN1UvhPw+4nOYXvHsBn2q3zVlV3VFV97TDPH8AOLifOJKk/vVS+PeoqvOqakN7Ox/oqwmeZPGk1ROA1dM9V5I0HL2Mxz+e5LnARe36icCPZtspyUU0UzYuSnI78NfAYUkOAApYSzMekCRphHop/C+guSzzTJqC/c/tthlV1Ymb2HzOnLKTJM27XiZi+Q5w7AhykSSNgGP1SFLHWPglqWMs/JLUMT0X/iSHJPlcki8lOX6YSUmShmemOXcfUlU/mLTpVTQneUNzZc8nhpybJGkIZrqq5/1JVgBvq6p/B34K/AmwkWbeXUnSFmjarp6qOh5YCVyR5Hk0E69vBHYC7OqRpC3UjH38VfUp4L8CuwGXAt+oqndX1fgokpMkzb+Zpl48NskXgc/RjKnzbOCEJBcl2W9UCUqS5tdMffxvAR4P7AhcWVUHA69Ksj/wVpoPAknSFmamwn8nTXHfEVg/sbGqbsWiL0lbrJn6+E+gOZG7geZqHknSVmC2qRffM8JcJEkj4JANktQxFn5J6hgLvyR1jIVfkjrGwi9JHWPhl6SOsfBLUsdY+CWpYyz8ktQxQyv8Sc5Nsj7J6knbHpTk6iS3tve7D+v4kqRNG2aL/3zgKVO2LQOuqar9gWvadUnSCA2t8FfVF4AfT9l8HHBBu3wBzuQlSSM36j7+PatqHUB7v8d0T0xySpLlSZaPjzvhlyTNl8325G5VnVVVS6tq6djY2EKnI0lbjVEX/juSLAZo79fP8nxJ0jwbdeG/HDipXT4J+OSIjy9JnTfMyzkvAr4MPDLJ7UlOBk4DjkpyK3BUuy5JGqGZ5twdSFWdOM1DRwzrmJKk2Q2t8GvLsGTZpwfaf+1px8xTJpJGZbO9qkeSNBwWfknqGAu/JHWMhV+SOsbCL0kdY+GXpI6x8EtSx1j4JaljLPyS1DEWfknqGAu/JHWMhV+SOsbCL0kdY+GXpI6x8EtSx1j4JaljLPyS1DHOwCXNwBnKtDWyxS9JHWPhl6SOsfBLUsdY+CWpYxbk5G6StcDPgHuADVW1dCHykKQuWsirep5UVT9cwONLUifZ1SNJHbNQhb+AzyZZkeSUTT0hySlJlidZPj4+PuL0JGnrtVCF/9CqOgh4KvCSJE+c+oSqOquqllbV0rGxsdFnKElbqQUp/FX1/fZ+PXAZcPBC5CFJXTTywp9k5yS7TiwDTwZWjzoPSeqqhbiqZ0/gsiQTx/9wVX1mAfKQpE4aeeGvqtuAx4z6uJKkhpdzSlLHWPglqWMs/JLUMRZ+SeoYZ+CSRsCZvLQ5scUvSR1j4ZekjrHwS1LH2Me/hbLPWFK/bPFLUsdY+CWpYyz8ktQx9vFLWxDP7Wg+2OKXpI6x8EtSx1j4JaljLPyS1DGe3JU6yJPE3WaLX5I6xsIvSR1j4ZekjrGPf8TsW9XWxPfzlskWvyR1zIK0+JM8BXgXsA1wdlWdthB5SNo8DPrNAfz2MBcjL/xJtgHeCxwF3A58JcnlVfW1UeciaeviB0hvFqLFfzDwb1V1G0CSjwDHAZt14bcvU+qO+foA2Vw/iFJV8x50xgMmzwSeUlUvbNefB/x+VZ065XmnAKe0q48EvjGklBYBPzSOcYxjnC04znT+U1WNTd24EC3+bGLbfT59quos4KyhJ5Msr6qlxjGOcYyzpcaZq4W4qud2YJ9J63sD31+APCSpkxai8H8F2D/Jvkm2B54NXL4AeUhSJ428q6eqNiQ5FbiK5nLOc6vqllHnMcl8dScZxzjGMc5CxZmTkZ/clSQtLH+5K0kdY+GXpI7pbOFPcm6S9UlWDxhnnyTXJlmT5JYkL+8zzg5JbkhyUxvnTQPmtU2Srya5YoAYa5PcnGRlkuUDxNktyceTfL39Oz2+jxiPbPOYuN2V5BV95vPK9m+8OslFSXboM87L2xi3zCWXTb33kjwoydVJbm3vd+8zzrPafDYm6ekywWnivK3991qV5LIku/UZ581tjJVJPpvkof3EmfTYa5JUkkV95vPGJN+b9D46ut98krw0yTfav/ff9ZnPRyflsjbJytnizIuq6uQNeCJwELB6wDiLgYPa5V2BfwV+u484AXZpl7cDrgcOGSCvVwEfBq4YIMZaYNE8/K0vAF7YLm8P7DZgvG2AH9D8OGWu++4FfAvYsV2/GPjTPuI8GlgN7ERzkcQ/Avv3+94D/g5Y1i4vA07vM86jaH7weB2wdIB8ngxs2y6fPkA+D5i0/DLg/f3EabfvQ3NRyLd7eV9Ok88bgdfM8d96U3Ge1P6b379d36Pf1zXp8TOAN8z1vdjPrbMt/qr6AvDjeYizrqpubJd/BqyhKS5zjVNV9fN2dbv21teZ9yR7A8cAZ/ez/3xK8gCaN/w5AFX1q6r66YBhjwC+WVXf7nP/bYEdk2xLU7j7+R3Jo4B/qapfVNUG4PPACb3sOM177ziaD0ja++P7iVNVa6pqTr9ynybOZ9vXBfAvNL+36SfOXZNWd6aH9/QM/zfPBP5nLzFmiTMn08R5MXBaVd3dPmf9IPkkCfBHwEWDZdubzhb+YUiyBDiQprXez/7btF/11gNXV1VfcYB30vwH2djn/hMK+GySFWmG0OjHw4Fx4Ly26+nsJDsPmNez6fM/SFV9D3g78B1gHXBnVX22j1CrgScmeXCSnYCj+c0fJs7VnlW1rs1xHbDHALHm2wuAf+h35yRvTfJd4DnAG/qMcSzwvaq6qd88Jjm17X46t5cutWk8AnhCkuuTfD7J4wbM6QnAHVV164BxemLhnydJdgEuAV4xpZXTs6q6p6oOoGldHZzk0X3k8TRgfVWt6CeHKQ6tqoOApwIvSfLEPmJsS/P19n1VdSDw/2i6MvqS5kd/xwIf63P/3Wla1/sCDwV2TvLcucapqjU0XSBXA58BbgI2zLjTFijJ62le14X9xqiq11fVPm2MU2d7/iZy2Al4PX1+aEzxPmA/4ACaD/4z+oyzLbA7cAjwP4CL21Z7v05kRK19sPDPiyTb0RT9C6vq0kHjtV0h1wFP6WP3Q4Fjk6wFPgIcnuRDfebx/fZ+PXAZzciqc3U7cPukby8fp/kg6NdTgRur6o4+9z8S+FZVjVfVr4FLgf/ST6CqOqeqDqqqJ9J8hR+ktXZHksUA7f2sXQfDluQk4GnAc6rthB7Qh4Fn9LHffjQf1De17+u9gRuTPGSugarqjraBtRH4AP29p6F5X1/adtHeQPPtetYTzpvSdjk+Hfhon7nMmYV/QO2n/DnAmqp6xwBxxiaunEiyI02B+vpc41TVX1bV3lW1hKZL5HNVNecWbZKdk+w6sUxzsm/OV0BV1Q+A7yZ5ZLvpCAYbgnvQltF3gEOS7NT+2x1Bc15mzpLs0d4/jOY/7iB5XQ6c1C6fBHxygFgDSzNZ0muBY6vqFwPE2X/S6rH0956+uar2qKol7fv6dpoLKn7QRz6LJ62eQB/v6dYngMPbmI+guWih31E2jwS+XlW397n/3I3iDPLmeKP5T7oO+DXNG+nkPuP8AU1f+CpgZXs7uo84vwd8tY2zmnk4uw8cRp9X9dD0zd/U3m4BXj9AHgcAy9vX9glg9z7j7AT8CHjggH+XN9EUoNXA39NemdFHnH+i+RC7CThikPce8GDgGppvDdcAD+ozzgnt8t3AHcBVfcb5N+C7k97TvVyNs6k4l7R/51XAp4C9+okz5fG19HZVz6by+Xvg5jafy4HFfcbZHvhQ+9puBA7v93UB5wN/Psh7eq43h2yQpI6xq0eSOsbCL0kdY+GXpI6x8EtSx1j4JaljLPxacO1oi2dMWn9NkjfOU+zzkzxzPmLNcpxnpRl59NphH0salIVfm4O7gaf3MtTuKCXZZg5PPxn4i6p60rDykeaLhV+bgw00c4++cuoDU1vsSX7e3h/WDo51cZJ/TXJakuekmdPg5iT7TQpzZJJ/ap/3tHb/bdKMOf+VdsCuF02Ke22SD9P80GdqPie28VcnOb3d9gaaH/K9P8nbpjz/fkn+Tztm+xVJrpx4Pe3464va5aVJrmuXd24HEPtKO7Ddce3232lf38o25/3b5346zTwOq5P8cfvcx7Z/nxVJrpo0HMTLknyt3f8jffxbaSsw8snWpWm8F1iVHia0mOQxNMMj/xi4DTi7qg5OMxnOS4GJyVGWAH9IM+bLtUl+C3g+zcicj0tyf+BLSSZG6TwYeHRVfWvywdJMInI68FjgJzQjlx5fVX+T5HCacd6nTljz9Pb4v0sz4uYa4NxZXtfraYbaeEE7jMcNSf4R+HPgXVV1YTtY3TY0o4J+v6qOaXN8YDt21HuA46pqvP0weCvNKJvLgH2r6u70MLmKtk62+LVZqGZE0w/STNbRq69UMx/C3cA3gYnCfTNNsZ1wcVVtrGbI29uA/0wz9tDz0wyDfT3NkAkT48rcMLXotx4HXFfNAG8TI1bONmLpHwAfa4//A6CXcwBPBpa1uV0H7AA8DPgy8Lokr6WZhOaX7Ws9MsnpSZ5QVXfSTMTyaODqNsZfce94+quAC9OMSLrVjSaq3tji1+bknTTjnpw3adsG2gZKO6ja9pMeu3vS8sZJ6xv5zff21HFJimbGs5dW1VWTH0hyGM3Q0ZvSz7C7M+3z/18bTXGfvM8z6r4TqqxJcj3NJDtXJXlhVX0uyWNpWv5/235ruQy4pao2NcXlMTQfVscC/yvJ79S9E66oI2zxa7NRVT+mmQrx5Emb19J0rUAzjv52fYR+VtvXvh/N4HPfoJnC78VttwhJHpHZJ4i5HvjDJIvaE78n0sy8NZMvAs9oj78nzcB5E9Zy72ubPFzxVcBL2w86khzY3j8cuK2q3k0zwNjvtd1Pv6iqD9FMMHNQ+/rG0s5tnGS79vzA/YB9qupamol6dgN2mSV/bYVs8Wtzcwa/OVnHB4BPJrmBZtTK6VrjM/kGTYHek2YUxH9PcjZNd9CNbYEdZ5bpDqtqXZK/pOmuCXBlVc02fPIlNEM/r6aZj/l64M72sTcB5yR5Hb85a9ubab79rGpzW0szLv4fA89N8muaOYf/hqb76W1JNtKM+vjiqvpVewL53UkeSPP//J3t8T/UbgtwZg0+Daa2QI7OKQ1Zkl2q6udJHgzcQDOz2ZzHkpfmiy1+afiuaK+g2R54s0VfC80WvyR1jCd3JaljLPyS1DEWfknqGAu/JHWMhV+SOuY/AK9dwnrB43dhAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%time show_metric(expectation, False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mean: 7.10 ± 1.79 guesses, worst: 19, N: 2,845\n",
      "cumulative: ≤3:1%, ≤4:4%, ≤5:13%, ≤6:36%, ≤7:69%, ≤8:86%, ≤9:92%, ≤10:96%\n",
      "CPU times: user 12.8 s, sys: 21.5 ms, total: 12.8 s\n",
      "Wall time: 12.8 s\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEGCAYAAABiq/5QAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAYu0lEQVR4nO3de5xcZX3H8c+XEOROwCwYIXQxRWqLEiBGbBQxgOViA3hppV5owcYbIKitUazFqq+GIqJYK41ys9xEAblWiAgqXgIbDEkwIopRAyFZL9yqRUJ+/eN5tgybndk5s3tmN3m+79drXnPmzPzO+c3umd8888yZ51FEYGZm5dhsrBMwM7PucuE3MyuMC7+ZWWFc+M3MCuPCb2ZWmM3HOoF2TJ48OXp7e8c6DTOzjcrixYt/FRE9g9dvFIW/t7eXvr6+sU7DzGyjIunnQ613V4+ZWWFc+M3MCuPCb2ZWGBd+M7PCuPCbmRXGhd/MrDAu/GZmhXHhNzMrjAu/mVlhNopf7trGrXfeDR3FrZx/5ChnYmbgFr+ZWXFc+M3MCuPCb2ZWGBd+M7PCuPCbmRXGhd/MrDAu/GZmhXHhNzMrjAu/mVlhXPjNzArjwm9mVpjaCr+kLSXdIeluSfdI+khev4ekRZLuk/QlSVvUlYOZmW2ozhb/E8DsiNgHmA4cJukA4Azg7IjYE/gtcEKNOZiZ2SC1Ff5IHs83J+ZLALOBr+T1FwFH15WDmZltqNY+fkkTJC0B1gILgZ8CD0fEuvyQVcCuTWLnSuqT1Nff319nmmZmRam18EfEUxExHdgNmAm8YKiHNYldEBEzImJGT09PnWmamRWlK2f1RMTDwG3AAcAkSQMTwOwGPNiNHMzMLKnzrJ4eSZPy8lbAIcAK4FbgdflhxwHX1JWDmZltqM6pF6cAF0maQHqDuSIirpf0Q+BySR8DfgCcV2MOZmY2SG2FPyKWAvsOsf5+Un+/mZmNAf9y18ysMC78ZmaFceE3MyuMC7+ZWWFc+M3MCuPCb2ZWGBd+M7PCuPCbmRXGhd/MrDAu/GZmhXHhNzMrjAu/mVlhXPjNzArjwm9mVhgXfjOzwrjwm5kVxoXfzKwwLvxmZoVx4TczK4wLv5lZYVz4zcwK48JvZlYYF34zs8LUVvglTZV0q6QVku6R9O68/nRJD0haki9H1JWDmZltaPMat70OeG9E3CVpO2CxpIX5vrMj4hM17tvMzJqorfBHxGpgdV5+TNIKYNe69mdmZu3pSh+/pF5gX2BRXnWipKWSzpe0Y5OYuZL6JPX19/d3I00zsyLUXvglbQtcCZwSEY8CnwOmAdNJnwjOGiouIhZExIyImNHT01N3mmZmxai18EuaSCr6l0TEVQARsSYinoqI9cDngZl15mBmZs9U51k9As4DVkTEJxvWT2l42DHA8rpyMDOzDdV5Vs8s4M3AMklL8roPAsdKmg4EsBJ4W405mJnZIHWe1XM7oCHuurGufZqZ2fD8y10zs8K48JuZFcaF38ysMHV+uWubiN55N3QUt3L+kaOciZmNBrf4zcwK48JvZlYYF34zs8K48JuZFcaF38ysMC78ZmaFceE3MyuMC7+ZWWFc+M3MCuPCb2ZWGBd+M7PCuPCbmRVm2MIvaZqkZ+XlgySdLGlS/amZmVkd2mnxXwk8JemPSXPo7gFcWmtWZmZWm3YK//qIWEeaGP1TEXEqMGWYGDMzG6faKfxPSjoWOA64Pq+bWF9KZmZWp3YK/98BLwU+HhE/k7QHcHG9aZmZWV3amYHr0Ig4eeBGLv6/rzEnMzOrUTst/uOGWPe3o5yHmZl1SdMWf+7X/xtgD0nXNty1HfDr4TYsaSrwReA5wHpgQUR8WtJOwJeAXmAl8FcR8dtOn4CZmVXTqqvnu8BqYDJwVsP6x4ClbWx7HfDeiLhL0nbAYkkLSZ8WbomI+ZLmAfOA93eSvJmZVde0qycifh4Rt0XES0kt84kR8U1gBbDVcBuOiNURcVdefizH7QocBVyUH3YRcPSInoGZmVXSzi93/x74CvCfedVuwFer7ERSL7AvsAjYJSJWQ3pzAHZuEjNXUp+kvv7+/iq7MzOzFtr5cvddwCzgUYCIuI8mxXookrYl/fr3lIh4tN24iFgQETMiYkZPT0+7YWZmNox2Cv8TEfGHgRuSNgeinY1Lmkgq+pdExFV59RpJU/L9U4C11VI2M7ORaKfwf1PSB4GtJB0KfBm4brggSSKN7bMiIj7ZcNe1PH2K6HHANdVSNjOzkWin8M8D+oFlwNuAG4EPtRE3C3gzMFvSknw5ApgPHCrpPuDQfNvMzLpk2F/uRsR64PP50raIuB1Qk7sPrrItMzMbPcMWfknL2LBP/xGgD/hYRAz7Yy4zMxs/2hmr57+Bp3h6DP435OtHgQuBvxz9tMzMrC7tFP5ZETGr4fYySd+JiFmS3lRXYmZmVo92vtzdVtJLBm5Imglsm2+uqyUrMzOrTTst/hOAC/IPsSCN1XOCpG2Af60tMzMzq0XLwi9pM+B5EfFCSTsAioiHGx5yRa3ZmZnZqGvZ1ZNP5TwxLz8yqOibmdlGqJ0+/oWS3idpqqSdBi61Z2ZmZrVop4//+Hz9roZ1ATxv9NMxM7O6tfPL3T26kYiZmXVHOy1+JO0N/Cmw5cC6iPhiXUmZmVl92hmy4Z+Bg0iF/0bgcOB20ny6Zma2kWnny93XkQZVeygi/g7YB3hWrVmZmVlt2unq+X1ErJe0TtL2pIlT/MWudU3vvBs6ils5/8hRzsRs09BO4e+TNIk0LPNi4HHgjlqzMjOz2rRzVs878+K5kr4GbB8RS+tNy8zM6jJsH7+kWwaWI2JlRCxtXGdmZhuXpi1+SVsCWwOTJe3I07NpbQ88twu5mZlZDVp19bwNOIVU5BfzdOF/FPhszXmZmVlNmhb+iPg08GlJJ0XEZ7qYk5mZ1WjYPn4XfTOzTUs7P+AyM7NNSNPCL2lWvvavdM3MNiGtWvzn5OvvdbJhSedLWitpecO60yU9IGlJvhzRybbNzKxzrc7qeVLSBcCuks4ZfGdEnDzMti8E/p0NB3M7OyI+USlLMzMbNa0K/6uBQ4DZpNM5K4mIb0nq7SwtMzOrS6vTOX8FXC5pRUTcPYr7PFHSW4A+4L0R8duhHiRpLjAXYPfddx/F3ZuZla2ds3p+Lenq3F+/RtKVknbrcH+fA6YB04HVwFnNHhgRCyJiRkTM6Onp6XB3ZmY2WDuF/wLgWtIveHcFrsvrKouINRHxVESsJ432ObOT7ZiZWefaKfw7R8QFEbEuXy4EOmqCS5rScPMYYHmzx5qZWT3aGY+/X9KbgMvy7WOBXw8XJOky0pSNkyWtAv4ZOEjSdCCAlaTxgMzMrIvaKfzHk07LPJtUsL+b17UUEccOsfq8StmZmdmoa2cill8Ac7qQi5mZdYHH6jEzK4wLv5lZYVz4zcwK03bhl3SApG9I+o6ko+tMyszM6tNqzt3nRMRDDaveQ/qSV6Qze75ac25mZlaDVmf1nCtpMXBmRPwv8DDwN8B60ry7Zma2EWra1RMRRwNLgOslvZk08fp6YGvAXT1mZhupln38EXEd8BfAJOAq4N6IOCci+ruRnJmZjb5WUy/OkXQ78A3SmDpvAI6RdJmkad1K0MzMRlerPv6PAS8FtgJujIiZwHsk7Ql8nPRGYGZmG5lWhf8RUnHfClg7sDIi7sNF38xso9Wqj/8Y0he560hn85iZ2SZguKkXP9PFXMzMrAs8ZIOZWWFc+M3MCuPCb2ZWGBd+M7PCuPCbmRXGhd/MrDAu/GZmhXHhNzMrjAu/mVlhaiv8ks6XtFbS8oZ1O0laKOm+fL1jXfs3M7Oh1dnivxA4bNC6ecAtEbEncEu+bWZmXVRb4Y+IbwG/GbT6KOCivHwRnsnLzKzrut3Hv0tErAbI1zs3e6CkuZL6JPX193vCLzOz0TJuv9yNiAURMSMiZvT09Ix1OmZmm4xuF/41kqYA5Ou1wzzezMxGWbcL/7XAcXn5OOCaLu/fzKx4dZ7OeRnwPWAvSasknQDMBw6VdB9waL5tZmZd1GrO3RGJiGOb3HVwXfu0ofXOu6GjuJXzjxzlTMxsPBi3X+6amVk9XPjNzArjwm9mVhgXfjOzwrjwm5kVxoXfzKwwLvxmZoVx4TczK4wLv5lZYVz4zcwK48JvZlYYF34zs8K48JuZFcaF38ysMC78ZmaFceE3MyuMC7+ZWWFc+M3MClPb1Itm44WnnjR7Jrf4zcwK48JvZlYYF34zs8K48JuZFWZMvtyVtBJ4DHgKWBcRM8YiDzOzEo3lWT2vjIhfjeH+zcyK5K4eM7PCjFXhD+BmSYslzR3qAZLmSuqT1Nff39/l9MzMNl1jVfhnRcR+wOHAuyQdOPgBEbEgImZExIyenp7uZ2hmtokak8IfEQ/m67XA1cDMscjDzKxEXS/8kraRtN3AMvAqYHm38zAzK9VYnNWzC3C1pIH9XxoRXxuDPMzMitT1wh8R9wP7dHu/ZmaW+HROM7PCuPCbmRXGhd/MrDAu/GZmhfEMXGbD6HQGL/AsXjY+ucVvZlYYF34zs8K48JuZFcZ9/OOc+5fNbLS5xW9mVhgXfjOzwrjwm5kVxoXfzKww/nLXrGb+gt7GG7f4zcwK48JvZlYYF34zs8K48JuZFcaF38ysMC78ZmaF8emcZuOcTwe10ebCXzO/aM1svHHhN9vEufFhg41JH7+kwyTdK+knkuaNRQ5mZqXqeotf0gTgs8ChwCrgTknXRsQPu52LmbWn008NA58YxjrenmksunpmAj+JiPsBJF0OHAWMy8Lvj8lmGz+/cTyTIqK7O5ReBxwWEW/Nt98MvCQiThz0uLnA3HxzL+DeGtKZDPxqjLfheMf7GHT8SI+BZv4oInoGrxyLFr+GWLfBu09ELAAW1JqI1BcRM8ZyG453vI9Bx4/0GKhqLL7cXQVMbbi9G/DgGORhZlaksSj8dwJ7StpD0hbAG4BrxyAPM7Midb2rJyLWSToRuAmYAJwfEfd0O49sNLqSRroNxzt+pMY6B8ePbXxlXf9y18zMxpYHaTMzK4wLv5lZYYos/JLOl7RW0vIO46dKulXSCkn3SHp3xfgtJd0h6e4c/5EO85gg6QeSru8gdqWkZZKWSOrrIH6SpK9I+lH+O7y0Yvxeed8Dl0clnVJxG6fmv99ySZdJ2rJi/Ltz7D3t7Huo40bSTpIWSrovX+9YMf71ef/rJbU8pa9J/Jn5f7BU0tWSJlWM/2iOXSLpZknPrZpDw33vkxSSJlfM4XRJDzQcC0dU3b+kk/IwMPdI+reK+/9Sw75XSlpSMX66pO8PvJYkzawYv4+k7+XX43WStm8WP2oiorgLcCCwH7C8w/gpwH55eTvgx8CfVogXsG1enggsAg7oII/3AJcC13cQuxKYPIK/4UXAW/PyFsCkEWxrAvAQ6ccm7cbsCvwM2CrfvgL42wrxewPLga1JJzl8Hdiz6nED/BswLy/PA86oGP8C0g8UbwNmdLD/VwGb5+UzOtj/9g3LJwPnVs0hr59KOmHj562OqyY5nA68r83/21Dxr8z/v2fl2ztXzb/h/rOAD1fc/83A4Xn5COC2ivF3Aq/Iy8cDH233OO70UmSLPyK+BfxmBPGrI+KuvPwYsIJUiNqNj4h4PN+cmC+VvmWXtBtwJPCFKnGjIbdIDgTOA4iIP0TEwyPY5MHATyPi5xXjNge2krQ5qYBX+T3IC4DvR8TvImId8E3gmFYBTY6bo0hvguTro6vER8SKiGjrV+lN4m/O+QN8n/S7mCrxjzbc3IZhjsMWr52zgX8cQXxbmsS/A5gfEU/kx6ztZP+SBPwVcFnF+AAGWuk70OI4bBK/F/CtvLwQeG2z+NFSZOEfTZJ6gX1JrfYqcRPyR8q1wMKIqBQPfIr0QltfMW5AADdLWqw0PEYVzwP6gQtyV9MXJG3TYR6QfsvR9MU2lIh4APgE8AtgNfBIRNxcYRPLgQMlPVvS1qSW2tRhYoayS0SszjmtBnbuYBuj5Xjgv6sGSfq4pF8CbwQ+3EH8HOCBiLi7amyDE3OX0/mtusuaeD7wckmLJH1T0os7zOHlwJqIuK9i3CnAmflv+AngAxXjlwNz8vLr6ew4rMSFfwQkbQtcCZwyqOU0rIh4KiKmk1poMyXtXWG/rwbWRsTiSgk/06yI2A84HHiXpAMrxG5O+rj6uYjYF/gfUjdHZUo/4psDfLli3I6k1vYewHOBbSS9qd34iFhB6hpZCHwNuBtY1zJoHJN0Gin/S6rGRsRpETE1x5443OMH7Xdr4DQ6eMNo8DlgGjCd9CZ+VsX4zYEdgQOAfwCuyK33qo6lYgMkewdwav4bnkr+JFzB8aTX4GJS1/EfOsihEhf+DkmaSCr6l0TEVZ1uJ3eR3AYcViFsFjBH0krgcmC2pIsr7vfBfL0WuJo0amq7VgGrGj6lfIX0RtCJw4G7ImJNxbhDgJ9FRH9EPAlcBfx5lQ1ExHkRsV9EHEj6+F21pQewRtIUgHzdtJuhLpKOA14NvDFyR3GHLqV6N8M00pvv3fl43A24S9Jz2t1ARKzJDaH1wOepdixCOh6vyl2od5A+BTf9gnkoubvwNcCXKu4b4DjS8QepAVMp/4j4UUS8KiL2J73x/LSDHCpx4e9Abk2cB6yIiE92EN8zcPaFpK1IRexH7cZHxAciYreI6CV1k3wjItpu7UraRtJ2A8ukLwjbPsMpIh4Cfilpr7zqYDofVrvTVtYvgAMkbZ3/HweTvmtpm6Sd8/XupBd9J3lcS3rhk6+v6WAbHZN0GPB+YE5E/K6D+D0bbs6hwnEIEBHLImLniOjNx+Mq0okPD1XIYUrDzWOocCxmXwVm5209n3SyQdXRLg8BfhQRqyrGQerTf0Venk3FBkTDcbgZ8CHg3A5yqKbub4/H44X0Al8NPEk6UE+oGP8yUh/5UmBJvhxRIf5FwA9y/HJanEXQxrYOouJZPaQ++rvz5R7gtA72Ox3oy8/hq8COHWxja+DXwA4dPvePkArVcuC/yGd1VIj/NukN627g4E6OG+DZwC2kF/stwE4V44/Jy08Aa4CbKsb/BPhlw3HY9KycJvFX5r/fUuA6YNeRvHYY5myxJjn8F7As53AtMKVi/BbAxfl53AXMrpo/cCHw9g6PgZcBi/NxtAjYv2L8u0lnBv4YmE8eUaHOi4dsMDMrjLt6zMwK48JvZlYYF34zs8K48JuZFcaF38ysMC78NubyiI5nNdx+n6TTR2nbF0p63Whsa5j9vF5plNJb696X2Ui58Nt48ATwmlbD+Y4FSRMqPPwE4J0R8cq68jEbLS78Nh6sI807eurgOwa32CU9nq8PygNyXSHpx5LmS3qj0jwHyyRNa9jMIZK+nR/36hw/QWks+zvz4GBva9jurZIuJf2oaHA+x+btL5d0Rl73YdKPeM6VdOagx28m6T+Uxom/XtKNA89Haez3yXl5hqTb8vI2ebCyO/MgeEfl9X+Wn9+SnPOe+bE3KM3tsFzSX+fH7p//Posl3dQwrMTJkn6Y4y/v4H9lm4CuT7Zu1sRngaVqMYnGEPYhDa/8G+B+4AsRMVNpYpyTSKMmAvSSflI/DbhV0h8DbyGN6PliSc8CviNpYHTPmcDeEfGzxp0pTVJyBrA/8FvS6KZHR8S/SJpNGlN+8KQ2r8n7fyFp5M4VwPnDPK/TSMNwHJ+H9rhD0teBtwOfjohL8uB2E0ijij4YEUfmHHfI40h9BjgqIvrzm8HHSYOBzQP2iIgn1GLSFtu0ucVv40Kk0U2/SJoMpF13Rpob4QnSwFYDhXsZqdgOuCIi1kcabvd+4E9I4xO9RWlo7EWkoRcGxq25Y3DRz15MmmSjP9IY+JeQ5iVo5WXAl/P+HwLa+Q7gVcC8nNttwJbA7sD3gA9Kej9p0prf5+d6iKQzJL08Ih4hje++N7Awb+NDPD1O/1LgEqWRTDfa0UhtZNzit/HkU6SxVi5oWLeO3EDJg7Ft0XDfEw3L6xtur+eZx/bgcUmCNAvaSRFxU+Mdkg4iDTM9lE6G+m0V8//PjVTcG2NeGxtO0LJC0iLSBDw3SXprRHxD0v6klv+/5k8tVwP3RMRQ02EeSXqzmgP8k6Q/i6cncrFCuMVv40ZE/IY0heIJDatXkrpWII2/P7GDTb8+97VPIw1Qdy9pmsB35G4RJD1fw08mswh4haTJ+YvfY0kzd7VyO/DavP9dSIPqDVjJ08+tcTjkm4CT8hsdkvbN188D7o+Ic0iDmb0odz/9LiIuJk0Csl9+fj3K8yBLmpi/H9gMmBoRt5Im8ZkEbDtM/rYJcovfxpuzeOZkIJ8HrpF0B2n0y2at8VbuJRXoXUgjMP6vpC+QuoPuygW2nxbTJkKaYUvSB0jdNQJujIjhhmG+kjRk9HLS6IuLgEfyfR8BzpP0QZ45g9tHSZ9+lubcVpLG2/9r4E2SniTNUfwvpO6nMyWtJ434+I6I+EP+AvkcSTuQXuefyvu/OK8TcHaMbMpM20h5dE6zmknaNiIel/Rs4A7S7Gdtj1dvNtrc4jer3/X5DJotgI+66NtYc4vfzKww/nLXzKwwLvxmZoVx4TczK4wLv5lZYVz4zcwK83/+8q3GQi5QPwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%time show_metric(neg_entropy, False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Comparing Metrics with Inconsistent Guesses Allowed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mean: 7.05 ± 0.98 guesses, worst: 10, N: 2,845\n",
      "cumulative: ≤3:0%, ≤4:1%, ≤5:6%, ≤6:24%, ≤7:69%, ≤8:95%, ≤9:100%, ≤10:100%\n",
      "CPU times: user 34 s, sys: 33.3 ms, total: 34 s\n",
      "Wall time: 34.1 s\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEGCAYAAABiq/5QAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAUn0lEQVR4nO3de7QlZX3m8e9DN8gtiEpDkCY2kh6iIQNoh+CQMYRLBgMDqJhAgmGEBJMgaIwrQSYziYmuwDhGwOWMQ0AgSwURNCAwEsLFjMYBm4tc7BAUW4fh0q1ykdEgLb/5o+qEw6H79G5C1enu9/tZ66y9q/au/XurL0/Veav2+6aqkCS1Y5O5boAkaVwGvyQ1xuCXpMYY/JLUGINfkhozf64bMIntttuuFi1aNNfNkKQNys033/ztqlowc/0GEfyLFi1i6dKlc90MSdqgJPnm6tbb1SNJjTH4JakxBr8kNcbgl6TGGPyS1BiDX5IaY/BLUmMMfklqjMEvSY3ZIL65K2nNFp1y5Wi1lp92yGi1NBzP+CWpMQa/JDXG4Jekxhj8ktQYg1+SGmPwS1JjDH5JaozBL0mNMfglqTEGvyQ1xuCXpMYY/JLUGINfkhpj8EtSYwx+SWqMwS9JjTH4Jakxgwd/knlJbk1yRb+8S5Ibk9yT5JNJNhu6DZKkp41xxv92YNm05dOBD1bVYuBh4PgR2iBJ6g0a/EkWAocA5/TLAfYHLunfcgFwxJBtkCQ909Bn/GcAfwA81S+/BHikqlb1y/cBO61uwyQnJFmaZOnKlSsHbqYktWOw4E9yKLCiqm6evno1b63VbV9VZ1fVkqpasmDBgkHaKEktmj/gZ+8LHJbkl4HNgW3ofgPYNsn8/qx/IXD/gG2QJM0w2Bl/Vb27qhZW1SLgKOC6qvp14HrgyP5txwKXDdUGSdKzzcV9/H8IvDPJ1+j6/M+dgzZIUrOG7Or5Z1V1A3BD//xeYO8x6kqSns1v7kpSYwx+SWqMwS9JjTH4JakxBr8kNcbgl6TGGPyS1BiDX5IaY/BLUmMMfklqjMEvSY0x+CWpMQa/JDXG4Jekxhj8ktQYg1+SGmPwS1JjDH5JaozBL0mNMfglqTEGvyQ1xuCXpMYY/JLUGINfkhpj8EtSYwx+SWqMwS9JjTH4JakxBr8kNcbgl6TGGPyS1BiDX5IaY/BLUmMMfklqjMEvSY0x+CWpMQa/JDVmsOBPsnmSm5J8JcldSd7Tr98lyY1J7knyySSbDdUGSdKzDXnG/wSwf1XtAewJHJxkH+B04INVtRh4GDh+wDZIkmYYLPir83i/uGn/U8D+wCX9+guAI4ZqgyTp2Qbt408yL8ltwArgGuDrwCNVtap/y33ATkO2QZL0TIMGf1X9qKr2BBYCewOvWN3bVrdtkhOSLE2ydOXKlUM2U5KastbgT7Jrkhf0z/dLcnKSbdelSFU9AtwA7ANsm2R+/9JC4P41bHN2VS2pqiULFixYl3KSpFlMcsZ/KfCjJD8JnAvsAnxibRslWTB1gEiyBXAgsAy4Hjiyf9uxwGXPod2SpOdo/trfwlNVtSrJ64EzqupDSW6dYLsdgQuSzKM7wFxcVVck+SpwUZL3ArfSHUwkSSOZJPifTHI03dn5v+/Xbbq2jarqdmCv1ay/l66/X5I0Bybp6nkL8BrgfVX1jSS7AB8btlmSpKFMcsZ/UFWdPLXQh/8PBmyTJGlAk5zxH7uadf/heW6HJGkkazzj7/v1fw3YJcnl0176MeA7QzdMkjSM2bp6/h54ANgO+MC09d8Dbh+yUZKk4awx+Kvqm8A3gdckeRmwuKr+tr8nfwu6A4AkaQMzyTd3f4tuULX/0a9aCPz1kI2SJA1nkou7JwL7Ao8BVNU9wPZDNkqSNJxJgv+Jqvrh1EI/zs5qB1aTJK3/Jgn+zyc5FdgiyUHAp4DPDtssSdJQJgn+U4CVwB3AW4GrgD8aslGSpOGs9Zu7VfUU8Jf9jyRpA7fW4E9yB8/u038UWAq8t6r8MpckbUAmGavnfwI/4ukx+I/qHx8DzufpETslNWTRKVeOWm/5aYeMWm9jNknw71tV+05bviPJF6tq3yTHDNUwSdIwJrm4u3WSn5taSLI3sHW/uGr1m0iS1leTnPEfD5yXZCrsvwccn2Qr4M8Ha5kkaRCzBn+STYCXV9XPJHkhkH7i9CkXD9o6SdLzbtaunv5Wzrf1zx+dEfqSpA3QJH381yR5V5Kdk7x46mfwlkmSBjFJH/9x/eOJ09YV8PLnvzmSpKFN8s3dXcZoiCRpHJOc8ZNkd+CVwOZT66rqr4ZqlCRpOJMM2fDHwH50wX8V8DrgC4DBL0kboEku7h4JHAA8WFVvAfYAXjBoqyRJg5kk+H/Q39a5Ksk2wAq8sCtJG6xJ+viXJtmWbljmm4HHgZsGbZUkaTCT3NXzu/3TjyT5HLBNVd0+bLMkSUNZa1dPkmunnlfV8qq6ffo6SdKGZY1n/Ek2B7YEtkvyIiD9S9sALx2hbZKkAczW1fNW4B10IX8zTwf/Y8CHB26XJGkgawz+qjoTODPJSVX1oRHbJEka0Fr7+A19Sdq4THIfvyRpI7LG4E+yb//ot3QlaSMy2xn/Wf3jl8ZoiCRpHLPd1fNkkvOAnZKcNfPFqjp5uGZJkoYyW/AfChwI7E93O6ckaSMw2+2c3wYuSrKsqr6yrh+cZGe6oZt/HHgKOLuqzuynbfwksAhYDvxKVT38HNouSXoOJrmr5ztJPpNkRZKHklyaZOEE260Cfr+qXgHsA5yY5JXAKcC1VbUYuLZfliSNZJLgPw+4nO4bvDsBn+3XzaqqHqiqW/rn3wOW9dsfDlzQv+0C4Ih1b7Yk6bmaJPi3r6rzqmpV/3M+sGBdiiRZBOwF3AjsUFUPQHdwALZfwzYnJFmaZOnKlSvXpZwkaRaTBP/KJMckmdf/HAN8Z9ICSbYGLgXeUVWPTbpdVZ1dVUuqasmCBet0nJEkzWKS4D8O+BXgQeABuqkYj5vkw5NsShf6H6+qT/erH0qyY//6jnQzekmSRjLJRCzfAg5b1w9OEuBcYFlV/cW0ly4HjgVO6x8vW9fPliQ9d5NMvfhc7Qu8GbgjyW39ulPpAv/iJMcD3wLeNGAbJEkzDBb8VfUFnh7Df6YDhqorSZqdo3NKUmMmDv4k+yS5LskXk3jvvSRtoGabc/fHq+rBaaveSXeRN8DfA389cNukDcaiU64crdby0w4ZrZY2TrP18X8kyc3A+6vqn4BHgF+jG3dn4vvxJUnrlzV29VTVEcBtwBVJ3kw38fpTwJY4zIIkbbBm7eOvqs8C/w7YFvg0cHdVnVVVjqEgSRuo2aZePCzJF4DrgDuBo4DXJ7kwya5jNVCS9PyarY//vcBrgC2Aq6pqb+CdSRYD76M7EEiSNjCzBf+jdOG+BdPG06mqezD0JWmDNVsf/+vpLuSuorubR5K0EVjb1IsfGrEtkqQROGSDJDXG4Jekxhj8ktQYg1+SGmPwS1JjDH5JaozBL0mNMfglqTEGvyQ1xuCXpMYY/JLUGINfkhpj8EtSYwx+SWqMwS9JjTH4JakxBr8kNcbgl6TGGPyS1BiDX5IaY/BLUmMMfklqjMEvSY0x+CWpMQa/JDXG4JekxgwW/Ek+mmRFkjunrXtxkmuS3NM/vmio+pKk1RvyjP984OAZ604Brq2qxcC1/bIkaUSDBX9V/R3w3RmrDwcu6J9fABwxVH1J0uqN3ce/Q1U9ANA/br+mNyY5IcnSJEtXrlw5WgMlaWO33l7craqzq2pJVS1ZsGDBXDdHkjYaYwf/Q0l2BOgfV4xcX5KaN3bwXw4c2z8/Frhs5PqS1Lwhb+e8EPgSsFuS+5IcD5wGHJTkHuCgflmSNKL5Q31wVR29hpcOGKqmJGnt1tuLu5KkYRj8ktQYg1+SGmPwS1JjDH5JaozBL0mNMfglqTEGvyQ1xuCXpMYY/JLUGINfkhpj8EtSYwx+SWqMwS9JjTH4JakxBr8kNcbgl6TGDDYDlzS2RadcOWq95acdMmo96fli8Eva4HiQ/5exq0eSGmPwS1JjDH5JaozBL0mNMfglqTEGvyQ1xuCXpMYY/JLUGINfkhpj8EtSYwx+SWqMwS9JjTH4JakxBr8kNcbgl6TGGPyS1BiDX5IaY/BLUmOcelHPK6fEk9Z/c3LGn+TgJHcn+VqSU+aiDZLUqtGDP8k84MPA64BXAkcneeXY7ZCkVs1FV8/ewNeq6l6AJBcBhwNfnYO2DGrMbg+7PKRxbAz/r1NVg3zwGgsmRwIHV9Vv9stvBn6uqt42430nACf0i7sBd4/aUNgO+PbINee6dov73GrtFve5xdovq6oFM1fOxRl/VrPuWUefqjobOHv45qxekqVVtaSl2i3uc6u1W9znlmvPNBcXd+8Ddp62vBC4fw7aIUlNmovg/zKwOMkuSTYDjgIun4N2SFKTRu/qqapVSd4GXA3MAz5aVXeN3Y4JzFk30xzWbnGfW63d4j63XPsZRr+4K0maWw7ZIEmNMfglqTEG/wxJPppkRZI7R667c5LrkyxLcleSt49Ye/MkNyX5Sl/7PWPV7uvPS3JrkitGrrs8yR1JbkuydOTa2ya5JMk/9H/nrxmp7m79/k79PJbkHWPU7uv/Xv9v7M4kFybZfKS6b+9r3jX0/q4uQ5K8OMk1Se7pH180ZBvWxuB/tvOBg+eg7irg96vqFcA+wIkjDmXxBLB/Ve0B7AkcnGSfkWoDvB1YNmK96X6xqvacg/urzwQ+V1U/BezBSPtfVXf3+7sn8Grg+8BnxqidZCfgZGBJVe1Od3PHUSPU3R34LbpRA/YADk2yeMCS5/PsDDkFuLaqFgPX9stzxuCfoar+DvjuHNR9oKpu6Z9/jy4IdhqpdlXV4/3ipv3PKFf9kywEDgHOGaPe+iDJNsBrgXMBquqHVfXIHDTlAODrVfXNEWvOB7ZIMh/YknG+w/MK4H9X1ferahXweeD1QxVbQ4YcDlzQP78AOGKo+pMw+NdDSRYBewE3jlhzXpLbgBXANVU1Vu0zgD8Anhqp3nQF/E2Sm/shQsbycmAlcF7fxXVOkq1GrD/lKODCsYpV1f8F/ivwLeAB4NGq+psRSt8JvDbJS5JsCfwyz/wS6Rh2qKoHoDvJA7Yfuf4zGPzrmSRbA5cC76iqx8aqW1U/6n/9Xwjs3f96PKgkhwIrqurmoWutwb5V9Sq6kWJPTPLakerOB14F/Peq2gv4f4z8q3//5cnDgE+NWPNFdGe+uwAvBbZKcszQdatqGXA6cA3wOeArdF2rzTL41yNJNqUL/Y9X1afnog19l8MNjHOdY1/gsCTLgYuA/ZN8bIS6AFTV/f3jCrp+7r1HKn0fcN+036ouoTsQjOl1wC1V9dCINQ8EvlFVK6vqSeDTwL8Zo3BVnVtVr6qq19J1w9wzRt1pHkqyI0D/uGLk+s9g8K8nkoSuz3dZVf3FyLUXJNm2f74F3X/Qfxi6blW9u6oWVtUium6H66pq8DNAgCRbJfmxqefAL9F1CQyuqh4E/k+S3fpVBzD+sORHM2I3T+9bwD5Jtuz/vR/ASBe1k2zfP/4E8AbG3/fLgWP758cCl41c/xmcenGGJBcC+wHbJbkP+OOqOneE0vsCbwbu6PvaAU6tqqtGqL0jcEE/Sc4mwMVVNeqtlXNgB+AzXf4wH/hEVX1uxPonAR/vu1zuBd4yVuG+n/sg4K1j1QSoqhuTXALcQtfVcivjDWNwaZKXAE8CJ1bVw0MVWl2GAKcBFyc5nu4A+Kah6k/CIRskqTF29UhSYwx+SWqMwS9JjTH4JakxBr8kNcbg15xLUkk+MG35XUn+5Hn67POTHPl8fNZa6rypH2Xz+qFrSf9SBr/WB08Ab0iy3Vw3ZLr+ew2TOh743ar6xaHaIz1fDH6tD1bRfZHn92a+MPOMPcnj/eN+ST6f5OIk/5jktCS/3s8rcEeSXad9zIFJ/lf/vkP77ecleX+SLye5Pclbp33u9Uk+AdyxmvYc3X/+nUlO79f9Z+DngY8kef+M92+S5L/148BfkeSqqf1JNx/Adv3zJUlu6J9v1Y/p/uV+ELfD+/U/3e/fbX2bF/fvvTLdXAp3JvnV/r2v7v98bk5y9bThAk5O8tV++4uew9+VNgJ+c1friw8Dtyf5L+uwzR50Q+5+l+7br+dU1d7pJrE5CZiacGMR8AvArsD1SX4S+A260SF/NskLgC8mmRopcm9g96r6xvRiSV5KN9jXq4GH6Ub2PKKq/jTJ/sC7qmrmhC5v6Ov/DN2IjMuAj65lv/4j3fAVx/VDadyU5G+B3wbOrKqpb/zOoxtp8v6qOqRv4wv7MZ8+BBxeVSv7g8H7gOPoBoPbpaqemBqmQ+3xjF/rhX4k0r+im6hjUl/u5zF4Avg6MBXcd9CF7ZSLq+qpqrqH7gDxU3Rj8/xGPzzGjcBLgKnJOW6aGfq9nwVu6AcZWwV8nG5c/dn8PPCpvv6DwCTXAH4JOKVv2w3A5sBPAF8CTk3yh8DLquoH/b4emOT0JP+2qh4FdgN2B67pP+OP6EZdBbidbqiIY2h8hMqWecav9ckZdOO4nDdt3Sr6E5R+YK/Npr32xLTnT01bfopn/tueOS5JAQFOqqqrp7+QZD+6YZJXJ2vdg3Xb5p/3jS7cp2/zxqq6e8b7lyW5kW7imquT/GZVXZfk1XRn/n/e/9byGeCuqlrddI6H0B2sDgP+U5Kf7g9iaohn/FpvVNV3gYvpLpROWU7XtQLdWO6bPoePflPf174r3SQodwNXA7/Td4uQ5F9l7ZOh3Aj8QpLt+gu/R9PN5jSbLwBv7OvvQDd415TlPL1vb5y2/mrgpP5AR5K9+seXA/dW1Vl0oz3+67776ftV9TG6SU5e1e/fgvTz+CbZtL8+sAmwc1VdTzf5zbbA1mtpvzZCnvFrffMB4G3Tlv8SuCzJTXRzla7pbHw2d9MF9A7Ab1fVPyU5h6476JY+YFeylunwquqBJO+m664JcFVVrW143Uvphh++E/hHuoPHo/1r7wHOTXIqz5xt7c/ofvu5vW/bcuBQ4FeBY5I8CTwI/Cld99P7kzxFN/Lk71TVD/sLyGcleSHd//Mz+vof69cF+OAcTfmoOebonNLAkmxdVY+nGxb4JrqZvx6c63apXZ7xS8O7or+DZjPgzwx9zTXP+CWpMV7claTGGPyS1BiDX5IaY/BLUmMMfklqzP8H30yOcPVXbYAAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%time show_metric(max_count, True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mean: 6.84 ± 0.95 guesses, worst: 10, N: 2,845\n",
      "cumulative: ≤3:0%, ≤4:1%, ≤5:7%, ≤6:32%, ≤7:78%, ≤8:97%, ≤9:100%, ≤10:100%\n",
      "CPU times: user 34.8 s, sys: 35.2 ms, total: 34.9 s\n",
      "Wall time: 34.9 s\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEGCAYAAABiq/5QAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAUnklEQVR4nO3de7hldX3f8feHm9yCIDNQZIyDhBINKaATgiU1hEuKgQIqNpBgqJBgEgTU+ES0aVOT2ECtEfGxtQQE8qgYBA0IVEIQTE0sMCByyYSgiJZymfECSDXION/+sdYJh8PMOZuRtc6Z+b1fz3Oevdfae+3vb83ls9f5rbV+v1QVkqR2bDLfDZAkjcvgl6TGGPyS1BiDX5IaY/BLUmM2m+8GTGLRokW1dOnS+W6GJG1Qbrnllm9W1eKZ6zeI4F+6dCnLly+f72ZI0gYlydfXtt6uHklqjMEvSY0x+CWpMQa/JDXG4Jekxhj8ktQYg1+SGmPwS1JjDH5JaswGceeupHVbesZVo9W678zDR6ul4XjEL0mNMfglqTEGvyQ1xuCXpMYY/JLUGINfkhpj8EtSYwx+SWqMwS9JjTH4JakxBr8kNcbgl6TGGPyS1BiDX5IaY/BLUmMMfklqjMEvSY0x+CWpMQa/JDXG4Jekxhj8ktQYg1+SGjN48CfZNMmXklzZL++W5MYk9yT58yRbDN0GSdJTxjjiPx1YMW35LOD9VbUH8B3gpBHaIEnqDRr8SZYAhwPn9csBDgIu7d9yEXD0kG2QJD3d0Ef8ZwO/C6zpl3cEHqmq1f3y/cCua9swyclJlidZvmrVqoGbKUntGCz4kxwBrKyqW6avXstba23bV9W5VbWsqpYtXrx4kDZKUos2G/CzDwCOTPJLwJbAdnS/AWyfZLP+qH8J8MCAbZAkzTDYEX9VvbOqllTVUuBY4HNV9avA9cAx/dtOAC4fqg2SpGeaj+v43wG8LclX6Pr8z5+HNkhSs4bs6vknVXUDcEP//F5gvzHqSpKeyTt3JakxBr8kNcbgl6TGGPyS1BiDX5IaY/BLUmMMfklqjMEvSY0x+CWpMQa/JDXG4Jekxhj8ktQYg1+SGmPwS1JjDH5JaozBL0mNMfglqTEGvyQ1xuCXpMYY/JLUGINfkhpj8EtSYwx+SWqMwS9JjTH4JakxBr8kNcbgl6TGGPyS1BiDX5IaY/BLUmMMfklqjMEvSY0x+CWpMQa/JDXG4Jekxhj8ktSYwYI/yZZJbkry5SR3JXl3v363JDcmuSfJnyfZYqg2SJKeac7gT7J7kuf1zw9MclqS7Sf47CeAg6pqb2Af4LAk+wNnAe+vqj2A7wAnrX/zJUnP1iRH/JcBP0zyE8D5wG7Ax+faqDqP94ub9z8FHARc2q+/CDj62TZakrT+Jgn+NVW1GngNcHZVvRXYZZIPT7JpktuAlcC1wFeBR/rPA7gf2HUd256cZHmS5atWrZqknCRpApME/5NJjgNOAK7s120+yYdX1Q+rah9gCbAf8NK1vW0d255bVcuqatnixYsnKSdJmsAkwf9G4JXAe6rqa0l2Az76bIpU1SPADcD+wPZJNutfWgI88Gw+S5L0o5kk+A+tqtOq6mKAqvoa8P25NkqyeOokcJKtgEOAFcD1wDH9204ALl+fhkuS1s8kwX/CWtb9uwm22wW4PsntwM3AtVV1JfAO4G1JvgLsSHfCWJI0ks3W9ULfr/8rwG5Jrpj20o8B35rrg6vqdmDftay/l66/X5I0D9YZ/MDfAg8Ci4D3TVv/XeD2IRslSRrOOoO/qr4OfB14ZZIXA3tU1V/1/fVb0X0BSJI2MJPcufsbdDdc/Y9+1RLgL4ZslCRpOJOc3D0FOAB4DKCq7gF2GrJRkqThTBL8T1TVD6YW+mvw13rTlSRp4Zsk+D+f5F3AVkkOBT4JfGbYZkmShjJJ8J8BrALuAN4EXA383pCNkiQNZ7bLOQGoqjXAn/Y/kqQN3JzBn+QOntmn/yiwHPijqprzZi5J0sIxZ/AD/xP4IU+NwX9s//gYcCHwb577ZkmShjJJ8B9QVQdMW74jyd9U1QFJjh+qYZKkYUxycnfbJD87tZBkP2DbfnH12jeRJC1UkxzxnwRckGQq7L8LnJRkG+CPB2uZJGkQswZ/kk2Al1TVTyd5PpB+UpUplwzaOknSc27W4K+qNUneDFxSVY+O1CZpg7P0jKtGq3XfmYePVksbp0n6+K9N8vYkL0rygqmfwVsmSRrEJH38J/aPp0xbV8BLnvvmSJKGNsmdu7uN0RBJ0jgmOeInyV7Ay4Atp9ZV1Z8N1ShJ0nAmGbLh94ED6YL/auDVwBcAg1+SNkCTnNw9BjgYeKiq3gjsDTxv0FZJkgYzSfB/vx+hc3WS7YCVeGJXkjZYk/TxL0+yPd2wzLcAjwM3DdoqSdJgJrmq57f7px9O8llgu6q6fdhmSZKGMmdXT5Lrpp5X1X1Vdfv0dZKkDcs6j/iTbAlsDSxKsgOQ/qXtgBeO0DZJ0gBm6+p5E/AWupC/haeC/zHgQwO3S5I0kHUGf1V9APhAklOr6oMjtkmSNKA5+/gNfUnauExyHb8kaSOyzuBPckD/6F26krQRme2I/5z+8YtjNESSNI7Zrup5MskFwK5Jzpn5YlWdNlyzJElDmS34jwAOAQ6iu5xTkrQRmO1yzm8Cn0iyoqq+PGKbJEkDmuSqnm8l+XSSlUkeTnJZkiWDt0ySNIhJgv8C4Aq6O3h3BT7Tr5tVPzn79UlWJLkryen9+hckuTbJPf3jDj/KDkiSnp1Jgn+nqrqgqlb3PxcCiyfYbjXwO1X1UmB/4JQkLwPOAK6rqj2A6/plSdJIJgn+VUmOT7Jp/3M88K25NqqqB6vq1v75d4EVdL8xHAVc1L/tIuDo9Wu6JGl9TBL8JwL/FngIeJBuKsYTn02RJEuBfYEbgZ2r6kHovhyAndaxzclJlidZvmrVqmdTTpI0i0kmYvkGcOT6FkiyLXAZ8JaqeizJXJtM1T0XOBdg2bJltb71JUlPN+hYPUk2pwv9j1XVp/rVDyfZpX99F7o5fCVJI5lkzt31ku7Q/nxgRVX9ybSXrgBOAM7sHy8fqg2ShrP0jKtGrXffmYePWm9jNljwAwcAbwDuSHJbv+5ddIF/SZKTgG8Arx+wDZKkGSYO/iT7A/8ZeB7w3qr6i9neX1Vf4KlZu2Y6eOIWSpKeU7PNufvPquqhaaveRneSN8DfArMGvyRpYZrtiP/DSW6hO7r/R+AR4FeANXTz7kqSNkDrvKqnqo4GbgOuTPIGuonX1wBb401XkrTBmvVyzqr6DPCvge2BTwF3V9U5VeUdVZK0gZpt6sUjk3wB+BxwJ3As8JokFyfZfawGSpKeW7P18f8R8EpgK+DqqtoPeFuSPYD30H0RSJI2MLMF/6N04b4V0+6urap7MPQlaYM1Wx//a+hO5K6mu5pHkrQRmGvqxQ+O2BZJ0ggGHaRNkrTwGPyS1BiDX5IaY/BLUmMMfklqjMEvSY0x+CWpMQa/JDXG4Jekxhj8ktQYg1+SGmPwS1JjDH5JaozBL0mNMfglqTEGvyQ1xuCXpMYY/JLUGINfkhpj8EtSYwx+SWqMwS9JjTH4JakxBr8kNcbgl6TGGPyS1BiDX5IaM1jwJ/lIkpVJ7py27gVJrk1yT/+4w1D1JUlrN+QR/4XAYTPWnQFcV1V7ANf1y5KkEQ0W/FX118C3Z6w+Criof34RcPRQ9SVJazd2H//OVfUgQP+408j1Jal5C/bkbpKTkyxPsnzVqlXz3RxJ2miMHfwPJ9kFoH9cua43VtW5VbWsqpYtXrx4tAZK0sZu7OC/Ajihf34CcPnI9SWpeUNeznkx8EVgzyT3JzkJOBM4NMk9wKH9siRpRJsN9cFVddw6Xjp4qJqSpLkt2JO7kqRhGPyS1BiDX5IaY/BLUmMMfklqjMEvSY0x+CWpMQa/JDXG4Jekxhj8ktQYg1+SGmPwS1JjDH5JaozBL0mNMfglqTGDjccvjW3pGVeNWu++Mw8ftZ70XPGIX5IaY/BLUmMMfklqjMEvSY0x+CWpMQa/JDXG4Jekxhj8ktQYg1+SGmPwS1JjDH5JaozBL0mNcZA2SRscB+T70XjEL0mNMfglqTEGvyQ1xuCXpMYY/JLUGINfkhpj8EtSY7yOX88pr6+WFr55OeJPcliSu5N8JckZ89EGSWrV6MGfZFPgQ8CrgZcBxyV52djtkKRWzUdXz37AV6rqXoAknwCOAv5uHtoyqDG7PWZ2ecxnbUkLW6pq3ILJMcBhVfXr/fIbgJ+tqjfPeN/JwMn94p7A3aM2FBYB3xy55nzXbnGfW63d4j63WPvFVbV45sr5OOLPWtY949unqs4Fzh2+OWuXZHlVLWupdov73GrtFve55dozzcfJ3fuBF01bXgI8MA/tkKQmzUfw3wzskWS3JFsAxwJXzEM7JKlJo3f1VNXqJG8GrgE2BT5SVXeN3Y4JzFs30zzWbnGfW63d4j63XPtpRj+5K0maXw7ZIEmNMfglqTEG/wxJPpJkZZI7R677oiTXJ1mR5K4kp49Ye8skNyX5cl/73WPV7utvmuRLSa4cue59Se5IcluS5SPX3j7JpUn+vv87f+VIdffs93fq57Ekbxmjdl//rf2/sTuTXJxky5Hqnt7XvGvo/V1bhiR5QZJrk9zTP+4wZBvmYvA/04XAYfNQdzXwO1X1UmB/4JQRh7J4AjioqvYG9gEOS7L/SLUBTgdWjFhvul+oqn3m4frqDwCfraqfBPZmpP2vqrv7/d0HeAXwPeDTY9ROsitwGrCsqvaiu7jj2BHq7gX8Bt2oAXsDRyTZY8CSF/LMDDkDuK6q9gCu65fnjcE/Q1X9NfDteaj7YFXd2j//Ll0Q7DpS7aqqx/vFzfufUc76J1kCHA6cN0a9hSDJdsCrgPMBquoHVfXIPDTlYOCrVfX1EWtuBmyVZDNga8a5h+elwP+uqu9V1Wrg88Brhiq2jgw5Criof34RcPRQ9Sdh8C9ASZYC+wI3jlhz0yS3ASuBa6tqrNpnA78LrBmp3nQF/GWSW/ohQsbyEmAVcEHfxXVekm1GrD/lWODisYpV1f8F/ivwDeBB4NGq+ssRSt8JvCrJjkm2Bn6Jp99EOoadq+pB6A7ygJ1Grv80Bv8Ck2Rb4DLgLVX12Fh1q+qH/a//S4D9+l+PB5XkCGBlVd0ydK11OKCqXk43UuwpSV41Ut3NgJcD/72q9gX+HyP/6t/fPHkk8MkRa+5Ad+S7G/BCYJskxw9dt6pWAGcB1wKfBb5M17XaLIN/AUmyOV3of6yqPjUfbei7HG5gnPMcBwBHJrkP+ARwUJKPjlAXgKp6oH9cSdfPvd9Ipe8H7p/2W9WldF8EY3o1cGtVPTxizUOAr1XVqqp6EvgU8C/HKFxV51fVy6vqVXTdMPeMUXeah5PsAtA/rhy5/tMY/AtEktD1+a6oqj8ZufbiJNv3z7ei+w/690PXrap3VtWSqlpK1+3wuaoa/AgQIMk2SX5s6jnwi3RdAoOrqoeA/5Nkz37VwYw/LPlxjNjN0/sGsH+Srft/7wcz0kntJDv1jz8OvJbx9/0K4IT++QnA5SPXfxqnXpwhycXAgcCiJPcDv19V549Q+gDgDcAdfV87wLuq6uoRau8CXNRPkrMJcElVjXpp5TzYGfh0lz9sBny8qj47Yv1TgY/1XS73Am8cq3Dfz30o8KaxagJU1Y1JLgVupetq+RLjDWNwWZIdgSeBU6rqO0MVWluGAGcClyQ5ie4L8PVD1Z+EQzZIUmPs6pGkxhj8ktQYg1+SGmPwS1JjDH5JaozBr3mXpJK8b9ry25P8p+fosy9Mcsxz8Vlz1Hl9P8rm9UPXkn5UBr8WgieA1yZZNN8Nma6/r2FSJwG/XVW/MFR7pOeKwa+FYDXdjTxvnfnCzCP2JI/3jwcm+XySS5L8Q5Izk/xqP6/AHUl2n/YxhyT5X/37jui33zTJe5PcnOT2JG+a9rnXJ/k4cMda2nNc//l3JjmrX/cfgZ8DPpzkvTPev0mS/9aPA39lkqun9ifdfACL+ufLktzQP9+mH9P95n4Qt6P69T/V799tfZv36N97Vbq5FO5M8sv9e1/R//nckuSaacMFnJbk7/rtP7Eef1faCHjnrhaKDwG3J/kvz2KbvemG3P023d2v51XVfukmsTkVmJpwYynw88DuwPVJfgL4NbrRIX8myfOAv0kyNVLkfsBeVfW16cWSvJBusK9XAN+hG9nz6Kr6gyQHAW+vqpkTury2r//TdCMyrgA+Msd+/Xu64StO7IfSuCnJXwG/CXygqqbu+N2UbqTJB6rq8L6Nz+/HfPogcFRVreq/DN4DnEg3GNxuVfXE1DAdao9H/FoQ+pFI/4xuoo5J3dzPY/AE8FVgKrjvoAvbKZdU1ZqquofuC+In6cbm+bV+eIwbgR2Bqck5bpoZ+r2fAW7oBxlbDXyMblz92fwc8Mm+/kPAJOcAfhE4o2/bDcCWwI8DXwTeleQdwIur6vv9vh6S5Kwk/6qqHgX2BPYCru0/4/foRl0FuJ1uqIjjaXyEypZ5xK+F5Gy6cVwumLZuNf0BSj+w1xbTXnti2vM105bX8PR/2zPHJSkgwKlVdc30F5IcSDdM8tpkzj14dtv8077Rhfv0bV5XVXfPeP+KJDfSTVxzTZJfr6rPJXkF3ZH/H/e/tXwauKuq1jad4+F0X1ZHAv8hyU/1X2JqiEf8WjCq6tvAJXQnSqfcR9e1At1Y7puvx0e/vu9r351uEpS7gWuA3+q7RUjyzzP3ZCg3Aj+fZFF/4vc4utmcZvMF4HV9/Z3pBu+ach9P7dvrpq2/Bji1/6Ijyb7940uAe6vqHLrRHv9F3/30var6KN0kJy/v929x+nl8k2zenx/YBHhRVV1PN/nN9sC2c7RfGyGP+LXQvA9487TlPwUuT3IT3Vyl6zoan83ddAG9M/CbVfWPSc6j6w66tQ/YVcwxHV5VPZjknXTdNQGurqq5hte9jG744TuBf6D78ni0f+3dwPlJ3sXTZ1v7Q7rffm7v23YfcATwy8DxSZ4EHgL+gK776b1J1tCNPPlbVfWD/gTyOUmeT/f//Oy+/kf7dQHeP09TPmqeOTqnNLAk21bV4+mGBb6Jbuavh+a7XWqXR/zS8K7sr6DZAvhDQ1/zzSN+SWqMJ3clqTEGvyQ1xuCXpMYY/JLUGINfkhrz/wE86IcsapAdRwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%time show_metric(expectation, True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mean: 6.82 ± 1.00 guesses, worst: 10, N: 2,845\n",
      "cumulative: ≤3:0%, ≤4:1%, ≤5:8%, ≤6:35%, ≤7:77%, ≤8:97%, ≤9:100%, ≤10:100%\n",
      "CPU times: user 35.8 s, sys: 28.5 ms, total: 35.9 s\n",
      "Wall time: 35.9 s\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEGCAYAAABiq/5QAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAXeUlEQVR4nO3de7hddX3n8feHAHITAXNgkIhByqCWDkGPKU46Frk4KDxcFKdQQUZpgy1XL1Mjc1FbfQpjEcTH0YkgpI8IRS4FgYIpFxmsE0gwhGCkUYwOEshR7qNFQj7zx/qd4XByLpuQtXaS3+f1POfZe6299/l+t4bPXue31/r9ZJuIiKjHZv1uICIiupXgj4ioTII/IqIyCf6IiMok+CMiKrN5vxvoxdSpUz19+vR+txERsVFZtGjRL20PjN6/UQT/9OnTWbhwYb/biIjYqEj62Vj7M9QTEVGZBH9ERGUS/BERlUnwR0RUJsEfEVGZBH9ERGUS/BERlUnwR0RUJsEfEVGZjeLK3YgY3/Q5N3RWa8XZh3VWK9qTI/6IiMok+CMiKpPgj4ioTII/IqIyCf6IiMok+CMiKpPgj4ioTII/IqIyCf6IiMq0HvySpkj6gaTry/YekhZIWi7p7yRt2XYPERHxgi6O+M8Alo3YPgc4z/ZewOPASR30EBERRavBL2kacBhwYdkWcCBwZXnKPOCoNnuIiIgXa/uI/3zgL4A1ZfvVwBO2V5fth4DdxnqhpNmSFkpaODQ01HKbERH1aC34JR0OrLK9aOTuMZ7qsV5ve67tQduDAwMDrfQYEVGjNqdlngUcIendwFbA9jR/AewgafNy1D8NeLjFHiIiYpTWjvhtf9L2NNvTgWOBW22/H7gNOKY87UTg2rZ6iIiItfXjPP5PAB+V9GOaMf+L+tBDRES1OlmBy/btwO3l/oPAzC7qRkTE2nLlbkREZRL8ERGVSfBHRFQmwR8RUZkEf0REZRL8ERGVSfBHRFQmwR8RUZkEf0REZRL8ERGVSfBHRFQmwR8RUZkEf0REZRL8ERGVSfBHRFSmzTV3t5J0l6R7Jd0v6TNl/yWSfippcfmZ0VYPERGxtjYXYnkWOND2M5K2AO6U9A/lsf9k+8oWa0dExDhaC37bBp4pm1uUH7dVLyIietPqGL+kKZIWA6uA+bYXlIc+J2mJpPMkvWKc186WtFDSwqGhoTbbjIioSqvBb/t52zOAacBMSfsAnwTeALwV2Ilm8fWxXjvX9qDtwYGBgTbbjIioSidn9dh+gmax9UNtr3TjWeBisvB6RESn2jyrZ0DSDuX+1sDBwI8k7Vr2CTgKWNpWDxERsbY2z+rZFZgnaQrNB8wVtq+XdKukAUDAYuDDLfYQERGjtHlWzxJgvzH2H9hWzYiImFyu3I2IqEyCPyKiMgn+iIjKJPgjIiqT4I+IqEyCPyKiMgn+iIjKJPgjIiqT4I+IqEyCPyKiMgn+iIjKJPgjIiqT4I+IqEyCPyKiMgn+iIjKtLkC11aS7pJ0r6T7JX2m7N9D0gJJyyX9naQt2+ohIiLW1uYR/7PAgbb3BWYAh0raHzgHOM/2XsDjwEkt9hAREaO0FvxlQfVnyuYW5cfAgcCVZf88mnV3IyKiI62O8UuaImkxsAqYD/wEeML26vKUh4DdxnntbEkLJS0cGhpqs82IiKq0Gvy2n7c9A5gGzATeONbTxnntXNuDtgcHBgbabDMioiqdnNVj+wngdmB/YAdJw4u8TwMe7qKHiIhotHlWz4CkHcr9rYGDgWXAbcAx5WknAte21UNERKxt88mfss52BeZJmkLzAXOF7esl/RC4XNJngR8AF7XYQ0REjNJa8NteAuw3xv4Hacb7IyKiD3LlbkREZRL8ERGVmTT4Je0p6RXl/gGSTh/+0jYiIjY+vRzxXwU8L+l3aL6I3QP4ZqtdRUREa3oJ/jXlStujgfNtf4TmjJ2IiNgI9RL8z0k6juac++vLvi3aaykiItrUS/B/EHgb8DnbP5W0B/CNdtuKiIi29HIe/yG2Tx/eKOH/mxZ7ioiIFvVyxH/iGPv+43ruIyIiOjLuEX8Z1/9jYA9J14146JXAr9puLCIi2jHRUM8/ASuBqcC5I/Y/DSxps6mIiGjPuMFv+2fAz4C3SXodsJftfywzbW5N8wEQEcD0OTd0Wm/F2Yd1Wi82Lb1cufunNEsl/s+yaxrw9202FRER7enly91TgFnAUwC2lwM7t9lURES0p5fgf9b2b4c3yupZYy6XGBERG75egv+7ks4CtpZ0CPAt4NuTvUjSayXdJmmZpPslnVH2f1rSLyQtLj/vfnlvISIiXopeLuCaA5wE3AecDNwIXNjD61YDH7N9j6RXAoskzS+PnWf7b9al4YiIeHkmDX7ba4CvlZ+e2V5Jczootp+WtAzYbV2ajIiI9WfS4Jd0H2uP6T8JLAQ+a3vSi7kkTadZhnEBzRfFp0r6QPkdH7P9+EtrOyIi1lUvY/z/ANwAvL/8fBu4A3gEuGSyF0vajmZO/zNtPwV8BdgTmEHzF8G547xutqSFkhYODQ310GZERPSilzH+WbZnjdi+T9L3bM+SdPxEL5S0BU3oX2r7agDbj454/Gu8MNXzi9ieC8wFGBwczFlEERHrSS9H/NtJ+v3hDUkzge3K5urxXiRJNCt2LbP9hRH7Ry7icjSw9CV1HBERL0svR/wnAReXIRtopmo4SdK2wF9P8LpZwAk0fyEsLvvOAo6TNIPme4MVNGcKRURERyYMfkmbAa+3/XuSXgXI9hMjnnLFeK+1fSegMR66cZ06jYiI9WLCoZ5yKuep5f6To0I/IiI2Qr2M8c+X9PFyJe5Owz+tdxYREa3oZYz/Q+X2lBH7DLx+/bcTERFt6+XK3T26aCQiIrrRyxE/kvYB3gRsNbzP9t+21VRERLSnlykbPgUcQBP8NwLvAu4EEvwRERuhXr7cPQY4CHjE9geBfYFXtNpVRES0ppfg/005rXO1pO2BVeSL3YiIjVYvY/wLJe1AMy3zIuAZ4K5Wu4qIiNb0clbPn5e7X5V0E7C97SXtthUREW2ZdKhH0i3D922vsL1k5L6IiNi4jHvEL2krYBtgqqQdeWHene2B13TQW0REtGCioZ6TgTNpQn4RLwT/U8CXW+4rIiJaMm7w2/4i8EVJp9n+Uoc9RUREiyYd40/oR0RsWno5jz8iIjYh4wa/pFnldp2u0i3TON8maZmk+yWdUfbvJGm+pOXldsd1az0iItbFREf8F5Tb76/j714NfMz2G4H9gVMkvQmYA9xiey/glrIdEREdmeisnuckXQzsJumC0Q/aPn2iX2x7JbCy3H9a0jJgN+BImknfAOYBtwOfeMmdR0TEOpko+A8HDgYOpDmdc51Jmg7sBywAdikfCtheKWnncV4zG5gNsPvuu7+c8hERMcJEp3P+Erhc0jLb965rAUnbAVcBZ9p+Shpr/fUx688F5gIMDg56XetHRMSL9XJWz68kXSNplaRHJV0laVovv1zSFjShf6ntq8vuRyXtWh7flWa2z4iI6EgvwX8xcB3NFby7Ad8u+yak5tD+ImCZ7S+MeOg64MRy/0Tg2pfScEREvDy9TMu8s+2RQX+JpDN7eN0s4ATgPkmLy76zgLOBKySdBPwceN9LaTgiNgzT59zQab0VZx/Wab1NWS/BPyTpeOCysn0c8KvJXmT7Tl6Y32e0g3prLyIi1rdehno+BPwH4BGa0zOPKfsiImIj1MtCLD8Hjuigl4iI6EDm6omIqEyCPyKiMgn+iIjK9Bz8kvaXdKuk70k6qs2mIiKiPROtufuvbD8yYtdHab7kFfBPwN+33FtERLRgorN6vippEfB52/8CPAH8MbCGZt3diIjYCI071GP7KGAxcL2kE2gWXl8DbANkqCciYiM14Ri/7W8D/x7YAbgaeMD2BbaHumguIiLWv4mWXjxC0p3ArcBS4FjgaEmXSdqzqwYjImL9mmiM/7PA24CtgRttzwQ+Kmkv4HM0HwQREbGRmSj4n6QJ960ZMWe+7eUk9CMiNloTjfEfTfNF7mqas3kiImITMNnSi1/qsJeIiOhAa1M2SPp6Wa5x6Yh9n5b0C0mLy8+726ofERFja3OunkuAQ8fYf57tGeXnxhbrR0TEGFoLftt3AI+19fsjImLd9GN2zlMlLSlDQTuO9yRJsyUtlLRwaCjXi0VErC9dB/9XgD2BGTTLOJ473hNtz7U9aHtwYGCgq/4iIjZ5nQa/7UdtP297DfA1YGaX9SMiouPgl7TriM2jaaaCiIiIDk262Pq6knQZcAAwVdJDwKeAAyTNAAysAE5uq35ERIytteC3fdwYuy9qq15ERPQma+5GRFQmwR8RUZkEf0REZRL8ERGVSfBHRFQmwR8RUZkEf0REZRL8ERGVSfBHRFQmwR8RUZkEf0REZRL8ERGVSfBHRFQmwR8RUZkEf0REZVoL/rKY+ipJS0fs20nSfEnLy+24i61HREQ72jzivwQ4dNS+OcAttvcCbinbERHRodaC3/YdwGOjdh8JzCv35wFHtVU/IiLG1vUY/y62VwKU253He6Kk2ZIWSlo4NDTUWYMREZu6DfbLXdtzbQ/aHhwYGOh3OxERm4yug/9RSbsClNtVHdePiKhe18F/HXBiuX8icG3H9SMiqtfm6ZyXAd8H9pb0kKSTgLOBQyQtBw4p2xER0aHN2/rFto8b56GD2qoZERGT22C/3I2IiHYk+CMiKpPgj4ioTII/IqIyCf6IiMok+CMiKpPgj4ioTII/IqIyCf6IiMq0duVuRNemz7mh03orzj6s03oR60uO+CMiKpPgj4ioTII/IqIyCf6IiMok+CMiKtOXs3okrQCeBp4HVtse7EcfERE16ufpnO+w/cs+1o+IqFKGeiIiKtOv4DfwHUmLJM3uUw8REVXq11DPLNsPS9oZmC/pR7bvGPmE8oEwG2D33XfvR48REZukvhzx23643K4CrgFmjvGcubYHbQ8ODAx03WJExCar8+CXtK2kVw7fB94JLO26j4iIWvVjqGcX4BpJw/W/afumPvQREVGlzoPf9oPAvl3XjYiIRk7njIioTII/IqIyCf6IiMok+CMiKpOlFyNio5NlNl+eHPFHRFQmwR8RUZkEf0REZRL8ERGVSfBHRFQmwR8RUZkEf0REZRL8ERGVSfBHRFQmV+7GepUrKiM2fDnij4ioTF+CX9Khkh6Q9GNJc/rRQ0RErTof6pE0BfgycAjwEHC3pOts/7DrXiIiXqpNYTizH2P8M4EflyUYkXQ5cCSwyQV/P/+BdFk74+wRGxfZ7ragdAxwqO0/KdsnAL9v+9RRz5sNzC6bewMPdNooTAV+2XHNfteu8T3XWrvG91xj7dfZHhi9sx9H/Bpj31qfPrbnAnPbb2dskhbaHqypdo3vudbaNb7nmmuP1o8vdx8CXjtiexrwcB/6iIioUj+C/25gL0l7SNoSOBa4rg99RERUqfOhHturJZ0K3AxMAb5u+/6u++hB34aZ+li7xvdca+0a33PNtV+k8y93IyKiv3LlbkREZRL8ERGVSfCPIunrklZJWtpx3ddKuk3SMkn3Szqjw9pbSbpL0r2l9me6qj2ihymSfiDp+o7rrpB0n6TFkhZ2WHcHSVdK+lH5//xtHdXdu7zX4Z+nJJ3ZRe1S/yPl39hSSZdJ2qqjumeUmvd38X7HyhFJO0maL2l5ud2x7T7Gk+Bf2yXAoX2ouxr4mO03AvsDp0h6U0e1nwUOtL0vMAM4VNL+HdUedgawrOOaw95he0bH51h/EbjJ9huAfenovdt+oLzXGcBbgF8D13RRW9JuwOnAoO19aE7uOLaDuvsAf0oza8C+wOGS9mq57CWsnSNzgFts7wXcUrb7IsE/iu07gMf6UHel7XvK/adpgmC3jmrb9jNlc4vy09m3/pKmAYcBF3ZVs58kbQ+8HbgIwPZvbT/Rh1YOAn5i+2cd1twc2FrS5sA2dHMNzxuB/23717ZXA98Fjm6z4Dg5ciQwr9yfBxzVZg8TSfBvgCRNB/YDFnRYc4qkxcAqYL7tzmoD5wN/AazpsOYwA9+RtKhME9KF1wNDwMVleOtCSdt2VHukY4HLuipm+xfA3wA/B1YCT9r+TgellwJvl/RqSdsA7+bFF5F2ZRfbK6E50AN27kMPQIJ/gyNpO+Aq4EzbT3VV1/bz5c//acDM8udx6yQdDqyyvaiLemOYZfvNwLtohtfe3kHNzYE3A1+xvR/wf+n4z/5y8eQRwLc6rLkjzVHvHsBrgG0lHd92XdvLgHOA+cBNwL00Q6vVSvBvQCRtQRP6l9q+uh89lCGH2+nue45ZwBGSVgCXAwdK+kZHtbH9cLldRTPWPbODsg8BD434q+pKmg+CLr0LuMf2ox3WPBj4qe0h288BVwP/tovCti+y/Wbbb6cZglneRd1RHpW0K0C5XdWHHoAE/wZDkmjGfJfZ/kLHtQck7VDub03zH+iPuqht+5O2p9meTjP0cKvt1o8CASRtK+mVw/eBd9IMC7TK9iPA/5G0d9l1EN1PS34cHQ7zFD8H9pe0Tfn3fhAdfaktaedyuzvwHrp/79BMTXNiuX8icG0fegCy5u5aJF0GHABMlfQQ8CnbF3VQehZwAnBfGWsHOMv2jR3U3hWYVxbJ2Qy4wnanp1X2yS7ANU0GsTnwTds3dVT7NODSMuTyIPDBjupSxrkPAU7uqiaA7QWSrgTuoRlq+QHdTWNwlaRXA88Bp9h+vM1iY+UIcDZwhaSTaD4E39dmDxP2lykbIiLqkqGeiIjKJPgjIiqT4I+IqEyCPyKiMgn+iIjKJPij7yRZ0rkjtj8u6dPr6XdfIumY9fG7JqnzvjLL5m1t14p4uRL8sSF4FniPpKn9bmSkcl1Dr04C/tz2O9rqJ2J9SfDHhmA1zYU8Hxn9wOgjdknPlNsDJH1X0hWS/lnS2ZLeX9YVuE/SniN+zcGS/ld53uHl9VMkfV7S3ZKWSDp5xO+9TdI3gfvG6Oe48vuXSjqn7PtvwB8AX5X0+VHP30zS/yjzwF8v6cbh96NmLYCp5f6gpNvL/W3LfO53l0ncjiz7f7e8v8Wl573Kc29Qs5bCUkl/VJ77lvK/zyJJN4+YKuB0ST8sr798Hf6/ik1ArtyNDcWXgSWS/vtLeM2+NFPuPkZz9euFtmeqWcTmNGB4wY3pwB8CewK3Sfod4AM0s0O+VdIrgO9JGp4pciawj+2fjiwm6TU0k329BXicZlbPo2z/paQDgY/bHr2Yy3tK/d+jmY1xGfD1Sd7Xf6aZuuJDZSqNuyT9I/Bh4Iu2h6/4nUIz0+TDtg8rPb6qzPn0JeBI20Plw+BzwIdoJoPbw/azw9N0RH1yxB8bhDIT6d/SLNTRq7vLOgbPAj8BhoP7PpqwHXaF7TW2l9N8QLyBZl6eD5TpMRYArwaGF+e4a3ToF28Fbi+TjK0GLqWZV38ifwB8q9R/BOjlO4B3AnNKb7cDWwG7A98HzpL0CeB1tn9T3uvBks6R9O9sPwnsDewDzC+/47/QzLoKsIRmqojjqXyGyprliD82JOfTzONy8Yh9qykHKGViry1HPPbsiPtrRmyv4cX/tkfPS2JAwGm2bx75gKQDaKZJHosmfQcv7TX//73RhPvI17zX9gOjnr9M0gKaRWtulvQntm+V9BaaI/+/Ln+1XAPcb3us5RwPo/mwOgL4r5J+t3yIRUVyxB8bDNuPAVfQfFE6bAXN0Ao0c7lvsQ6/+n1lrH1PmkVQHgBuBv6sDIsg6V9r8sVQFgB/KGlq+eL3OJrVnCZyJ/DeUn8Xmom7hq3ghff23hH7bwZOKx90SNqv3L4eeND2BTQzPf6bMvz0a9vfoFnk5M3l/Q2orOMraYvy/cBmwGtt30az8M0OwHaT9B+boBzxx4bmXODUEdtfA66VdBfNOqXjHY1P5AGagN4F+LDtf5F0Ic1w0D0lYIeYZCk82yslfZJmuEbAjbYnm1r3Kprph5cC/0zz4fFkeewzwEWSzuLFq639Fc1fP0tKbyuAw4E/Ao6X9BzwCPCXNMNPn5e0hmbmyT+z/dvyBfIFkl5F89/5+aX+N8o+Aef1acnH6LPMzhnRMknb2X5GzbTAd9Gs+vVIv/uKeuWIP6J915czaLYE/iqhH/2WI/6IiMrky92IiMok+CMiKpPgj4ioTII/IqIyCf6IiMr8Px40mtlheb0MAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%time show_metric(neg_entropy, True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Jotto Summary\n",
    "\n",
    "Here's a table of results:\n",
    "\n",
    "|<br><br>Algorithm|Consistent<br><u>Only</u><br>Mean (Max)|Inconsistent<br><u>Allowed</u><br>Mean (Max)|\n",
    "|--|--|--|\n",
    "|baseline random guesser|7.36 (17)|  |\n",
    "|minimize max_counts|7.15 (18)|7.05 (10)|\n",
    "|minimize expectation|7.14 (17)|6.84 (10)|\n",
    "|minimize neg_entropy|7.10 (19)|6.82 (10)|\n",
    "\n",
    "So we started out with a mean of 7.36 and a worst score of 17 with the random guesser, and were able to improve by half a guess to a mean of 6.81 and a worst score of 10 using  minimization of negative entropy over partition counts, with inconsistent guesses allowed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Wordle\n",
    "\n",
    "[Wordle](https://www.powerlanguage.co.uk/wordle/) is a [suddenly-popular](https://www.nytimes.com/2022/01/03/technology/wordle-word-game-creator.html) variant of Jotto (with some Mastermind thrown in) with these differences:\n",
    "- Words with repeated letters are allowed, as are anagrams.\n",
    "- The reply to a guess consists of 5 trits of information, one for each position in the guess:\n",
    "  - *Green* if the guess letter is in the correct spot.\n",
    "  - *Yellow* if the guess letter is in the word but in the wrong spot.\n",
    "  - *Miss* if the letter is not in the word in any spot.\n",
    "  \n",
    "Since repeated letters and anagrams are allowed, I can use all of `sgb_words` as my list of allowable Wordle words.\n",
    "\n",
    "There seems to be an ambiguity in the rules. Assume the guess is *etude* and the target is *poems*. I think the correct reply should be that one letter *e* is *yellow* and the other is a *miss*, although a strict reading of the rules would say they both should be *yellow*, because both instances of *e* are \"in the word but in the wrong spot.\" I decided that in cases like this I would report the first one as yellow and the second as a miss."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "Green, Yellow, Miss = 'GY.' # A reply is 5 characters, each one of 'GY.'\n",
    "\n",
    "def wordle_reply_for(guess, target) -> str: \n",
    "    \"The five-character reply for this guess on this target in Wordle.\"\n",
    "    # We'll start by having each reply be either Green (exact match) or Miss ...\n",
    "    pairs = list(zip(guess, target))\n",
    "    reply = [Green if g == t else Miss for g, t in pairs]\n",
    "    counts = Counter(t for g, t in pairs if g != t)\n",
    "    # ... then we'll put in the replies that should be yellow\n",
    "    for i in range(5):\n",
    "        if reply[i] == Miss and counts[guess[i]] > 0:\n",
    "            counts[guess[i]] -= 1\n",
    "            reply[i] = Yellow\n",
    "    return ''.join(reply)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that in Jotto, `reply_for` was symmetric; `reply_for(g, t) == reply_for(t, g)`. But that is not true for Wordle. I had to check my code to make sure I hadn't inadvertently reversed arguments anywhere. Here are some tests for `wordle_reply_for`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "assert wordle_reply_for('treat', 'truss') == 'GG...'\n",
    "assert wordle_reply_for('palls', 'splat') == 'YYG.Y'\n",
    "assert wordle_reply_for('splat', 'palls') == 'YYGY.'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The right thing to do now would be to refactor the code to allow for the injection of a different `reply_for` function. However, I'm not going to do that; instead I'm going to \"cheat\" and just redefine `reply_for` to be `wordle_reply_for`. So if you want to go back in this notebook and re-run some Jotto cells, you'll have to re-run the \"`def reply_for`\" cell first."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "reply_for = wordle_reply_for"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can test the new `reply_for` on a partition (using every 200th word in `sgb_words`):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "defaultdict(list,\n",
       "            {'Y....': ['which'],\n",
       "             '.YY..': ['dance'],\n",
       "             '.Y...': ['ended', 'rivet', 'vowed', 'coupe', 'unfed'],\n",
       "             '....G': ['trips'],\n",
       "             '.Y..G': ['poets', 'runes'],\n",
       "             '.....': ['drunk', 'oxbow'],\n",
       "             '...Y.': ['plump'],\n",
       "             '...G.': ['folly'],\n",
       "             '..G.G': ['gnats'],\n",
       "             '.YY.G': ['canes'],\n",
       "             'GGGGG': ['heals'],\n",
       "             '...GG': ['rills'],\n",
       "             '..YGG': ['palls'],\n",
       "             'YY..G': ['wheys'],\n",
       "             '..YG.': ['amply'],\n",
       "             '..Y.Y': ['gassy'],\n",
       "             '..Y..': ['kabob', 'outta'],\n",
       "             'Y.Y..': ['mahua'],\n",
       "             'Y...G': ['withs'],\n",
       "             '.Y.Y.': ['lurer'],\n",
       "             '.GG.G': ['teaks'],\n",
       "             'Y...Y': ['shirr']})"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "partition('heals', sgb_words[::200])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That looks good. Notice that there are many more possible replies in Wordle than the 6 possible replies in Jotto, so the target words are partitioned into smaller branches. It should take fewer guesses to solve a Wordle than a Jotto. How many possible replies are there? There are 3 responses at each of five positions, and  3<sup>5</sup> = 243, but five of those replies are impossible: you can't have four Greens and one Yellow, because if four letters of the guess are in the right place then the fifth must be either in the right place or a miss. \n",
    "\n",
    "Let's see what a game with a random guesser looks like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Guess 1: \"vivid\" Reply: ..... right; Consistent targets: 3262\n",
      "Guess 2: \"shrug\" Reply: Y.... right; Consistent targets: 524\n",
      "Guess 3: \"blabs\" Reply: .YY.G right; Consistent targets: 37\n",
      "Guess 4: \"palls\" Reply: .GG.G right; Consistent targets: 10\n",
      "Guess 5: \"calms\" Reply: .GGYG right; Consistent targets: 2\n",
      "Guess 6: \"males\" Reply: GGGGG right; Consistent targets: 1\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "6"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "play(random_guesser, wordlist=sgb_words, verbose=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I could compare all six metrics like I did with Jotto, but with a bigger word list the computations will take longer, so I'm only going to try minimizing negative entropy with inconsistent guesses allowed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mean: 3.82 ± 0.71 guesses, worst: 7, N: 5,756\n",
      "cumulative: ≤3:32%, ≤4:87%, ≤5:98%, ≤6:100%, ≤7:100%, ≤8:100%, ≤9:100%, ≤10:100%\n",
      "CPU times: user 9min 59s, sys: 738 ms, total: 10min\n",
      "Wall time: 10min\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEGCAYAAABiq/5QAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAUcUlEQVR4nO3dfbRddX3n8fcHgoIgIiYwlIBBzDBaO0pNqa50WgRxaGEgVmyhhWaETmyroGVcNTKdcfrgKtRaEZczDoKIyweKIAWBESkErdYBEkoBGylKo8PwkPjA04xFQ77zx9m3XEPuuSch+55783u/1jrrnL3P3md/Lw+f8zu/vffvl6pCktSOncZdgCRpZhn8ktQYg1+SGmPwS1JjDH5Jasy8cRcwivnz59eiRYvGXYYkzSlr1qz5TlUt2Hz9nAj+RYsWsXr16nGXIUlzSpJvbWm9XT2S1BiDX5IaY/BLUmMMfklqjMEvSY0x+CWpMQa/JDXG4Jekxhj8ktSYOXHnrrS9LFp5zbhLGGrd2ceMuwQ1wBa/JDXG4Jekxhj8ktQYg1+SGmPwS1JjDH5JaozBL0mNMfglqTEGvyQ1xuCXpMYY/JLUGINfkhpj8EtSY3odnTPJOuAx4ElgY1UtSbI38BfAImAd8CtV9f0+65AkPWUmWvyvqapXVNWSbnklcENVLQZu6JYlSTNkHF09xwMXd68vBpaNoQZJalbfwV/AF5KsSbKiW7dvVT0A0D3v03MNkqRJ+p6Ba2lV3Z9kH+D6JF8fdcfui2IFwIEHHthXfZLUnF5b/FV1f/e8HrgCOAx4KMl+AN3z+in2Pb+qllTVkgULFvRZpiQ1pbfgT7J7kudOvAZeB9wFXAUs7zZbDlzZVw2SpKfrs6tnX+CKJBPH+VRVfT7JrcClSU4Dvg28sccaJEmb6S34q+pe4OVbWP9d4Mi+jitJGs47dyWpMQa/JDXG4Jekxhj8ktQYg1+SGmPwS1JjDH5JaozBL0mNMfglqTEGvyQ1xuCXpMYY/JLUGINfkhpj8EtSYwx+SWqMwS9JjTH4JakxBr8kNcbgl6TGGPyS1BiDX5IaY/BLUmMMfklqjMEvSY0x+CWpMQa/JDXG4Jekxhj8ktQYg1+SGtN78CfZOcnfJrm6Wz4oyc1J7knyF0me1XcNkqSnzESL/23A2knL5wDvr6rFwPeB02agBklSp9fgT7IQOAa4oFsOcARwWbfJxcCyPmuQJP24vlv85wK/B2zqll8APFxVG7vl+4D9t7RjkhVJVidZvWHDhp7LlKR29Bb8SY4F1lfVmsmrt7BpbWn/qjq/qpZU1ZIFCxb0UqMktWhej5+9FDguyS8BuwJ7MvgFsFeSeV2rfyFwf481SJI201uLv6reVVULq2oRcCJwY1X9OrAKOKHbbDlwZV81SJKebhzX8b8TODPJNxj0+V84hhokqVl9dvX8s6q6Cbipe30vcNhMHFeS9HTeuStJjTH4JakxBr8kNcbgl6TGGPyS1BiDX5IaY/BLUmMMfklqzLTBn+TgJM/uXh+e5Iwke/VfmiSpD6O0+C8HnkzyYgbDKxwEfKrXqiRJvRkl+Dd1I2m+Hji3qn4X2K/fsiRJfRkl+H+U5CQGI2le3a3bpb+SJEl9GiX43wS8GnhPVf1jkoOAT/RbliSpL6OMznlUVZ0xsdCF/w96rEmS1KNRWvzLt7Du32/nOiRJM2TKFn/Xr/9rwEFJrpr01nOB7/ZdmCSpH8O6ev4GeACYD7xv0vrHgDv6LEqS1J8pg7+qvgV8C3h1khcCi6vqr5LsBuzG4AtAkjTHjHLn7n8ALgP+R7dqIfCXfRYlSerPKCd33wIsBR4FqKp7gH36LEqS1J9Rgv+JqvrhxEKSeUD1V5IkqU+jBP8Xk5wF7JbkKOAzwOf6LUuS1JdRgn8lsAG4E3gzcC3w+30WJUnqz7R37lbVJuAj3UOSNMdNG/xJ7uTpffqPAKuBP64qb+aSpDlklLF6/ifwJE+NwX9i9/wo8DHg323/siRJfRkl+JdW1dJJy3cm+UpVLU1ycl+FSZL6McrJ3T2S/OzEQpLDgD26xY29VCVJ6s0oLf7TgIuSTIT9Y8BpSXYH/qS3yiRJvRga/El2Al5UVT+V5HlAqurhSZtcOmTfXYEvAc/ujnNZVb27m8jlEmBv4DbglMk3iGluWbTymnGXMNS6s48ZdwnSrDO0q6e7lPOt3etHNgv96TwBHFFVLwdeARyd5FXAOcD7q2ox8H0GvygkSTNklD7+65O8I8kBSfaeeEy3Uw083i3u0j0KOILBoG8AFwPLtqVwSdK2GaWP/9Tu+S2T1hXwoul2TLIzsAZ4MfAh4JvAw1U1cVL4PmD/KfZdAawAOPDAA0coU5I0ilHu3D1oWz+8qp4EXpFkL+AK4CVb2myKfc8HzgdYsmSJg8JJ0nYySoufJC8DXgrsOrGuqj4+6kGq6uEkNwGvAvZKMq9r9S8E7t+qiiVJz8goE7G8G/hg93gN8KfAcSPst6Br6dPN2vVaYC2wCjih22w5cOU2VS5J2iajnNw9ATgSeLCq3gS8nMElmtPZD1iV5A7gVuD6qroaeCdwZpJvAC8ALtymyiVJ22SUrp4fVNWmJBuT7AmsZ4QTu1V1B3DoFtbfCxy21ZVKkraLUYJ/dddl8xEGV+g8DtzSa1WSpN6MclXP73QvP5zk88CeXWtekjQHjXJy94aJ11W1rqrumLxOkjS3TNni78baeQ4wP8nzgXRv7Qn8xAzUJknqwbCunjcDb2cQ8mt4KvgfZXAXriRpDpoy+KvqA8AHkpxeVR+cwZokST2ato/f0JekHcsoN3BJknYgUwZ/kqXd8yh36UqS5ohhLf7zuuevzkQhkqSZMeyqnh8luQjYP8l5m79ZVWf0V5YkqS/Dgv9YBiNqHsHgck5J0g5g2OWc3wEuSbK2qv5uBmuSJPVolKt6vpvkiiTrkzyU5PIkC3uvTJLUi1GC/yLgKgZ38O4PfK5bJ0mag0YJ/n2q6qKq2tg9PgYs6LkuSVJPRgn+DUlOTrJz9zgZ+G7fhUmS+jFK8J8K/ArwIPAAg6kYT+2zKElSf0aZiOXbjDC5uiRpbnCsHklqjMEvSY0x+CWpMSMHf5JXJbkxyVeSLOuzKElSf4bNufsvqurBSavOZHCSN8DfAH/Zc22SpB4Mu6rnw0nWAO+tqn8CHgZ+DdjEYN5dSdIcNGVXT1UtA24Hrk5yCoOJ1zcBzwHs6pGkOWpoH39VfQ74t8BewGeBu6vqvKraMBPFSZK2v2FTLx6X5MvAjcBdwInA65N8OsnBM1WgJGn7GtbH/8fAq4HdgGur6jDgzCSLgfcw+CKQJM0xw7p6HmEQ7icC6ydWVtU9VTVt6Cc5IMmqJGuTfC3J27r1eye5Psk93fPzn+kfIUka3bDgfz2DE7kbGVzNs7U2Av+xql4CvAp4S5KXAiuBG6pqMXBDtyxJmiHTTb34wW394Kp6gMFonlTVY0nWMpjI5Xjg8G6zi4GbgHdu63EkSVtnRoZsSLIIOBS4Gdi3+1KY+HLYZ4p9ViRZnWT1hg1eRCRJ20vvwZ9kD+By4O1VNfKNX1V1flUtqaolCxY44ZckbS+9Bn+SXRiE/ier6rPd6oeS7Ne9vx+TThxLkvrXW/AnCXAhsLaq/nzSW1cBy7vXy4Er+6pBkvR0087A9QwsBU4B7kxye7fuLOBs4NIkpwHfBt7YYw2SpM30FvxV9WUGI3luyZF9HVeSNJwTsUhSYwx+SWqMwS9JjTH4JakxBr8kNcbgl6TGGPyS1BiDX5IaY/BLUmMMfklqjMEvSY0x+CWpMQa/JDXG4Jekxhj8ktQYg1+SGmPwS1JjDH5JaozBL0mNMfglqTEGvyQ1xuCXpMYY/JLUGINfkhpj8EtSYwx+SWrMvHEXIGlqi1ZeM+4Shlp39jHjLkHbwBa/JDXG4JekxvQW/Ek+mmR9krsmrds7yfVJ7umen9/X8SVJW9Zni/9jwNGbrVsJ3FBVi4EbumVJ0gzqLfir6kvA9zZbfTxwcff6YmBZX8eXJG3ZTPfx71tVDwB0z/tMtWGSFUlWJ1m9YcOGGStQknZ0s/bkblWdX1VLqmrJggULxl2OJO0wZjr4H0qyH0D3vH6Gjy9JzZvp4L8KWN69Xg5cOcPHl6Tm9Xk556eBrwKHJLkvyWnA2cBRSe4BjuqWJUkzqLchG6rqpCneOrKvY0qSpjdrT+5Kkvph8EtSYwx+SWqMwS9JjTH4JakxBr8kNcbgl6TGGPyS1BiDX5IaY/BLUmMMfklqjMEvSY0x+CWpMQa/JDXG4Jekxhj8ktQYg1+SGmPwS1JjDH5JaozBL0mNMfglqTEGvyQ1xuCXpMYY/JLUGINfkhpj8EtSY+aNuwBJO5ZFK68ZdwlTWnf2MeMuYVawxS9JjbHFP8vM5tYS2GKSdgRjafEnOTrJ3Um+kWTlOGqQpFbNePAn2Rn4EPCLwEuBk5K8dKbrkKRWjaOr5zDgG1V1L0CSS4Djgb8fQy2SGjebu1f76lpNVfXywVMeMDkBOLqqfrNbPgX42ap662bbrQBWdIuHAHfPaKFTmw98Z9xFbAXr7Zf19muu1Quzq+YXVtWCzVeOo8WfLax72rdPVZ0PnN9/OVsnyeqqWjLuOkZlvf2y3n7NtXphbtQ8jpO79wEHTFpeCNw/hjokqUnjCP5bgcVJDkryLOBE4Kox1CFJTZrxrp6q2pjkrcB1wM7AR6vqazNdxzMw67qfpmG9/bLefs21emEO1DzjJ3clSePlkA2S1BiDX5IaY/CPKMlHk6xPcte4axlFkgOSrEqyNsnXkrxt3DUNk2TXJLck+buu3j8Yd02jSLJzkr9NcvW4a5lOknVJ7kxye5LV465nOkn2SnJZkq93/x2/etw1TSXJId0/14nHo0nePu66pmIf/4iS/DzwOPDxqnrZuOuZTpL9gP2q6rYkzwXWAMuqalbeIZ0kwO5V9XiSXYAvA2+rqv815tKGSnImsATYs6qOHXc9wyRZByypqtlyc9FQSS4G/rqqLuiuAHxOVT087rqm0w1L838Y3Jj6rXHXsyW2+EdUVV8CvjfuOkZVVQ9U1W3d68eAtcD+461qajXweLe4S/eY1a2SJAuBY4ALxl3LjibJnsDPAxcCVNUP50Lod44EvjlbQx8M/iYkWQQcCtw83kqG67pNbgfWA9dX1ayuFzgX+D1g07gLGVEBX0iyphsSZTZ7EbABuKjrSrsgye7jLmpEJwKfHncRwxj8O7gkewCXA2+vqkfHXc8wVfVkVb2Cwd3chyWZtV1qSY4F1lfVmnHXshWWVtVPMxgZ9y1d9+VsNQ/4aeC/V9WhwP8FZv0Q7l2X1HHAZ8ZdyzAG/w6s6yu/HPhkVX123PWMqvtJfxNw9JhLGWYpcFzXb34JcESST4y3pOGq6v7ueT1wBYORcmer+4D7Jv3qu4zBF8Fs94vAbVX10LgLGcbg30F1J0svBNZW1Z+Pu57pJFmQZK/u9W7Aa4Gvj7eqqVXVu6pqYVUtYvDT/saqOnnMZU0pye7dSX66LpPXAbP2CrWqehD430kO6VYdydwYuv0kZnk3Dzj14siSfBo4HJif5D7g3VV14XirGmopcApwZ9dvDnBWVV07xpqG2Q+4uLsiYifg0qqa9ZdIziH7AlcM2gPMAz5VVZ8fb0nTOh34ZNd9ci/wpjHXM1SS5wBHAW8edy3T8XJOSWqMXT2S1BiDX5IaY/BLUmMMfklqjMEvSY0x+DV2SSrJ+yYtvyPJf91On/2xJCdsj8+a5jhv7EaQXNX3saRnyuDXbPAE8MtJ5o+7kMm6ewpGdRrwO1X1mr7qkbYXg1+zwUYG85T+7uZvbN5iT/J493x4ki8muTTJPyQ5O8mvd2P635nk4Ekf89okf91td2y3/85J3pvk1iR3JHnzpM9dleRTwJ1bqOek7vPvSnJOt+6/AD8HfDjJezfbfqck/62bY+DqJNdO/D3d+Pjzu9dLktzUvd49g/kfbu0GKDu+W/+T3d93e1fz4m7bazKYx+CuJL/abfvK7p/PmiTXdcN0k+SMJH/f7X/JNvy70g7AO3c1W3wIuCPJn27FPi8HXsJguOx7gQuq6rAMJp05HZiYCGMR8AvAwcCqJC8GfgN4pKp+Jsmzga8k+UK3/WHAy6rqHycfLMlPAOcArwS+z2Cky2VV9YdJjgDeUVWbT3Dyy93xfwrYh8Hw2B+d5u/6TwyGgDi1G8biliR/BfwW8IGqmribdWfgl4D7q+qYrsbndWM0fRA4vqo2dF8G7wFOZTDQ2UFV9cTEEBlqjy1+zQrdyKEfB87Yit1u7eYdeAL4JjAR3HcyCNsJl1bVpqq6h8EXxL9iMFbNb3TDWdwMvABY3G1/y+ah3/kZ4Kaq2lBVG4FPMhgzfpifAz7THf9BYJRzAK8DVna13QTsChwIfBU4K8k7gRdW1Q+6v/W1Sc5J8m+q6hHgEOBlwPXdZ/w+gxFPAe5gMAzCyQx+aalBtvg1m5wL3AZcNGndRroGSjfw3LMmvffEpNebJi1v4sf/2958XJICApxeVddNfiPJ4QyGAN6STPsXbN0+//y3MQj3yfu8oaru3mz7tUluZjD5y3VJfrOqbkzySgYt/z/pfrVcAXytqrY0VeExDL6sjgP+c5Kf7L7E1BBb/Jo1qup7wKUMTpROWMegawXgeAYzc22tN3Z97QczmODjbuA64Le7bhGS/MtMP9HHzcAvJJnfnfg9CfjiNPt8GXhDd/x9GQz0N2EdT/1tb5i0/jrg9O6LjiSHds8vAu6tqvOAq4B/3XU//b+q+gTwZwyGLr4bWJBujtoku3TnB3YCDqiqVQwmkNkL2GOa+rUDssWv2eZ9wFsnLX8EuDLJLcANTN0aH+ZuBgG9L/BbVfVPSS5g0B10WxewG4Blwz6kqh5I8i4G3TUBrq2qK6c59uUMhhS+C/gHBl8ej3Tv/QFwYZKz+PHZ0f6Iwa+fO7ra1gHHAr8KnJzkR8CDwB8y6H56b5JNwI+A366qH3YnkM9L8jwG/5+f2x3/E926AO+fQ9MZajtydE6pZ0n26CaRfwFwC4OZsB4cd11qly1+qX9Xd1fQPAv4I0Nf42aLX5Ia48ldSWqMwS9JjTH4JakxBr8kNcbgl6TG/H+br1vTsGpqzgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%time show_metric(neg_entropy, True, sgb_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's pretty good. The Wordle web site challenges you to solve each puzzle in six guesses; this guesser can do that almost all the time (the output says \"`≤6:100%`\" but that's rounded off; it is actually only 99.8%)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Wordle Words and Wordle Helper\n",
    "\n",
    "Wordle does not use `sgb-words.txt`, so the results above are good for comparing Jotto to Wordle, but imperfect for actually playing Wordle. I made an attempt to extract the official word list from the [Wordle site](https://www.powerlanguage.co.uk/wordle/). The javascript code there actually contains two word lists, but the code is obfuscated so I'm not sure what's going on with them. Possibly the target word is chosen from the smaller list, and the larger list defines the allowable guesses. Here are the two files:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    2315 wordle1.txt\n",
      "   10657 wordle2.txt\n",
      "   12972 total\n"
     ]
    }
   ],
   "source": [
    "wordle1 = open('wordle1.txt').read().split()\n",
    "wordle2 = open('wordle2.txt').read().split()\n",
    "\n",
    "!wc -w wordle*.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Twice as many words as `sgb-words.txt`.\n",
    "\n",
    "Here's a function, `helper`, to interactively make suggestions for the day's puzzle. First I'll search for top-scoring first guesses:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'raise slate crate irate trace arise stare snare arose least alert crane stale saner alter later react leant trade learn cater roast aisle trice scare parse saute heart alone store alien share grate trail siren snore caste scale atone renal'"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "first_guesses = sorted(wordle1, key=lambda g: neg_entropy(partition_counts(g, wordle1)))\n",
    "' '.join(first_guesses[:50])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "def helper(words=wordle1 + wordle2, n=15) -> str:\n",
    "    \"\"\"Type your guess into Wordle, then type the guess and reply here and get suggestions for the next guess.\"\"\"\n",
    "    print(helper.__doc__)\n",
    "    good = first_guesses[:n]\n",
    "    while len(words) > 1:\n",
    "        print(f'{len(words)} possible words. Some good ones:\\n{\" \".join(good[:n])}.')\n",
    "        response = input(f'your guess and reply (e.g. \"{good[0]} G...Y\")?').split()\n",
    "        guess, reply = response[0].lower(), response[1].upper()\n",
    "        words = [w for w in words if wordle_reply_for(guess, w) == reply]\n",
    "        good = sorted(words, key=lambda g: neg_entropy(partition_counts(g, words))) \n",
    "    print('It must be:', words[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Next Steps\n",
    "\n",
    "There are many directions you could take this if you are interested:\n",
    "- Do the refactoring so that the code can neatly handle multiple different games with different replies, etc.\n",
    "- Run the computations to figure out the best strategy for Wordle.\n",
    "- Rerun the computations with the larger Wordle word lists. If necessary, optimize code first.\n",
    "- Consider game variant where each reply consists of two numbers: the number of letters in common with the target, and the number of letters that are in the exact correct position (as in Mastermind).\n",
    "- Implement Mastermind with 6 colors and 4 pegs, and with other combinations.\n",
    "- What's the best strategy for a chooser who is trying to make the guesser get a bad score.  Is there a strategy equilibrium?\n",
    "- Our `make_tree` function is **greedy** in that it guesses the word that minimizes some metric of the current situation without looking ahead to future branches in the tree. Can you get better performance by doing some **look-ahead**? Perhaps with a beam search?\n",
    "- Can you improve a tree by editing it? Given a tree, look for interior nodes that end up with a worse-than-expected average score, and see if the node can be replaced with something better (covering the same target words). Correcting a few bad nodes might be faster than carefully searching for good nodes in the first place.\n",
    "- Research what other computer scientists have done with [Jotto](https://arxiv.org/abs/1107.3342) or [Mastermind](http://serkangur.freeservers.com/).\n",
    "- What else can you explore?"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}