From 81abe96fcf5eac1269e7b796493993064a51b260 Mon Sep 17 00:00:00 2001 From: Peter Norvig Date: Mon, 6 Jul 2020 14:47:56 -0700 Subject: [PATCH] Add files via upload --- ipynb/jotto.ipynb | 1153 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 1153 insertions(+) create mode 100644 ipynb/jotto.ipynb diff --git a/ipynb/jotto.ipynb b/ipynb/jotto.ipynb new file mode 100644 index 0000000..987e7a9 --- /dev/null +++ b/ipynb/jotto.ipynb @@ -0,0 +1,1153 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "
Peter Norvig
April 2020
\n", + "\n", + "# Jotto\n", + "\n", + "[Jotto](https://en.wikipedia.org/wiki/Jotto) is a game in which a **guesser** tries to guess a **target** word, which is chosen from a list of permissible words, in a minimum number of guesses. The **reply** to each guess is the number of letters in common between the guess word and the target word, regardless of the positions of the letters. \n", + "\n", + "Here is an example Jotto game, where I show the guesses, the replies, the number of possible targets that are consistent with all the guesses/replies seen so far, and finally the letters that matched (this is as an aid to you, the reader; these matching letters are not known to the guesser). In this game, the guesser gets to the target word, \"wonky\", in 7 guesses. \n", + "\n", + " Guess 1: stoma, Reply: 1, Possible targets: 1118 (Matched: \"o\")\n", + " Guess 2: bairn, Reply: 1, Possible targets: 441 (Matched: \"n\")\n", + " Guess 3: swipe, Reply: 1, Possible targets: 197 (Matched: \"w\")\n", + " Guess 4: lurks, Reply: 1, Possible targets: 87 (Matched: \"k\")\n", + " Guess 5: rowdy, Reply: 3, Possible targets: 14 (Matched: \"owy\")\n", + " Guess 6: roved, Reply: 1, Possible targets: 2 (Matched: \"o\")\n", + " Guess 7: wonky, Reply: 5, Possible targets: 1 (Matched: \"wonky\")\n", + "\n", + "\n", + "\n", + "There are several variants of the game; here are my answers to four key questions:\n", + "\n", + "1. How many letters can each word be?
My answer is **only five-letter words**.\n", + "2. Can a guess be a word that is not in the word list?
My answer is **no**.\n", + "1. What is the reply for a word that has the same letter twice, like the `s` in `stars`?
My answer is to **disallow such words**.\n", + "2. What if the reply is `5`, but the guess is not the target? For example, if the guess is `parse` the\n", + "target might be `apres`, `asper`, `pares`, `parse`, `pears`, `reaps`, `spare`, or `spear`.
My answer is to **disallow two or more words that are anagrams** of each other; only one such word is allowed in the word list. \n", + "\n", + "Typically, the **guesser** plays against a **chooser**, who chooses the target word. This introduces a game-theoretic aspect: the chooser chooses a word that should be difficult for the guesser, but knowing that, the guesser modifies their strategy, and knowing that, the chooser further modifies their choice, etc. To avoid these complications, I will instead use a scenario that eliminates the chooser: the guesser plays against *every* possible target, and we measure the scores from all those games. (This is appropriate when the guesser is a program, but would be extremely tedious for a human guesser.)\n", + "\n", + "We can make a Jotto word list by reading in a file of words, `sgb-words.txt`, and keeping only those words that have five distinct letters, and only one of the words that share a common anagram form. (But first some imports.)" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": {}, + "outputs": [], + "source": [ + "import matplotlib.pyplot as plt\n", + "import random\n", + "from typing import List\n", + "from statistics import mean, stdev\n", + "from collections import namedtuple\n", + "from math import log\n", + "\n", + "! [ -e sgb-words.txt ] || curl -O https://norvig.com/ngrams/sgb-words.txt" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "2845" + ] + }, + "execution_count": 2, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "def jotto_words(words) -> List[str]:\n", + " \"\"\"Build a list of permissible Jotto words from an iterable of words.\"\"\"\n", + " anagram_table = {anagram_form(w): w \n", + " for w in words if len(w) == 5 == len(set(w))}\n", + " return list(anagram_table.values())\n", + "\n", + "def anagram_form(word) -> str: return ''.join(sorted(word))\n", + "\n", + "wordlist = jotto_words(open('sgb-words.txt').read().split())\n", + "len(wordlist)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We see there are 2845 permissible words." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Playing a Game\n", + "\n", + "We will define the function `play_jotto` to play a game and optionally produce the output shown above in the example game (without the \"(Matched: ...)\"), and finally return the number of guesses made. To avoid a possible infinite loop with a poor guesser, we impose an upper limit of 1000 guesses for a game.\n", + "\n", + "The first argument to `play_jotto` is a `guesser`. I choose to implement a guesser as a `callable` (e.g., a function) that is passed the current state of the game and returns a guess word. So how do we represent the state of the game? `play_jotto` is keeping track of the `targets` that are still possible given previous guesses and replies, so we might as well pass that in. In addition, we pass in the reply from the previous guess. Note that a guess need not be one of the possible targets, but it must be one of the words in the word list. (If it is not, the guesser forfeits the game, and the score is recorded as the limit.) " + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [], + "source": [ + "def play_jotto(guesser, target=None, wordlist=wordlist, limit=1000, verbose=False) -> int:\n", + " \"\"\"How many guesses does it take for the guesser to guess the Jotto target word?\"\"\"\n", + " target = target or random.choice(wordlist)\n", + " targets = wordlist\n", + " reply = None\n", + " for guesses in range(1, limit + 1):\n", + " guess = guesser(reply, targets)\n", + " if guess not in wordlist:\n", + " return limit\n", + " reply = reply_for(target, guess)\n", + " targets = [w for w in targets if reply_for(guess, w) == reply]\n", + " if verbose: \n", + " print(f'Guess {guesses}: {guess}, Reply: {reply}, Possible targets: {len(targets)}')\n", + " if guess == target or guesses == limit: \n", + " return guesses\n", + " \n", + "def reply_for(target, guess) -> int: \n", + " \"The number of letters in common between the target and guess\"\n", + " return len(set(target).intersection(guess))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Random Guesser\n", + "\n", + "One simple guesser function is `random_guesser`: from the possible targets, pick one at random. That sounds naive, but it is actually a decent strategy:" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Guess 1: blunt, Reply: 0, Possible targets: 683\n", + "Guess 2: exams, Reply: 2, Possible targets: 290\n", + "Guess 3: vised, Reply: 2, Possible targets: 120\n", + "Guess 4: heard, Reply: 3, Possible targets: 38\n", + "Guess 5: redox, Reply: 2, Possible targets: 30\n", + "Guess 6: pawed, Reply: 4, Possible targets: 8\n", + "Guess 7: waked, Reply: 3, Possible targets: 3\n", + "Guess 8: paged, Reply: 4, Possible targets: 2\n", + "Guess 9: caped, Reply: 5, Possible targets: 1\n" + ] + }, + { + "data": { + "text/plain": [ + "9" + ] + }, + "execution_count": 4, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "def random_guesser(_, targets) -> str: return random.choice(targets)\n", + "\n", + "play_jotto(random_guesser, verbose=True)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Evaluating Guessers\n", + "\n", + "That was just one sample game. How well will this guesser do averaged over all targets? The function `play_jottos` plays a game against every target and collects the scores. Then, `show` displays a histogram and the mean, standard deviation, and maximum values of the scores. " + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [], + "source": [ + "def play_jottos(guesser, targets=wordlist, wordlist=wordlist) -> List[int]:\n", + " \"\"\"Scores for this guesser on all targets.\"\"\"\n", + " return [play_jotto(guesser, target, wordlist, verbose=False)\n", + " for target in targets]\n", + " \n", + "def show(scores):\n", + " \"\"\"Show a histogram and statistics for these scores.\"\"\"\n", + " scores = list(scores)\n", + " bins = range(min(scores), max(scores) + 2) \n", + " weights = [1/len(scores)] * len(scores)\n", + " plt.hist(scores, align='left', rwidth=0.9, bins=bins, weights=weights)\n", + " plt.xticks(bins[:-1])\n", + " plt.xlabel('Number of guesses to find target'); \n", + " plt.ylabel('Proportion of targets')\n", + " print(f'mean = {mean(scores):.2f} ± {stdev(scores):.2f}, max = {max(scores)}')" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Since `random_guesser` has a random component, let's evaluate each target multiple times to reduce variance. Let's say 5 times:" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "mean = 7.37 ± 1.71, max = 18\n" + ] + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEGCAYAAAB/+QKOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAexElEQVR4nO3de7hWZZ3/8fcnFM8iIk4mGGioWZmHrVYampqiNNDBA53GZiwny7K67CeOZYozhTkdxjLD0nLKxGPKKIUm0OEqlYOoIJGIpFs0Kaw8pQLf3x/3vXW5WXvv9Tx7r7038Hld17qedbrv57ue0/dZp/tWRGBmZtbeq/o6ADMz65+cIMzMrJQThJmZlXKCMDOzUk4QZmZWapO+DqCn7LDDDjFixIi+DsPMbL0yb968P0fE0LJlG0yCGDFiBHPnzu3rMMzM1iuS/tjRMh9iMjOzUk4QZmZWygnCzMxKOUGYmVkpJwgzMyvlBGFmZqWcIMzMrJQThJmZlXKCMDOzUhvMndS2YRox8Zamyy6fPLYHIzHb+HgPwszMSjlBmJlZKScIMzMr5QRhZmalnCDMzKyUE4SZmZVygjAzs1JOEGZmVsoJwszMSjlBmJlZKScIMzMrVWuCkDRG0hJJSyVNLFn+OUn3S7pX0u2SXltYtkbSgjxMqzNOMzNbV22N9UkaAFwMvBNoBeZImhYR9xdWuxtoiYhnJZ0KfBU4MS97LiL2qSs+MzPrXJ17EAcCSyNiWUS8AEwFxhdXiIhZEfFsnrwDGFZjPGZm1oA6E8TOwCOF6dY8ryMnAz8rTG8uaa6kOyS9u6yApFPyOnNXrlzZ/YjNzOwldfYHoZJ5Ubqi9CGgBTi0MHuXiFghaVdgpqT7IuLBV1QWcSlwKUBLS0tp3WZm1pw69yBageGF6WHAivYrSToSOBsYFxHPt82PiBX5cRkwG9i3xljNzKydOhPEHGCUpJGSBgITgFdcjSRpX2AKKTk8UZg/WNJmeXwH4GCgeHLbzMxqVtshpohYLek0YAYwALg8IhZJmgTMjYhpwIXA1sC1kgAejohxwOuBKZLWkpLY5HZXP5mZWc1q7ZM6IqYD09vNO6cwfmQH5X4LvKnO2MzMrHO+k9rMzEo5QZiZWSknCDMzK+UEYWZmpZwgzMysVK1XMZn1ByMm3tJ02eWTx/ZgJGbrF+9BmJlZKScIMzMr5QRhZmalnCDMzKyUE4SZmZVygjAzs1JOEGZmVsoJwszMSjlBmJlZKScIMzMr5QRhZmalnCDMzKyUE4SZmZVygjAzs1JOEGZmVsoJwszMSjlBmJlZKScIMzMr5QRhZmalnCDMzKyUE4SZmZVygjAzs1JdJghJx0vaJo9/QdINkvarPzQzM+tLVfYgvhgRT0k6BDgauAK4pErlksZIWiJpqaSJJcs/J+l+SfdKul3SawvLTpL0QB5OqrpBZmbWM6okiDX5cSxwSUTcBAzsqpCkAcDFwDHAXsD7Je3VbrW7gZaI2Bu4DvhqLrs98CXgIOBA4EuSBleI1czMekiVBPGopCnACcB0SZtVLHcgsDQilkXEC8BUYHxxhYiYFRHP5sk7gGF5/GjgtohYFRFPArcBYyo8p5mZ9ZAqP/QnADOAMRHxV2B74PMVyu0MPFKYbs3zOnIy8LNGyko6RdJcSXNXrlxZISQzM6uqSoKYEhE3RMQDABHxGPDhCuVUMi9KV5Q+BLQAFzZSNiIujYiWiGgZOnRohZDMzKyqKgniDcWJfG5h/wrlWoHhhelhwIr2K0k6EjgbGBcRzzdS1szM6tNhgpB0lqSngL0l/V3SU3n6CeCmCnXPAUZJGilpIDABmNbuOfYFppCSwxOFRTOAoyQNzienj8rzzMysl3SYICLiKxGxDXBhRGwbEdvkYUhEnNVVxRGxGjiN9MO+GLgmIhZJmiRpXF7tQmBr4FpJCyRNy2VXAeeTkswcYFKeZ2ZmvWSTCuucnc8RjIyI8yUNB3aKiLu6KhgR04Hp7eadUxg/spOylwOXV4jPzMxqUOUcxMXAW4EP5Omn8zwzM9uAVdmDOCgi9pN0N0BEPJnPKZiZ2Qasyh7Ei/nKpQCQNBRYW2tUZmbW56okiIuAnwI7Svov4DfAl2uNyszM+lyXh5gi4kpJ84AjSDewvTsiFtcemZmZ9akuE0RuOO8J4KrCvE0j4sU6AzMzs75V5RDTfGAl8AfggTz+kKT5kqrcUW1mZuuhKgni58CxEbFDRAwhNd99DfAJ4Dt1BmdmZn2nSoJoiYiXmrmIiFuB0RFxB7BZbZGZmVmfqnIfxCpJZ5L6cwA4EXgyX/rqy13NzDZQVfYgPkBqTfXGPAzP8waQ+oowM7MNUKd7EHkv4cyI+FQHqyzt+ZDMzKw/6HQPIiLWUK3vBzMz28BUOQdxd26G+1rgmbaZEXFDbVGZmVmfq5Igtgf+AhxemBeAE4SZ2QasSlMb/9obgZiZWf9SpamNzYGTSX1Tb942PyL+rca4bAMwYuItTZddPnlsD0ZiZs2ocpnrj4BXA0cDvyRd8vpUnUGZmVnfq5IgXhcRXwSeiYgrgLHAm+oNy8zM+lqlDoPy418lvREYBIyoLSIzM+sXqlzFdKmkwcAXgGnA1sAXa43KzMz6XJUEcXtEPAn8CtgVQNLIWqMyM7M+V+UQ0/Ul867r6UDMzKx/6XAPQtKepEtbB0l6b2HRthQudzUzsw1TZ4eY9gDeBWwH/HNh/lPAx+oMyszM+l6HCSIibgJukvTWiPhdL8ZkZmb9QJfnIJwczMw2TlVOUpuZ2UaowwQh6fT8eHDvhWNmZv1FZ3sQba24fqvZyiWNkbRE0lJJE0uWj5Y0X9JqSce1W7ZG0oI8TGs2BjMza05nVzEtlrQcGCrp3sJ8ARERe3dWce6u9GLgnUArMEfStIi4v7Daw8BHgDNKqnguIvbpehPMzKwOnV3F9H5JrwZmAOOaqPtAYGlELAOQNBUYD7yUICJieV62ton6zcysRl31Sf14RLwZeAzYJg8rIuKPFereGXikMN2a51W1uaS5ku6Q9O4GypmZWQ+o0mHQocD/AstJh5eGSzopIn7VVdGSedFAbLtExApJuwIzJd0XEQ+2i+0U4BSAXXbZpYGqzcysK1Uuc/06cFREHBoRo0kdB32jQrlWYHhhehiwompgEbEiPy4DZgP7lqxzaUS0RETL0KFDq1ZtZmYVVEkQm0bEkraJiPgDsGmFcnOAUZJGShoITCA1F94lSYMlbZbHdwAOpnDuwszM6lclQcyVdJmkw/LwPWBeV4UiYjVwGukk92LgmohYJGmSpHEAkg6Q1AocD0yRtCgXf31+3nuAWcDkdlc/mZlZzar0B3Eq8Eng06TzCr8CvlOl8oiYDkxvN++cwvgc0qGn9uV+i7s1NTPrU10miIh4nnQe4uv1h2NmZv2F22IyM7NSThBmZlbKCcLMzEpVuVFud+DzwGuL60fE4TXGZWZmfazKVUzXAt8FvgesqTccMzPrL6okiNURcUntkZiZWb9S5RzE/0n6hKSdJG3fNtQemZmZ9akqexAn5cfPF+YFsGvPh2NmZv1FlRvlRvZGIGZm1r9UuYppU1JzG6PzrNnAlIh4sca4zMysj1U5xHQJqfXWtvaXPpznfbSuoMzMrO9VSRAH5F7l2szMrayamdkGrMpVTGsk7dY2kXt48/0QZmYbuCp7EJ8HZklaRmru+7XAv9YalZmZ9bkqVzHdLmkUsAcpQfw+NwFuZmYbsA4ThKTDI2KmpPe2W7SbJCLihppjM+s3Rky8pemyyyeP7cFIzHpPZ3sQhwIzgX8uWRaAE4SZ2QaswwQREV/Ko5Mi4qHiMkm+ec7MbANX5Sqm60vmXdfTgZiZWf/S2TmIPYE3AIPanYfYFti87sDMzKxvdXYOYg/gXcB2vPI8xFPAx+oMyszM+l5n5yBuknQzcGZEfLkXYzIzs36g03MQEbEGeGcvxWJmZv1IlTupfyvp28DVwDNtMyNifm1RmZlZn6uSIN6WHycV5gVweM+HY2Zm/UWVpjbe0RuBmJlZ/9LlfRCSBkn6uqS5efiapEG9EZyZmfWdKjfKXU66tPWEPPwd+EGdQZmZWd+rcg5it4h4X2H6PEkL6grIzMz6hyp7EM9JOqRtQtLBwHNVKpc0RtISSUslTSxZPlrSfEmrJR3XbtlJkh7Iw0lVns/MzHpOlT2IU4Er8nkHAauALn+wJQ0ALibdR9EKzJE0LSLuL6z2MPAR4Ix2ZbcHvgS0kK6YmpfLPlkhXjMz6wFVrmJaALxZ0rZ5+u8V6z4QWBoRywAkTQXGAy8liIhYnpetbVf2aOC2iFiVl98GjAGuqvjcZmbWTVWuYhoi6SJgNqnr0f+RNKRC3TsDjxSmW/O8KiqVlXRK29VVK1eurFi1mZlVUeUcxFRgJfA+4Lg8fnWFciqZFxXjqlQ2Ii6NiJaIaBk6dGjFqs3MrIoqCWL7iDg/Ih7Kw3+SWnjtSiswvDA9DFhRMa7ulDUzsx5QJUHMkjRB0qvycAJQpYPeOcAoSSMlDQQmANMqxjUDOErSYEmDgaPyPDMz6yVVEsS/Az8BXsjDVOBzkp6S1OEJ64hYDZxG+mFfDFwTEYskTZI0DkDSAZJageOBKZIW5bKrgPNJSWYOqdvTVc1upJmZNa7KVUzbNFt5REwHprebd05hfA7p8FFZ2ctJd3GbmVkfqHIfBPkf/+g8OTsibq4vJDMz6w+qXOY6GTiddP/C/cDpeZ6ZmW3AquxBHAvsExFrASRdAdwNrNN0hpmZbTiqnKSGV17W6qa+zcw2AlX2IL4C3C1pFukGttHAWbVGZWZmfa7TBCFJwG+AtwAHkBLEmRHxeC/EZmZmfajTBBERIenGiNif6je5mZnZBqDKOYg7JB1QeyRmZtavVDkH8Q7g45KWA8+QDjNFROxdZ2BmZta3qiSIY2qPwszM+p0OE4SkzYGPA68D7gMuy+0rmZnZRqCzPYgrgBeBX5P2IvYi3VFtG4ERE6s02Ftu+eSxPRiJmfWVzhLEXhHxJgBJlwF39U5IZmbWH3R2FdOLbSM+tGRmtvHpbA/izYX+HgRskafbrmLatvbozMysz3SYICJiQG8GYmZm/UvVxvrMzGwj4wRhZmalnCDMzKyUE4SZmZVygjAzs1JOEGZmVsoJwszMSjlBmJlZKScIMzMr5QRhZmalnCDMzKyUE4SZmZVygjAzs1K1JghJYyQtkbRU0sSS5ZtJujovv1PSiDx/hKTnJC3Iw3frjNPMzNbVWX8Q3SJpAHAx8E6gFZgjaVpE3F9Y7WTgyYh4naQJwAXAiXnZgxGxT13xmZlZ5+rcgzgQWBoRyyLiBWAqML7dOuNJfV8DXAccIUk1xmRmZhXVmSB2Bh4pTLfmeaXr5G5N/wYMyctGSrpb0i8lvb3sCSSdImmupLkrV67s2ejNzDZydSaIsj2BqLjOY8AuEbEv8DngJ5LW6eI0Ii6NiJaIaBk6dGi3AzYzs5fVmSBageGF6WHAio7WkbQJMAhYFRHPR8RfACJiHvAgsHuNsZqZWTu1naQG5gCjJI0EHgUmAB9ot8404CTgd8BxwMyICElDSYlijaRdgVHAshpjNavdiIm3NF12+eSxPRiJWTW1JYiIWC3pNGAGMAC4PCIWSZoEzI2IacBlwI8kLQVWkZIIwGhgkqTVwBrg4xGxqq5YzcxsXXXuQRAR04Hp7eadUxj/B3B8SbnrgevrjM3MzDrnO6nNzKyUE4SZmZVygjAzs1JOEGZmVsoJwszMSjlBmJlZKScIMzMr5QRhZmalnCDMzKyUE4SZmZVygjAzs1JOEGZmVqrWxvqsb7hZaTPrCd6DMDOzUk4QZmZWygnCzMxKOUGYmVkpn6Q2W4/4AgTrTd6DMDOzUk4QZmZWygnCzMxKOUGYmVkpJwgzMyvlBGFmZqWcIMzMrJTvgzDbyPheCqvKexBmZlbKexD9jP/dmVl/4T0IMzMr5T0IM2uY93Q3DrXuQUgaI2mJpKWSJpYs30zS1Xn5nZJGFJadlecvkXR0nXGamdm6atuDkDQAuBh4J9AKzJE0LSLuL6x2MvBkRLxO0gTgAuBESXsBE4A3AK8BfiFp94hYU1e8Zta7vBfS/9V5iOlAYGlELAOQNBUYDxQTxHjg3Dx+HfBtScrzp0bE88BDkpbm+n5XY7zd5g+8We/qie+cv7cdU0TUU7F0HDAmIj6apz8MHBQRpxXWWZjXac3TDwIHkZLGHRHx4zz/MuBnEXFdu+c4BTglT+4BLKllY2AH4M+uw3W4DtexHtfRkddGxNCyBXXuQahkXvts1NE6VcoSEZcClzYeWmMkzY2IFtfhOlyH61hf62hGnSepW4HhhelhwIqO1pG0CTAIWFWxrJmZ1ajOBDEHGCVppKSBpJPO09qtMw04KY8fB8yMdMxrGjAhX+U0EhgF3FVjrGZm1k5th5giYrWk04AZwADg8ohYJGkSMDcipgGXAT/KJ6FXkZIIeb1rSCe0VwOf7OMrmHriMJbrcB2uw3X0ZR0Nq+0ktZmZrd/c1IaZmZVygjAzs1JOEJ2QdLmkJ/L9Gs3WMVzSLEmLJS2SdHoTdWwu6S5J9+Q6zutGPAMk3S3p5ibLL5d0n6QFkuY2Wcd2kq6T9Pv8ury1wfJ75OdvG/4u6TNNxPHZ/HoulHSVpM2bqOP0XH5R1RjKPleStpd0m6QH8uPgJuo4PsexVlKXl0R2UMeF+X25V9JPJW3XRB3n5/ILJN0q6TWN1lFYdoakkLRDE3GcK+nRwufk2GbikPQppSZ/Fkn6ahNxXF2IYbmkBU3UsY+kO9q+d5IO7KyOHhMRHjoYgNHAfsDCbtSxE7BfHt8G+AOwV4N1CNg6j28K3Am8pcl4Pgf8BLi5yfLLgR26+bpeAXw0jw8EtutGXQOAx0k3+zRSbmfgIWCLPH0N8JEG63gjsBDYknTBxy+AUc18roCvAhPz+ETggibqeD3phtHZQEuTcRwFbJLHL2gyjm0L458GvttoHXn+cNJFLn/s6jPXQRznAmc08H6W1fGO/L5ulqd3bGZbCsu/BpzTRBy3Asfk8WOB2Y18VpsdvAfRiYj4Fenqqu7U8VhEzM/jTwGLST9OjdQREfF0ntw0Dw1fXSBpGDAW+H6jZXuKpG1JX4DLACLihYj4azeqPAJ4MCL+2ETZTYAt8j04W9L4vTavJ93x/2xErAZ+Cbynq0IdfK7GkxIn+fHdjdYREYsjonJrAh3UcWveFoA7SPcgNVrH3wuTW9HFZ7WT79k3gP/XVfku6qisgzpOBSZHavaHiHii2TgkCTgBuKqJOgLYNo8PopfuC3OC6EVKrdXuS9oDaLTsgLxr+gRwW0Q0XAfwTdIXbm0TZdsEcKukeUpNnTRqV2Al8IN8qOv7krbqRjwT6OILVyYiHgX+G3gYeAz4W0Tc2mA1C4HRkoZI2pL0z254F2U68k8R8ViO7TFgxybr6Un/BvysmYKS/kvSI8AHgXOaKD8OeDQi7mnm+QtOy4e7Lu/qsF0HdgfertTa9C8lHdCNWN4O/CkiHmii7GeAC/Nr+t/AWd2IozIniF4iaWvgeuAz7f5hVRIRayJiH9I/ugMlvbHB538X8EREzGv0uds5OCL2A44BPilpdIPlNyHtPl8SEfsCz5AOqTRM6QbMccC1TZQdTPrXPpLUYvBWkj7USB0RsZh0GOY24OfAPaT7dtZ7ks4mbcuVzZSPiLMjYnguf1pX67d77i2Bs2kisbRzCbAbsA/pT8DXmqhjE2Aw8Bbg88A1eU+gGe+niT8z2anAZ/Nr+lnyHnjdnCB6gaRNScnhyoi4oTt15cMxs4ExDRY9GBgnaTkwFThc0o+beP4V+fEJ4KekVnYb0Qq0FvaAriMljGYcA8yPiD81UfZI4KGIWBkRLwI3AG9rtJKIuCwi9ouI0aTDAs38OwT4k6SdAPJjp4cy6iTpJOBdwAcjH/Tuhp8A72uwzG6kxH1P/rwOA+ZLenUjlUTEn/Ifq7XA92j8swrp83pDPsx7F2nvu9MT5mXyYcz3Alc3EQOkFifafjuupbltaZgTRM3yv43LgMUR8fUm6xjadjWJpC1IP26/b6SOiDgrIoZFxAjSYZmZEdHQP2ZJW0napm2cdEKzoSu8IuJx4BFJe+RZR/DKJuAb0Z1/ZA8Db5G0ZX6PjiCdH2qIpB3z4y6kH4Bm4yk2O3MScFOT9XSLpDHAmcC4iHi2yTpGFSbH0fhn9b6I2DEiRuTPayvpQo/HG4xjp8Lke2jws5rdCBye69uddFFFM62qHgn8PnLL1U1YARyaxw+n+T8ijemNM+Hr60D6sj8GvEj6kJ7cRB2HkI7b3wssyMOxDdaxN3B3rmMhXVwFUaG+w2jiKibS+YN78rAIOLvJ598HmJu350ZgcBN1bAn8BRjUjdfhPNKP10LgR+QrVRqs49ekBHcPcESznytgCHA76Yt/O7B9E3W8J48/D/wJmNFEHUuBRwqf1a6uQCqr4/r8mt4L/B+wc6N1tFu+nK6vYiqL40fAfTmOacBOTdQxEPhx3p75wOHNbAvwQ+Dj3fh8HALMy5+zO4H9m/3cNzK4qQ0zMyvlQ0xmZlbKCcLMzEo5QZiZWSknCDMzK+UEYWZmpZwgrFO5Jc2vFabPkHRuD9X9Q0nH9URdXTzP8Uqtxs6q+7nqlFv07LRF0g7KXZWbm/ispEmSjmyw/PKy1lQl/UejsTSj2e227nOCsK48D7y3q+aWe5ukAQ2sfjLwiYh4R13x9JJ9SO09VZbvPn5bROwdEd+IiHMi4hc9FE/DCaLB961Nw9ttPcMJwrqymtQf7mfbL2i/ByDp6fx4WG7Y7BpJf5A0WdIHlfq0uE/SboVqjpT067zeu3L5AUr9EszJ/3z/vVDvLEk/Id0A1T6e9+f6F0q6IM87h3ST0XclXdhu/VdJ+o5SO/83S5retj3Ff82SWiTNzuNb5Ybf5uTGBsfn+W/I27cgxzwqr3uLUj8eCyWdmNfdP78+8yTNKDSx8WlJ9+fyU9vFOhCYBJyYn+NEpT4kbszr3yFp75L371Zgx1zm7cX3LG/jeZLm59dtzzx/iFI/DndLmkJqbr79az2Z1BLuAklX5nk35m1apEJDjpKeznsudwJvlXSsUp8Tv5F0kXLfJGWvbdl2l2yj1aU37sbzsP4OwNOkZoaXk5oZPgM4Ny/7IXBccd38eBjwV1JfGJsBjwLn5WWnA98slP856Y/KKNJdo5sDpwBfyOtsRrrremSu9xlgZEmcryE1nzGU1MDaTODdedlsSvpHAI4DpufnfzXwZNv2ULh7F2ght78PfBn4UB7fjtS/x1bAt0htF0G6+3YLUhtE3ys83yBSU+2/BYbmeScCl+fxFbzc78A6fWQAHwG+XZj+FvClPH44sKCkzAhe2a/AS+9Z3sZP5fFPAN/P4xeR79YnNQ8flNzJ3PZ+F6a3z49bkO48HpKnAzghj29OulN7ZJ6+inxXfyev7Su220PvDd6DsC5Fan32f0mdv1Q1J1JfGM8DD5L+yUL65z+isN41EbE2UhPIy4A9SW08/YtS8+Z3kpqhaGvf566IeKjk+Q4g/YivjNSfwZWkfic6cwhwbX7+x4Eq5yiOAibm2GaTfvB2AX4H/IekM0mdFz2Xt/VISRdIentE/I3Uoc8bgdtyHV/g5T4X7gWuVGpVtkqrsIeQmpMgImYCQyQNqlCuqK0BuHm8/L6MJjUvQUTcQkqcVXxa0j2kfiSG8/J7tobU/Aak93dZ4T0stl3V0WtrfWSTvg7A1hvfJLVF84PCvNXkw5SSRPrn3Ob5wvjawvRaXvm5a9/WS5AOaXwqImYUF0g6jLQHUaaZJpg7K/PStpF+qIpl3hfrdsyzOB9CGQvMkPTRiJgpaX/S8fOvSLqV1ALuoogo62Z1LOnHeRzwRUlviJc776kaf6Nt57S9L2vo/H3pVH5vjgTeGhHP5kNyba/bPyJiTduqnVVDyWsr6aBGYrGe4z0IqyQiVpG65Ty5MHs5sH8eH086fNKo4/O5gN1IjQEuIXUzeapSM+lI2l1ddyp0J3CopB2UToS+n9TDW2d+A7wvP/8/kQ5htVnOy9tWbK56BvCpnBCRtG9+3JX0z/giUsNweyv1xfxsRPyY1MnLfnn7hir3wy1p03z+4lXA8IiYRerUaTtg63bxPkXqtrbNr0gd8rT9QP85muhrpESx3mNI/SGUebHtPSIdPnsyJ4c9Sf0nlPk9sKtS51mQDrG1KX1tWXe7rZc4QVgjvsYr28L/HulH+S7gIDr+d9+ZJaQf8p+RWrv8B6lL1PtJfQAsBKbQxd5upF7YziIdJrqH1E9EV01mX08679H2HHcCf8vLzgP+R9KvSf+u25xPSoT35tjOz/NPBBbmwyN7kg7JvQm4K887G/jPiHiBdO7jgnw4ZgGpH4oBwI8l3UdqufcbsW5XrLOAvQona88FWiTdC0zm5ebCu+s8Uk9580mHfR7uYL1LSa/DlaRzSZvkWM4nHWZaRz709gng55J+Q2p1tu017+i1bb/d1kvcmqtt1CRtHRFPSxoC3EXqMa+hfgesMYXXXMDFwAMR8Y2+jsvW5XMQtrG7WakzpoHA+U4OveJjSr3WDSTtLU3p43isA96DMDOzUj4HYWZmpZwgzMyslBOEmZmVcoIwM7NSThBmZlbq/wOZcI0AKz1uaQAAAABJRU5ErkJggg==\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], + "source": [ + "show(play_jottos(random_guesser, targets=5*wordlist))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The average is 7.38 guesses, and most of the time it will take from 6 to 9 guesses. Can we do better than that?\n", + "\n", + "# Strategies that Partition Targets\n", + "\n", + "It seems that a key idea in guessing is to reduce the number of possible targets. We can think of each guess as **partitioning** the possible targets into different **branches** of a tree, each branch corresponding to a different reply number. Once you get the reply, you can discard all the other branches. \n", + "\n", + "I will define `reply_branches(guess, targets)[i]` to give all the target words that yield a reply of `i`, for any `i` from 0 to 5. Similarly, `reply_counts(guess, targets)[i]` gives the number of targets that yield a reply of `i`. The reply counts for every guess (with the full wordlist as targets) are cached in `reply_counts_cache`." + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": {}, + "outputs": [], + "source": [ + "def reply_branches(guess, targets) -> List[str]:\n", + " \"\"\"A list of [[targets_with_reply_i], ...] for i from 0 to 5.\"\"\"\n", + " result = [[] for _ in range(6)]\n", + " for target in targets:\n", + " result[reply_for(target, guess)].append(target)\n", + " return result\n", + "\n", + "def reply_counts(guess, targets) -> List[int]: \n", + " \"A list of [number_of_targets_with_reply_i, ...] for i from 0 to 5.\"\n", + " counts = [0] * 6\n", + " for target in targets:\n", + " counts[reply_for(guess, target)] += 1\n", + " return counts\n", + "\n", + "reply_counts_cache = {guess: reply_counts(guess, wordlist) for guess in wordlist}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "I don't want to show `reply_branches` with all 2,845 words, so here it is for the first 20 words of the word list, with the guess `'after'`:" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "[['would', 'cloud', 'sound'],\n", + " ['sword', 'think', 'fondu', 'might'],\n", + " ['about', 'girth', 'place', 'ethos', 'nuder'],\n", + " ['their', 'throe', 'write', 'rifts', 'resay'],\n", + " ['water', 'grate'],\n", + " ['after']]" + ] + }, + "execution_count": 8, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "reply_branches('after', wordlist[:20])" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We see that `'after'` does a pretty good job of partitioning the words into roughly-equal branches." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Below are the reply counts over the full wordlist for two possible guesses, `ouija` and `coder`:" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "[175, 1848, 755, 65, 1, 1]" + ] + }, + "execution_count": 9, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "reply_counts_cache['ouija']" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "[433, 1030, 1014, 327, 40, 1]" + ] + }, + "execution_count": 10, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "reply_counts_cache['coder']" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We see that for the guess `'ouija'`, there are 1848 possible targets in the `1` branch (the second entry in the counts, because the first entry is for `0`). There are far fewer possible targets in the other branches, so that means that for the majority of targets, you will be left with 1848 possible targets.\n", + "\n", + "For the guess `coder`, the reply will be `1` or `2` for (1030 + 1014) / 2845 = 72% of the targets, and in any case, there will never be more than 1030 possible targets to deal with. That suggests that `coder` is a better first guess, and it suggests a general strategy: **choose the guess that best partitions the possible targets into small branches.**\n", + "\n", + "What's a proper metric for measuring how \"small\" the branches are? \n", + "\n", + "- The simplest metric is to **minimize the maximum number in the reply counts**. That is, we should deduce that `coder`, with its `max` reply count of 1030, is a better guess than `ouija`, with its `max` of 1848. \n", + "\n", + "- A slightly more sophisticated metric is to **minimize the expected number in the reply counts**: the expectation is the sum, over all branches, of the probability of ending up in the branch times the size of the branch.\n", + "\n", + "- Information theory provides a suggestion to **maximize the entropy in the reply counts**. Entropy is similar to expectation, except that it weights each branch size by its base 2 logarithm (whereas expectation weights it by its actual size). Note that we want to minimize *negative* entropy." + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "metadata": {}, + "outputs": [], + "source": [ + "def expectation(counts) -> float:\n", + " \"Given a list of counts, give the expected count.\"\n", + " def P(x, scale=1/sum(counts)): return scale * x\n", + " return sum(P(x) * x for x in counts)\n", + "\n", + "def entropy(counts) -> float: \n", + " \"\"\"Information theoretic entropy of a list of counts.\"\"\"\n", + " def P(x, scale=1/sum(counts)): return scale * x\n", + " return - sum(P(x) * log(P(x), 2) for x in counts if x)\n", + "\n", + "def negative_entropy(counts) -> float: return - entropy(counts)\n", + "\n", + "def top(words, metric, n=15) -> str: \n", + " \"\"\"The top n words according to metric(reply_counts_cache[w]).\"\"\"\n", + " return ' '.join(sorted(words, key=lambda w: metric(reply_counts_cache[w]))[:n])" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Here are the top 15 words according to each of the three metrics:" + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "'wader cadre armed diner coder padre rayed raved delta drone garde eland heard tired debar'" + ] + }, + "execution_count": 12, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# minimize the maximum number in the reply counts\n", + "top(wordlist, metric=max)" + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "'raved debar roved orbed wader armed fader dater alder cadre garde padre deign gored laved'" + ] + }, + "execution_count": 13, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# minimize the expected number in the reply counts\n", + "top(wordlist, metric=expectation)" + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "'debar alder raved dater cadre armed garde wader lased padre fader dears drone diner rayed'" + ] + }, + "execution_count": 14, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# maximize the entropy (by minimizing negative entropy) in the reply counts\n", + "top(wordlist, metric=negative_entropy)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The three metrics yield a lot of overlap; `wader`, `raved`, and `debar` appear near the top in all three. Which metric is best? We have the tools to answer that: we could use `play_jottos` to get scores for each metric. But that would take a long time, because many computations would be repeated for each target word. We were able to create the `reply_counts_cache` for the words in the complete wordlist, but we really could use a similar cache for all the cases where we have eliminated some of the possible targets." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Guesser Trees \n", + "\n", + "My approach to the cache problem will be to build a **guesser tree**: a tree that partitions target words into branches all the way to the end, where each partition holds a single word. Think of this as a universal strategy that we can precompute once, and then apply to every possible target word with no need for additional complex computation—just branch-following in the tree. I define a tree as either:\n", + "- A **leaf**, which is a string, such as `'oiuja'`, indicating that this is the sole remaining possible target. Every word in the word list should appear as a leaf in exactly one place in the guesser tree.\n", + "- An **interior node**, such as `Node(guess='wader', branches={1: 'oiuja', 5: 'wader'})`, consisting of a guess and branches which are `{reply: tree}` pairs. The node shown here means that we should guess `'wader'` first, and if the reply is `1`, then we're left with the single target `'oiuja'` to guess next, and if the reply is `5`, then we finished the game; `'wader'` was the target.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": {}, + "outputs": [], + "source": [ + "Leaf = str\n", + "Node = namedtuple('Node', 'guess, branches')\n", + "Tree = (Node, Leaf) # A Tree is a Node or a Leaf" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The function `make_tree` builds a tree that, at every node, chooses the guess that minimizes the value of a metric applied to the reply counts for the guess and the targets. It takes two arguments:\n", + "- `metric`, a function that takes a reply count list as input and returns a number;\n", + "- `targets`, the list of remaining possible target words. " + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "metadata": {}, + "outputs": [], + "source": [ + "def make_tree(metric, targets=wordlist) -> Tree:\n", + " \"\"\"Make a tree that guesses to minimize or maximize metric(reply_counts(guess, targets)).\"\"\"\n", + " if len(targets) == 1:\n", + " return targets[0]\n", + " else:\n", + " guess = min(targets, key=lambda guess: metric(reply_counts(guess, targets))) \n", + " branches = reply_branches(guess, targets)\n", + " return Node(guess, {i: make_tree(metric, branches[i]) \n", + " for i in range(6) if branches[i]})" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Consider the case where there are five possible targets: `purge`, `bites`, `sulky`, `patsy`, and `hayed`.\n", + "Here is a tree to cover those words by, at each node in the tree, making the guess that **minimizes** the **maximum** number in the reply counts list:" + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "Node(guess='bites', branches={1: Node(guess='purge', branches={1: Node(guess='sulky', branches={1: 'hayed', 5: 'sulky'}), 5: 'purge'}), 2: 'patsy', 5: 'bites'})" + ] + }, + "execution_count": 17, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "words5 = 'purge bites sulky patsy hayed'.split()\n", + "tree5 = make_tree(max, words5)\n", + "tree5" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "That tree is easier to understand if we reformat it like this:\n", + "\n", + " Node(guess='bites', \n", + " branches={1: Node(guess='purge', \n", + " branches={1: Node(guess='sulky', \n", + " branches={1: 'hayed', \n", + " 5: 'sulky'}), \n", + " 5: 'purge'}), \n", + " 2: 'patsy', \n", + " 5: 'bites'})\n", + " \n", + "This says that the first guess is `'bites'`, and if the reply is `1` there is a complex subtree to consider (starting with the guess `'purge'`), but if the reply is `2` the target must be `'patsy'` and of course if the reply is `5` then `'bites'` was the target." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Turning a Tree into a Callable Guesser\n", + "\n", + "Now that we've made a tree, we need to use it as a component of a guesser that `play_jotto` can call upon. A `TreeGuesser` is an object that stores a tree in the `self.root` and `self.tree` attributes, and while the game is being played, it follows branches in the tree, choosing the guess indicated at each node. This is done within the `__call__` method, because `play_jotto` expects a callable. Note that whenever the reply is `None`, that indicates the first turn of a new game, in which case we reset `self.tree` to be the root tree. For subsequent turns, we follow the branch indicated by the reply. If the resulting `self.tree` is a `Node` object, then the `.guess` attribute is the guess; if it is a leaf, then the leaf itself is the guess." + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "metadata": {}, + "outputs": [], + "source": [ + "class TreeGuesser:\n", + " \"\"\"Given a guesser tree, use it to create a callable that can play_jotto.\"\"\"\n", + " def __init__(self, tree): self.root = self.tree = tree\n", + " \n", + " def __call__(self, reply, _) -> str:\n", + " self.tree = self.root if reply is None else self.tree.branches[reply]\n", + " return guess(self.tree)\n", + " \n", + "def guess(tree) -> str: return tree.guess if isinstance(tree, Node) else tree " + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Here's how we play a game with the five-word list:" + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Guess 1: bites, Reply: 1, Possible targets: 3\n", + "Guess 2: purge, Reply: 1, Possible targets: 2\n", + "Guess 3: sulky, Reply: 5, Possible targets: 1\n" + ] + }, + { + "data": { + "text/plain": [ + "3" + ] + }, + "execution_count": 19, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "play_jotto(TreeGuesser(tree5), target='sulky', wordlist=words5, verbose=True)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "And here we build a tree over the whole word list and use it to play a game:" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Guess 1: wader, Reply: 0, Possible targets: 466\n", + "Guess 2: tings, Reply: 1, Possible targets: 142\n", + "Guess 3: hypos, Reply: 0, Possible targets: 8\n", + "Guess 4: climb, Reply: 5, Possible targets: 1\n" + ] + }, + { + "data": { + "text/plain": [ + "4" + ] + }, + "execution_count": 20, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "tree = make_tree(max)\n", + "play_jotto(TreeGuesser(tree), verbose=True)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Guessing a Nontarget Word\n", + "\n", + "So far, we have always guessed one of the possible targets. That seems reasonable; why waste a guess on a word that could not possibly be the target? But it turns out that in some cases it *is* a good strategy to guess such a word.\n", + "\n", + "I will redefine `make_tree` so that it is passed both the list of possible target words and the complete word list. It also has a flag, `nontargets`. When this flag is true, any word in the wordlist can be considered as a guess; when false, only target words are considered, as before. (One hack: if there are only three targets or fewer remaining, then it makes no sense to consider a nontarget guess, because it can't possibly decrease the mean score.)" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [], + "source": [ + "def make_tree(metric, targets=wordlist, wordlist=wordlist, nontargets=False) -> Tree:\n", + " \"\"\"Make a tree that guesses to minimize or maximize metric(reply_counts(guess, targets)).\n", + " Allow nontarget guesses if `nontargets` is true.\"\"\"\n", + " if len(targets) == 1:\n", + " return targets[0]\n", + " else:\n", + " candidates = wordlist if nontargets and len(targets) > 3 else targets\n", + " guess = min(candidates, key=lambda guess: metric(reply_counts(guess, targets))) \n", + " branches = reply_branches(guess, targets)\n", + " return Node(guess, {i: make_tree(metric, branches[i], wordlist, nontargets) \n", + " for i in range(6) if branches[i]})" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Here we see that by default, `make_tree` behaves just as it did before:" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "Node(guess='bites', branches={1: Node(guess='purge', branches={1: Node(guess='sulky', branches={1: 'hayed', 5: 'sulky'}), 5: 'purge'}), 2: 'patsy', 5: 'bites'})" + ] + }, + "execution_count": 22, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "make_tree(max, words5)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "But when allowed to guess a nontarget word, it comes up with a different tree:" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "Node(guess='dashy', branches={0: 'purge', 1: 'bites', 2: 'sulky', 3: 'patsy', 4: 'hayed'})" + ] + }, + "execution_count": 23, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "tree5b = make_tree(max, words5, nontargets=True)\n", + "tree5b" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "This tree guesses a nontarget word, `dashy` with the first guess. There is no chance that this is the target, but it sets us up so that we will always be able to get the target on the second guess (so the average score, and the maximum score, is 2):" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "[2, 2, 2, 2, 2]" + ] + }, + "execution_count": 24, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "play_jottos(TreeGuesser(tree5b), words5)" + ] + }, + { + "cell_type": "code", + "execution_count": 25, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "(2, 2)" + ] + }, + "execution_count": 25, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "mean(_), max(_)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "In contrast, the guesser that guesses only target words has an average score of 2.4, and can take up to four guesses:" + ] + }, + { + "cell_type": "code", + "execution_count": 26, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "[2, 1, 3, 2, 4]" + ] + }, + "execution_count": 26, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "play_jottos(TreeGuesser(tree5), words5)" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "(2.4, 4)" + ] + }, + "execution_count": 27, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "mean(_), max(_)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "So the tree with a nontarget guess is better both in the mean (2.0 versus 2.4) and in the maximum (2 versus 4).\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Comparing Metrics" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Now we will compare six guessers derived from `make_tree`:\n", + "- Minimizing either `max`, `expectation`, or `negative_entropy`.\n", + "- For each of the above, guessing either targets only or targets plus nontargets.\n", + "\n", + "The function `show_metric` makes the appropriate tree and calls `play_jottos` and `show` to display results:" + ] + }, + { + "cell_type": "code", + "execution_count": 28, + "metadata": {}, + "outputs": [], + "source": [ + "def show_metric(metric, nontargets=False):\n", + " \"\"\"Show statistics and histogram for a guesser that minimizes `metric` over reply counts.\"\"\"\n", + " guesser = TreeGuesser(make_tree(metric, nontargets=nontargets))\n", + " show(play_jottos(guesser))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Comparing Metrics for Target Guesses Only" + ] + }, + { + "cell_type": "code", + "execution_count": 29, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "mean = 7.15 ± 1.81, max = 18\n", + "CPU times: user 12.7 s, sys: 28.7 ms, total: 12.8 s\n", + "Wall time: 12.8 s\n" + ] + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEGCAYAAAB/+QKOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAf9klEQVR4nO3deZgdVZnH8e/PQNh3mhEhMQEDCIIsDaggIkQN4BBUkOAy0UEzoijKg0MYFCGMyCLuiKBEUYHIJmQwGhCIyqNAEghLgpEQIjQBiQYFBIGEd/44p6G4qe6ue7uru5P8Ps9zn67tnHrv0ve9darqHEUEZmZmjV410AGYmdng5ARhZmalnCDMzKyUE4SZmZVygjAzs1JrDHQAfWXzzTePESNGDHQYZmYrldmzZ/81ItrK1q0yCWLEiBHMmjVroMMwM1upSPpzV+vcxGRmZqWcIMzMrJQThJmZlXKCMDOzUk4QZmZWygnCzMxKOUGYmVkpJwgzMyvlBGFmZqVWmTupbdU0YuIvWi676MxD+jASs9WPjyDMzKyUE4SZmZVygjAzs1JOEGZmVsoJwszMSjlBmJlZKScIMzMr5QRhZmalnCDMzKyUE4SZmZVygjAzs1JOEGZmVsoJwszMStWaICSNkTRf0gJJE0vWf0LSPZLmSLpF0o6FdSflcvMlvavOOM3MbEW1JQhJQ4DzgIOAHYGjigkguzQido6IXYGzga/lsjsC44CdgDHAd3N9ZmbWT+o8gtgLWBARCyPieWAKMLa4QUQ8WZhdD4g8PRaYEhHPRcSDwIJcn5mZ9ZM6BwzaCni4MN8B7N24kaRPAccDQ4EDCmVvbSi7VUnZCcAEgOHDh/dJ0GZmltR5BKGSZbHCgojzImJb4ETgC02WvTAi2iOiva2trVfBmpnZK9WZIDqAYYX5rYHF3Ww/BTisxbJmZtbH6kwQM4FRkkZKGko66Ty1uIGkUYXZQ4D78/RUYJyktSSNBEYBt9cYq5mZNajtHERELJN0LDAdGAJMjoi5kiYBsyJiKnCspNHAC8ATwPhcdq6ky4F5wDLgUxGxvK5YzcxsRXWepCYipgHTGpadUpg+rpuyXwa+XF90ZmbWHd9JbWZmpZwgzMyslBOEmZmVcoIwM7NSThBmZlbKCcLMzEo5QZiZWSknCDMzK+UEYWZmpZwgzMyslBOEmZmVcoIwM7NSThBmZlbKCcLMzEo5QZiZWSknCDMzK+UEYWZmpZwgzMyslBOEmZmVcoIwM7NSThBmZlbKCcLMzErVmiAkjZE0X9ICSRNL1h8vaZ6kuyXdKOm1hXXLJc3Jj6l1xmlmZitao66KJQ0BzgPeAXQAMyVNjYh5hc3uBNoj4hlJxwBnA0fmdc9GxK51xWdmZt2r8whiL2BBRCyMiOeBKcDY4gYRcXNEPJNnbwW2rjEeMzNrQp0JYivg4cJ8R17WlaOBXxbm15Y0S9Ktkg4rKyBpQt5m1pIlS3ofsZmZvaTHBCHpCEkb5OkvSLpa0u4V6lbJsuhiHx8C2oFzCouHR0Q78AHgG5K2XaGyiAsjoj0i2tva2iqEZGZmVVU5gvhiRDwlaV/gXcDFwPkVynUAwwrzWwOLGzeSNBo4GTg0Ip7rXB4Ri/PfhcAMYLcK+zQzsz5SJUEsz38PAc6PiGuBoRXKzQRGSRopaSgwDnjF1UiSdgMuICWHxwvLN5G0Vp7eHNgHKJ7cNjOzmlW5iukRSRcAo4Gz8hd3j4klIpZJOhaYDgwBJkfEXEmTgFkRMZXUpLQ+cIUkgIci4lDg9cAFkl7M+zqz4eonMzOrWZUE8X5gDPDViPi7pC2Bz1epPCKmAdMalp1SmB7dRbnfAztX2YeZmdWjShPTBRFxdUTcDxARjwIfrjcsMzMbaFUSxE7FmXwD3B71hGNmZoNFlwlC0kmSngJ2kfSkpKfy/OPAtf0WoZmZDYguE0REfCUiNgDOiYgNI2KD/NgsIk7qxxjNzGwAVGliOlnShyR9EUDSMEl71RyXmZkNsCoJ4jzgzaQ7mgGezsvMzGwVVuUy170jYndJdwJExBP5xjczM1uFVTmCeCFfuRQAktqAF2uNyszMBlyVBPEt4OfAFpK+DNwCnFFrVGZmNuB6bGKKiEskzQYOJPXQelhE3Fd7ZGZmNqB6TBCSNiXd+3BZYdmaEfFCnYGZmdnAqtLEdAewBPgTcH+eflDSHZJ8R7WZ2SqqSoL4FXBwRGweEZsBBwGXA58EvltncGZmNnCqJIj2iJjeORMR1wP7RcStwFq1RWZmZgOqyn0QSyWdCEzJ80cCT+RLX325q3VpxMRftFx20ZmH9GEkZtaKKkcQHyANF3pNfgzLy4aQxoowM7NVULdHEPko4cSI+HQXmyzo+5DMzGww6PYIIiKW47EfzMxWS1XOQdwpaSpwBfDPzoURcXVtUZmZ2YCrkiA2Bf4GHFBYFoAThJnZKqxKVxsf7Y9AzMxscKnS1cbawNGksanX7lweEf9ZY1xmZjbAqlzm+hPg1cC7gN+QLnl9qs6gzMxs4FVJEK+LiC8C/4yIi4FDgJ2rVC5pjKT5khZImliy/nhJ8yTdLelGSa8trBsv6f78GF/1CZmZWd+oNGBQ/vt3SW8ANgJG9FQo30NxHqnvph2BoyTt2LDZnaSuPHYBrgTOzmU3Bb4E7A3sBXxJ0iYVYjUzsz5SJUFcmL+cvwBMBeYBZ1UotxewICIWRsTzpK46xhY3iIibI+KZPHsrqfkKUnPWDRGxNCKeAG4AxlTYp5mZ9ZEql7nemL+kfwtsAyBpZIVyWwEPF+Y7SEcEXTka+GU3ZbdqLCBpAjABYPjw4RVCMjOzqqocQVxVsuzKCuVUsixKN5Q+BLQD5zRTNiIujIj2iGhva2urEJKZmVXV5RGEpB1Il7ZuJOm9hVUbUrjctRsdpI79Om0NLC7Zz2jgZOBtEfFcoez+DWVnVNinmZn1ke6amLYH3g1sDPx7YflTwMcr1D0TGJWbox4BxpF6gX2JpN2AC4AxEfF4YdV04IzCiel3AidV2KeZmfWRLhNERFwLXCvpzRHxh2Yrjohlko4lfdkPASZHxFxJk4BZETGV1KS0PnCFJICHIuLQiFgq6XRSkgGYFBFLm43BzMxaV6WrjaaTQ6HsNGBaw7JTCtOjuyk7GZjc6r7NzKx3qpykNjOz1VCXCULScfnvPv0XjpmZDRbdNTF9FPgm8G1g9/4Jx6zveWxss9Z0lyDuk7QIaJN0d2G5gMjdY5iZ2Sqqu6uYjpL0atJVSIf2X0hmZjYYdHsVU0Q8BrxR0lBgu7x4fkS80E0xMzNbBVQZMOhtwI+BRaTmpWGSxkfEb2uOzczMBlCVzvq+BrwzIuYDSNoOuAzYo87AzMxsYFW5D2LNzuQAEBF/AtasLyQzMxsMqhxBzJJ0EWnoUYAPArPrC8nMzAaDKgniGOBTwGdI5yB+C3y3zqDMzGzgVemL6TnSeYiv1R+OmZkNFu6LyczMSjlBmJlZKScIMzMrVeVGue2AzwOvLW4fEQfUGJeZmQ2wKlcxXQF8D/g+sLzecMzMbLCokiCWRcT5tUdiZmaDSpVzEP8n6ZOStpS0aeej9sjMzGxAVTmCGJ//fr6wLIBt+j4cMzMbLKrcKDeyPwIxM7PBpcpVTGuSutvYLy+aAVzgMSHMzFZtVZqYzif13trZ/9KH87KP1RWUmZkNvConqfeMiPERcVN+fBTYs0rlksZImi9pgaSJJev3k3SHpGWSDm9Yt1zSnPyYWu3pmJlZX6lyBLFc0rYR8QCApG2ocD+EpCHAecA7gA5gpqSpETGvsNlDwEeAE0qqeDYidq0Qn5mZ1aBKgvg8cLOkhaTuvl8LfLRCub2ABRGxEEDSFGAs8FKCiIhFed2LzYVtZmZ1q3IV042SRgHbkxLEH3MX4D3ZCni4MN8B7N1EbGtLmgUsA86MiGsaN5A0AZgAMHz48CaqNjOznnSZICQdEBE3SXpvw6ptJRERV/dQt0qWRROxDY+IxblJ6yZJ93Q2c71UWcSFwIUA7e3tzdRtZmY96O4I4m3ATcC/l6wLoKcE0QEMK8xvDSyuGlhELM5/F0qaAewGPNBtITMz6zNdJoiI+FKenBQRDxbXSapy89xMYFTe9hFgHPCBKkFJ2gR4JiKek7Q5sA9wdpWyZmbWN6pc5npVybIreyoUEcuAY4HpwH3A5RExV9IkSYcCSNpTUgdwBHCBpLm5+OuBWZLuAm4mnYOYt+JezMysLt2dg9gB2AnYqOE8xIbA2lUqj4hpwLSGZacUpmeSmp4ay/0e2LnKPszMrB7dnYPYHng3sDGvPA/xFPDxOoMyM7OB1905iGslXQecGBFn9GNMZmY2CHR7DiIilpPuhDYzs9VMlTupfy/pO8DPgH92LoyIO2qLyszMBlyVBPGW/HdSYVkAB/R9OGZmNlhU6Wrj7f0RiJmZDS493gchaSNJX5M0Kz/OlbRRfwRnZmYDp8qNcpNJl7a+Pz+eBH5YZ1BmZjbwqpyD2DYi3leYP03SnLoCMjOzwaHKEcSzkvbtnJG0D/BsfSGZmdlgUOUI4hjg4nzeQcBSYHytUZmZ2YCrchXTHOCNkjbM80/WHpWZmQ24KlcxbSbpW8AM0tCj35S0We2RmZnZgKpyDmIKsAR4H3B4nv5ZnUGZmdnAq3IOYtOIOL0w/7+SDqsrIDMzGxyqHEHcLGmcpFflx/uBX9QdmJmZDawqCeK/gEuB5/NjCnC8pKck+YS1mdkqqspVTBv0RyBmZja4VDkHQR5Der88OyMirqsvJDMzGwyqXOZ6JnAcMC8/jsvLzMxsFVblCOJgYNeIeBFA0sXAncDEOgMzM7OBVeUkNcDGhWl39W1mthqokiC+Atwp6Uf56GE2cEaVyiWNkTRf0gJJKxxxSNpP0h2Slkk6vGHdeEn354f7fjIz62fdNjFJEnAL8CZgT1JnfSdGxGM9VSxpCHAe8A6gA5gpaWpEzCts9hDwEeCEhrKbAl8C2knDm87OZZ+o+LzMzKyXuk0QERGSromIPYCpTda9F7AgIhYCSJoCjCWd6O6sf1Fe92JD2XcBN0TE0rz+BmAMcFmTMZiZWYuqNDHdKmnPFureCni4MN+Rl/VZWUkTOodCXbJkSQshmplZV6okiLeTksQDku6WdI+kuyuUU8myqBhXpbIRcWFEtEdEe1tbW8WqzcysiiqXuR7UYt0dwLDC/NbA4ibK7t9QdkaLcZiZWQu6TBCS1gY+AbwOuAe4KCKWNVH3TGCUpJHAI8A44AMVy04HzpC0SZ5/J3BSE/s2M7Ne6q6J6WLSVUT3kI4izm2m4pxMjiV92d8HXB4RcyVNyl13IGlPSR3AEcAFkubmskuB00lJZiYwqfOEtZmZ9Y/umph2jIidASRdBNzebOURMQ2Y1rDslML0TFLzUVnZycDkZvdpZmZ9o7sjiBc6J5psWjIzs1VAd0cQbyyM9yBgnTwv0i0SG9YenQ2YERNbHxNq0ZmH9GEkZjZQukwQETGkPwMxM7PBpWpnfWZmtppxgjAzs1JOEGZmVsoJwszMSjlBmJlZKScIMzMr5QRhZmalnCDMzKyUE4SZmZVygjAzs1JOEGZmVsoJwszMSjlBmJlZKScIMzMr5QRhZmalnCDMzKyUE4SZmZXqbshRM8s8BKutjnwEYWZmpZwgzMysVK0JQtIYSfMlLZA0sWT9WpJ+ltffJmlEXj5C0rOS5uTH9+qM08zMVlTbOQhJQ4DzgHcAHcBMSVMjYl5hs6OBJyLidZLGAWcBR+Z1D0TErnXFZ2Zm3avzCGIvYEFELIyI54EpwNiGbcYCF+fpK4EDJanGmMzMrKI6E8RWwMOF+Y68rHSbiFgG/APYLK8bKelOSb+R9NayHUiaIGmWpFlLlizp2+jNzFZzdSaIsiOBqLjNo8DwiNgNOB64VNKGK2wYcWFEtEdEe1tbW68DNjOzl9WZIDqAYYX5rYHFXW0jaQ1gI2BpRDwXEX8DiIjZwAPAdjXGamZmDepMEDOBUZJGShoKjAOmNmwzFRifpw8HboqIkNSWT3IjaRtgFLCwxljNzKxBbVcxRcQySccC04EhwOSImCtpEjArIqYCFwE/kbQAWEpKIgD7AZMkLQOWA5+IiKV1xWpmZiuqtauNiJgGTGtYdkph+l/AESXlrgKuqjM2MzPrnu+kNjOzUk4QZmZWygnCzMxKOUGYmVkpJwgzMyvlAYPM+okHHbKVjY8gzMyslBOEmZmVchPTKshNGWbWF3wEYWZmpZwgzMyslBOEmZmVcoIwM7NSThBmZlbKVzGZrUR8hZr1Jx9BmJlZKScIMzMr5QRhZmalfA7CbDXj8xhWlY8gzMyslBOEmZmVchPTIOPDfzMbLJwgzKxp/iGzeqg1QUgaA3wTGAL8ICLObFi/FvBjYA/gb8CREbEorzsJOBpYDnwmIqbXGauZ9S8nmcGvtgQhaQhwHvAOoAOYKWlqRMwrbHY08EREvE7SOOAs4EhJOwLjgJ2A1wC/lrRdRCyvK14zW/k4ydSrziOIvYAFEbEQQNIUYCxQTBBjgVPz9JXAdyQpL58SEc8BD0pakOv7Q43x9po/rGYrn774vx0sdfQ1RUQ9FUuHA2Mi4mN5/sPA3hFxbGGbe/M2HXn+AWBvUtK4NSJ+mpdfBPwyIq5s2McEYEKe3R6YX8uTgc2Bv7oO1+E6XMdKXEdXXhsRbWUr6jyCUMmyxmzU1TZVyhIRFwIXNh9acyTNioh21+E6XIfrWFnraEWd90F0AMMK81sDi7vaRtIawEbA0oplzcysRnUmiJnAKEkjJQ0lnXSe2rDNVGB8nj4cuClSm9dUYJyktSSNBEYBt9cYq5mZNaitiSkilkk6FphOusx1ckTMlTQJmBURU4GLgJ/kk9BLSUmEvN3lpBPay4BPDfAVTH3RjOU6XIfrcB0DWUfTajtJbWZmKzf3xWRmZqWcIMzMrJQTRDckTZb0eL5fo9U6hkm6WdJ9kuZKOq6FOtaWdLuku3Idp/UiniGS7pR0XYvlF0m6R9IcSbNarGNjSVdK+mN+Xd7cZPnt8/47H09K+mwLcXwuv573SrpM0tot1HFcLj+3agxlnytJm0q6QdL9+e8mLdRxRI7jRUk9XhLZRR3n5Pflbkk/l7RxC3WcnsvPkXS9pNc0W0dh3QmSQtLmLcRxqqRHCp+Tg1uJQ9KnJc3Pr+3ZLcTxs0IMiyTNaaGOXSXd2vl/J2mv7uroMxHhRxcPYD9gd+DeXtSxJbB7nt4A+BOwY5N1CFg/T68J3Aa8qcV4jgcuBa5rsfwiYPNevq4XAx/L00OBjXtR1xDgMdLNPs2U2wp4EFgnz18OfKTJOt4A3AusS7rg49fAqFY+V8DZwMQ8PRE4q4U6Xk+6YXQG0N5iHO8E1sjTZ7UYx4aF6c8A32u2jrx8GOkilz/39JnrIo5TgROaeD/L6nh7fl/XyvNbtPJcCuvPBU5pIY7rgYPy9MHAjGY+q60+fATRjYj4Lenqqt7U8WhE3JGnnwLuI305NVNHRMTTeXbN/Gj66gJJWwOHAD9otmxfkbQh6R/gIoCIeD4i/t6LKg8EHoiIP7dQdg1gnXwPzro0f6/N60l3/D8TEcuA3wDv6alQF5+rsaTESf57WLN1RMR9EVG5N4Eu6rg+PxeAW0n3IDVbx5OF2fXo4bPazf/Z14H/7ql8D3VU1kUdxwBnRur2h4h4vNU4JAl4P3BZC3UEsGGe3oh+ui/MCaIfSRoB7EY6Ami27JB8aPo4cENENF0H8A3SP9yLLZTtFMD1kmYrdXXSrG2AJcAPc1PXDySt14t4xtHDP1yZiHgE+CrwEPAo8I+IuL7Jau4F9pO0maR1Sb/shvVQpiv/FhGP5tgeBbZosZ6+9J/AL1spKOnLkh4GPgic0kL5Q4FHIuKuVvZfcGxu7prcU7NdF7YD3irpNkm/kbRnL2J5K/CXiLi/hbKfBc7Jr+lXgZN6EUdlThD9RNL6wFXAZxt+YVUSEcsjYlfSL7q9JL2hyf2/G3g8ImY3u+8G+0TE7sBBwKck7ddk+TVIh8/nR8RuwD9JTSpNU7oB81DgihbKbkL61T6S1GPwepI+1EwdEXEfqRnmBuBXwF2k+3ZWepJOJj2XS1opHxEnR8SwXP7YnrZv2Pe6wMm0kFganA9sC+xK+hFwbgt1rAFsArwJ+DxweT4SaMVRtPBjJjsG+Fx+TT9HPgKvmxNEP5C0Jik5XBIRV/emrtwcMwMY02TRfYBDJS0CpgAHSPppC/tfnP8+Dvyc1MtuMzqAjsIR0JWkhNGKg4A7IuIvLZQdDTwYEUsi4gXgauAtzVYSERdFxO4RsR+pWaCVX4cAf5G0JUD+221TRp0kjQfeDXwwcqN3L1wKvK/JMtuSEvdd+fO6NXCHpFc3U0lE/CX/sHoR+D7Nf1YhfV6vzs28t5OOvrs9YV4mN2O+F/hZCzFA6nGi87vjClp7Lk1zgqhZ/rVxEXBfRHytxTraOq8mkbQO6cvtj83UEREnRcTWETGC1CxzU0Q09YtZ0nqSNuicJp3QbOoKr4h4DHhY0vZ50YG8sgv4ZvTmF9lDwJskrZvfowNJ54eaImmL/Hc46Qug1XiK3c6MB65tsZ5eURrk60Tg0Ih4psU6RhVmD6X5z+o9EbFFRIzIn9cO0oUejzUZx5aF2ffQ5Gc1uwY4INe3HemiilZ6VR0N/DFyz9UtWAy8LU8fQOs/RJrTH2fCV9YH6Z/9UeAF0of06Bbq2JfUbn83MCc/Dm6yjl2AO3Md99LDVRAV6tufFq5iIp0/uCs/5gInt7j/XYFZ+flcA2zSQh3rkkYh3KgXr8NppC+ve4GfkK9UabKO35ES3F3Aga1+roDNgBtJ//g3Apu2UMd78vRzwF+A6S3UsQB4uPBZ7ekKpLI6rsqv6d3A/wFbNVtHw/pF9HwVU1kcPwHuyXFMBbZsoY6hwE/z87kDOKCV5wL8CPhELz4f+wKz8+fsNmCPVj/3zTzc1YaZmZVyE5OZmZVygjAzs1JOEGZmVsoJwszMSjlBmJlZKScI61buSfPcwvwJkk7to7p/JOnwvqirh/0codRr7M1176tOuUfPbnsk7aLcZbm7ic9JmiRpdJPlF5X1pirpf5qNpRWtPm/rPScI68lzwHt76m65v0ka0sTmRwOfjIi31xVPP9mV1N9TZfnu47dExC4R8fWIOCUift1H8TSdIJp83zo1/bytbzhBWE+WkcbD/VzjisYjAElP57/7547NLpf0J0lnSvqg0pgW90jatlDNaEm/y9u9O5cfojQuwcz8y/e/CvXeLOlS0g1QjfEcleu/V9JZedkppJuMvifpnIbtXyXpu0r9/F8naVrn8yn+apbULmlGnl4vd/w2M3c2ODYv3yk/vzk55lF5218ojeNxr6Qj87Z75NdntqTphS42PiNpXi4/pSHWocAk4Mi8jyOVxpC4Jm9/q6RdSt6/64Etcpm3Ft+z/BxPk3RHft12yMs3UxrH4U5JF5C6m298rc8k9YQ7R9Iledk1+TnNVaEjR0lP5yOX24A3SzpYacyJWyR9S3lskrLXtux5lzxHq0t/3I3nx8r7AJ4mdTO8iNTN8AnAqXndj4DDi9vmv/sDfyeNhbEW8AhwWl53HPCNQvlfkX6ojCLdNbo2MAH4Qt5mLdJd1yNzvf8ERpbE+RpS9xltpA7WbgIOy+tmUDI+AnA4MC3v/9XAE53Ph8Ldu0A7uf994AzgQ3l6Y9L4HusB3yb1XQTp7tt1SH0Qfb+wv41IXbX/HmjLy44EJufpxbw87sAKY2QAHwG+U5j/NvClPH0AMKekzAheOa7AS+9Zfo6fztOfBH6Qp79Fvluf1D18UHInc+f7XZjfNP9dh3Tn8WZ5PoD35+m1SXdqj8zzl5Hv6u/mtX3F8/aj/x4+grAeRep99sekwV+qmhlpLIzngAdIv2Qh/fIfUdju8oh4MVIXyAuBHUh9PP2HUvfmt5G6oejs3+f2iHiwZH97kr7El0Qaz+AS0rgT3dkXuCLv/zGgyjmKdwITc2wzSF94w4E/AP8j6UTS4EXP5uc6WtJZkt4aEf8gDejzBuCGXMcXeHnMhbuBS5R6la3SK+y+pO4kiIibgM0kbVShXFFnB3Czefl92Y/UvQQR8QtS4qziM5LuIo0jMYyX37PlpO43IL2/CwvvYbHvqq5eWxsgawx0ALbS+AapL5ofFpYtIzdTShLpl3On5wrTLxbmX+SVn7vGvl6C1KTx6YiYXlwhaX/SEUSZVrpg7q7MS8+N9EVVLPO+WHFgnvtyE8ohwHRJH4uImyTtQWo//4qk60k94M6NiLJhVg8hfTkfCnxR0k7x8uA9VeNvtu+czvdlOd2/L93K781o4M0R8Uxukut83f4VEcs7N+2uGkpeW0l7NxOL9R0fQVglEbGUNCzn0YXFi4A98vRYUvNJs47I5wK2JXUGOJ80zOQxSt2kI2k79Tyo0G3A2yRtrnQi9CjSCG/duQV4X97/v5GasDot4uXnVuyuejrw6ZwQkbRb/rsN6Zfxt0gdw+2iNBbzMxHxU9IgL7vn59emPA63pDXz+YtXAcMi4mbSoE4bA+s3xPsUadjaTr8lDcjT+QX912hhrJESxXoPIo2HUOaFzveI1Hz2RE4OO5DGTyjzR2AbpcGzIDWxdSp9bVnxeVs/cYKwZpzLK/vC/z7pS/l2YG+6/nXfnfmkL/Jfknq7/BdpSNR5pDEA7gUuoIej3UijsJ1Eaia6izRORE9dZl9FOu/RuY/bgH/kdacB35T0O9Kv606nkxLh3Tm20/PyI4F7c/PIDqQmuZ2B2/Oyk4H/jYjnSec+zsrNMXNI41AMAX4q6R5Sz71fjxWHYr0Z2LFwsvZUoF3S3cCZvNxdeG+dRhop7w5Ss89DXWx3Iel1uIR0LmmNHMvppGamFeSmt08Cv5J0C6nX2c7XvKvXtvF5Wz9xb662WpO0fkQ8LWkz4HbSiHlNjTtgzSm85gLOA+6PiK8PdFy2Ip+DsNXddUqDMQ0FTndy6BcfVxq1bijpaOmCAY7HuuAjCDMzK+VzEGZmVsoJwszMSjlBmJlZKScIMzMr5QRhZmal/h8pAQ+lWWck8AAAAABJRU5ErkJggg==\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], + "source": [ + "%time show_metric(max, False)" + ] + }, + { + "cell_type": "code", + "execution_count": 30, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "mean = 7.14 ± 1.82, max = 17\n", + "CPU times: user 14.2 s, sys: 69.5 ms, total: 14.2 s\n", + "Wall time: 14.5 s\n" + ] + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEGCAYAAAB/+QKOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAfnUlEQVR4nO3de7hVVb3/8fcnFO8XVDyVYKChpmletlpZVmqGUmBlil1+dH6Wp5L05GM/8VimeCqs0z0zKSkrlbyl+yiFplj5lAooomAkIimiSWGpaSr4/f0xxtbJYu6951p7z7238Hk9z3zWvI2xvuuy93eNeRlDEYGZmVmjV/R3AGZmNjA5QZiZWSknCDMzK+UEYWZmpZwgzMys1Ab9HUBv2W677WLEiBH9HYaZ2cvK3Llz/xoRQ8u2rTMJYsSIEcyZM6e/wzAze1mR9OfOtvkQk5mZlXKCMDOzUk4QZmZWygnCzMxKOUGYmVkpJwgzMyvlBGFmZqWcIMzMrJQThJmZlVpn7qS29cuISdf1qPzSKWN6KRKzdZdbEGZmVsoJwszMSjlBmJlZKScIMzMr5QRhZmalnCDMzKyUE4SZmZVygjAzs1JOEGZmVsoJwszMStWaICSNlrRI0mJJk0q2f0LS3ZLmSbpF0u6FbafncoskvavOOM3MbG21JQhJg4DzgCOA3YHjigkguyQi9oyIvYGvAF/PZXcHxgN7AKOB7+X6zMysj9TZgjgAWBwRSyLiOWA6MK64Q0Q8UVjcDIg8Pw6YHhHPRsQDwOJcn5mZ9ZE6e3PdAXiosLwMOLBxJ0knAqcAg4FDCmVvbSi7Q0nZE4ATAHbcccdeCdrMzJI6WxAqWRdrrYg4LyJ2Bk4DPtdk2akR0RYRbUOHDu1RsGZmtqY6E8QyYHhheRiwvIv9pwNHtVjWzMx6WZ0JYjYwStJISYNJJ53biztIGlVYHAPcl+fbgfGSNpI0EhgF3F5jrGZm1qC2cxARsUrSRGAmMAiYFhELJE0G5kREOzBR0mHA88DjwIRcdoGky4CFwCrgxIhYXVesZma2tlqHHI2IGcCMhnVnFuZP7qLsF4Ev1hedmZl1xXdSm5lZKScIMzMr5QRhZmalnCDMzKyUE4SZmZVygjAzs1JOEGZmVsoJwszMSjlBmJlZKScIMzMr5QRhZmalnCDMzKyUE4SZmZVygjAzs1JOEGZmVsoJwszMSjlBmJlZKScIMzMr5QRhZmalnCDMzKyUE4SZmZVygjAzs1JOEGZmVqrWBCFptKRFkhZLmlSy/RRJCyXNl3SjpNcUtq2WNC9P7XXGaWZma9ugroolDQLOA94JLANmS2qPiIWF3e4E2iLiaUmfBL4CHJu3PRMRe9cVn5mZda3OFsQBwOKIWBIRzwHTgXHFHSJiVkQ8nRdvBYbVGI+ZmTWhzgSxA/BQYXlZXteZ44FfFpY3ljRH0q2SjiorIOmEvM+cFStW9DxiMzN7UW2HmACVrIvSHaUPA23A2wqrd4yI5ZJ2Am6SdHdE3L9GZRFTgakAbW1tpXWbmVlr6mxBLAOGF5aHAcsbd5J0GHAGMDYinu1YHxHL8+MS4GZgnxpjNTOzBt0mCEkfkLRFnv+cpKsk7Vuh7tnAKEkjJQ0GxgNrXI0kaR/gAlJyeKywfoikjfL8dsBBQPHktpmZ1axKC+LzEfGkpLcA7wIuAs7vrlBErAImAjOBe4HLImKBpMmSxubdvgpsDlzecDnr64A5ku4CZgFTGq5+MjOzmlU5B7E6P44Bzo+IaySdVaXyiJgBzGhYd2Zh/rBOyv0e2LPKc5iZWT2qtCAelnQBcAwwIx/68R3YZmbruCr/6I8hHSYaHRF/B7YBPltrVGZm1u+qJIgLIuKqiLgPICIeAT5Sb1hmZtbfqiSIPYoLuQuN/eoJx8zMBopOE4Sk0yU9Cewl6QlJT+blx4Br+ixCMzPrF50miIj4ckRsAXw1IraMiC3ytG1EnN6HMZqZWT+ocojpDEkflvR5AEnDJR1Qc1xmZtbPqiSI84A3AR/My0/ldWZmtg6rcqPcgRGxr6Q7ASLi8dx1hlnTRky6rkfll04Z00uRmFl3qrQgns9XLgWApKHAC7VGZWZm/a5Kgvg28Atge0lfBG4BvlRrVGZm1u+6PcQUERdLmgscShrj4aiIuLf2yMzMrF91myAkbUO69+HSwroNI+L5OgMzM7P+VeUQ0x3ACuBPwH15/gFJd0jyHdVmZuuoKgniV8CREbFdRGwLHAFcBnwK+F6dwZmZWf+pkiDaImJmx0JEXA8cHBG3AhvVFpmZmfWrKvdBrJR0GjA9Lx8LPJ4vffXlrmZm66gqLYgPAsOAq/M0PK8bRBorwszM1kFdtiByK+G0iPh0J7ss7v2QzMxsIOiyBRERq/HYD2Zm66Uq5yDulNQOXA78s2NlRFxVW1RmZtbvqiSIbYC/AYcU1gXgBGFmtg6r0tXGv/dFIGZmNrB0exWTpI0lnSjpe5KmdUxVKpc0WtIiSYslTSrZfoqkhZLmS7pR0msK2yZIui9PE5p7WWZm1lNVLnP9KfBK4F3Ab0iXvD7ZXaF8BdR5pDuvdweOk7R7w253km7E2wu4AvhKLrsN8AXgQOAA4AuShlR5QWZm1juqJIjXRsTngX9GxEXAGGDPCuUOABZHxJKIeI50o9244g4RMSsins6Lt5KSD6RkdENErIyIx4EbgNEVntPMzHpJpQGD8uPfJb0e2AoYUaHcDsBDheVleV1njgd+2UxZSSdImiNpzooVKyqEZGZmVVVJEFPz4Z3PAe3AQuDcCuVUsi5Kd5Q+DLQBX22mbERMjYi2iGgbOnRohZDMzKyqKpe53pgP8/wW2AlA0sgK5ZaRuuXoMAxY3riTpMOAM4C3RcSzhbJvbyh7c4XnNDOzXlKlBXFlyborKpSbDYySNFLSYGA8qQXyIkn7ABcAYyPiscKmmcDhkobk1svheZ2ZmfWRTlsQknYD9gC2kvS+wqYtgY27qzgiVkmaSPrHPgiYFhELJE0G5kREO+mQ0ubA5ZIAHoyIsRGxUtI5pCQDMDkiVrbw+szMrEVdHWLaFXg3sDXwnsL6J4GPV6k8ImYAMxrWnVmYP6yLstOASvdbmJlZ7+s0QUTENcA1kt4UEX/ow5jMzGwA6PYchJODmdn6qcpJajMzWw91miAknZwfD+q7cMzMbKDoqgXR0Yvrd/oiEDMzG1i6uorpXklLgaGS5hfWC4jcwZ6Zma2jurqK6ThJryTdxzC270IyM7OBoMuuNiLiUeAN+U7oXfLqRRHxfBfFzMxsHdBtX0yS3gb8BFhKOrw0XNKEiPhtzbGZmVk/qtJZ39eBwyNiEYCkXYBLgf3qDMzMzPpXlfsgNuxIDgAR8Sdgw/pCMjOzgaBKC2KOpAtJQ48CfAiYW19IZmY2EFRJEJ8ETgROIp2D+C3wvTqDMjOz/tdtgsiD+Hw9T2brlBGTrutR+aVTxvRSJGYDj/tiMjOzUk4QZmZWygnCzMxKVblRbhfgs8BrivtHxCE1xmVmZv2sylVMlwPfB34ArK43HDMzGyiqJIhVEXF+7ZGYmdmAUuUcxP9K+pSkV0napmOqPTIzM+tXVVoQE/LjZwvrAtip98MxM7OBosqNciP7IhAzMxtYuj3EJGlDSSdJuiJPEyVV6qxP0mhJiyQtljSpZPvBku6QtErS0Q3bVkual6f26i/JzMx6Q5VDTOeTem/t6H/pI3ndx7oqJGkQcB7wTmAZMFtSe0QsLOz2IPBR4NSSKp6JiL0rxGdmZjWokiD2j4g3FJZvknRXhXIHAIsjYgmApOnAOODFBBERS/O2FypHbGZmfaLKVUyrJe3csSBpJ6rdD7ED8FBheVleV9XGkuZIulXSUU2UMzOzXlClBfFZYJakJaTuvl8D/HuFcipZF03EtmNELM8J6SZJd0fE/Ws8gXQCcALAjjvu2ETVZmbWnSpXMd0oaRSwK+mf/h9zF+DdWQYMLywPA5ZXDSwilufHJZJuBvYB7m/YZyowFaCtra2Z5GNmZt3o9BCTpEPy4/uAMcBrgZ2BMXldd2YDoySNlDQYGA9UuhpJ0hBJG+X57YCDKJy7MDOz+nXVgngbcBPwnpJtAVzVVcURsUrSRGAmMAiYFhELJE0G5kREu6T9gV8AQ4D3SDo7IvYAXgdckE9evwKY0nD1k5mZ1azTBBERX8izkyPigeI2SZVunouIGcCMhnVnFuZnkw49NZb7PbBnlecwM7N6VLmK6cqSdVf0diBmZjawdNqCkLQbsAewVcM5hy2BjesOzMzM+ldX5yB2Bd4NbM2a5yGeBD5eZ1BmZtb/ujoHcY2ka4HTIuJLfRiTmZkNAF2eg4iI1aS+lMzMbD1T5U7q30v6LvBz4J8dKyPijtqiMjOzflclQbw5P04urAvgkN4Px8zMBooqXW28oy8CMTOzgaXKgEFbSfp67ll1jqSvSdqqL4IzM7P+U+VGuWmkS1uPydMTwI/qDMrMzPpflXMQO0fE+wvLZ0uaV1dAZmY2MFRpQTwj6S0dC5IOAp6pLyQzMxsIqrQgPglclM87CFgJTKg1KjMz63dVrmKaB7xB0pZ5+YnaozIzs35X5SqmbSV9G7iZNPTotyRtW3tkZmbWr6qcg5gOrADeDxyd539eZ1BmZtb/qpyD2CYiziks/7eko+oKyMzMBoYqLYhZksZLekWejgGuqzswMzPrX1USxH8AlwDP5Wk6cIqkJyX5hLWZ2TqqylVMW/RFIGZmNrBUOQeBpLHAwXnx5oi4tr6QzMxsIKhymesU4GRgYZ5OzuvMzGwdVqUFcSSwd0S8ACDpIuBOYFKdgZmZWf+qcpIaYOvCfOWuviWNlrRI0mJJayUUSQdLukPSKklHN2ybIOm+PLlrDzOzPlalBfFl4E5Js0h9MR0MnN5dIUmDgPNIY1ovA2ZLao+IhYXdHgQ+CpzaUHYb4AtAG2n0urm57OMV4jUzs17QZYKQJOAW4I3A/qQEcVpEPFqh7gOAxRGxJNc1HRhHOo8BQEQszdteaCj7LuCGiFiZt98AjAYurfC8ZmbWC7pMEBERkq6OiP2A9ibr3gF4qLC8DDiwB2V3aPL5zcysB6qcg7hV0v4t1K2SddGbZSWd0DEU6ooVK5oKzszMulYlQbyDlCTulzRf0t2S5lcotwwYXlgeBiyvGFelshExNSLaIqJt6NChFas2M7MqqpykPqLFumcDoySNBB4GxgMfrFh2JvAlSUPy8uFUODFuZma9p9MEIWlj4BPAa4G7gQsjYlXViiNilaSJpH/2g4BpEbFA0mRgTkS050NXvwCGAO+RdHZE7BERKyWdQ0oyAJM7TlibmVnf6KoFcRHwPPA7Uitid9Id1ZVFxAxgRsO6Mwvzs0mHj8rKTgOmNfN8ZmbWe7pKELtHxJ4Aki4Ebu+bkMzMbCDoKkE83zGTDxf1QTg2UI2Y1LMhQJZOGdNLkZhZX+kqQbyhMN6DgE3yski3SGxZe3RmZtZvOk0QETGoLwMxM7OBpWpnfWZmtp5xgjAzs1JOEGZmVsoJwszMSjlBmJlZKScIMzMr5QRhZmalnCDMzKyUE4SZmZVygjAzs1JOEGZmVsoJwszMSjlBmJlZKScIMzMr1dV4EGZWkQdUsnWRWxBmZlbKCcLMzEo5QZiZWSknCDMzK+UEYWZmpWpNEJJGS1okabGkSSXbN5L087z9Nkkj8voRkp6RNC9P368zTjMzW1ttl7lKGgScB7wTWAbMltQeEQsLux0PPB4Rr5U0HjgXODZvuz8i9q4rPjMz61qdLYgDgMURsSQingOmA+Ma9hkHXJTnrwAOlaQaYzIzs4rqTBA7AA8VlpfldaX7RMQq4B/AtnnbSEl3SvqNpLeWPYGkEyTNkTRnxYoVvRu9mdl6rs4EUdYSiIr7PALsGBH7AKcAl0jacq0dI6ZGRFtEtA0dOrTHAZuZ2UvqTBDLgOGF5WHA8s72kbQBsBWwMiKejYi/AUTEXOB+YJcaYzUzswZ1JojZwChJIyUNBsYD7Q37tAMT8vzRwE0REZKG5pPcSNoJGAUsqTFWMzNrUNtVTBGxStJEYCYwCJgWEQskTQbmREQ7cCHwU0mLgZWkJAJwMDBZ0ipgNfCJiFhZV6xmZra2WntzjYgZwIyGdWcW5v8FfKCk3JXAlXXGZmZmXfOd1GZmVsoJwszMSjlBmJlZKY8oZzaAeGQ6G0jcgjAzs1JOEGZmVsoJwszMSjlBmJlZKZ+kXsf5pKeZtcotCDMzK+UEYWZmpZwgzMyslM9BmK2DfO7JeoNbEGZmVsoJwszMSjlBmJlZKScIMzMr5QRhZmalfBWTmXXKV0Ot39yCMDOzUk4QZmZWyoeYBig37W1d4u/zy5NbEGZmVqrWFoSk0cC3gEHADyNiSsP2jYCfAPsBfwOOjYiledvpwPHAauCkiJhZZ6xmNvC5JdK3aksQkgYB5wHvBJYBsyW1R8TCwm7HA49HxGsljQfOBY6VtDswHtgDeDXwa0m7RMTquuI1s/WHE001dbYgDgAWR8QSAEnTgXFAMUGMA87K81cA35WkvH56RDwLPCBpca7vDzXG2yv8xTNbf/TW3/tA/b+hiKinYuloYHREfCwvfwQ4MCImFva5J++zLC/fDxxIShq3RsTP8voLgV9GxBUNz3ECcEJe3BVYVMuLSbYD/up6XI/rcT0v43rKvCYihpZtqLMFoZJ1jdmos32qlCUipgJTmw+teZLmRESb63E9rsf1vFzraVadVzEtA4YXlocByzvbR9IGwFbAyoplzcysRnUmiNnAKEkjJQ0mnXRub9inHZiQ548Gbop0zKsdGC9pI0kjgVHA7TXGamZmDWo7xBQRqyRNBGaSLnOdFhELJE0G5kREO3Ah8NN8EnolKYmQ97uMdEJ7FXDiALiCqbcOZbke1+N6XE9/1dOU2k5Sm5nZy5vvpDYzs1JOEGZmVsoJohuSpkl6LN+z0WodwyXNknSvpAWSTm6xno0l3S7prlzP2a3GlOsbJOlOSdf2oI6lku6WNE/SnB7Us7WkKyT9Mb9Pb2qhjl1zHB3TE5L+s8V4PpPf43skXSpp4xbrOTnXsaDZWMq+e5K2kXSDpPvy45AW6/lAjukFSZUun+yknq/mz2y+pF9I2rrFes7JdcyTdL2kV7dST2HbqZJC0nYtxnOWpIcL36UjW41H0qclLcrv91dajOfnhViWSprXXT29IiI8dTEBBwP7Avf0oI5XAfvm+S2APwG7t1CPgM3z/IbAbcAbexDXKcAlwLU9qGMpsF0vvM8XAR/L84OBrXtY3yDgUdJNQM2W3QF4ANgkL18GfLSFel4P3ANsSrog5NfAqJ5894CvAJPy/CTg3BbreR3p5tKbgbYexHM4sEGeP7cH8WxZmD8J+H4r9eT1w0kXx/y5ynezk3jOAk5t8vMuq+cd+XPfKC9v3+rrKmz/GnBms9/HVia3ILoREb8lXWHVkzoeiYg78vyTwL2kf0LN1hMR8VRe3DBPLV1lIGkYMAb4YSvle5OkLUl/FBcCRMRzEfH3HlZ7KHB/RPy5xfIbAJvk+3M2pbX7cF5H6hHg6YhYBfwGeG/Vwp1898aRkin58ahW6omIeyOiqZ4HOqnn+vzaAG4l3bPUSj1PFBY3o8L3uou/zW8A/69KHd3U05RO6vkkMCVSt0FExGM9iUeSgGOAS3sWbTVOEH1M0ghgH9Kv/1bKD8rNy8eAGyKipXqAb5L+iF5osXyHAK6XNFep65NW7ASsAH6UD3n9UNJmPYxrPC3+EUXEw8D/AA8CjwD/iIjrW6jqHuBgSdtK2hQ4kjVvAG3Fv0XEIznOR4Dte1hfb/q/wC9bLSzpi5IeAj4EnNliHWOBhyPirlbjKJiYD3tNq3IorxO7AG+VdJuk30jav4cxvRX4S0Tc18N6KnGC6EOSNgeuBP6z4RdTZRGxOiL2Jv1SO0DS61uI493AYxExt5UYGhwUEfsCRwAnSjq4hTo2IDWpz4+IfYB/kg6ftETpxsyxwOUtlh9C+qU+ktSb8GaSPtxsPRFxL+mwyw3Ar4C7SPf1rHMknUF6bRe3WkdEnBERw3MdE7vbvySGTYEzaDG5NDgf2BnYm/Qj4Wst1rMBMAR4I/BZ4LLcCmjVcfRR6wGcIPqMpA1JyeHiiLiqp/XlQzA3A6NbKH4QMFbSUmA6cIikn7UYx/L8+BjwC1Kvu81aBiwrtIauICWMVh0B3BERf2mx/GHAAxGxIiKeB64C3txKRRFxYUTsGxEHkw4b9PSX318kvQogP3Z7yKJukiYA7wY+FPkgeQ9dAry/hXI7k5L6Xfm7PQy4Q9Irm60oIv6Sf4y9APyA1r7XkL7bV+XDw7eTWuzdnjgvkw93vg/4eYuxNM0Jog/kXwwXAvdGxNd7UM/QjqtEJG1C+kf2x2briYjTI2JYRIwgHYq5KSKa/oUsaTNJW3TMk05YNn21V0Q8Cjwkade86lDW7Ba+WT39lfUg8EZJm+bP7lDSeaOmSdo+P+5I+uPu6a+/Yvc0E4BrelhfjygNCnYaMDYinu5BPaMKi2Np7Xt9d0RsHxEj8nd7GenikEdbiOdVhcX30sL3OrsaOCTXuQvpAoxWe2U9DPhj5N6v+0RfnAl/OU+kP+hHgOdJX7jjW6jjLaRj9fOBeXk6soV69gLuzPXcQy9cyQC8nRavYiKdO7grTwuAM3oQx97AnPzargaGtFjPpqTRCbfq4ftyNumf1D3AT8lXobRQz+9Iye4u4NCefveAbYEbSS2RG4FtWqznvXn+WeAvwMwW61kMPFT4Xle5+qisnivzez0f+F9gh1bqadi+lGpXMZXF81Pg7hxPO/CqFusZDPwsv7Y7gENafV3Aj4FP9OR73ezkrjbMzKyUDzGZmVkpJwgzMyvlBGFmZqWcIMzMrJQThJmZlXKCsC7lHjG/Vlg+VdJZvVT3jyUd3Rt1dfM8H1DqIXZW3c9VJ0l7V+lVtKTcpbnLiM9ImizpsCbLLy3rFVXSfzUbSytafd3Wc04Q1p1ngfdV6Ta5L0ka1MTuxwOfioh31BVPH9mb1J9TZfku4jdHxF4R8Y2IODMift1L8TSdIJr83Do0/bqtdzhBWHdWkcbD/UzjhsYWgKSn8uPbc8dkl0n6k6Qpkj6kNJbF3ZJ2LlRzmKTf5f3encsPUhpnYHb+5fsfhXpnSbqEdBNTYzzH5frvkXRuXncm6UbF70v6asP+r5D0PaV++q+VNKPj9RR/NUtqk3Rznt8sd942O3csOC6v3yO/vnk55lF53+uUxu+4R9Kxed/98vszV9LMQtcZJ0lamMtPb4h1MDAZODY/x7FKY0Ncnfe/VdJeJZ/f9cD2ucxbi59Zfo1nS7ojv2+75fXbKo3JcKekC0jdzDe+11NIvd3Ok3RxXnd1fk0LVOi4UdJTueVyG/AmSUcqjSFxi6RvK49HUvbelr3uktdodenLu/I8vfwm4ClgS9JdqVsBpwJn5W0/Bo4u7psf3w78nTQOxkbAw8DZedvJwDcL5X9F+qEyinTX6MbACcDn8j4bke6wHpnr/ScwsiTOV5O6yBhK6iDtJuCovO1mSsY8AI4GZuTnfyXweMfroXAXLtAG3JznvwR8OM9vTRrbYzPgO6S+iCDdPbsJqT+hHxSebytSF+2/B4bmdccC0/L8cl4aN2Ct8TCAjwLfLSx/B/hCnj8EmFdSZgRrjk/w4meWX+On8/yngB/m+W+T79IndQkflNyR3PF5F5a3yY+bkO4c3jYvB3BMnt+YdOf1yLx8KflO/i7e2zVet6e+m9yCsG5F6nn2J6SBXKqaHWkcjGeB+0m/ZCH98h9R2O+yiHghUvfFS4DdSH06/R+lbs1vI3Uv0dFXz+0R8UDJ8+1P+ie+ItL4BBeTxpjoyluAy/PzPwpUOUdxODApx3Yz6R/ejsAfgP+SdBppkKJn8ms9TNK5kt4aEf8gDdLzeuCGXMfneGkMhfnAxUo9x1bp9fUtpC4hiIibgG0lbVWhXFFHx5FzeelzOZjUPQQRcR0pcVZxkqS7SONCDOelz2w1qSsNSJ/vksJnWOybqrP31vrJBv0dgL1sfJPUl8yPCutWkQ9TShLpl3OHZwvzLxSWX2DN711jXy9BOqTx6YiYWdwg6e2kFkSZVrpQ7qrMi6+N9I+qWOb9sfZgO/fmQyhjgJmSPhYRN0naj3T8/MuSrif1eLsgIsqGVB1D+uc8Fvi8pD3ipcF4qsbfbN85HZ/Larr+XLqUP5vDgDdFxNP5kFzH+/aviFjdsWtX1VDy3ko6sJlYrPe4BWGVRMRK0tCbxxdWLwX2y/PjSIdPmvWBfC5gZ1Lnf4tIw0V+UqmLdCTtou4HELoNeJuk7ZROhB5HGsGtK7cA78/P/2+kQ1gdlvLSayt2PT0T+HROiEjaJz/uRPpl/G1S5257KY2r/HRE/Iw0ANG++fUNVR5zW9KG+fzFK4DhETGLNJDT1sDmDfE+SRqytsNvSYPrdPyD/mu0OM5Ig2K9R5DGMyjzfMdnRDp89nhODruRxj8o80dgJ6WBsyAdYutQ+t6y9uu2PuIEYc34Gmv2Zf8D0j/l24ED6fzXfVcWkf6R/5LUU+W/SMOgLiT15X8PcAHdtHYjja52Oukw0V2k8SC66wr7StJ5j47nuA34R952NvAtSb8j/brucA4pEc7PsZ2T1x8L3JMPj+xGOiS3J3B7XncG8N8R8Rzp3Me5+XDMPNJYE4OAn0m6m9Rj7zdi7WFXZwG7F07WngW0SZoPTOGlbsB76mzSSHh3kA77PNjJflNJ78PFpHNJG+RYziEdZlpLPvT2KeBXkm4h9STb8Z539t42vm7rI+7N1dZrkjaPiKckbQvcThohr+nxA6y6wnsu4Dzgvoj4Rn/HZWvzOQhb312rNAjTYOAcJ4c+8XGlUegGk1pLF/RzPNYJtyDMzKyUz0GYmVkpJwgzMyvlBGFmZqWcIMzMrJQThJmZlfr/+VH9UgPwYj4AAAAASUVORK5CYII=\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], + "source": [ + "%time show_metric(expectation, False)" + ] + }, + { + "cell_type": "code", + "execution_count": 31, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "mean = 7.09 ± 1.79, max = 19\n", + "CPU times: user 12.9 s, sys: 59.3 ms, total: 13 s\n", + "Wall time: 13 s\n" + ] + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEGCAYAAAB/+QKOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3de7RVdbn/8fcnFO938VSKgYaaZnnZaqVpqRlqgZUmVufQyfJkWZbDfmKWKZ7My7E6lZmUlKdUvGakFJpK5igVUJSLkoikiCaFpaap4PP74/vdOd3Mvfaciz3ZW/i8xlhjz9sz57Muez1r3r5fRQRmZmZdvaavEzAzs/7JBcLMzEq5QJiZWSkXCDMzK+UCYWZmpdbo6wR6y+abbx5Dhgzp6zTMzF5Vpk+f/peIGFQ2b5UpEEOGDGHatGl9nYaZ2auKpD91N8+HmMzMrJQLhJmZlXKBMDOzUi4QZmZWygXCzMxKuUCYmVkpFwgzMyvlAmFmZqVcIMzMrNQqcye1vboNGXN927ELzjq0FzMxs07egzAzs1IuEGZmVsoFwszMSrlAmJlZKRcIMzMr5QJhZmalXCDMzKyUC4SZmZVygTAzs1IuEGZmVsoFwszMSjVaICQNlzRX0jxJY0rmf1rSTEkzJN0macfCvJNz3FxJ720yTzMzW15jBULSAOB84GBgR+CoYgHILo2InSNiF+Ac4Js5dkdgFLATMBz4fl6fmZmtJE3uQewJzIuI+RHxAjABGFlcICKeKoyuB0QeHglMiIjnI+IhYF5en5mZrSRNNve9JfBIYXwhsFfXhSR9FjgBGAjsX4i9vUvsls2kaWZmZZrcg1DJtFhuQsT5EbEtcBLwlTqxko6RNE3StMWLF69QsmZm9kpNFoiFwODC+FbAohbLTwAOqxMbEeMioiMiOgYNGrSC6ZqZWVGTBWIqMEzSUEkDSSedJxYXkDSsMHoo8EAengiMkrSWpKHAMODOBnM1M7MuGjsHERFLJR0HTAYGAOMjYrakscC0iJgIHCfpQOBF4ElgdI6dLekKYA6wFPhsRCxrKlczM1teo31SR8QkYFKXaacWho9vEft14OvNZWdmZq34TmozMyvlAmFmZqVcIMzMrJQLhJmZlXKBMDOzUi4QZmZWygXCzMxKuUCYmVkpFwgzMyvlAmFmZqVcIMzMrJQLhJmZlXKBMDOzUi4QZmZWygXCzMxKuUCYmVkpFwgzMyvlAmFmZqVcIMzMrJQLhJmZlXKBMDOzUi4QZmZWygXCzMxKNVogJA2XNFfSPEljSuafIGmOpHsl3STpDYV5yyTNyI+JTeZpZmbLW6OpFUsaAJwPvAdYCEyVNDEi5hQWuxvoiIhnJR0LnAMcmec9FxG7NJWfmZm11uQexJ7AvIiYHxEvABOAkcUFIuKWiHg2j94ObNVgPmZmVkOTBWJL4JHC+MI8rTtHA78qjK8taZqk2yUdVhYg6Zi8zLTFixeveMZmZvYvjR1iAlQyLUoXlD4GdAD7FSZvHRGLJG0D3CxpZkQ8+IqVRYwDxgF0dHSUrtvMzNrT4x6EpCMkbZCHvyLpGkm7VVj3QmBwYXwrYFHJ+g8ETgFGRMTzndMjYlH+Ox+YAuxaYZtmZtZLqhxi+mpEPC1pH+C9wMXABRXipgLDJA2VNBAYBbziaiRJuwIXkorDE4Xpm0haKw9vDuwNFE9um5lZw6oUiGX576HABRHxC2BgT0ERsRQ4DpgM3AdcERGzJY2VNCIvdi6wPnBll8tZ3wRMk3QPcAtwVpern8zMrGFVzkE8KulC4EDg7PzLvtLJ7YiYBEzqMu3UwvCB3cT9Hti5yjbMzKwZVb7oP0zaCxgeEX8DNgW+1GhWZmbW56oUiAsj4pqIeAAgIh4D/r3ZtMzMrK9VKRA7FUfyHdK7N5OOmZn1F92eg5B0MvBlYB1JT/HyfQ0vkO89MOs0ZMz1bccuOOvQXszEzHpLt3sQEfGNiNgAODciNoyIDfJjs4g4eSXmaGZmfaDKIaZTJH1M0lcBJA2WtGfDeZmZWR+rUiDOB94OfCSPP5OnmZnZKqzKfRB7RcRuku4GiIgn853RZma2CquyB/FivnIpACQNAl5qNCszM+tzVQrEd4CfA1tI+jpwG3Bmo1mZmVmf6/EQU0RcImk6cADpUtfDIuK+xjMzM7M+1WOBkLQp8ARwWWHamhHxYpOJmZlZ36pyiOkuYDHwR+CBPPyQpLsk+Y5qM7NVVJUC8WvgkIjYPCI2Aw4GrgA+A3y/yeTMzKzvVCkQHRExuXMkIm4A9o2I24G1GsvMzMz6VJX7IJZIOgmYkMePBJ7Ml776clczs1VUlT2Ij5D6k742PwbnaQNIfUWYmdkqqOUeRN5LOCkiPtfNIvN6PyUzM+sPWu5BRMQy3PeDmdlqqco5iLslTQSuBP7ROTEirmksKzMz63NVCsSmwF+B/QvTAnCBMDNbhVVpauM/V0YiZmbWv1RpamNt4GhS39Rrd06PiE80mJeZmfWxKpe5/hR4LfBe4LekS16frrJyScMlzZU0T9KYkvknSJoj6V5JN0l6Q2HeaEkP5Mfoak/HzMx6S5UC8caI+Crwj4i4GDgU2LmnoHyJ7Pmkpjl2BI6StGOXxe4m3an9FuAq4JwcuynwNWAvYE/ga5I2qfaUzMysN1TqMCj//ZukNwMbAUMqxO0JzIuI+RHxAulO7JHFBSLiloh4No/eTto7gbS3cmNELImIJ4EbgeEVtmlmZr2kSoEYl3+9fwWYCMwBzq4QtyXwSGF8YZ7WnaOBX9WJlXSMpGmSpi1evLhCSmZmVlWVy1xvyr/ibwW2AZA0tEKcSqZF6YLSx4AOYL86sRExDhgH0NHRUbpuMzNrT5U9iKtLpl1VIW4hqd2mTlsBi7ouJOlA4BRgREQ8XyfWzMya0+0ehKQdSJe2biTpg4VZG1K43LWFqcCwvLfxKDCK1MhfcRu7AhcCwyPiicKsycCZhRPTBwEnV9immZn1klaHmLYH3gdsDLy/MP1p4FM9rTgilko6jvRlPwAYHxGzJY0FpkXEROBcYH3gSkkAD0fEiIhYIukMUpEBGBsRS2o+NzMzWwHdFoiI+AXwC0lvj4g/tLPyiJgETOoy7dTC8IEtYscD49vZrpmZrbgez0G0WxzMzOzVrcpJajMzWw11WyAkHZ//7r3y0jEzs/6i1R5EZyuu310ZiZiZWf/S6iqm+yQtAAZJurcwXUDk9pPMzGwV1eoqpqMkvZZ0meqIlZeSmZn1By2b2oiIx4G3ShoIbJcnz42IF1uEmZnZKqBKh0H7Af8HLCAdXhosaXRE3NpwbmZm1oeqNNb3TeCgiJgLIGk74DJg9yYTMzOzvlXlPog1O4sDQET8EVizuZTMzKw/qLIHMU3SRaSuRwE+CkxvLiUzM+sPqhSIY4HPAp8nnYO4Ffh+k0mZmVnf67FA5D4avpkfZma2mqiyB2HW7w0Zc33bsQvOOrQXMzFbdbixPjMzK+UCYWZmparcKLcd8CXgDcXlI2L/BvMyM7M+VuUcxJXAD4AfAsuaTcfMzPqLKgViaURc0HgmZmbWr1Q5B/FLSZ+R9DpJm3Y+Gs/MzMz6VJU9iNH575cK0wLYpvfTMTOz/qLKjXJDV0YiZmbWv1S5imlNUnMb++ZJU4AL3SeEmdmqrco5iAtITXt/Pz92z9N6JGm4pLmS5kkaUzJ/X0l3SVoq6fAu85ZJmpEfE6tsz8zMek+VcxB7RMRbC+M3S7qnpyBJA4DzgfcAC4GpkiZGxJzCYg8DHwdOLFnFcxGxS4X8zMysAVX2IJZJ2rZzRNI2VLsfYk9gXkTMj4gXgAnAyOICEbEgIu4FXqqRs5mZrQRV9iC+BNwiaT6pue83AP9ZIW5L4JHC+EJgrxq5rS1pGrAUOCsiru26gKRjgGMAtt566xqrNjOznlS5iukmScOA7UkF4v7cBHhPVLa6GrltHRGL8h7LzZJmRsSDXXIbB4wD6OjoqLNuMzPrQbcFQtL+EXGzpA92mbWtJCLimh7WvRAYXBjfClhUNbGIWJT/zpc0BdgVeLBlkJmZ9ZpWexD7ATcD7y+ZF0BPBWIqMEzSUOBRYBTwkSpJSdoEeDYinpe0ObA3cE6VWDMz6x3dFoiI+FoeHBsRDxXn5S/9liJiqaTjgMnAAGB8RMyWNBaYFhETJe0B/BzYBHi/pNMjYifgTcCFkl4inUg/q8vVT2Zm1rAqJ6mvBnbrMu0q0v0QLUXEJGBSl2mnFoankg49dY37PbBzhdzMzKwhrc5B7ADsBGzU5TzEhsDaTSdmZmZ9q9UexPbA+4CNeeV5iKeBTzWZlJmZ9b1W5yB+Iek64KSIOHMl5mRmZv1AyzupI2IZqakMMzNbzVQ5Sf17Sd8DLgf+0TkxIu5qLCszM+tzVQrEO/LfsYVpAezf++mYmVl/UaWpjXevjETMzKx/6bE1V0kbSfqmpGn5cZ6kjVZGcmZm1neqNPc9nnRp64fz4yngx00mZWZmfa/KOYhtI+JDhfHTJc1oKiEzM+sfquxBPCdpn84RSXsDzzWXkpmZ9QdV9iCOBS7O5x0ELAFGN5qVmZn1uSpXMc0A3ippwzz+VONZmZlZn6tyFdNmkr4DTCF1Pfq/kjZrPDMzM+tTVc5BTAAWAx8CDs/DlzeZlJmZ9b0q5yA2jYgzCuP/LemwphIyM7P+ocoexC2SRkl6TX58GLi+6cTMzKxvVSkQ/wVcCryQHxOAEyQ9LcknrM3MVlFVrmLaYGUkYmZm/UuVcxBIGgHsm0enRMR1zaVkZmb9QZXLXM8Cjgfm5MfxeZqZma3CquxBHALsEhEvAUi6GLgbGNNkYmZm1reqnKQG2Lgw7Ka+zcxWA1UKxDeAuyX9JO89TAfOrLJyScMlzZU0T9JyexyS9pV0l6Slkg7vMm+0pAfyw20/mZmtZC0PMUkScBvwNmAPUmN9J0XE4z2tWNIA4HzgPcBCYKqkiRExp7DYw8DHgRO7xG4KfA3oIHVvOj3HPlnxeZmZ2QpqWSAiIiRdGxG7AxNrrntPYF5EzAeQNAEYSTrR3bn+BXneS11i3wvcGBFL8vwbgeHAZTVzMDOzNlU5xHS7pD3aWPeWwCOF8YV5Wq/FSjqmsyvUxYsXt5GimZl1p0qBeDepSDwo6V5JMyXdWyFOJdOiYl6VYiNiXER0RETHoEGDKq7azMyqqHKZ68FtrnshMLgwvhWwqEbsu7rETmkzDzMza0O3BULS2sCngTcCM4GLImJpjXVPBYZJGgo8CowCPlIxdjJwpqRN8vhBwMk1tm1mZiuo1SGmi0lXEc0k7UWcV2fFuZgcR/qyvw+4IiJmSxqbm+5A0h6SFgJHABdKmp1jlwBnkIrMVGBs5wlrMzNbOVodYtoxInYGkHQRcGfdlUfEJGBSl2mnFoankg4flcWOB8bX3aaZmfWOVgXixc6BiFiabomwVdWQMe138bHgrEN7MRMz6y9aFYi3Fvp7ELBOHhfpFokNG8/OzMz6TLcFIiIGrMxEzMysf6naWJ+Zma1mXCDMzKyUC4SZmZVygTAzs1IuEGZmVsoFwszMSrlAmJlZKRcIMzMr5QJhZmalXCDMzKyUC4SZmZVygTAzs1IuEGZmVsoFwszMSrlAmJlZqVYdBpmtNtyjntnyvAdhZmalXCDMzKyUC4SZmZVygTAzs1KNFghJwyXNlTRP0piS+WtJujzPv0PSkDx9iKTnJM3Ijx80maeZmS2vsauYJA0AzgfeAywEpkqaGBFzCosdDTwZEW+UNAo4Gzgyz3swInZpKj8zM2utyT2IPYF5ETE/Il4AJgAjuywzErg4D18FHCBJDeZkZmYVNVkgtgQeKYwvzNNKl4mIpcDfgc3yvKGS7pb0W0nvLNuApGMkTZM0bfHixb2bvZnZaq7JAlG2JxAVl3kM2DoidgVOAC6VtOFyC0aMi4iOiOgYNGjQCidsZmYva7JALAQGF8a3AhZ1t4ykNYCNgCUR8XxE/BUgIqYDDwLbNZirmZl10WSBmAoMkzRU0kBgFDCxyzITgdF5+HDg5ogISYPySW4kbQMMA+Y3mKuZmXXR2FVMEbFU0nHAZGAAMD4iZksaC0yLiInARcBPJc0DlpCKCMC+wFhJS4FlwKcjYklTuZqZ2fIabawvIiYBk7pMO7Uw/E/giJK4q4Grm8zNzMxa853UZmZWygXCzMxKuUCYmVkpFwgzMyvlAmFmZqXc5ahZL2m321J3WWr9lfcgzMyslAuEmZmV8iGmVYQPb5hZb/MehJmZlXKBMDOzUi4QZmZWygXCzMxK+SS1WT/hCw2sv/EehJmZlXKBMDOzUi4QZmZWygXCzMxKuUCYmVkpFwgzMyvly1zNVhG+TNZ6mwtEP+F/bjPrb1wgzAzwjxRbXqPnICQNlzRX0jxJY0rmryXp8jz/DklDCvNOztPnSnpvk3mamdnyGtuDkDQAOB94D7AQmCppYkTMKSx2NPBkRLxR0ijgbOBISTsCo4CdgNcDv5G0XUQsaypfM1sx7e6BQNoL6et4W16Th5j2BOZFxHwASROAkUCxQIwETsvDVwHfk6Q8fUJEPA88JGleXt8fGsx3hXj33OzVzQVmeYqIZlYsHQ4Mj4hP5vF/B/aKiOMKy8zKyyzM4w8Ce5GKxu0R8bM8/SLgVxFxVZdtHAMck0e3B+Y28mRgc+Avju+z+P6Qg+NX7/j+kENvPIcyb4iIQWUzmtyDUMm0rtWou2WqxBIR44Bx9VOrR9K0iOhwfN/E94ccHL96x/eHHHrjOdTV5EnqhcDgwvhWwKLulpG0BrARsKRirJmZNajJAjEVGCZpqKSBpJPOE7ssMxEYnYcPB26OdMxrIjAqX+U0FBgG3NlgrmZm1kVjh5giYqmk44DJwABgfETMljQWmBYRE4GLgJ/mk9BLSEWEvNwVpBPaS4HP9vEVTCt6GMvxK66vc3D86h3fH3Jo/HB6V42dpDYzs1c3N9ZnZmalXCDMzKyUC0QLksZLeiLfr9FO/GBJt0i6T9JsScfXjF9b0p2S7snxp7eZxwBJd0u6ro3YBZJmSpohaVob8RtLukrS/fl1eHuN2O3zdjsfT0n6Qs3tfzG/drMkXSZp7Zrxx+fY2VW3Xfa5kbSppBslPZD/blIz/oicw0uSWl7q2E38ufk9uFfSzyVtXDP+jBw7Q9INkl5fJ74w70RJIWnzmts/TdKjhc/CIXW3L+lzSk33zJZ0Ts3tX17Y9gJJM2rG7yLp9s7/I0l7dhffYh1vlfSH/P/4S0kbtlpHr4gIP7p5APsCuwGz2ox/HbBbHt4A+COwY414Aevn4TWBO4C3tZHHCcClwHVtxC4ANl+B1/Bi4JN5eCCwcZvrGQA8Trqpp2rMlsBDwDp5/Arg4zXi3wzMAtYlXdDxG2BYO58b4BxgTB4eA5xdM/5NpJtBpwAdbWz/IGCNPHx2G9vfsDD8eeAHdeLz9MGki1b+1Ooz1c32TwNOrPi+lcW/O79/a+XxLermX5h/HnBqze3fABychw8BprTxHKYC++XhTwBnVP0st/vwHkQLEXEr6eqqduMfi4i78vDTwH2kL62q8RERz+TRNfOj1lUFkrYCDgV+VCeuN+RfOPuSrlYjIl6IiL+1uboDgAcj4k8149YA1sn32axLvftp3kS6o//ZiFgK/Bb4QE9B3XxuRpKKJfnvYXXiI+K+iKjUUkA38Tfk5wBwO+neojrxTxVG16PF57DF/823gP/XKraH+Eq6iT8WOCtS8z1ExBPtbF+SgA8Dl9WMD6DzF/9G9PA57GYd2wO35uEbgQ+1WkdvcIFYSZRaqt2VtBdQJ25A3p19ArgxImrFA98m/VO+VDOuUwA3SJqu1LRJHdsAi4Ef50NcP5K0Xpt5jKLFP2WZiHgU+B/gYeAx4O8RcUONVcwC9pW0maR1Sb/8BvcQ051/i4jHcl6PAVu0uZ7e8AngV3WDJH1d0iPAR4FTa8aOAB6NiHvqbrfguHyYa3yrQ3Td2A54p1Kr0b+VtEebObwT+HNEPFAz7gvAufn1+x/g5Da2PQsYkYePoP3PYmUuECuBpPWBq4EvdPkl1qOIWBYRu5B+8e0p6c01tvs+4ImImF4r4VfaOyJ2Aw4GPitp3xqxa5B2ky+IiF2Bf5AOr9SidKPlCODKmnGbkH65DyW1CryepI9VjY+I+0iHY24Efg3cQ7ov51VL0imk53BJ3diIOCUiBufY43pavrDNdYFTqFlUurgA2BbYhVTsz6sZvwawCfA24EvAFXlvoK6jqPlDJTsW+GJ+/b5I3quu6ROk/8HppEPWL7SxjlpcIBomaU1ScbgkIq5pdz350MwUYHiNsL2BEZIWABOA/SX9rOZ2F+W/TwA/J7WqW9VCYGFhr+cqUsGo62Dgroj4c824A4GHImJxRLwIXAO8o84KIuKiiNgtIvYl7fLX/eXY6c+SXgeQ/3Z7iKMpkkYD7wM+GvlAdpsupd7hjW1JRfqe/FncCrhL0murriAi/px/LL0E/JB6n0NIn8Vr8mHbO0l71N2eKC+TD1N+ELi85rYhtRjR+f9/JfXzJyLuj4iDImJ3UpF6sI08anGBaFD+hXIRcF9EfLON+EGdV5tIWof0hXd/1fiIODkitoqIIaRDNDdHROVf0JLWk7RB5zDpRGflK7oi4nHgEUnb50kH8Mrm3qtq91fbw8DbJK2b34sDSOeBKpO0Rf67NenLoZ084JXNyowGftHmetoiaThwEjAiIp5tI35YYXQE9T6HMyNii4gYkj+LC0kXbzxeY/uvK4x+gBqfw+xaYP+8ru1IF0zUbRn1QOD+yK1P17QI2C8P708bPzQKn8XXAF8BftBGHvU0fRb81fwgfRk8BrxI+lAfXTN+H9Ix/HuBGflxSI34twB35/hZtLhyosK63kXNq5hI5xDuyY/ZwCltbHcXYFp+DtcCm9SMXxf4K7BRm8/7dNKX2Szgp+SrWGrE/45U1O4BDmj3cwNsBtxE+mK4Cdi0ZvwH8vDzwJ+ByTXj5wGPFD6Hra5CKou/Or+G9wK/BLZs9/+GHq6M62b7PwVm5u1PBF5XM34g8LP8HO4C9q+bP/AT4NNtvv/7ANPz5+gOYPc21nE86UrIPwJnkVvCaPLhpjbMzKyUDzGZmVkpFwgzMyvlAmFmZqVcIMzMrJQLhJmZlXKBsJZyy5vnFcZPlHRaL637J5IO74119bCdI5Rakr2l6W01KbcI2m0rpi3iLstNVHxR0lhJB9aMX1DW+qqkL9fNpR3tPm9bcS4Q1pPngQ+2ap65L0gaUGPxo4HPRMS7m8pnJdmF1B5UZflu5XdExFsi4lsRcWpE/KaX8qldIGq+b51qP2/rHS4Q1pOlpL5wv9h1Rtc9AEnP5L/vyg2iXSHpj5LOkvRRpb4tZkratrCaAyX9Li/3vhw/QKn/gqn5l+9/FdZ7i6RLSTdNdc3nqLz+WZLOztNOJd2k9ANJ53ZZ/jWSvq/UP8B1kiZ1Pp/ir2ZJHZKm5OH1cmNxU3MDhCPz9J3y85uRcx6Wl71eqT+PWZKOzMvunl+f6ZImF5rg+LykOTl+QpdcBwJjgSPzNo5U6mPi2rz87ZLeUvL+3QBskWPeWXzP8nM8XdJd+XXbIU/fTKnPh7slXUhqdr7ra30WqZXcGZIuydOuzc9ptgoNO0p6Ju+53AG8XdIhSn1T3CbpO8r9lJS9tmXPu+Q5WlOavhPPj1f3A3iG1EzxAlIzxScCp+V5PwEOLy6b/74L+BupP4y1gEeB0/O844FvF+J/TfqhMox0x+jawDHAV/Iya5HuxB6a1/sPYGhJnq8nNa0xiNQw283AYXneFEr6UAAOBybl7b8WeLLz+VC42xfoILffD5wJfCwPb0y6q3U94LukNo4g3bW7Dqm9oh8WtrcRqcn23wOD8rQjgfF5eBEv91ewXL8ZwMeB7xXGvwt8LQ/vD8woiRnCK/sU+Nd7lp/j5/LwZ4Af5eHvkO/aJzUVH5Tc+dz5fhfGN81/1yHdsbxZHg/gw3l4bdId3UPz+GXkO/xbvLaveN5+rLyH9yCsR5FaoP0/UkcxVU2N1B/G86RGxTqb2Z5J+tLqdEVEvBSp+eT5wA6kNp/+Q6mZ8ztIzVR0tgV0Z0Q8VLK9PUhf4osj9XtwCakvilb2Aa7M238cqHKO4iBgTM5tCukLb2vgD8CXJZ1E6tToufxcD5R0tqR3RsTfSW36vxm4Ma/jK7zcN8O9wCVKLc5WaTV2H1ITFETEzcBmkjaqEFfU2YDcdF5+X/YlNUtBRFxPKpxVfF7SPaT+Jgbz8nu2jNRUB6T3d37hPSy2bdXda2t9ZI2+TsBeNb5NasPmx4VpS8mHKSWJ9Mu50/OF4ZcK4y/xys9d17ZegnRI43MRMbk4Q9K7SHsQZdppurlVzL+eG+mLqhjzoVi+85778iGUQ4HJkj4ZETdL2p10/Pwbkm4gtYg7OyLKul49lPTlPAL4qqSd4uVOfqrmX7ftnM73ZRmt35eW8ntzIPD2iHg2H5LrfN3+GRHLOhdttRpKXltJe9XJxXqP9yCskohYQuqy8+jC5AXA7nl4JOnwSV1H5HMB25IaB5xL6pbyWKWm0pG0nXruaOgOYD9JmyudCD2K1ANcK7cBH8rb/zfSIaxOC3j5uRWbtp4MfC4XRCTtmv9uQ/pl/B1SY3JvUeq3+dmI+Bmpk5jd8vMbpNw3t6Q18/mL1wCDI+IWUgdPGwPrd8n3aVI/AJ1uJXXe0/kF/Zeo2d9IN4rrPZjUj0KZFzvfI9LhsydzcdiB1O9CmfuBbZQ60IJ0iK1T6WvL8s/bVhIXCKvjPF7Zhv4PSV/KdwJ70f2v+1bmkr7If0VqKfOfpO5R55D6DJgFXEgPe7uRemk7mXSY6B5S/xE9Nal9Nem8R+c27gD+nuedDvyvpN+Rfl13OoNUCO/NuZ2Rpx8JzMqHR3YgHZLbGbgzTzsF+O+IeIF07uPsfDhmBqmPigHAzyTNJLXg+61YvnvWW4AdCydrTwM6JN1Lat1zNL3jdFJPeneRDvs83M1y40ivwyWkc0lr5FzOIB1mWk4+9PYZ4NeSbiO1TNv5mnf32nZ93raSuDVXW61JWj8inpG0GXAnqQe9yv0UWH2F11IiEHIAAABUSURBVFzA+cADEfGtvs7LludzELa6u06pU6aBwBkuDivFp5R6txtI2lu6sI/zsW54D8LMzEr5HISZmZVygTAzs1IuEGZmVsoFwszMSrlAmJlZqf8PMQn1oTpPcw4AAAAASUVORK5CYII=\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], + "source": [ + "%time show_metric(negative_entropy, False)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Comparing Metrics with Nontarget Guesses Allowed" + ] + }, + { + "cell_type": "code", + "execution_count": 32, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "mean = 7.05 ± 0.98, max = 10\n", + "CPU times: user 32.3 s, sys: 47.2 ms, total: 32.3 s\n", + "Wall time: 32.4 s\n" + ] + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAZmklEQVR4nO3de7wdVXn/8c+XcJVLkCTeIBDACI3I9XATGhGpDYQGfnJXLG1RVK5trSWoUAn9KepPUBQpUVCqQOSm5AcpwUIi5dUCuXANGImYQkAkVpSbAglP/1hr487JPvtMDpl1yJnv+/XarzMze/Y8z+yd7GfPmpm1FBGYmVlzrTXYCZiZ2eByITAzazgXAjOzhnMhMDNrOBcCM7OGW3uwE1hVI0eOjDFjxgx2GmZma5R58+b9OiJGdXpujSsEY8aMYe7cuYOdhpnZGkXSf/f1nJuGzMwazoXAzKzhXAjMzBrOhcDMrOFcCMzMGs6FwMys4VwIzMwazoXAzKzhXAjMzBpujbuz2Mz6NmbyjcViLT53YrFYVi8fEZiZNZwLgZlZw7kQmJk1nAuBmVnDuRCYmTWcC4GZWcO5EJiZNZwLgZlZw7kQmJk1nAuBmVnDuRCYmTWcC4GZWcO5EJiZNZwLgZlZw7kQmJk1nAuBmVnDuRCYmTVcrYVA0gRJCyUtkjS5y3qHSwpJPXXmY2ZmK6utEEgaBlwIHAiMA46RNK7DehsDpwJ31pWLmZn1rc4jgj2ARRHxSES8BEwDDumw3jnAl4A/1JiLmZn1oc5CsDnwWNv8krzsVZJ2AUZHxA3dNiTpBElzJc1dunTp6s/UzKzB6iwE6rAsXn1SWgs4H/hkfxuKiKkR0RMRPaNGjVqNKZqZWZ2FYAkwum1+C+CJtvmNgR2A2ZIWA3sB033C2MysrDoLwRxgrKStJa0LHA1Mbz0ZEb+LiJERMSYixgB3AJMiYm6NOZmZWS+1FYKIWAacDMwEHgKuiogFkqZImlRXXDMzWzVr17nxiJgBzOi17Kw+1t2vzlzMzKwz31lsZtZwLgRmZg3nQmBm1nAuBGZmDedCYGbWcC4EZmYN50JgZtZwLgRmZg3nQmBm1nAuBGZmDedCYGbWcC4EZmYN50JgZtZwLgRmZg3nQmBm1nAuBGZmDedCYGbWcC4EZmYN50JgZtZwLgRmZg3nQmBm1nAuBGZmDddvIZB0hKSN8/RnJV0nadf6UzMzsxKqHBGcGRHPStoX+HPgMuCietMyM7NSqhSC5fnvROCiiLgeWLe+lMzMrKQqheBxSRcDRwIzJK1X8XVmZrYGqPKFfiQwE5gQEb8FNgM+VWtWZmZWTJVCcHFEXBcRDwNExC+BD9eblpmZlVKlELyzfUbSMGC3etIxM7PS+iwEks6Q9Cywo6RnJD2b558Cri+WoZmZ1arPQhARX4iIjYEvR8QmEbFxfoyIiDMK5mhmZjWq0jT0GUnHSjoTQNJoSXvUnJeZmRVSpRBcCOwNfDDPP5eXmZnZELB2hXX2jIhdJd0NEBFPS/INZWZmQ0SVI4KX85VCASBpFPBKrVmZmVkxVQrBBcAPgTdJ+r/A7cDna83KzMyK6bdpKCIulzQPeB8g4NCIeKj2zMzMrIgq3VBvRrp34ErgCuBXktapsnFJEyQtlLRI0uQOz39c0v2S7pF0u6Rxq7oDZmb22lRpGpoPLAV+Bjycp38hab6kPu8wzucVLgQOBMYBx3T4or8iIt4VETsDXwLOG8A+mJnZa1ClENwEHBQRIyNiBOmL/SrgROCbXV63B7AoIh6JiJeAacAh7StExDNtsxuST0ibmVk5VQpBT0TMbM1ExM3A+Ii4A1ivy+s2Bx5rm1+Sl61A0kmSfk46Iji1UtZmZrbaVCkEv5F0uqSt8uMfgadz00+3y0jVYdlKv/gj4sKI2BY4Hfhsxw1JJ0iaK2nu0qVLK6RsZmZVVSkEHwS2AH6UH6PzsmGksQr6siSv27IF8ESX9acBh3Z6IiKmRkRPRPSMGjWqQspmZlZV18tH86/+0yPilD5WWdTl5XOAsZK2Bh4HjuaP3VS0tj+2Nc4BaSjMhzEzs6K6FoKIWN7tyqB+XrtM0smk0c2GAZdGxAJJU4C5ETEdOFnSAcDLwNPAcQOJZWZmA1elr6G7JU0Hrgaeby2MiOv6e2FEzABm9Fp2Vtv0adVTNTOzOlQpBJsB/wPs37YsgH4LgZmZvf5V6WLir0skYmZmg6PfQiBpfeB40tjF67eWR8Tf1JiXmZkVUuXy0e8BbwH+HPgJ6TLQZ+tMyszMyqlSCN4eEWcCz0fEZaTLPN9Vb1pmZlZKpYFp8t/fStoBGA6MqS0jMzMrqspVQ1MlvZHU/cN0YCPgzFqzMjOzYqoUglsi4mngNmAbgHy3sJmZDQFVmoau7bDsmtWdiJmZDY4+jwgkbU+6ZHS4pA+0PbUJbZeRmpnZmq1b09B2wMHApsBftC1/FvhonUmZmVk5fRaCiLgeuF7S3hHxXwVzMjOzgvo9R+AiYGY2tFW5asjMrKsxk28sGm/xuROLxhvq+jwikHRa/rtPuXTMzKy0bk1DrV5Hv14iETMzGxzdmoYekrQYGCXpvrblAiIidqw1MzMzK6LbVUPHSHoLaajJSeVSMjOzkvobs/hJYCdJ6wLvyIsXRsTLXV5mZmZrkCoD07wH+FdgMalZaLSk4yLitppzMzOzAqpcPnoe8P6IWAgg6R3AlcBudSZmZmZlVOl0bp1WEQCIiJ8B69SXkpmZlVTliGCupEtIQ1YCfAiYV19KZmZWUpVC8AngJOBU0jmC24Bv1pmUmZmV028hiIgXSecJzqs/HTMzK63KOQIzMxvCXAjMzBrOhcDMrOGq3FD2DuBTwFbt60fE/jXmZWZmhVS5auhq4F+AbwHL603HzMxKq1IIlkXERbVnYmZmg6LKOYL/L+lESW+VtFnrUXtmZmZWRJUjguPy30+1LQtgm9WfjpmZlVblhrKtSyRiZmaDo8pVQ+uQupkYnxfNBi72mARmZkNDlaahi0i9jbb6F/pwXvaRupIyM7NyqhSC3SNip7b5WyXdW1dCZmZWVpWrhpZL2rY1I2kbfD+BmdmQUaUQfAqYJWm2pJ8AtwKfrLJxSRMkLZS0SNLkDs//vaQHJd0n6RZJW61a+mZm9lpVuWroFkljge1I4xH8NHdN3ZWkYcCFwJ8BS4A5kqZHxINtq90N9ETEC5I+AXwJOGoA+2FmZgPUZyGQtH9E3CrpA72e2lYSEXFdP9veA1gUEY/k7U0DDgFeLQQRMatt/TuAY1cpezMze826HRG8h9QM9Bcdngugv0KwOfBY2/wSYM8u6x8P/FunJySdAJwAsOWWW/YT1szMVkWfhSAi/ilPTomIX7Q/J6nKTWbqtNmOK0rHAj2k4tMpl6nAVICenp6O2zAzs4GpcrL42g7LrqnwuiXA6Lb5LYAneq8k6QDgM8CkKucezMxs9ep2jmB74J3A8F7nCTYB1q+w7TnA2Hz08DhwNPDBXjF2AS4GJkTEU6uYu5mZrQbdzhFsBxwMbMqK5wmeBT7a34YjYpmkk4GZwDDg0ohYIGkKMDcipgNfBjYCrpYE8GhETBrQnpiZ2YB0O0dwvaQbgNMj4vMD2XhEzABm9Fp2Vtv0AQPZrpmZrT5dzxFExHLSfQBmZjZEVelr6D8lfQP4AfB8a2FEzK8tKzMzK6ZKIXh3/julbVkAHrzerIMxk28sGm/xuROLxrOhp0oXE+8tkYiZmQ2Ofu8jkDRc0nmS5ubHVyQNL5GcmZnVr8oNZZeSLhk9Mj+eAb5TZ1JmZlZOlXME20bEYW3zZ0u6p66EzMysrCpHBL+XtG9rRtI+wO/rS8nMzEqqckTwCeCyfF5AwG+A42rNyszMiqly1dA9wE6SNsnzz9SelZmZFVPlqqERki4AZpOGrPyapBG1Z2ZmZkVUOUcwDVgKHAYcnqd/UGdSZmZWTpVzBJtFxDlt8/8s6dC6EjIzs7KqHBHMknS0pLXy40ig7D30ZmZWmyqF4GPAFcBL+TEN+HtJz0ryiWMzszVclauGNi6RiJmZDY4q5wiQNAkYn2dnR8QN9aVkZmYlVbl89FzgNODB/DgtLzMzsyGgyhHBQcDOEfEKgKTLgLuByXUmZmZmZVQ5WQxpAPsWd0FtZjaEVDki+AJwt6RZpL6GxgNn1JqVmZkV07UQSBJwO7AXsDupEJweEU8WyM3MzAroWggiIiT9KCJ2A6YXysnMzAqqco7gDkm7156JmZkNiirnCN4LfFzSYuB5UvNQRMSOdSZmZmZlVCkEB9aehZmZDZo+C4Gk9YGPA28H7gcuiYhlpRIzM7Myup0juAzoIRWBA4GvFMnIzMyK6tY0NC4i3gUg6RLgrjIpmZlZSd2OCF5uTbhJyMxs6Op2RLBT23gDAjbI862rhjapPTszM6tdn4UgIoaVTMTMzAZH1U7nzMxsiHIhMDNrOBcCM7OGcyEwM2s4FwIzs4artRBImiBpoaRFklYa2lLSeEnzJS2TdHiduZiZWWe1FQJJw4ALSd1TjAOOkTSu12qPAn8FXFFXHmZm1l2V3kcHag9gUUQ8AiBpGnAI8GBrhYhYnJ97pcY8zMysizqbhjYHHmubX5KXrTJJJ0iaK2nu0qVLV0tyZmaW1FkI1GFZDGRDETE1InoiomfUqFGvMS0zM2tXZyFYAoxum98CeKLGeGZmNgB1FoI5wFhJW0taFzgamF5jPDMzG4DaCkHuuvpkYCbwEHBVRCyQNEXSJABJu0taAhwBXCxpQV35mJlZZ3VeNUREzABm9Fp2Vtv0HFKTkZmZDRLfWWxm1nAuBGZmDedCYGbWcC4EZmYN50JgZtZwLgRmZg3nQmBm1nAuBGZmDedCYGbWcC4EZmYN50JgZtZwLgRmZg3nQmBm1nAuBGZmDedCYGbWcC4EZmYN50JgZtZwtY5QZjZYxky+sWi8xedOLBrPbHVyITCzNZqL/mvnpiEzs4ZzITAzazgXAjOzhnMhMDNrOBcCM7OGcyEwM2s4FwIzs4ZzITAzazgXAjOzhnMhMDNrOBcCM7OGcyEwM2s4FwIzs4ZzITAzazgXAjOzhnMhMDNrOBcCM7OGcyEwM2s4D1VptSo5jOBQHELQrIRajwgkTZC0UNIiSZM7PL+epB/k5++UNKbOfMzMbGW1FQJJw4ALgQOBccAxksb1Wu144OmIeDtwPvDFuvIxM7PO6mwa2gNYFBGPAEiaBhwCPNi2ziHA5/L0NcA3JCkiosa8BkXJJhJwM4lZCUPl/7Xq+s6VdDgwISI+kuc/DOwZESe3rfNAXmdJnv95XufXvbZ1AnBCnt0OWFhL0n0bCfy637Uce02P69jNidvE2FtFxKhOT9R5RKAOy3pXnSrrEBFTgamrI6mBkDQ3Inoce2jHdWx/1k2I3UmdJ4uXAKPb5rcAnuhrHUlrA8OB39SYk5mZ9VJnIZgDjJW0taR1gaOB6b3WmQ4cl6cPB24diucHzMxez2prGoqIZZJOBmYCw4BLI2KBpCnA3IiYDlwCfE/SItKRwNF15fMaDVqzVENjN3Gfmxq7ifs82LFXUtvJYjMzWzO4iwkzs4ZzITAzazgXgi4kXSrpqXy/Q8m4oyXNkvSQpAWSTisYe31Jd0m6N8c+u1TsthyGSbpb0g2F4y6WdL+keyTNLRh3U0nXSPpp/sz3LhR3u7yvrcczkv62ROwc/+/yv7EHJF0paf2CsU/LcRfUvc+dvkckbSbpx5Iezn/fWGcO/XEh6O67wIRBiLsM+GRE/AmwF3BSh+456vIisH9E7ATsDEyQtFeh2C2nAQ8Vjtny3ojYufA13l8DboqI7YGdKLTvEbEw7+vOwG7AC8APS8SWtDlwKtATETuQLigpcrGIpB2Aj5J6P9gJOFjS2BpDfpeVv0cmA7dExFjgljw/aFwIuoiI2xiE+xoi4pcRMT9PP0v6Yti8UOyIiOfy7Dr5UeyKAklbABOBb5eKOZgkbQKMJ11BR0S8FBG/HYRU3gf8PCL+u2DMtYEN8j1Eb2Dl+4zq8ifAHRHxQkQsA34C/J+6gvXxPXIIcFmevgw4tK74VbgQvM7lHll3Ae4sGHOYpHuAp4AfR0Sx2MBXgX8EXikYsyWAmyXNy92alLANsBT4Tm4O+7akDQvFbnc0cGWpYBHxOPD/gEeBXwK/i4ibC4V/ABgvaYSkNwAHseLNryW8OSJ+CemHH/CmwvFX4ELwOiZpI+Ba4G8j4plScSNieW4u2ALYIx9K107SwcBTETGvRLwO9omIXUk95p4kaXyBmGsDuwIXRcQuwPMUbibIN3xOAq4uGPONpF/FWwNvAzaUdGyJ2BHxEKmn4x8DNwH3kppjG8uF4HVK0jqkInB5RFw3GDnkJorZlDtPsg8wSdJiYBqwv6TvF4pNRDyR/z5Faivfo0DYJcCStqOua0iFoaQDgfkR8auCMQ8AfhERSyPiZeA64N2lgkfEJRGxa0SMJzXbPFwqdvYrSW8FyH+fKhx/BS4Er0OSRGozfigizisce5SkTfP0BqT/sD8tETsizoiILSJiDKmp4taIKPIrUdKGkjZuTQPvJzUh1CoingQek7RdXvQ+VuyqvYRjKNgslD0K7CXpDfnf+/soeIGApDflv1sCH6D8/rd3r3MccH3h+CvwUJVdSLoS2A8YKWkJ8E8RcUmB0PsAHwbuz231AJ+OiBkFYr8VuCwPLLQWcFVEFL2Mc5C8Gfhh+k5ibeCKiLipUOxTgMtzE80jwF8XiktuI/8z4GOlYgJExJ2SrgHmk5pl7qZstwvXShoBvAycFBFP1xWo0/cIcC5wlaTjSUXxiLriV+EuJszMGs5NQ2ZmDedCYGbWcC4EZmYN50JgZtZwLgRmZg3nQmAASApJX2mb/wdJn1tN2/6upMNXx7b6iXNE7r1zVt2x6iRpZ0kHDeB1V0q6L/fqOUXSAav4+sWSRnZY/ulVzWUgBrrf9tq5EFjLi8AHOn0RDKZ8P0NVxwMnRsR768qnkJ1J/d9UJuktwLsjYseIOD8izoqIf19N+axyIVjFz61llffbVg8XAmtZRrqh5+96P9H7F72k5/Lf/ST9RNJVkn4m6VxJH1Iaz+B+Sdu2beYASf+R1zs4v36YpC9LmpN/yX6sbbuzJF0B3N8hn2Py9h+Q9MW87CxgX+BfJH251/prSfpm7nv+BkkzWvvT/itYUo+k2Xl6Q6V+5OfkzuAOycvfmffvnpzz2LzujUpjODwg6ai87m75/ZknaWZblwKnSnowv35ar1zXBaYAR+UYRyn1Xf+jvP4dknbs8PndDLwpv+ZP2z+zvI9nS5qf37ft8/IRkm7O+3cxoA7v9bmkHkLvkXR5XvajvE8L1NY5n6Tn8pHIncDekg5SGmPhdkkXKI8v0em97bTfHfbR6hIRfvgB8BywCbAYGA78A/C5/Nx3gcPb181/9wN+S7obeT3gceDs/NxpwFfbXn8T6YfHWFL/OusDJwCfzeusB8wldUK2H6nzta075Pk20p2Yo0h3AN8KHJqfm03q3773aw4HZuT4bwGebu1P3t+ReboHmJ2nPw8cm6c3BX4GbAh8HfhQXr4usAFwGPCttnjDSd13/ycwKi87Crg0Tz8BrNfadod8/wr4Rtv810l3tQPsD9zT4TVjgAfa5l/9zPI+npKnTwS+nacvAM7K0xNJva+O7PRvo9f8ZvnvBqRuOEbk+QCOzNPrA4+1PkNSFw439PPerrDffpR7+IjAXhWph9N/JQ0YUtWcSOMnvAj8nPTLFNIv+TFt610VEa9ExMOkbhS2J/Xn85dK3WjcCYwgFQqAuyLiFx3i7U76sl4aqS/5y0n9+XezL3B1jv8kUOUcwvuByTm32aQvti2B/wI+Lel0YKuI+H3e1wMkfVHSn0bE74DtgB2AH+dtfJbUmyvAfaQuJY6lWq+X+wLfA4iIW4ERkoZXeF27VseF8/jj5zIe+H7e7o2kAlnFqZLuBe4gdd/c+syWkzpKhPT5PtL2Gbb35dPXe2uDxH0NWW9fJfX/8p22ZcvIzYiSRPol3PJi2/QrbfOvsOK/r959mQSpKeKUiJjZ/oSk/UhHBJ2s1HxRQbfXvLpvpC+k9tccFhELe63/UG76mAjMlPSRiLhV0m6k9u0vSLqZ1HvpgojoNOzkRNKX8CTgTEnvzEVtVfJf1b5hWp/Lcrp/Ll3lz+YAYO+IeCE3pbXetz9ExPLWqt02Q4f3VtKeq5KLrT4+IrAVRMRvgKtIJ15bFpOGMoTUh/w6A9j0EbmtflvSYCwLgZnAJ5S63EbSO9T/oCx3Au+RNFLphOQxpBGmurkdOCzHfzOp6allMX/ct8Pals8ETsmFD0m75L/bkH7pXkDqQXJHSW8DXoiI75MGW9k1798o5fGHJa2Tzy+sBYyOiFmkAXg2BTbqle+zwMZt87cBH8rb2Q/4daye8Snat3sg0Ne4uS+3PiNSs9fTuQhsTxpKtZOfAtsoDawEqWmspeN7y8r7bYW4EFgnXwHarx76FunL9y5gT/r+td7NQtIX9r8BH4+IP5CGo3wQmK80sPfF9HOUGmk0pzNIzTv3kvrR768L32tJ5yVaMe4EfpefOxv4mqT/IP1abjmHVPDuy7mdk5cfBTyQmzW2JzWlvQu4Ky/7DPDPEfES6dzEF3Mzyj2k/vaHAd+XdD+px83zY+WhKWcB49pOmn4O6JF0H6nXyuNYPc4mjdQ1n9Rc82gf600lvQ+Xk871rJ1zOYfUPLSS3GR2InCTpNuBX/HH97yv97b3flsh7n3UGkHSRhHxnFLXw3eRRiN7crDzGsra3nMBFwIPR8T5g52XrcznCKwpblAacGdd4BwXgSI+Kuk40nt+N+lozF6HfERgZtZwPkdgZtZwLgRmZg3nQmBm1nAuBGZmDedCYGbWcP8L2FhFPDSIBY0AAAAASUVORK5CYII=\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], + "source": [ + "%time show_metric(max, True)" + ] + }, + { + "cell_type": "code", + "execution_count": 33, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "mean = 6.84 ± 0.95, max = 10\n", + "CPU times: user 34 s, sys: 66.9 ms, total: 34.1 s\n", + "Wall time: 34.3 s\n" + ] + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAZkUlEQVR4nO3de7wdVX338c+XcJVLkCTeIBDACI3I9XATGhGpDYSClbviQ31QVK6trSWoUAl9FPURFEVKFJQqELkpKaQEH0ikvFogF8IlxEjEVAIi8RElgAKBX/9Ya+POyT77TA6ZdciZ7/v12q8zM3v2/H6zd7J/e9bMrKWIwMzMmmudwU7AzMwGlwuBmVnDuRCYmTWcC4GZWcO5EJiZNdy6g53A6ho5cmSMGTNmsNMwM1urzJ079zcRMarTc2tdIRgzZgxz5swZ7DTMzNYqkv67r+fcNGRm1nAuBGZmDedCYGbWcC4EZmYN50JgZtZwLgRmZg3nQmBm1nAuBGZmDedCYGbWcGvdncVm1rcxk24uFmvJ+ROLxbJ6+YjAzKzhXAjMzBrOhcDMrOFcCMzMGs6FwMys4VwIzMwazoXAzKzhXAjMzBrOhcDMrOFcCMzMGs6FwMys4VwIzMwazoXAzKzhXAjMzBrOhcDMrOFcCMzMGs6FwMys4VwIzMwazoXAzKzhXAjMzBrOhcDMrOFqLQSSJkhaJGmxpEld1jtSUkjqqTMfMzNbVW2FQNIw4GLgYGAccJykcR3W2xQ4Hbi7rlzMzKxvdR4R7AUsjohHIuIFYCpweIf1zgO+BPyxxlzMzKwPdRaCLYFH2+aX5mWvkLQbMDoibuq2IUknSZojac6yZcvWfKZmZg1WZyFQh2XxypPSOsCFwN/3t6GImBIRPRHRM2rUqDWYopmZ1VkIlgKj2+a3Ah5vm98U2AmYJWkJsA8wzSeMzczKqrMQzAbGStpW0vrAscC01pMR8fuIGBkRYyJiDHAXcFhEzKkxJzMz66W2QhARK4BTgRnAQuCaiFggabKkw+qKa2Zmq2fdOjceEdOB6b2WndPHugfUmYuZmXXmO4vNzBrOhcDMrOFcCMzMGs6FwMys4VwIzMwazoXAzKzhXAjMzBrOhcDMrOFcCMzMGs6FwMys4VwIzMwazoXAzKzhXAjMzBrOhcDMrOFcCMzMGs6FwMys4VwIzMwazoXAzKzh+i0Eko6StGme/qykGyTtXn9qZmZWQpUjgrMjYrmk/YG/BK4ALqk3LTMzK6VKIXgp/50IXBIRNwLr15eSmZmVVKUQPCbpUuBoYLqkDSq+zszM1gJVvtCPBmYAEyLid8AWwKdqzcrMzIqpUggujYgbIuJhgIj4FfChetMyM7NSqhSCt7fPSBoG7FFPOmZmVlqfhUDSWZKWAztLelrS8jz/JHBjsQzNzKxWfRaCiPhCRGwKfDkiNouITfNjREScVTBHMzOrUZWmoc9IOl7S2QCSRkvaq+a8zMyskCqF4GJgX+ADef6ZvMzMzIaAdSuss3dE7C7pXoCIeEqSbygzMxsiqhwRvJivFAoASaOAl2vNyszMiqlSCC4Cfgi8QdL/Ae4EPl9rVmZmVky/TUMRcaWkucB7AAHvi4iFtWdmZmZF9FsIJG1Bunfg6rZl60XEi3UmZmZmZVRpGpoHLAN+Bjycp38haZ4k32FsZraWq1IIbgEOiYiRETECOBi4BjgZ+GadyZmZWf2qFIKeiJjRmomIW4HxEXEXsEG3F0qaIGmRpMWSJnV4/uOSHpA0X9Kdksat9h6YmdmrUqUQ/FbSmZK2yY9/BJ7Kl5T2eRlpfv5i0hHEOOC4Dl/0V0XEOyJiV+BLwAUD2w0zMxuoKoXgA8BWwI/yY3ReNow0VkFf9gIWR8QjEfECMBU4vH2FiHi6bXZj8r0KZmZWTterhvKv+jMj4rQ+Vlnc5eVbAo+2zS8F9u4Q4xTgk6ThLw/sI4+TgJMAtt56624pm5nZaup6RBARLzHwsQfUaZMdYlwcEdsDZwKf7SOPKRHRExE9o0aNGmA6ZmbWSZW+hu6VNA24Fni2tTAibujndUtJzUgtWwGPd1l/KnBJhXzMzGwNqlIItgD+Pys32wTQXyGYDYyVtC3wGHAsf+rBFABJY1tDYAITSfcpmJlZQVW6mPjwQDYcESsknUoa+H4YcHlELJA0GZgTEdOAUyUdBLwIPAWcMJBYZmY2cFW6mNgQOJE0dvGGreUR8b/7e21ETAem91p2Ttv0GauTrJmZrXlVLh/9HvAm4C+Bn5Da+pfXmZSZmZVTpRC8NSLOBp6NiCtIbfnvqDctMzMrpdLANPnv7yTtBAwHxtSWkZmZFVXlqqEpkl5PusZ/GrAJcHatWZmZWTFVCsFtEfEUcAewHUC+JNTMzIaAKk1D13dYdt2aTsTMzAZHn0cEknYkXTI6XNL7257ajLbLSM3MbO3WrWloB+BQYHPgr9qWLwc+WmdSZmZWTp+FICJuBG6UtG9E/FfBnMzMrKB+zxG4CJiZDW1VThabmdkQ1mchkHRG/rtfuXTMzKy0bkcErV5Hv14iETMzGxzdrhpaKGkJMErS/W3LBURE7FxrZmZmVkS3q4aOk/Qm0ngCh5VLyWztNmbSzUXjLTl/YtF4NvR07WIiIp4AdpG0PvC2vHhRRLzY5WVmZrYWqTIwzbuAfwWWkJqFRks6ISLuqDk3MzMroEqncxcA742IRQCS3gZcDexRZ2JmZlZGlfsI1msVAYCI+BmwXn0pmZlZSVWOCOZIuow0ZCXAB4G59aVkZmYlVSkEnwBOAU4nnSO4A/hmnUmZmVk5/RaCiHiedJ7ggvrTMTOz0tzXkJlZw7kQmJk1nAuBmVnDVbmh7G3Ap4Bt2tePiANrzMvMzAqpctXQtcC/AN8CXqo3HTMzK61KIVgREZfUnomZmQ2KKucI/k3SyZLeLGmL1qP2zMzMrIgqRwQn5L+falsWwHZrPh0zMyutyg1l25ZIxMzMBkeVq4bWI3UzMT4vmgVc6jEJzMyGhipNQ5eQehtt9S/0obzsI3UlZWZm5VQpBHtGxC5t87dLuq+uhMzMrKwqVw29JGn71oyk7fD9BGZmQ0aVI4JPATMlPULqhnob4MO1ZmVmZsVUuWroNkljgR1IheCnuWtqMzMbAvpsGpJ0YP77fmAi8FZge2BiXtYvSRMkLZK0WNKkDs9/UtJDku6XdJukbQa2G2ZmNlDdjgjeBdwO/FWH5wK4oduGJQ0DLgb+AlgKzJY0LSIealvtXqAnIp6T9AngS8Axq5G/mZm9Sn0Wgoj4pzw5OSJ+0f6cpCo3me0FLI6IR/JrpgKHA68UgoiY2bb+XcDxFfM2M7M1pMpVQ9d3WHZdhddtCTzaNr80L+vLicC/d3pC0kmS5kias2zZsgqhzcysqj6PCCTtCLwdGN7rnMBmwIYVtq0Oy6KPWMcDPaTmqFVfFDEFmALQ09PTcRtmZjYw3c4R7AAcCmzOyucJlgMfrbDtpcDotvmtgMd7ryTpIOAzwLt8NZLZ2mnMpJuLxlty/sSi8Ya6bucIbpR0E3BmRHx+ANueDYzN5xMeA44FPtC+gqTdgEuBCRHx5ABimJnZq9T1HEFEvES66me1RcQK4FRgBrAQuCYiFkiaLOmwvNqXgU2AayXNlzRtILHMzGzgqtxZ/J+SvgH8AHi2tTAi5vX3woiYDkzvteyctumDqqdqZmZ1qFII3pn/Tm5bFoAHrzczGwKqdDHx7hKJmJnZ4Oj3PgJJwyVd0LqOX9JXJA0vkZyZmdWvyg1ll5MuGT06P54GvlNnUmZmVk6VcwTbR8QRbfPnSppfV0JmZlZWlSOCP0javzUjaT/gD/WlZGZmJVU5IvgEcEU+LyDgt8AJtWZlZmbFVLlqaD6wi6TN8vzTtWdlZmbFVLlqaISki4BZpCErvyZpRO2ZmZlZEVXOEUwFlgFHAEfm6R/UmZSZmZVT5RzBFhFxXtv8P0t6X10JmZlZWVWOCGZKOlbSOvlxNFC2z1kzM6tNlULwMeAq4IX8mAp8UtJyST5xbGa2lqty1dCmJRIxM7PBUeUcAXn8gPF5dlZE3FRfSmZmVlKVy0fPB84AHsqPM/IyMzMbAqocERwC7BoRLwNIugK4F5hUZ2JmZlZGlZPFkAawb3EX1GZmQ0iVI4IvAPdKmknqa2g8cFatWZmZWTFdC4EkAXcC+wB7kgrBmRHxRIHczMysgK6FICJC0o8iYg9gWqGczMysoCrnCO6StGftmZiZ2aCoco7g3cDHJS0BniU1D0VE7FxnYmZmVkaVQnBw7VmYmdmg6bMQSNoQ+DjwVuAB4LKIWFEqMTMzK6PbOYIrgB5SETgY+EqRjMzMrKhuTUPjIuIdAJIuA+4pk5KZmZXU7YjgxdaEm4TMzIaubkcEu7SNNyBgozzfumpos9qzMzOz2vVZCCJiWMlEzMxscFTtdM7MzIYoFwIzs4ZzITAzazgXAjOzhnMhMDNrOBcCM7OGq7UQSJogaZGkxZJWGeNY0nhJ8yStkHRknbmYmVlntRUCScOAi0n9FI0DjpM0rtdqvwT+BriqrjzMzKy7Kt1QD9RewOKIeARA0lTgcOCh1goRsSQ/93KNeZiZWRd1Ng1tCTzaNr80L1ttkk6SNEfSnGXLlq2R5MzMLKmzEKjDshjIhiJiSkT0RETPqFGjXmVaZmbWrs5CsBQY3Ta/FfB4jfHMzGwA6iwEs4GxkraVtD5wLDCtxnhmZjYAtRWCPIbBqcAMYCFwTUQskDRZ0mEAkvaUtBQ4CrhU0oK68jEzs87qvGqIiJgOTO+17Jy26dmkJiMzMxskvrPYzKzhXAjMzBrOhcDMrOFcCMzMGs6FwMys4VwIzMwazoXAzKzhXAjMzBrOhcDMrOFcCMzMGs6FwMys4VwIzMwazoXAzKzhXAjMzBrOhcDMrOFqHY/AbLCMmXRz0XhLzp9YNJ7ZmuQjAjOzhnMhMDNrOBcCM7OGcyEwM2s4FwIzs4ZzITAzazgXAjOzhnMhMDNrOBcCM7OGcyEwM2s4FwIzs4ZzITAzazh3OmdmazV3MPjq+YjAzKzhXAjMzBrOhcDMrOFcCMzMGs6FwMys4VwIzMwazoXAzKzhfB+B1arkNd5D8fpusxJqPSKQNEHSIkmLJU3q8PwGkn6Qn79b0pg68zEzs1XVVggkDQMuBg4GxgHHSRrXa7UTgaci4q3AhcAX68rHzMw6q7NpaC9gcUQ8AiBpKnA48FDbOocDn8vT1wHfkKSIiBrzGhSDeRu8m2fMrBvV9Z0r6UhgQkR8JM9/CNg7Ik5tW+fBvM7SPP/zvM5vem3rJOCkPLsDsKiWpPs2EvhNv2s59toe17GbE7eJsbeJiFGdnqjziEAdlvWuOlXWISKmAFPWRFIDIWlORPQ49tCO69j+rJsQu5M6TxYvBUa3zW8FPN7XOpLWBYYDv60xJzMz66XOQjAbGCtpW0nrA8cC03qtMw04IU8fCdw+FM8PmJm9ltXWNBQRKySdCswAhgGXR8QCSZOBORExDbgM+J6kxaQjgWPryudVGrRmqYbGbuI+NzV2E/d5sGOvoraTxWZmtnZwFxNmZg3nQmBm1nAuBF1IulzSk/l+h5JxR0uaKWmhpAWSzigYe0NJ90i6L8c+t1TsthyGSbpX0k2F4y6R9ICk+ZLmFIy7uaTrJP00f+b7Foq7Q97X1uNpSX9bInaO/3f539iDkq6WtGHB2GfkuAvq3udO3yOStpD0Y0kP57+vrzOH/rgQdPddYMIgxF0B/H1E/BmwD3BKh+456vI8cGBE7ALsCkyQtE+h2C1nAAsLx2x5d0TsWvga768Bt0TEjsAuFNr3iFiU93VXYA/gOeCHJWJL2hI4HeiJiJ1IF5QUuVhE0k7AR0m9H+wCHCppbI0hv8uq3yOTgNsiYixwW54fNC4EXUTEHQzCfQ0R8auImJenl5O+GLYsFDsi4pk8u15+FLuiQNJWwETg26ViDiZJmwHjSVfQEREvRMTvBiGV9wA/j4j/LhhzXWCjfA/R61j1PqO6/BlwV0Q8FxErgJ8Af11XsD6+Rw4HrsjTVwDvqyt+FS4Er3G5R9bdgLsLxhwmaT7wJPDjiCgWG/gq8I/AywVjtgRwq6S5uVuTErYDlgHfyc1h35a0caHY7Y4Fri4VLCIeA/4v8EvgV8DvI+LWQuEfBMZLGiHpdcAhrHzzawlvjIhfQfrhB7yhcPyVuBC8hknaBLge+NuIeLpU3Ih4KTcXbAXslQ+layfpUODJiJhbIl4H+0XE7qQec0+RNL5AzHWB3YFLImI34FkKNxPkGz4PA64tGPP1pF/F2wJvATaWdHyJ2BGxkNTT8Y+BW4D7SM2xjeVC8BolaT1SEbgyIm4YjBxyE8Usyp0n2Q84TNISYCpwoKTvF4pNRDye/z5Jaivfq0DYpcDStqOu60iFoaSDgXkR8euCMQ8CfhERyyLiReAG4J2lgkfEZRGxe0SMJzXbPFwqdvZrSW8GyH+fLBx/JS4Er0GSRGozXhgRFxSOPUrS5nl6I9J/2J+WiB0RZ0XEVhExhtRUcXtEFPmVKGljSZu2poH3kpoQahURTwCPStohL3oPK3fVXsJxFGwWyn4J7CPpdfnf+3soeIGApDfkv1sD76f8/rd3r3MCcGPh+CvxUJVdSLoaOAAYKWkp8E8RcVmB0PsBHwIeyG31AJ+OiOkFYr8ZuCIPLLQOcE1EFL2Mc5C8Efhh+k5iXeCqiLilUOzTgCtzE80jwIcLxSW3kf8F8LFSMQEi4m5J1wHzSM0y91K224XrJY0AXgROiYin6grU6XsEOB+4RtKJpKJ4VF3xq3AXE2ZmDeemITOzhnMhMDNrOBcCM7OGcyEwM2s4FwIzs4ZzITAAJIWkr7TN/4Okz62hbX9X0pFrYlv9xDkq9945s+5YdZK0q6RDBvC6qyXdn3v1nCzpoNV8/RJJIzss//Tq5jIQA91ve/VcCKzleeD9nb4IBlO+n6GqE4GTI+LddeVTyK6k/m8qk/Qm4J0RsXNEXBgR50TE/1tD+ax2IVjNz61ltffb1gwXAmtZQbqh5+96P9H7F72kZ/LfAyT9RNI1kn4m6XxJH1Qaz+ABSdu3beYgSf+R1zs0v36YpC9Lmp1/yX6sbbszJV0FPNAhn+Py9h+U9MW87Bxgf+BfJH251/rrSPpm7nv+JknTW/vT/itYUo+kWXl6Y6V+5GfnzuAOz8vfnvdvfs55bF73ZqUxHB6UdExed4/8/syVNKOtS4HTJT2UXz+1V67rA5OBY3KMY5T6rv9RXv8uSTt3+PxuBd6QX/Pn7Z9Z3sdzJc3L79uOefkISbfm/bsUUIf3+nxSD6HzJV2Zl/0o79MCtXXOJ+mZfCRyN7CvpEOUxli4U9JFyuNLdHpvO+13h320ukSEH34APANsBiwBhgP/AHwuP/dd4Mj2dfPfA4Dfke5G3gB4DDg3P3cG8NW2199C+uExltS/zobAScBn8zobAHNInZAdQOp8bdsOeb6FdCfmKNIdwLcD78vPzSL1b9/7NUcC03P8NwFPtfYn7+/IPN0DzMrTnweOz9ObAz8DNga+DnwwL18f2Ag4AvhWW7zhpO67/xMYlZcdA1yepx8HNmhtu0O+fwN8o23+66S72gEOBOZ3eM0Y4MG2+Vc+s7yPp+Xpk4Fv5+mLgHPy9ERS76sjO/3b6DW/Rf67EakbjhF5PoCj8/SGwKOtz5DUhcNN/by3K+23H+UePiKwV0Tq4fRfSQOGVDU70vgJzwM/J/0yhfRLfkzbetdExMsR8TCpG4UdSf35/C+lbjTuBkaQCgXAPRHxiw7x9iR9WS+L1Jf8laT+/LvZH7g2x38CqHIO4b3ApJzbLNIX29bAfwGflnQmsE1E/CHv60GSvijpzyPi98AOwE7Aj/M2PkvqzRXgflKXEsdTrdfL/YHvAUTE7cAIScMrvK5dq+PCufzpcxkPfD9v92ZSgazidEn3AXeRum9ufWYvkTpKhPT5PtL2Gbb35dPXe2uDxH0NWW9fJfX/8p22ZSvIzYiSRPol3PJ82/TLbfMvs/K/r959mQSpKeK0iJjR/oSkA0hHBJ2s0nxRQbfXvLJvpC+k9tccERGLeq2/MDd9TARmSPpIRNwuaQ9S+/YXJN1K6r10QUR0GnZyIulL+DDgbElvz0VtdfJf3b5hWp/LS3T/XLrKn81BwL4R8VxuSmu9b3+MiJdaq3bbDB3eW0l7r04utub4iMBWEhG/Ba4hnXhtWUIayhBSH/LrDWDTR+W2+u1Jg7EsAmYAn1DqchtJb1P/g7LcDbxL0kilE5LHkUaY6uZO4Igc/42kpqeWJfxp345oWz4DOC0XPiTtlv9uR/qlexGpB8mdJb0FeC4ivk8abGX3vH+jlMcflrRePr+wDjA6ImaSBuDZHNikV77LgU3b5u8APpi3cwDwm1gz41O0b/dgoK9xc19sfUakZq+nchHYkTSUaic/BbZTGlgJUtNYS8f3llX32wpxIbBOvgK0Xz30LdKX7z3A3vT9a72bRaQv7H8HPh4RfyQNR/kQME9pYO9L6ecoNdJoTmeRmnfuI/Wj318XvteTzku0YtwN/D4/dy7wNUn/Qfq13HIeqeDdn3M7Ly8/BngwN2vsSGpKewdwT172GeCfI+IF0rmJL+ZmlPmk/vaHAd+X9ACpx80LY9WhKWcC49pOmn4O6JF0P6nXyhNYM84ljdQ1j9Rc88s+1ptCeh+uJJ3rWTfnch6peWgVucnsZOAWSXcCv+ZP73lf723v/bZC3PuoNYKkTSLiGaWuh+8hjUb2xGDnNZS1vecCLgYejogLBzsvW5XPEVhT3KQ04M76wHkuAkV8VNIJpPf8XtLRmL0G+YjAzKzhfI7AzKzhXAjMzBrOhcDMrOFcCMzMGs6FwMys4f4HX0w5FRT/m8sAAAAASUVORK5CYII=\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], + "source": [ + "%time show_metric(expectation, True)" + ] + }, + { + "cell_type": "code", + "execution_count": 34, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "mean = 6.81 ± 1.00, max = 10\n", + "CPU times: user 34.3 s, sys: 20.8 ms, total: 34.3 s\n", + "Wall time: 34.3 s\n" + ] + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEGCAYAAAB/+QKOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAf5UlEQVR4nO3de5xdVX338c/XcLPInbFWCCZgAEEFZAxaFBG5BLHER1GC0idtqRQFobW1QFWUUBX18dqikkoqVTFy8ZIHoxEFtLwskAEiEDASQgoxWKYGBQWBhG//2GvwZLJnZiedfWZIvu/X67zO3mvvtddvn5Oc3+zbWrJNRETEYM8Y6wAiImJ8SoKIiIhaSRAREVErCSIiImolQURERK3NxjqA0bLzzjt70qRJYx1GRMTTyk033fTftnvqlm00CWLSpEn09fWNdRgREU8rkv5zqGU5xRQREbWSICIiolYSRERE1EqCiIiIWkkQERFRKwkiIiJqJUFEREStJIiIiKiVBBEREbU2miepI2Jok876dlfbW37+MV1tL9qRI4iIiKiVBBEREbWSICIiolYSRERE1EqCiIiIWkkQERFRKwkiIiJqJUFEREStJIiIiKjVaoKQNE3SEklLJZ01zHrHSbKk3o6ys0u9JZKOajPOiIhYV2tdbUiaAFwAHAGsABZKmmf7jkHrbQOcDtzQUbYPMAPYF3gu8H1Je9pe01a8ERGxtjaPIKYCS20vs/04MBeYXrPeecBHgd91lE0H5tp+zPY9wNKyvYiI6JI2E8QuwH0d8ytK2VMkHQBMtH3l+tYt9U+W1Cepr7+/f3SijogIoN0EoZoyP7VQegbwSeBv17fuUwX2bNu9tnt7eno2ONCIiFhXm919rwAmdszvCqzsmN8GeCFwrSSA5wDzJB3boG5ERLSszSOIhcAUSZMlbUF10XnewELbv7a9s+1JticB1wPH2u4r682QtKWkycAU4MYWY42IiEFaO4KwvVrSacACYAIwx/ZiSbOAPtvzhqm7WNKlwB3AauDU3MEUEdFdrY4oZ3s+MH9Q2TlDrHvooPkPAh9sLbiIiBhWnqSOiIhaSRAREVErCSIiImolQURERK0kiIiIqJUEERERtZIgIiKiVhJERETUSoKIiIhaSRAREVErCSIiImolQURERK0kiIiIqJUEERERtZIgIiKiVqsJQtI0SUskLZV0Vs3yUyTdJmmRpOsk7VPKJ0l6tJQvkvT5NuOMiIh1tTZgkKQJwAXAEVRjTC+UNM/2HR2rXWL782X9Y4FPANPKsrtt799WfBERMbw2jyCmAkttL7P9ODAXmN65gu2HOma3BtxiPBERsR7aTBC7APd1zK8oZWuRdKqku4GPAqd3LJos6RZJP5T0yhbjjIiIGm0mCNWUrXOEYPsC23sAZwLvLcX3A7vZPgB4F3CJpG3XaUA6WVKfpL7+/v5RDD0iItpMECuAiR3zuwIrh1l/LvB6ANuP2f5lmb4JuBvYc3AF27Nt99ru7enpGbXAIyKi3QSxEJgiabKkLYAZwLzOFSRN6Zg9BrirlPeUi9xI2h2YAixrMdaIiBiktbuYbK+WdBqwAJgAzLG9WNIsoM/2POA0SYcDTwAPAjNL9UOAWZJWA2uAU2yvaivWiIhYV2sJAsD2fGD+oLJzOqbPGKLeFcAVbcYWERHDy5PUERFRKwkiIiJqJUFEREStJIiIiKiVBBEREbWSICIiotaICULSmyRtU6bfK+nrkl7SfmgRETGWmhxBvM/2w5JeARwFXAx8rt2wIiJirDVJEGvK+zHA52x/C9iivZAiImI8aJIgfi7pQuDNwHxJWzasFxERT2NNfujfTNWf0jTbvwJ2BN7dalQRETHmmiSIC21/3fZdALbvB/603bAiImKsNUkQ+3bOlG64D2wnnIiIGC+GTBCSzpb0MPBiSQ9JerjMPwB8q2sRRkTEmBgyQdj+sO1tgI/Z3tb2NuW1k+2zuxhjRESMgSanmN4j6URJ7wOQNFHS1JbjioiIMdYkQVwAvBx4S5n/TSkbkaRpkpZIWirprJrlp0i6TdIiSddJ2qdj2dml3hJJRzVpLyIiRk+TBHGQ7VOB3wHYfpAGD8qVi9kXAEcD+wAndCaA4hLbL7K9P/BR4BOl7j5UY1jvC0wDPjswRnVERHRHkwTxRPlxNoCkHuDJBvWmAkttL7P9ODAXmN65gu2HOma3HmijrDfX9mO27wGWlu1FRESXNEkQnwG+ATxb0geB64APNai3C3Bfx/yKUrYWSadKupvqCOL09ax7sqQ+SX39/f0NQoqIiKZGTBC2vwL8PfBh4H7g9bYva7Bt1W2uZvsX2N4DOBN473rWnW2713ZvT09Pg5AiIqKpzUZaQdKOVM8+fLWjbHPbT4xQdQUwsWN+V2DlMOvP5fe9xK5v3YiIGGVNTjHdDPQDPwPuKtP3SLpZ0nBPVC8EpkiaLGkLqovO8zpXkDSlY/aYsn3KejMkbSlpMjAFuLHJDkVExOgY8QgC+C7wDdsLACQdSXVn0aXAZ4GD6irZXi3pNKqO/iYAc2wvljQL6LM9DzhN0uHAE8CDwMxSd7GkS4E7gNXAqbbX1LUTERHtaJIgem2fMjBj+3uSPmT7XaXr7yHZng/MH1R2Tsf0GcPU/SDwwQbxRUREC5okiFWSzqS6RgBwPPBgufW1ye2uERHxNNTkGsRbqC4Sf7O8JpayCVRjRURExEZo2COIcpRwpu13DrHK0tEPKSIixoNhjyDKheGM/RARsQlqcg3iFknzgMuA3w4U2v56a1FFRMSYa5IgdgR+CRzWUWYgCSIiYiM2YoKw/efdCCQiIsaXJl1tbAWcRNX19lYD5bb/osW4IiJijDW5zfVLwHOAo4AfUt3y+nCbQUVExNhrkiCeb/t9wG9tX0zVZ9KL2g0rIiLGWpOL1AO9tv5K0guBXwCTWosoYiM16axvd7W95ecf09X2YuPTJEHMlrQD1VgN84BnAe9rNaqIiBhzTRLED8o41D8CdgcoXXBHRMRGrMk1iCtqyi4f7UAiImJ8GfIIQtLeVLe2bifpDR2LtqXjdteIiNg4DXeKaS/gdcD2wJ90lD8MvK3JxiVNAz5N1fPrF2yfP2j5u4C/pBoUqB/4C9v/WZatAW4rq95r+9gmbUZExOgYMkHY/hbwLUkvt/0f67vh0hPsBcARVGNML5Q0z/YdHavdQjUg0SOS3g58lGq8CYBHbe+/vu1GRMToGPEaxIYkh2IqsNT2MtuPUw04NH3Qtq+x/UiZvZ7qIbyIiBgHmlyk3lC7APd1zK8oZUM5CfhOx/xWkvokXS/p9XUVJJ1c1unr7+//30ccERFPGTJBSDqjvB+8gdtWTZmHaOtEoBf4WEfxbrZ7qUav+5SkPdbZmD3bdq/t3p6eng0MMyIi6gx3BDHQi+s/beC2V1ANTzpgV2Dl4JUkHQ68BzjW9mMD5bZXlvdlwLXAARsYR0REbIDh7mK6U9JyoEfSrR3lAmz7xSNseyEwpTxU93NgBtXRwO83JB0AXAhMs/1AR/kOwCO2H5O0M3Aw1QXsiIjokuHuYjpB0nOABcB632Jqe7Wk00r9CcAc24slzQL6bM+jOqX0LOAySfD721lfAFwo6Umqo5zzB939FBERLRu2qw3bvwD2k7QFsGcpXmL7iWGqddafD8wfVHZOx/ThQ9T7MekxNiJiTDUZMOhVwL8By6lOL02UNNP2j1qOLSIixlCTzvo+ARxpewmApD2BrwIHthlYRESMrSbPQWw+kBwAbP8M2Ly9kCIiYjxocgTRJ+kiqqFHAd4K3NReSBERMR40SRBvB04FTqe6BvEj4LNtBhUREWNvxARRHl77RHlFRMQmos2+mCIi4mksCSIiImolQURERK0mD8rtCbwbeF7n+rYPazGuiIgYY03uYroM+DzwL8CadsOJiIjxokmCWG37c61HEhER40qTaxD/X9I7JP2RpB0HXq1HFhERY6rJEcTM8v7ujjIDu49+OBERMV40eVBucjcCiYiI8WXEU0ySNpd0uqTLy+s0SY0665M0TdISSUslnVWz/F2S7pB0q6QfSHpex7KZku4qr5mD60ZERLuaXIP4HFXX3p8trwNL2bAkTQAuAI4G9gFOkLTPoNVuAXrL8KWXU4YVLdc43g8cBEwF3l+GIY2IiC5pcg3ipbb365i/WtJPGtSbCiy1vQxA0lxgOvDU0KG2r+lY/3rgxDJ9FHCV7VWl7lXANKpxKCIioguaHEGskbTHwIyk3Wn2PMQuwH0d8ytK2VBOAr6zPnUlnSypT1Jff39/g5AiIqKpJkcQ7waukbSMqrvv5wF/3qCeaspcu6J0ItALvGp96tqeDcwG6O3trd12RERsmCZ3Mf1A0hRgL6of7p+WLsBHsgKY2DG/K7By8EqSDgfeA7yqY7srgEMH1b22QZsRETFKhkwQkg6zfbWkNwxatIckbH99hG0vBKZImgz8HJgBvGVQGwcAFwLTbD/QsWgB8KGOC9NHAmePvDsRETFahjuCeBVwNfAnNcsMDJsgbK+WdBrVj/0EYI7txZJmAX225wEfA54FXCYJ4F7bx9peJek8qiQDMGvggnVEPL1MOuvbXW1v+fnHdLW9jdmQCcL2+8vkLNv3dC4rRwUjsj0fmD+o7JyO6cOHqTsHmNOknYiIGH1N7mK6oqbs8tEOJCIixpfhrkHsDewLbDfoOsS2wFZtBxYREWNruGsQewGvA7Zn7esQDwNvazOoiIgYe8Ndg/iWpCuBM21/qIsxRUTEODDsNQjba4AjuhRLRESMI02epP6xpH8Gvgb8dqDQ9s2tRRUREWOuSYL44/I+q6PMwGGjH05ERIwXTbraeHU3AomIiPGlyYBB20n6xECvqZI+Lmm7bgQXERFjp8mDcnOobm19c3k9BPxrm0FFRMTYa3INYg/bb+yYP1fSorYCioiI8aHJEcSjkl4xMCPpYODR9kKKiIjxoMkRxNuBi8t1BwGrgJmtRhUREWOuyV1Mi4D9JG1b5h9qPaqIiBhzTe5i2knSZ6hGdLtG0qcl7dR6ZBERMaaaXIOYC/QDbwSOK9Nfa7JxSdMkLZG0VNJZNcsPkXSzpNWSjhu0bI2kReU1r0l7ERExeppcg9jR9nkd8/8o6fUjVZI0AbiAqi+nFcBCSfNs39Gx2r3AnwF/V7OJR23v3yC+iIhoQZMjiGskzZD0jPJ6M9BkDMGpwFLby2w/TnUkMr1zBdvLbd8KPLnekUdERKuaJIi/Ai4BHi+vucC7JD0sabgL1rsA93XMryhlTW1Vnty+fqgjFkknDzzh3d/fvx6bjoiIkTS5i2mbDdy26ja3HvV3s71S0u7A1ZJus333oNhmA7MBent712fbERExgibXIJB0LHBImb3W9pUNqq0AJnbM7wqsbBqY7ZXlfZmka4EDgLuHrRQREaOmyW2u5wNnAHeU1xmlbCQLgSmSJkvaApgBNLobSdIOkrYs0zsDB5e2IyKiS5ocQbwW2N/2kwCSLgZuAda5bbWT7dWSTgMWABOAObYXS5oF9NmeJ+mlwDeAHYA/kXSu7X2BFwAXSnqSKomdP+jup4iIaFmjU0zA9lRdbAA07urb9nxg/qCyczqmF1Kdehpc78fAi5q2ExERo69JgvgwcIuka6guPB8CnN1qVBERMeaGTRCSBFwHvAx4KVWCONP2L7oQW0REjKFhE4RtS/qm7QNpeIE5IiI2Dk0elLu+XEyOiIhNSJNrEK8GTpG0HPgt1Wkm235xm4FFRMTYapIgjm49ioiIGHeGTBCStgJOAZ4P3AZcZHt1twKLiIixNdw1iIuBXqrkcDTw8a5EFBER48Jwp5j2sf0iAEkXATd2J6SIiBgPhjuCeGJgIqeWIiI2PcMdQezXMd6DgGeW+YG7mLZtPbqIiBgzQyYI2xO6GUhERIwvTR6Ui4iITVASRERE1EqCiIiIWq0mCEnTJC2RtFTSOgMMSTpE0s2SVks6btCymZLuKq+ZbcYZERHrai1BSJoAXED1kN0+wAmS9hm02r3AnwGXDKq7I/B+4CBgKvB+STu0FWtERKyrzSOIqcBS28tsPw7MBaZ3rmB7ue1bgScH1T0KuMr2KtsPAlcB01qMNSIiBmkzQewC3Ncxv6KUjVpdSSdL6pPU19/fv8GBRkTEutpMEKop82jWtT3bdq/t3p6envUKLiIihtdmglgBTOyY3xVY2YW6ERExCtpMEAuBKZImS9oCmEHzYUsXAEdK2qFcnD6ylEVERJe0liBKB3+nUf2w3wlcanuxpFmSjgWQ9FJJK4A3ARdKWlzqrgLOo0oyC4FZpSwiIrqkyYhyG8z2fGD+oLJzOqYXUp0+qqs7B5jTZnwRETG0PEkdERG1kiAiIqJWEkRERNRKgoiIiFpJEBERUSsJIiIiaiVBRERErSSIiIiolQQRERG1kiAiIqJWq11tRIxHk876dtfaWn7+MV1rK2K05QgiIiJqJUFEREStJIiIiKiVBBEREbVaTRCSpklaImmppLNqlm8p6Wtl+Q2SJpXySZIelbSovD7fZpwREbGu1u5ikjQBuAA4gmqM6YWS5tm+o2O1k4AHbT9f0gzgI8DxZdndtvdvK76IiBhem0cQU4GltpfZfhyYC0wftM504OIyfTnwGklqMaaIiGiozQSxC3Bfx/yKUla7ThnD+tfATmXZZEm3SPqhpFe2GGdERNRo80G5uiMBN1znfmA327+UdCDwTUn72n5orcrSycDJALvtttsohBwREQPaPIJYAUzsmN8VWDnUOpI2A7YDVtl+zPYvAWzfBNwN7Dm4Aduzbffa7u3p6WlhFyIiNl1tJoiFwBRJkyVtAcwA5g1aZx4ws0wfB1xt25J6ykVuJO0OTAGWtRhrREQM0topJturJZ0GLAAmAHNsL5Y0C+izPQ+4CPiSpKXAKqokAnAIMEvSamANcIrtVW3FGhER62q1sz7b84H5g8rO6Zj+HfCmmnpXAFe0GVtERAwvT1JHREStJIiIiKiVBBEREbWSICIiolZGlIuIjVZGD/zfyRFERETUSoKIiIhaSRAREVErCSIiImolQURERK0kiIiIqJUEERERtZIgIiKiVhJERETUypPUMSa6+YQrbJxPuUa0LUcQERFRq9UEIWmapCWSlko6q2b5lpK+VpbfIGlSx7KzS/kSSUe1GWdERKyrtVNMZUzpC4AjgBXAQknzbN/RsdpJwIO2ny9pBvAR4HhJ+1ANP7ov8Fzg+5L2tL2mrXgjIkbLxnIKtc1rEFOBpbaXAUiaC0wHOhPEdOADZfpy4J8lqZTPtf0YcE8Zs3oq8B8txjsmxvIf0sbyjzgi2iHb7WxYOg6YZvsvy/yfAgfZPq1jndvLOivK/N3AQVRJ43rbXy7lFwHfsX35oDZOBk4us3sBS1rZmeHtDPz3JtTuptr2prjPm2rbm9o+P892T92CNo8gVFM2OBsNtU6TutieDcxe/9BGj6Q+272bSrubatub4j5vqm1vivs8lDYvUq8AJnbM7wqsHGodSZsB2wGrGtaNiIgWtZkgFgJTJE2WtAXVRed5g9aZB8ws08cBV7s65zUPmFHucpoMTAFubDHWiIgYpLVTTLZXSzoNWABMAObYXixpFtBnex5wEfClchF6FVUSoax3KdUF7dXAqeP4DqaxOsU1lqfWNsW2N8V93lTb3hT3uVZrF6kjIuLpLU9SR0RErSSIiIiolQSxASTNkfRAeY6j221PlHSNpDslLZZ0Rhfb3krSjZJ+Uto+t1ttl/YnSLpF0pVdbne5pNskLZLU1+W2t5d0uaSflu/85V1qd6+yvwOvhyT9dZfa/pvy7+t2SV+VtFU32i1tn1HaXdz2/tb9jkjaUdJVku4q7zu0GcNIkiA2zBeBaWPU9mrgb22/AHgZcGrpmqQbHgMOs70fsD8wTdLLutQ2wBnAnV1sr9Orbe8/Bveofxr4ru29gf3o0v7bXlL2d3/gQOAR4BtttytpF+B0oNf2C6lucJnRdrul7RcCb6PqtWE/4HWSprTY5BdZ93fkLOAHtqcAPyjzYyYJYgPY/hHVXVdj0fb9tm8u0w9T/WDs0qW2bfs3ZXbz8urKXQ6SdgWOAb7QjfbGA0nbAodQ3e2H7cdt/2oMQnkNcLft/+xSe5sBzyzPRv0B3XsG6gVUPTg8Yns18EPg/7TV2BC/I9OBi8v0xcDr22q/iSSIp7HS++0BwA1dbHOCpEXAA8BVtrvV9qeAvwee7FJ7nQx8T9JNpXuXbtkd6Af+tZxa+4KkrbvY/oAZwFe70ZDtnwP/D7gXuB/4te3vdaNt4HbgEEk7SfoD4LWs/cBuN/yh7fuh+mMQeHaX219LEsTTlKRnAVcAf237oW61a3tNOe2wKzC1HJa3StLrgAds39R2W0M42PZLgKOpTukd0qV2NwNeAnzO9gHAb+nyKYfykOuxwGVdam8Hqr+iJ1P15Ly1pBO70bbtO6l6lL4K+C7wE6pTupusJIinIUmbUyWHr9j++ljEUE51XEt3rsUcDBwraTkwFzhM0pe70C4AtleW9weozsNP7VLTK4AVHUdpl1MljG46GrjZ9n91qb3DgXts99t+Avg68MddahvbF9l+ie1DqE7/3NWttov/kvRHAOX9gS63v5YkiKeZ0h36RcCdtj/R5bZ7JG1fpp9J9Z/5p223a/ts27vankR1uuNq2135q1LS1pK2GZgGjqQ6FdE6278A7pO0Vyl6DWt3l98NJ9Cl00vFvcDLJP1B+bf+Grp4Y4KkZ5f33YA30N19h7W7H5oJfKvL7a8lY1JvAElfBQ4Fdpa0Ani/7Yu61PzBwJ8Ct5VrAQD/YHt+F9r+I+BiVYNBPQO41HZXbzkdA38IfKP6rWIz4BLb3+1i++8EvlJO9SwD/rxbDZfz8EcAf9WtNm3fIOly4Gaq0zu30N3uJ66QtBPwBFUXPw+21VDd7whwPnCppJOokuWb2mq/iXS1ERERtXKKKSIiaiVBRERErSSIiIiolQQRERG1kiAiIqJWEkQMS5Ilfbxj/u8kfWCUtv1FSceNxrZGaOdNpSfUa9puq02S9pf02g2o91VJt5ZeUmdJOnw96y+XtHNN+T+sbywbYkP3O/73kiBiJI8Bb6j7gRhL5VmMpk4C3mH71W3F0yX7U/UP1Jik5wB/bPvFtj9p+xzb3x+leNY7Qazn9zZgvfc7RkcSRIxkNdWDSn8zeMHgIwBJvynvh0r6oaRLJf1M0vmS3qpqLInbJO3RsZnDJf17We91pf4ESR+TtLD85ftXHdu9RtIlwG018ZxQtn+7pI+UsnOAVwCfl/SxQes/Q9JnS9//V0qaP7A/nX81S+qVdG2Z3lpVP/4LSwd600v5vmX/FpWYp5R1v61q/IzbJR1f1j2wfD43SVrQ0bXC6ZLuKPXnDop1C2AWcHxp43hVYwd8s6x/vaQX13x/3wOeXeq8svM7K/t4rqSby+e2dynfSdL3yv5dCKjmsz6fqsfVRZK+Usq+WfZpsTo6NZT0m3LkcgPwckmvVTW+xXWSPqMyvkfdZ1u33zX7GG2xnVdeQ76A3wDbAsuB7YC/Az5Qln0ROK5z3fJ+KPArqievtwR+Dpxblp0BfKqj/nep/lCZQtX30FbAycB7yzpbAn1UnbcdStVh3eSaOJ9L9eRpD9UTz1cDry/LrqUaX2BwneOA+aX95wAPDuxP2d+dy3QvcG2Z/hBwYpneHvgZsDXwT8BbS/kWwDOBNwL/0tHedlRdpP8Y6CllxwNzyvRKYMuBbdfE+2fAP3fM/xPVU/wAhwGLaupMAm7vmH/qOyv7+M4y/Q7gC2X6M8A5ZfoYqt5sd677tzFofsfy/kyq7kh2KvMG3lymtwLuG/gOqbqyuHKEz3at/c6re68cQcSIXPUW+29UA7k0tdDV2BWPAXdT/SUL1V/+kzrWu9T2k7bvoupKYm+q/o7+r6quRG4AdqJKIAA32r6npr2XUv2I97vqy/8rVGMpDOcVwGWl/V8ATa5RHAmcVWK7luoHbzfgP4B/kHQm8Dzbj5Z9PVzSRyS90vavgb2AFwJXlW28l6pnXIBbqbrVOJFmvYi+AvgSgO2rgZ0kbdegXqeBzh5v4vffyyHAl8t2v02VOJs4XdJPgOupuske+M7WUHUuCdX3u6zjO+zs62iozzbGSPpiiqY+RdU/zr92lK2mnKaUJKq/nAc81jH9ZMf8k6z9725wXy+mOqXxTtsLOhdIOpTqCKLOOqdBGhiuzlP7RvVD1VnnjbaXDFr/znIK5RhggaS/tH21pAOpzp9/WNL3qHqDXWy7bujQY6h+nI8F3idp35Ls1if+9e07Z+B7WcPw38uwyndzOPBy24+UU3IDn9vvbK8ZWHW4zVDz2Uo6aH1iidGTI4hoxPYq4FKqC74DllMNRwlVH/6bb8Cm31SuBexBNUDOEmAB8HZV3ZojaU+NPFDODcCrJO2s6kLoCVQjgg3nOuCNpf0/pDqFNWA5v9+3N3aULwDeWRIikg4o77tT/WX8GaoeOV8s6bnAI7a/TDUIzkvK/vWojC0tafNy/eIZwETb11ANjLQ98KxB8T4MbNMx/yPgrWU7hwL/7dEZG6Rzu0cDQ42L/MTAd0R1+uzBkhz2phoOt85Pgd1VDXYF1Sm2AbWfLevud3RJEkSsj48DnXcz/QvVj/KNwEEM/df9cJZQ/ZB/BzjF9u+ohhW9A7hZ1YDuFzLC0a6r0bfOpjpN9BOqMQxG6ir5CqrrHgNt3AD8uiw7F/i0pH+n+ut6wHlUifDWEtt5pfx44PZyemRvqlNyLwJuLGXvAf7R9uNU1z4+Uk7HLKIa72AC8GVJt1H1YPpJrzu86DXAPh0Xaz8A9Eq6laoX0JmMjnOpRla7meq0z71DrDeb6nP4CtW1pM1KLOdRnWZaRzn19g7gu5KuA/6L33/mQ322g/c7uiS9ucYmTdKzbP9GVRfPN1KNHveLsY5rY9bxmQu4ALjL9ifHOq5YV65BxKbuSlWDIG0BnJfk0BVvkzST6jO/heroLcahHEFEREStXIOIiIhaSRAREVErCSIiImolQURERK0kiIiIqPU/G4A5RncSI24AAAAASUVORK5CYII=\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], + "source": [ + "%time show_metric(negative_entropy, True)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Jotto Summary\n", + "\n", + "Here's a table of results:\n", + "\n", + "|
Algorithm|Targets Only
Mean (Max)|Nontargets
Mean (Max)|\n", + "|--|--|--|\n", + "|baseline random|7.37 (18)| |\n", + "|min(max(reply_counts))|7.15 (18)|7.05 (10)|\n", + "|min(expectation(reply_counts))|7.14 (17)|6.84 (10)|\n", + "|max(entropy(reply_counts))|7.09 (19)|6.81 (10)|\n", + "\n", + "So we started out with a mean of 7.37 and a worst score of 18 with the random guesser, and were able to improve by half a guess to a mean of 6.81 and a worst score of 10 using maximization of entropy over reply counts, with nontarget guesses allowed." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Next Steps\n", + "\n", + "There are many directions you could take this if you are interested:\n", + "- Try different length words.\n", + "- Allow words with repeated letters, like `stars`.\n", + "- Have each reply consist of two numbers: the number of letters in common with the target, and the number of letters that are in the exact correct position.\n", + "- Apply the same techniques to **Mastermind**, which uses the two-number reply system over an alphabet of colors, not letters.\n", + "- Examine the strategy that a chooser would use, and how the guesser responds to that. Is there an equilibrium?\n", + "- Our `make_tree` function is **greedy** in that it chooses the guess that minimizes (or maximizes) some metric of the current situation without looking ahead to future branches in the tree. Can you get better performance by doing some **look-ahead**?\n", + "- Can you improve a tree by editing it? Given a tree, look for interior nodes that have a worse-than-expected average score, and see if the node can be replaced with something better (covering the same target words).\n", + "- Research what other computer scientists have done with [Jotto](https://arxiv.org/abs/1107.3342) or [Mastermind](http://serkangur.freeservers.com/).\n", + "- What else can you explore?" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.7.6" + } + }, + "nbformat": 4, + "nbformat_minor": 4 +}