First draft.
This commit is contained in:
Danilo Enoque Ferreira de Lima committed
1 parent
29f46c859a
commit
65893bbabf
26 files changed
+11352
No files matched your search
@@ -0,0 +1,317 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "1bba0128",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Simple fits\n",
|
||||
"\n",
|
||||
"Here we are fitting a line from scratch.\n",
|
||||
"In the next notebook, we will do fancier fits with neural networks, but let's start with a basic problem and complicate it as we go along.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"id": "23feddde",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from typing import Tuple\n",
|
||||
"\n",
|
||||
"import pandas as pd\n",
|
||||
"import numpy as np\n",
|
||||
"import matplotlib.pyplot as plt\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "bb1286f0",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We start by generating some fake dataset, which is simple enough that we can visualize the results easily. For this reason, the dataset will contain only two variables.\n",
|
||||
"\n",
|
||||
"The simulated example data will be $f(x) = 3 x + \\epsilon$, where $\\epsilon \\sim \\mathcal{N}(\\mu=0, \\sigma=0.5)$.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"id": "5d457cd8",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def generate_data(N: int) -> np.ndarray:\n",
|
||||
" x = 2*np.random.randn(N, 1)\n",
|
||||
" epsilon = 0.5*np.random.randn(N, 1)\n",
|
||||
" z = 3*x + epsilon\n",
|
||||
" return np.concatenate((x, z), axis=1).astype(np.float32)\n",
|
||||
"\n",
|
||||
"data = generate_data(N=1000)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"id": "bc7938ee-c442-428a-984b-47975605b850",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"data = np.concatenate([data,\n",
|
||||
" np.array([[10.0, -4.0],\n",
|
||||
" [5.0, -5.0]])\n",
|
||||
" ], axis=0)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "48433f6f",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We can fit this line from scratch, assuming $y = f(x) = \\beta x + \\alpha + \\epsilon$, where $\\epsilon$ is a zero-mean Gaussian noise."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "6a170344-34ff-4af6-9a6c-daa44b95408e",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We can do this in two ways. In both cases, we start assuming the knowledge of $f(x)$ and use the Bayes Theorem:\n",
|
||||
"\n",
|
||||
"$ p(\\alpha, \\beta|\\mathcal{D}) \\, \\propto p(\\mathcal{D}|\\alpha, \\beta) \\, p(\\alpha) \\, p(\\beta)$\n",
|
||||
"\n",
|
||||
"We assume $p(\\alpha)$ and $p(\\beta)$ are constants, so we just need to maximize the likelihood $p(\\mathcal{D}|\\alpha, \\beta)$.\n",
|
||||
"That likelihood is a Gaussian distribution, because we assumed $y$ is exactly equals to a line plus some Gaussian noise coming from $\\epsilon$.\n",
|
||||
"\n",
|
||||
"$p(\\alpha,\\beta|\\mathcal{D}) \\propto \\prod \\mathcal{N}(y_i|\\text{mean}=f(x_i), \\text{std. dev.}=\\sigma_\\epsilon)$\n",
|
||||
"\n",
|
||||
"This leads to minimizing $L = \\sum_i (y_i - \\beta x_i - \\alpha)^2$, as seen in the lecture (simply take $- \\log p$ and to maximize $p$, one has to minimize $-\\log p$).\n",
|
||||
"\n",
|
||||
"We can try finding out $\\beta$ and $\\alpha$ by taking the derivative of $L$ and setting it to zero:\n",
|
||||
"\n",
|
||||
"$\\frac{d L}{d\\beta} = 2 \\sum_i (y_i - \\beta x_i - \\alpha) x_i = 0$\n",
|
||||
"\n",
|
||||
"$\\frac{d L}{d\\alpha} = -2 \\sum_i (y_i - \\beta x_i - \\alpha) = 0$\n",
|
||||
"\n",
|
||||
"Re-arranging:\n",
|
||||
"\n",
|
||||
"$2 \\sum_i (y_i x_i - \\beta x_i^2 - \\alpha x_i) = 0$\n",
|
||||
"\n",
|
||||
"$2 \\sum_i (y_i - \\beta x_i - \\alpha) = 0$\n",
|
||||
"\n",
|
||||
"This leads to:\n",
|
||||
"\n",
|
||||
"$ \\beta \\sum_i x_i^2 = \\sum_i (y_i - \\alpha) x_i$\n",
|
||||
"\n",
|
||||
"$ \\alpha = 1/N \\sum_i y_i - \\beta x_i = \\mathbb{E}[y - \\beta x]$\n",
|
||||
"\n",
|
||||
"We can then substitute $\\alpha$ in $\\beta$:\n",
|
||||
"\n",
|
||||
"$ \\beta \\sum_i x_i^2 = \\sum_i y_i x_i - 1/N \\sum_i \\sum_j (y_j - \\beta x_j) x_i$\n",
|
||||
"\n",
|
||||
"And expand it:\n",
|
||||
"\n",
|
||||
"$ \\beta (\\sum_i x_i^2 - 1/N \\sum_i \\sum_j x_i x_j) = \\sum_i y_i x_i - 1/N \\sum_i \\sum_j y_j x_i$\n",
|
||||
"\n",
|
||||
"Taking $x_j$ from a sum that does not depend on $j$, we see that the second term is just the square of the sum over $x_i$:\n",
|
||||
"\n",
|
||||
"$ \\beta (\\sum_i x_i^2 - 1/N(\\sum_{i} x_i)^2 ) = \\sum_i y_i x_i - 1/N \\sum_i \\sum_j y_j x_i$\n",
|
||||
"\n",
|
||||
"Dividing by the number of points $N$, several sums reduce to averages, which make this much easier to interpret if we use some standard identities from statistics.\n",
|
||||
"For example $1/N \\sum_i x_i^2 - 1/N^2 (\\sum_i x_i)^2$ is a known expression for the variance of $X$. \n",
|
||||
"\n",
|
||||
"$ \\beta \\text{var}[X] = \\mathbb{E}[XY] - \\mathbb{E}[X]\\mathbb{E}[Y]$\n",
|
||||
"\n",
|
||||
"And the right-hand-side is a known expression for the covariance between $X$ and $Y$:\n",
|
||||
"\n",
|
||||
"$ \\beta \\text{var}[X] = \\text{cov}[X,Y]$\n",
|
||||
"\n",
|
||||
"This leads to:\n",
|
||||
"\n",
|
||||
"$\\beta = \\frac{\\text{cov}[X,Y]}{\\text{var}[X]}$\n",
|
||||
"\n",
|
||||
"This, together with the expression from above for $\\alpha$, allows us to easily fit a line in a few lines of code:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 9,
|
||||
"id": "30205402",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"cov = np.cov(data[:,0], data[:,1])[0,1]\n",
|
||||
"varX = np.std(data[:,0])**2\n",
|
||||
"beta = cov/varX\n",
|
||||
"alpha = np.mean(data[:,1] - beta*data[:,0])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"id": "cabdfc12-3033-4a2f-96ed-4ab1ea8312ab",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<matplotlib.legend.Legend at 0x7fdf8b28ccd0>"
|
||||
]
|
||||
},
|
||||
"execution_count": 10,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAj8AAAGwCAYAAABGogSnAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAABfZ0lEQVR4nO3de1zT9f4H8NdAGBdhynWoGCiWkVc0FTXv90ual8q08pKlaWVq5iVF8kKaWXlJTVM8WmnmLc0w72mhlGiKqHnBvA1QiQ1QBm7f88dkOrltY/e9no/HHj/32ffy3u8kvPx8PxeRIAgCiIiIiJyEi7ULICIiIrIkhh8iIiJyKgw/RERE5FQYfoiIiMipMPwQERGRU2H4ISIiIqfC8ENEREROpZK1C7BFarUaN2/ehI+PD0QikbXLISIiIj0IgoCcnBxUq1YNLi6l9+8w/JTg5s2bCA0NtXYZREREZIRr166hRo0apX7O8FMCHx8fAJr/5/n6+lq5GiIicmp37wK9egGnTgGrVwPPP2/timyWQqFAaGio9vd4aRh+SlD0qMvX15fhh4iIrMvXF/jlF+D8eaB1a2tXYxfKG7LC8ENERGRL9u0DatUCwsMftgUGal5kEpztRUREZCvi44Fu3YAePYD//rN2NQ6L4YeIiMjaBAGYORMYNgy4fx84dw744gtrV+Ww+NiLiIjImgoKgDffBNaufdj2zjvAjBnWq8nBMfwQERFZi1wO9OsH7N+veS8SAZ99Bowbp/kzmQXDDxERkTVcvaoZ23PmjOa9hwewfj3Qv79163ICDD9ERESWduIE0LMnIJNp3gcEAD/9BERHW7cuJ8HwQ0REZGm7dj0MPhERmnV8IiKsW5MTYfghIiKytKlTgQsXgH/+0fT4BARYuyKnwvBDRERUCpVaQFJaFjJz8hHk44Fm4X5wdTHBQGSRCPj6a0ClAjw9K349MgjDDxERUQkSUmSI3ZEKmTxf2xYi8UBM70h0qxei/4WUSuCttzRr+LRt+7Dd3d2E1ZIhuMghERHRYxJSZBi9Plkn+ABAujwfo9cnIyFFpt+FsrKALl00a/j07QucPWv6YslgDD9WolILSLx0B9tP3kDipTtQqQWz3m/o0KEQiUQQiURwc3NDcHAwOnfujNWrV0OtVut9nfj4eFSpUsV8hRIRWZlKLSB2RypK+qlc1Ba7I7X8n9tpaUDLlsBvv2neK5XAv/+aslQyEh97WYHJulIN1K1bN6xZswYqlQoZGRlISEjAe++9hx9//BE//fQTKlXifw5ERElpWcV6fB4lAJDJ85GUloXo2v4lH/Tnn0CvXkBmpuZ9UBCwcyfw7LOmL5gMxp4fCzNZV6oRxGIxpFIpqlevjqioKEydOhXbt2/HL7/8gvj4eADAwoULUb9+fXh7eyM0NBRvv/02cnNzAQAHDx7EsGHDIJfLtb1IM2fOBACsW7cOTZs2hY+PD6RSKV555RVkFv2lJyKyI5k5pQcfvY776SfN2J6in4F16wJHjzL42BCGHwsyWVeqCXXo0AENGzbEli1bAAAuLi5YtGgRzpw5g7Vr12L//v2YNGkSAKBly5b44osv4OvrC5lMBplMhokTJwIACgsLMWvWLPz999/Ytm0brly5gqFDh1rsexARmUqQj4fxxy1ZArzwAnDvnuZ9mzbAH38A4eEmrJAqis85LMgkXalmULduXZw6dQoAMG7cOG17WFgYZs+ejVGjRuGrr76Cu7s7JBIJRCIRpFKpzjWGDx+u/XOtWrWwaNEiPPvss8jNzUXlypUt8j2IiEyhWbgfQiQeSJfnl/iPVREAqUQz7V3HtGnA3LkP37/yCrB6NSAWm7NcMgJ7fiyowl2pZiIIAkQPNtDbu3cvOnbsiOrVq8PHxwevvvoq7ty5g7t375Z5jePHj6N3796oWbMmfHx80PbBdM6rV6+avX4iIlNydREhpnckAE3QeVTR+5jekcXX+4mKergZ6dSpwLp1DD42iuHHgirUlWpGZ8+eRXh4OK5cuYJevXqhQYMG2Lx5M44fP46lS5cCAAoKCko9Py8vD127doWvry++/fZb/Pnnn9i6dWu55xER2apu9UKwbEgUpBLdn8dSiQeWDYkqeXJK//7AwoXAihXAnDmAC3/F2io+9rIgo7tSzWj//v04ffo03n//fRw/fhxqtRqfffYZXB78pf3hhx90jnd3d4dKpdJpO3fuHO7cuYNPPvkEoaGhAIC//vrLMl+AiMhMutULQedIaekrPN+5A/g/NkThkaEDZLsYSy3I6K5UE1EqlUhPT8eNGzeQnJyMuXPnok+fPujVqxdee+01REREoLCwEIsXL8bly5exbt06LF++XOcaYWFhyM3Nxb59+3D79m3cvXsXNWvWhLu7u/a8n376CbNmzTLLdyAisiRXFxGia/ujT6PqiK7t//Dn8x9/AE89BTz2M5LsA8OPhRnVlWoiCQkJCAkJQVhYGLp164YDBw5g0aJF2L59O1xdXdGwYUMsXLgQ8+bNQ7169fDtt98iLi5O5xotW7bEqFGj8NJLLyEwMBDz589HYGAg4uPjsWnTJkRGRuKTTz7BggULzPY9iIis6scfgQ4dND0/Y8YAe/dauyIykEgQBMvNq7YTCoUCEokEcrkcvr6+ZrmH2TbLIyIivRj8c1gQgM8/ByZO1PwZADp2BDZvBiQSyxRNZdL39zfH/FhJUVcqERFZnsEr7atUmvE8S5Y8bHv9dc3O7Nyg1O7wsRcRETkVg1faz8sD+vXTDT4xMcCaNQw+dsquwk9cXByeffZZ+Pj4ICgoCH379sX58+d1jsnPz8eYMWPg7++PypUro3///sjIyLBSxUREZEsMXmk/IwNo106zZQUAVKqkCT0zZz5c04fsjl2Fn0OHDmHMmDE4evQo9uzZg8LCQnTp0gV5eXnaY95//33s2LEDmzZtwqFDh3Dz5k3069fPilUTEZGtMGSlfQgCMHAgULR0h68v8MsvALfusXt2NeYnISFB5318fDyCgoJw/PhxtGnTBnK5HN988w2+++47dOjQAQCwZs0aPP300zh69ChatGhR4nWVSiWUSqX2vUKhMN+XICIiqzFopX2RCFi8GHjuOc2A5l27gPr1zVwhWYJd9fw8Ti6XAwD8/DSLAh4/fhyFhYXo1KmT9pi6deuiZs2aSExMLPU6cXFxkEgk2lfRQn1ERORYDF5pv2FDYOdOza7sDD4Ow27Dj1qtxrhx49CqVSvUq1cPAJCeng53d3dUqVJF59jg4GCkp6eXeq0pU6ZALpdrX9euXTNn6UREZCVFK+2XOFpHENDr7G+oUdlNd6X9Nm2A6tUtVSJZgF099nrUmDFjkJKSgiNHjlT4WmKxGGJuPkdE5PCKVtofvT4ZIjwc5OyqVmHWr8vwyt8JuCqWwVXU2ZplkpnZZc/P2LFjsXPnThw4cAA1atTQtkulUhQUFCA7O1vn+IyMDEilUgtXaV9EIhG2bdtm9vt8/fXXCA0NhYuLC7744gvMnDkTjRo1Mvt9iYiKPL7SvrfyLlZt/hiv/K0ZV1pz0zrNYy5yWHa1wrMgCHjnnXewdetWHDx4EHXq1NH5XC6XIzAwEN9//z369+8PADh//jzq1q2LxMTEUgc8P84SKzxb2tChQ5GdnV1qwElPT0fVqlXN2gOmUCgQEBCAhQsXon///pBIJFCr1VAqlfB/sDlgeXUSEZmKSi3gxNEziBgxCFXOpWga3d01U9lfecW6xZFRHHKF5zFjxuC7777D9u3b4ePjox3HI5FI4OnpCYlEghEjRmD8+PHw8/ODr68v3nnnHURHR+sdfJyVJXrGrl69isLCQvTs2RMhIQ9XUK1cubLZ701E9DjX1DNo+nIPoGicZ5UqwLZtQNu21iyLLMCuHnstW7YMcrkc7dq1Q0hIiPa1ceNG7TGff/45evXqhf79+6NNmzaQSqXYsmWLFau2D48+9rpy5QpEIhG2bNmC9u3bw8vLCw0bNiw2Y+7IkSN47rnn4OnpidDQULz77rs6ay49Kj4+HvUfzJSoVasWRCIRrly5ovPYa+bMmVi7di22b98OkUgEkUiEgwcPmusrE5Ez27cPaNXqYfAJC9Ps1M7g4xTsKvwIglDia+gjC055eHhg6dKlyMrKQl5eHrZs2cLxPkaaNm0aJk6ciJMnT+LJJ5/EoEGDcP/+fQDApUuX0K1bN/Tv3x+nTp3Cxo0bceTIEYwdO7bEa7300kvY+2Dn46SkJMhksmJLCkycOBEvvvgiunXrBplMBplMhpYtW5r3SxKR8/nf/4Bu3YCiNd2aNgUSE4Gnn7ZuXWQxdvXYy6YtXKh5lScq6uEy6UWefx5ITi7/3PHjNS8LmThxInr27AkAiI2NxTPPPIOLFy+ibt26iIuLw+DBgzFu3DgAQJ06dbBo0SK0bdsWy5Ytg4eH7loanp6e2nE9gYGBJQbSypUrw9PTE0qlkoGViMxDpQKWLwce/EMOvXsD338PeHtbty6yKIYfU1EogBs3yj+upAUUb93S71wLrzzdoEED7Z+LxuhkZmaibt26+Pvvv3Hq1Cl8++232mMEQYBarUZaWhqe5r+giMgWuboC27cD0dGa3p8vv9S0kVNh+DEVX1/9FsEKDCy5TZ9zLTzzzM3NTftn0YMN/NRqNQAgNzcXb731Ft59991i59WsWdMyBRIRGSMwEEhKAqpW5eakTorhx1Qq8kjq8cdgdiAqKgqpqamIiIgw6XXd3d2hUqlMek0icmLXrml+Nq9YAfg9smrzo38mp8Pw40TkcjlOnjyp0+bv72/UXmYffvghWrRogbFjx+KNN96At7c3UlNTsWfPHixZssToGsPCwrB7926cP38e/v7+kEgkOj1QROQcVGoBSWlZyMzJR5CPB5qF+8HVxcBempMngZ49gZs3NcMLdu8GuJo/geHHqRw8eBCNGzfWaRsxYgRWrVpl8LUaNGiAQ4cOYdq0aXjuuecLine truncated
|
||||
"text/plain": [
|
||||
"<Figure size 640x480 with 1 Axes>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"plt.scatter(data[:,0], data[:,1], label=\"Data\")\n",
|
||||
"plt.axline((0, alpha), slope=beta, c='r', lw=2, ls='--', label=\"Line fit\")\n",
|
||||
"plt.xlabel(r\"$x$\")\n",
|
||||
"plt.ylabel(r\"$y$\")\n",
|
||||
"plt.legend(frameon=False)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "bd74e2d4-b75b-4235-8541-d5565ec4d7a8",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Try comparing the fit result (values of $\\alpha$ and $\\beta$) with the simulated values in the beginning of the notebook. Are they close? Which criteria should you use to establish that they are close?"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 11,
|
||||
"id": "c6a1998f-81eb-4a32-8d00-04639ca777c2",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"-0.06067522680624462"
|
||||
]
|
||||
},
|
||||
"execution_count": 11,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"alpha"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 12,
|
||||
"id": "ec33c149-7d6f-433b-872a-ca97c0b7d7d6",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"2.89487707822029"
|
||||
]
|
||||
},
|
||||
"execution_count": 12,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"beta"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "91f0020f-6be7-4d3f-898c-2b062f825959",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Why is this a terrible idea?\n",
|
||||
"\n",
|
||||
"The previous calculation assumes that $p(\\alpha)$ and $p(\\beta)$, that is, the priors, are uniform. That means that any value of $\\alpha$ and $\\beta$ is acceptable. As we have discussed in the previous lecture, this leads to a fit that is very vulnerable to outliers. In the next notebook we will try to fix this by changing the priors to give a higher weight to values of $\\alpha$ and $\\beta$ close to zero, by setting $p(\\alpha)$ and $p(\\beta)$ to a Gaussian distribution, making an assumption on $\\alpha$ and $\\beta$ before looking at data. This method is called *regularization* and it starts from the assumption that, if the data has been previously *standardized*, we tend not to get wildly large values of a single parameter.\n",
|
||||
"\n",
|
||||
"Nevertheless, the calculation above can be changed to consider a Gaussian $p(\\alpha)$ and $p(\\beta)$ without using the methods described in the next notebook. Feel free to try redoing the calculation at home considering that.\n",
|
||||
"\n",
|
||||
"There are many other regularization techniques that make different assumptions on the parameters before looking at data. The choice of this method is strongly dependent on the problem at hand and it applies not only to line fitting, but fits in general.\n",
|
||||
"\n",
|
||||
"Try for example to add a data point in the variable `data` that is very far off to simulate an outlier. What happens with the fit? Are the fit parameters still close?"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "6cc1b58d-08fc-4f83-8aa4-e2979aa92c89",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "2bdcc1c0-427c-4f6c-83d3-2f39a498cfa1",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Another problem with the procedure above is that it does not calculate the errors on $\\alpha$ and $\\beta$. We will also cover this in the next notebook.\n",
|
||||
"\n",
|
||||
"One way of estimating uncertainties from the previous calculations is by making the approximation that the posterior probability $p(\\alpha, \\beta|\\mathcal{D})$ is approximately Gaussian around the best values of $\\alpha$ and $\\beta$ (this is called the *Laplace approximation*). In the example above, this is exactly true, but depending on the likelihood and priors this is not always so, although the approximation tends to be relatively good in several common cases. If we assume the posterior is Gaussian, the variance of the posterior can be taken as a measure of the uncertainty squared of $\\alpha$ and $\\beta$. How could we calculate that variance?\n",
|
||||
"\n",
|
||||
"One theorem called the Cramer Rao Lower Bound shows that the covariance matrix of the \n",
|
||||
"It can be shown, furthermore, that the covariance matrix of the posterior, $C$ satisfies:\n",
|
||||
"\n",
|
||||
"$C - I^{-1}$ is semi-positive definite,\n",
|
||||
"\n",
|
||||
"where $I_{ij} = -\\mathbb{E}[\\frac{\\partial^2 L}{\\partial \\theta_i \\partial \\theta_j}]$ (this is called the *Fisher information matrix*) and $L$ is the negative log-posterior, as above.\n",
|
||||
"\n",
|
||||
"This means, within the context of the calculation above, that the variance of the posterior could be said to have a *lower bound* of at least $-(\\frac{\\partial^2 L}{\\partial^2 \\theta})^{-1}$, calculated at the optimal values of the parameters.\n",
|
||||
"\n",
|
||||
"By taking this second derivative we could get estimates to the variances of the parameters. This is often how curve-fitting algorithms establish approximate uncertainties for the parameters."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "0812975c-a8fc-4d9e-98bc-d3851e75be70",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.10.12"
|
||||
},
|
||||
"widgets": {
|
||||
"application/vnd.jupyter.widget-state+json": {
|
||||
"state": {},
|
||||
"version_major": 2,
|
||||
"version_minor": 0
|
||||
}
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
@@ -0,0 +1,587 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "1bba0128",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Bayesian fits\n",
|
||||
"\n",
|
||||
"This is a notebook showing a visualization of a Byesian interpretation for the line fit.\n",
|
||||
"The idea is that we generate all possible line fits and reject them if they are not compatible with the data.\n",
|
||||
"Naturally, this is not how most fits are done, but it illustrates the logic behind the Bayes Theorem when applied for fitting lines.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"id": "23feddde",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from typing import Tuple\n",
|
||||
"\n",
|
||||
"import pandas as pd\n",
|
||||
"import numpy as np\n",
|
||||
"import matplotlib.pyplot as plt\n",
|
||||
"import matplotlib\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "bb1286f0",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We start by generating some fake dataset, which is simple enough that we can visualize the results easily. For this reason, the dataset will contain only two variables.\n",
|
||||
"\n",
|
||||
"The simulated example data will be $f(x) = 0.5 x + 0.3 + \\epsilon$, where $\\epsilon \\sim \\mathcal{N}(\\mu=0, \\sigma=0.5)$.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"id": "aef54435-78af-4b13-aee2-2bc576c773cd",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"true_alpha = 0.3\n",
|
||||
"true_beta = 0.5\n",
|
||||
"true_eps = 0.5\n",
|
||||
"def generate_data(N: int) -> np.ndarray:\n",
|
||||
" x = 2*np.random.randn(N, 1)\n",
|
||||
" epsilon = true_eps*np.random.randn(N, 1)\n",
|
||||
" z = true_beta*x + true_alpha + epsilon\n",
|
||||
" return np.concatenate((x, z), axis=1).astype(np.float32)\n",
|
||||
"\n",
|
||||
"data = generate_data(N=100)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "04432613-39cd-4209-8833-7af585e7eed9",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Given some parameters $\\alpha$ and $\\beta$, we don't know the true $f(x)$, but we assume it is something like $f(x) = \\beta x + \\alpha$."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"id": "a795239a-03eb-483a-8b07-635e3d9dbc55",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def f(x: np.ndarray, alpha: float, beta: float) -> np.ndarray:\n",
|
||||
" return beta*x + alpha"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "a2fba887-dc1a-4400-ab40-59fec452dc83",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Let's plot it for reference:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"id": "fead461d-4d3a-48f2-a525-c518be4f1cf8",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"image/png": "iVBORw0KGgoAAAANSUhEUgAAArIAAAINCAYAAADC5ZwgAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAAB41klEQVR4nO3deXiU5fn28e9kZ0kGAoGwExbRgIjsWxVQNCgIWm1VrKLVVqpVtL4KtgpYFa3auiPys27YSq1VFjWIAirKJhEVEZRVDGEPCUSSkMy8fzxOyDLLM8lsz8z5OQ6ONjP3zNxJhJy55nqu2+Z0Op2IiIiIiFhMXLg3ICIiIiJSHwqyIiIiImJJCrIiIiIiYkkKsiIiIiJiSQqyIiIiImJJCrIiIiIiYkkKsiIiIiJiSQqyIiIiImJJCeHeQKg5HA727NlDamoqNpst3NsRERERkVqcTidHjx6lbdu2xMV5rrvGXJDds2cPHTp0CPc2RERERMSH3bt30759e4/3x1yQTU1NBYwvTFpaWph3IyIiIiK1FRcX06FDh6rc5knMBVlXO0FaWpqCrIiIiEgE89UGqou9RERERMSSFGRFRERExJIUZEVERETEkhRkRURERMSSFGRFRERExJIUZEVERETEkhRkRURERMSSFGRFRERExJIUZEVERETEkhRkRURERMSSFGRFRERExJIUZKPEpEmTsNls2Gw2EhMTad26NaNHj+af//wnDofDr+caMWIEU6ZMCc5GRURERAJEQTYIKh1OVm07xIIN+azadohKhzMkr5uTk0NBQQE7d+7kvffeY+TIkdx6662MHTuWioqKkOyhunXr1vGb3/yGU045haZNm9KtWzcuueQSRo0axSWXXMLbb78d8j2JiIhI9FCQDbDcjQUMf3gZV8xdza2vb+CKuasZ/vAycjcWBP21k5OTyczMpF27dvTt25e7776bBQsW8N577/HSSy8Z+8vNZfjw4TRr1owWLVowduxYtm3bVvUckyZN4qOPPuKJJ56oqvDu3LnT5+NqW7t2LVu2bGHOnDlMmTKFkpIS3n33Xf73v/+xbNkyJkyYwMUXX8zTTz8d7C+LiIiIRCmb0+kMTbkwQhQXF2O32ykqKiItLS2gz527sYDJ8/Ko/QW1/fy/s6/qS06vNgF9TZdJkyZx5MgRt1XOPn360LZtW959913efPNNbDYbvXv35tixY9x7773s3LmTDRs2EBcXR1FREWPGjKFXr17cd999AGRkZPD22297fZw3EydO5Msvv2Tjxo01bh86dCjHjx/niy++CNjXQURERILA4QAfP+8DyWxeSwjZjqJcpcPJzEWb6oRYACdGmJ25aBOjszOJj7O5WRU8p556Kl999RUAv/zlL2vc989//pOMjAw2bdpEr169sNvtJCUl0bhxYzIzM6vW+XqcJydOnOCdd97hpptuqnNfx44dWbduXUM+NREREQmmzz6DNWvg6FFo2RIuugjatw/3rqqotSBA1u44TEFRqcf7nUBBUSlrdxwO3aZcr+10YrMZ4fn777/niiuuoEuXLqSlpdG5c2cAfvjhB6/PUd/HLV++nKKiIiZMmFDnvh9++IEuXbr4/fmIiIhIkB0/DjNmwPvvQ1GRUZHdvx9eeQXKysK9uyqqyAbI/qOeQ2x91gXSt99+S1ZWFgDjxo2jU6dOzJ07l7Zt2+JwOOjVqxfl5eVen6O+j1uwYAHt27enf//+NW7fuXMna9as4cknn2zYJyciIiKBtWYNvPee+/vKy2HXLjjllNDuyQMF2QBplZoS0HWBsmzZMr7++mtuu+02Dh06xJYtW5g7dy6/+MUvAFi5cmWdxyQlJVFZWVn1sdnH1eZ0Olm4cCEXXXRRVUUYoLy8nBtuuIHTTjuN66+/vqGfooiIiARCWRnMmuV7nd0e/L2YpCAbIAOz0mljT2FvUanbPlkbkGlPYWBWetD2UFZWxt69e6msrGTfvn3k5uYya9Ysxo4dy9VXX43NZqNFixY8//zztGnThh9++IGpU6fWeZ7OnTuzZs0adu7cSdOmTUlPTzf1uNpWrVrFjz/+yLhx4wAoKSlh8eLF3HPPPSQnJ/POO++QnJwc8K+DiIiI+Gn9eli0yPe6M8+EVq2Cvx+T1CMbIPFxNqaPywZOTilwcX08fVx2UC/0ys3NpU2bNnTu3JmcnByWL1/Ok08+yYIFC4iPjycuLo7XX3+d9evX06tXL2677TYeeeSROs9zxx13EB8fT3Z2NhkZGfzwww+mHlfdpk2buOKKKwD43e9+R69evcjOzubll19m2rRp5OXl0alTp6B8HURERMSk8nKjF9ZMiG3cGMaOBVtoL1r3RuO3Aix3YwEzF22qceFXG3sK08dlB230loiIiIjfvvwS3nrL3NrLLoOePYO7n2o0fitMcnq1YXR2Jmt3HGb/0VJapRrtBKEeuSUiIiLi1okT8NBDUO16GI8SE+GuuyAhMiNjZO7K4uLjbAzp2iLc2xARERGpaeNG+O9/za295BLo3Tu4+2kgBVkRERGRaFdRAY8+CqUmx4D++c9GNTbCKciKiIiIRLPNm+H1182tvegi6Ns3uPsJIAVZERERkWhUWQmPP24cL2vGtGlgsbGYCrIiIiIi0eb77+G118ytvfBCGDAguPsJEgVZERERkWjhcMBTT0Fhobn1U6dCSmhPHQ0kBVkRERGRaLB9O7zyirm1558PQ4YEdz8hoCArIiIiYmUOB8yZA/v2mVt/553GKV1RQEFWRERExKp27YIXXzS3dtQoOOus4O4nxOLCvQFpOJvN5vXPjBkzwr1FERERCSSnE+bONR9i/9//i7oQC6rIRoWCgoKq/z9//nzuvfdetmzZUnVb06ZN6zymvLycpKSkkOwvmo0YMYJJkyYxadKkcG9FRERixY8/wv/9n7m1Z58NI0cGdz9hpCDrycKFsH9/ePfQqpUxmNiHzMzMqv9vt9ux2Ww1bgMjcPXq1YuEhATmzZvH6aefzvLly+ncuTNTpkxhypQpVWv79OnDhAkTmDFjBg6Hg4cffpjnn3+evXv3csopp3DPPfdw6aWXetxP+/btufvuu/nDH/5Qddtnn33Gueeey7fffkunTp38+CJ4NnjwYK688kpuueUWAC6//HLmz5/P8ePHSUlJYffu3XTt2pWNGzdyyimneHyef//731x33XVs376dNm3aAHDttdeyfv16PvnkE+x2e8TsVUREYpjTCS+/DDt3mlv/pz9BampQtxRuCrKe7N9v/MYTRV5++WUmT57Mp59+avoxs2bNYt68eTz33HN0796djz/+mKuuuoqMjAzOPvtst48ZNGgQ69atq/rY6XQyZcoUbrvttjoh9sEHH+TBBx/0uodNmzbRsWPHOrc3a9aMoz8Ped69ezfvv/8+TZo04ciRI2RmZjJnzhxGjx7tMxhefvnlPPTQQzz44IM89dRTTJ8+nQ8++IDVq1cHJMQGcq8iIhKj9uyB5583t3boUDjvvODuJ0IoyMaQ7t2787e//c30+rKyMh588EE++OADhvw8oqNLly6sXLmSOXPmeAyygwcP5uWXX676+NVXX2X37t1Mmzatztobb7yRX/3qV1730bZtW7e3Vw+HTz/9NFdddRULFy6ksLCQ9PR05s6dy6uvvgrAxRdfzIoVKzjnnHP473//W+N5bDYbDzzwAJdeeimZmZk89dRTfPLJJ7Rr187rvvzhz14XL17Mn/70JxwOB3fddRfXX399wPYhIiIW43TCv/5lHHBgxm23QYCKMFagIBtD+vXr59f6rVu38tNPPzF69Ogat5eXl3PmmWd6fNzgwYOZOnUqx44dw2azcffdd3P//fe77dVNT08nPT3dr325uMJhSUkJL7zwAqtXr+ajjz6isLCQ//73v7Ro0aJq77feeivXXXddjYBd3dixY8nOzua+++7j/fffp2fPnm7X1a4gHz9+nNWrV3PzzTdX3eaugmx2rxUVFdx+++0sX74cu91Ov379uPjii2nRokW9vkYiImJh+/bB7Nnm1g4cCBdcENz9RCAF2RjSpEmTOrfFxcXhdDpr3HbixAkAjh07BsA777xTpzqZ7OUs5n79+hEXF0deXh4ffPABGRkZXHvttW7XNrS1YM+ePbz88ssMHTqUbt26kZaWRmFhIc888wy33HILNpsNMHqEV6xY4fE1cnNz2bx5M5WVlbRu3drjutoV5IkTJ/LLX/6SSy65pOo2dxVks3tdu3YtPXv2rPp6jxkzhvfff58rrrjC69dIRESizH/+A5s2mVt7663QvHlw9xOhFGQ9adUq3DsIyR4yMjJqTD0oLi5mx44dAGRnZ5OcnMwPP/zgsY3AncaNG3P66afz5ptvMnfuXN59913i4txPemtoa8G3337LE088wbPPPgsYF7stX76cb7/9lquvvtrUfvPy8vjVr37FCy+8wEsvvcQ999zDG2+84XZt7Qpyo0aNaNWqFd26dfP6Gmb3umfPnhq/NLRr1478/HxTn4eIiESBgwfh6afNrT3zTBg/Prj7ASodTtbuOMz+o6W0Sk1hYFY68XG2oL+uGQqynpiYFhANRo0axUsvvcS4ceNo1qwZ9957L/Hx8QCkpqZyxx13cNttt+FwOBg+fDhFRUV8+umnpKWlcc0113h83sGDB/PUU08xfvx4RowY4XFdQ1sLli1bRlZWFueccw4AaWlpPPfcc/zhD3+gsYlTS3bu3MmFF17I3XffzRVXXEGXLl0YMmQIeXl59O3bt177CtZeRUQkyr31Fnz5pbm1f/wjhKDtLHdjATMXbaKgqLTqtjb2FKaPyyanV5ugv74vCrIxbtq0aezYsYOxY8dit9v561//WlWRBfjrX/9KRkYGs2bNYvv27TRr1oy+ffty9913e33eM844g8TERB555JGg7b1Zs2YcO3aMW2+9teo2u91OaWkpN910k8/HHz58mJycHMaPH8/UqVMBY+LLine truncated
|
||||
"text/plain": [
|
||||
"<Figure size 800x600 with 1 Axes>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"fig = plt.figure(figsize=(8, 6))\n",
|
||||
"plt.scatter(data[:,0], data[:,1], label=r\"Data $\\mathcal{D}$\")\n",
|
||||
"plt.plot(data[:,0], f(data[:,0], true_alpha, true_beta), c='r', ls='-', alpha=0.5, lw=4, label=r\"True $y=w_1 x + w_0$\")\n",
|
||||
"xs = np.linspace(-2, 2, 10)\n",
|
||||
"plt.plot(xs, f(xs, true_alpha, true_beta) - 2*true_eps, c='r', ls='--', alpha=0.5, lw=4)\n",
|
||||
"plt.plot(xs, f(xs, true_alpha, true_beta) + 2*true_eps, c='r', ls='--', alpha=0.5, lw=4)\n",
|
||||
"xa = 2\n",
|
||||
"ya = f(xa, true_alpha, true_beta)\n",
|
||||
"plt.annotate(xy=(xa, ya + 2*true_eps), xytext=(xa, ya - 2*true_eps), arrowprops=dict(arrowstyle='|-|', lw=3, alpha=0.5, color=\"green\"), text=r\"\")\n",
|
||||
"plt.text(xa+0.5, ya, s=r\"$\\epsilon$ noise\", size=16)\n",
|
||||
"plt.legend(frameon=False)\n",
|
||||
"plt.xlabel(r\"$x$\")\n",
|
||||
"plt.ylabel(r\"$y$\")\n",
|
||||
"plt.savefig(\"data.png\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "00e098e8-0082-4451-927a-073cf242787e",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Let's show several random lines and reject the ones that do not agree with the likelihood.\n",
|
||||
"\n",
|
||||
"Note that we are producing lines to show in the plots below randomly by taking samples for the posterior and therefore they would differ every time the notebook is executed, but they should still be likely samples that match the data generated above."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"id": "08d9daaf-25c5-4d06-b4a6-6cecaaa35651",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import matplotlib.pylab as pylab\n",
|
||||
"params = {'legend.fontsize': 'x-large',\n",
|
||||
" 'figure.figsize': (15, 5),\n",
|
||||
" 'axes.labelsize': 'x-large',\n",
|
||||
" 'axes.titlesize':'x-large',\n",
|
||||
" 'xtick.labelsize':'x-large',\n",
|
||||
" 'ytick.labelsize':'x-large'}\n",
|
||||
"pylab.rcParams.update(params)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"id": "4c7002b2-5cb5-4929-8e01-964144ee60b6",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def get_log_prior(A, B):\n",
|
||||
" #return 0.25*A/A/1/5\n",
|
||||
" return -0.5*A**2/1**2 - 0.5*B**2/1**2 #- (2*np.pi*p_width**2)\n",
|
||||
"def sample_prior() -> Tuple[float, float]:\n",
|
||||
" #return (2*np.random.rand() - 1)*1, (2*np.random.rand() - 1)*5\n",
|
||||
" return 1*np.random.randn(), 1*np.random.randn()\n",
|
||||
"def get_log_likelihood(x, y, A, B, sigma):\n",
|
||||
" error = f(x, A, B) - y\n",
|
||||
" return np.sum(-0.5*error**2/sigma**2, axis=0) # - (2*np.pi*sigma**2)\n",
|
||||
"def plot(I: int):\n",
|
||||
" # how many lines to show in last column\n",
|
||||
" K = 20\n",
|
||||
" # assumed noise std. dev.\n",
|
||||
" sigma=0.5\n",
|
||||
" # region of alpha and beta to plot\n",
|
||||
" R = 4\n",
|
||||
" # take a subset of data points\n",
|
||||
" D = data[:I,:]\n",
|
||||
" # make plot\n",
|
||||
" fig = plt.figure(figsize=(4*5, 4))\n",
|
||||
" gs = matplotlib.gridspec.GridSpec(1, 7)\n",
|
||||
"\n",
|
||||
" # create alpha and beta grid\n",
|
||||
" Ax = np.linspace(-R, R, 50)\n",
|
||||
" Bx = np.linspace(-R, R, 50)\n",
|
||||
" A, B = np.meshgrid(Ax, Bx)\n",
|
||||
"\n",
|
||||
" # calculate the log prior\n",
|
||||
" log_prior = get_log_prior(A, B)\n",
|
||||
" # calculate the log likelihood\n",
|
||||
" log_likelihood = get_log_likelihood(D[:, 0, np.newaxis, np.newaxis],\n",
|
||||
" D[:, 1, np.newaxis, np.newaxis],\n",
|
||||
" A[np.newaxis, ...],\n",
|
||||
" B[np.newaxis, ...],\n",
|
||||
" sigma=sigma)\n",
|
||||
" # the log posterior is, except for a constant factor, just the sum of those\n",
|
||||
" log_posterior = log_prior + log_likelihood\n",
|
||||
"\n",
|
||||
" gsp = matplotlib.gridspec.GridSpecFromSubplotSpec(2, 2, width_ratios=[3,1], height_ratios=[3,1], subplot_spec=gs[0,0], wspace=0, hspace=0)\n",
|
||||
" ax = plt.subplot(gsp[0,0])\n",
|
||||
" axy = plt.subplot(gsp[0,1], sharey=ax)\n",
|
||||
" axx = plt.subplot(gsp[1,0], sharex=ax)\n",
|
||||
" axy.plot(np.exp(log_prior).sum(axis=1), Ax)\n",
|
||||
" axx.plot(Bx, np.exp(log_prior).sum(axis=0))\n",
|
||||
" axx.set(xlabel=r\"$w_1$\")\n",
|
||||
" axx.get_xaxis().set_major_locator(matplotlib.ticker.MultipleLocator(2))\n",
|
||||
" ax.get_yaxis().set_major_locator(matplotlib.ticker.MultipleLocator(2))\n",
|
||||
" ax.set(title=r\"$p(w_0, w_1)$\", ylabel=r\"$w_0$\")\n",
|
||||
" ax.get_xaxis().set_visible(False)\n",
|
||||
" axy.get_yaxis().set_visible(False)\n",
|
||||
" axy.get_xaxis().set_visible(False)\n",
|
||||
" axx.get_yaxis().set_visible(False)\n",
|
||||
" m=ax.imshow(np.exp(log_prior), extent=(-R, R, -R, R), aspect=\"auto\", origin='lower')\n",
|
||||
" #plt.colorbar(m,ax=ax, orientation=\"horizontal\")\n",
|
||||
"\n",
|
||||
" gsp = matplotlib.gridspec.GridSpecFromSubplotSpec(2, 2, width_ratios=[3,1], height_ratios=[3,1], subplot_spec=gs[0,1], wspace=0, hspace=0)\n",
|
||||
" ax = plt.subplot(gsp[0,0])\n",
|
||||
" axy = plt.subplot(gsp[0,1], sharey=ax)\n",
|
||||
" axx = plt.subplot(gsp[1,0], sharex=ax)\n",
|
||||
" axy.plot(np.exp(log_likelihood).sum(axis=1), Ax)\n",
|
||||
" axx.plot(Bx, np.exp(log_likelihood).sum(axis=0))\n",
|
||||
" axx.set(xlabel=r\"$w_1$\")\n",
|
||||
" axx.get_xaxis().set_major_locator(matplotlib.ticker.MultipleLocator(2))\n",
|
||||
" ax.get_yaxis().set_major_locator(matplotlib.ticker.MultipleLocator(2))\n",
|
||||
" ax.set(title=r\"$p(\\mathcal{D}|w_0,w_1)$\", ylabel=r\"$w_0$\")\n",
|
||||
" ax.get_xaxis().set_visible(False)\n",
|
||||
" axy.get_yaxis().set_visible(False)\n",
|
||||
" axy.get_xaxis().set_visible(False)\n",
|
||||
" axx.get_yaxis().set_visible(False)\n",
|
||||
" m=ax.imshow(np.exp(log_likelihood), extent=(-R, R, -R, R), aspect=\"auto\", origin='lower')\n",
|
||||
" #plt.colorbar(m,ax=ax, orientation=\"horizontal\")\n",
|
||||
"\n",
|
||||
" gsp = matplotlib.gridspec.GridSpecFromSubplotSpec(2, 2, width_ratios=[3,1], height_ratios=[3,1], subplot_spec=gs[0,2], wspace=0, hspace=0)\n",
|
||||
" ax = plt.subplot(gsp[0,0])\n",
|
||||
" axy = plt.subplot(gsp[0,1], sharey=ax)\n",
|
||||
" axx = plt.subplot(gsp[1,0], sharex=ax)\n",
|
||||
" axy.plot(np.exp(log_posterior).sum(axis=1), Ax)\n",
|
||||
" axx.plot(Bx, np.exp(log_posterior).sum(axis=0))\n",
|
||||
" axx.set(xlabel=r\"$w_1$\")\n",
|
||||
" axx.get_xaxis().set_major_locator(matplotlib.ticker.MultipleLocator(2))\n",
|
||||
" ax.get_yaxis().set_major_locator(matplotlib.ticker.MultipleLocator(2))\n",
|
||||
" ax.set(title=r\"$p(w_1, w_0|\\mathcal{D})$\", ylabel=r\"$w_0$\")\n",
|
||||
" ax.get_xaxis().set_visible(False)\n",
|
||||
" axy.get_yaxis().set_visible(False)\n",
|
||||
" axy.get_xaxis().set_visible(False)\n",
|
||||
" axx.get_yaxis().set_visible(False)\n",
|
||||
" m=ax.imshow(np.exp(log_posterior), extent=(-R, R, -R, R), aspect=\"auto\", origin='lower')\n",
|
||||
" #plt.colorbar(m,ax=ax, orientation=\"horizontal\")\n",
|
||||
"\n",
|
||||
" ax = plt.subplot(gs[0,3])\n",
|
||||
"\n",
|
||||
" # sample alpha and beta\n",
|
||||
" def get_line():\n",
|
||||
" a, b = sample_prior()\n",
|
||||
" if len(D) == 0:\n",
|
||||
" return a, b\n",
|
||||
" for _ in range(1000):\n",
|
||||
" ap, bp = a + 0.1*np.random.randn(), b + 0.1*np.random.randn()\n",
|
||||
" A = min(1, np.exp(get_log_likelihood(D[:, 0], D[:, 1], ap, bp, sigma=sigma) - get_log_likelihood(D[:, 0], D[:, 1], a, b, sigma=sigma) ))\n",
|
||||
" if np.random.rand() <= A:\n",
|
||||
" a = ap\n",
|
||||
" b = bp\n",
|
||||
" return a, b\n",
|
||||
" for _ in range(K):\n",
|
||||
" a, b = get_line()\n",
|
||||
" ax.axline((0, a), slope=b, lw=2, ls='-', c='r', alpha=0.3)\n",
|
||||
" if I > 0:\n",
|
||||
" ax.scatter(D[:, 0], D[:,1], label=\"Data\", alpha=1.0)\n",
|
||||
" ax.set(xlabel=\"x\", ylabel=\"y\", title=\"Data\", xlim=(-5, 5), ylim=(-5, 5))\n",
|
||||
"\n",
|
||||
" #plt.legend(frameon=False)\n",
|
||||
" plt.tight_layout()\n",
|
||||
" plt.savefig(f\"bayes_reg_{I}.png\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"id": "693827d3-c5ac-4b6d-b019-233799cd5d78",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"image/png": "iVBORw0KGgoAAAANSUhEUgAABHkAAAGGCAYAAADvkF2zAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAAEAAElEQVR4nOy9d3xb5fn+f0myLG95x3bileXY2TskEAgUCrSMlk7KLNBSWmbHFzogdMCnn1/7KdDNCquFlpYW2lIgQEIG2Xt6O3acxPHeli3p/P64cvwceQ/ZGr7fr5deOZLOeHQsPznn8nVft0nTNA2CIAiCIAiCIAiCIAhCQGP29QAEQRAEQRAEQRAEQRCE0SMijyAIgiAIgiAIgiAIQhAgIo8gCIIgCIIgCIIgCEIQICKPIAiCIAiCIAiCIAhCECAijyAIgiAIgiAIgiAIQhAgIo8gCIIgCIIgCIIgCEIQICKPIAiCIAiCIAiCIAhCECAijyAIgiAIgiAIgiAIQhAgIo8gCIIgCIIgCIIgCEIQICKPIAiCIAiCIAiCIAhCECAijyAIgiAIgiAIgiAIQhAgIs8E4eabb0ZycjJaW1t9PRSfs2fPHphMJjz77LO+HoogCIIgCIIgCIIgeA0ReSYAu3btwssvv4wHH3wQkZGRvh6Oz1m8eDGuvfZa/OhHP0JLS4uvhyMIAcFYCcVr166FyWRCWVmZV/cbqIgILUx05I9SnozVHClzjSAEFiaTyeNhs9mQlJSERYsW4fbbb8d///tfuFwurxzrhRdegMlkwgsvvOCV/Qnjj4g8E4Af/OAHiImJwTe+8Q1fD8VveOihh3DmzBk89dRTvh6KIPg9IhSPHyJCCxMZmWvGD5lrBCEweeSRR/DII4/ge9/7Hr70pS8hNjYWL7/8Mq688kqsWLECBQUFvh6i4AeE+HoAwthSUFCA999/H7fffjvCw8N9PRy/YdmyZZg1axb++Mc/4sEHH4TZLHqnIPTHQEJxZWUlnnnmGXz00UcoKipCbW0tHA4Hpk6diujoaLS3t2Py5MlYvnw5vvSlL2Hu3Lk++ASBxUMPPYTly5fjqaeewve//31fD0cQxg35o9TwGO38K3ONIAQea9eu7fVaVVUV7r77brz++uv4xCc+gd27dyM5OXn8Byf4DXJnG2Bs27YNJpMJd999N1599VVccMEFsNvtsNlsWLZsGTZs2OCx/vPPPw9N0/DFL36x174++OADmEwmPPzwwx6v79ixo9sK2NMefMMNN8BsNiM/P9/rn228x/OlL30J5eXlWL9+/WiGLQhBjS4Uf+ELX/AQijVNwy9+8Qv87//+Ly666CI8+eSTiIyMRHt7O6677joUFhZi7969OHToEL7zne/gH//4B+bNm4dvf/vbPvw0gYFRhHa73b4ejiCMC/3NNUJvvDX/ylwjCMHBpEmT8Nprr+Giiy5CRUUFHnvsMY/39+zZg3vvvRfz589HfHw8wsLCMGPGDHz7299GfX29x7oXXXQRbr31VgDArbfe6lEipt+HnTp1Cj/+8Y+xatUqpKSkIDQ0FGlpabj++utx9OjRcfnMwsCIyBNg7NmzBwDw/vvv48Ybb0R8fDzuvPNOrF69Grt27cKVV16J8vLy7vXff/99WCwWrFixote+4uLiAADNzc0er//85z/vXq6rq+tePnXqFP7617/iqquuQk5Ojlc/ly/Gs2rVKgAQkUeYUHhLKD5z5gw++clP4sknn8RFF12EefPmISSE5tAbbrihe72QkBBcfvnl2LBhA+x2O/7v//5vSL9zIkKLCC0ENvJHqbEbjzfnX5lrBCE4MJvN+OEPfwgAePXVV6FpWvd7zzzzDF577TXk5OTg1ltvxTe+8Q2kpqbi//7v/7Bq1SqPe69bbrkF11xzDQDgmmuu6S4Pe+SRRxAbGwsA2LRpE/7nf/4HsbGxuO6663D//fdjxYoV+Nvf/oZly5bhwIED4/fBhb7RhIDi1ltv1QBoMTEx2ubNmz3eu/vuuzUA2re//W1N0zStpaVFs1gs2pw5c/rcV0lJiQZAu+2227pfy8/P18xms3bttddqALT333+/+73vf//7GoBex/UW4z2ehoYGDYC2dOnSUY9dEAKFX//61xoAbdasWZrFYtGuvvpq7Xvf+572iU98QgOghYWFaSdOnOhef/HixZrFYtFaWloG3G9xcXH39q2trX2uc9ttt2kAtDvuuKP7tUceeUQDoJWWlnqsu2fPHg2Adt9993m8/pnPfEYDoAHQ9uzZ0/16ZWWlZrVatauvvnqop2JYjPd41q9f7zGfC0Kg4c25ZiLPB/3NkUZGOv9qmsw1ghAo6HPLQHR0dGghISEaAK2kpKT79bKyMs3pdPZa/9lnn9UAaP/zP//j8fq6des0ANq6dev6PE5VVZXW1NTU6/X9+/drkZGR2uWXXz6ETySMJeLkCTD27t0LAPj1r3+N888/3+O922+/HQC6bXKVlZVwuVxITU3tc199OWd+8YtfIDo6Gt/73vcAKOdMR0cHnn76aSxfvrzXcb3FeI/HbrcjLCzMw/kkCMGOPoecOnUKGzduxJtvvomf//znWL9+Pe6++250dHR0B5K3trZi//79yM3NHTQE9a233gIAXHLJJYiIiOhznalTpwIAKioqBh1nX/NBQUEB3nzzTVx77bUA4GEx/u1vf4uuri5897vfHXTfI2G8x7N06VIA/GuZIAQi3pxrJvp8MBijmX9lrhGE4MFmsyEhIQEAUF1d3f16ZmYmLBZLr/W/+tWvIiYmBu++++6wjpOcnIzo6Oher8+fPx8XX3wxNmzYgK6urmGOXvAmIvIEEA6HA0ePHkV6erqHHVdH/6XWf6lqa2sBqIuRntjtdpjN5u6LlKqqKrz00kv4+te/jvT0dABKVHn55ZdRU1OD73znOx77+N3vfofs7GyEhYVh8eLF2Lx584g/32jHs2nTJlx99dWYPHnykNv+xcfHo6amZsRjFoRAw5tCsZF//etfAIBPf/rT/a6jd3AJDQ0ddH8iQosILQQ28kep8RvPaOZfmWsEIbjQzpVpmUym7te6urrwm9/8Bueffz7i4+NhsVhgMplgNpvR1NSEysrKYR/nP//5D6666iqkpqbCarV2l6r+61//gsPhkPsrHyMiTwBx6NAhdHV14ZOf/GSf3aD0+u+MjAwA6A4u7Ojo6HN/JpMJdru9+yLlySefhKZpuPfee2G32wGov0Q9+eSTmDZtGj772c92b/+Xv/wF9957L77//e9j3759WLlyJa644ooRXyiMdjwtLS2YM2cOnnzyySGHNra3t0vAozBh8LZQrNPU1NQt8A50k3HixAkAwOzZswcdq4jQIkILgYu//VFqJL9/A+FP85M35l+ZawQhOOjo6Oiee5KSkrpf/+IXv4i7774bp0+fxjXXXIPvfe973Tk7drsdDodjWMd58skn8elPfxpbt27F6tWrcd999+Hhhx/GI488gvnz5wPAsPcpeBdpoR5A6H8Vy8rK6vP9//znPwCAyy67DAC6W+fpF099ERcXh+bmZrS0tOD3v/89rr/+eqSlpQEALBYL6urq8N577+HIkSP4zW9+4yEu/d///R9uueUW3HHHHQD417p33nkHv//97/H444+P6DOOZjxXXnklrrzySgAMDRsMt9uNhoYGZGdnj2isghBoeFso1nnnnXfQ1dWFhQsXYsqUKX2u43K5uu3An/rUpwYd60Cir24RHooI/bvf/Q7nn38+fve73+GKK67A0aNHuz/fcBjteHQR+qabbsJNN900pGOKCC0EKuP5R6mx+v0bCH+an7wx/8pcIwjBwZYtW+B0OjFp0qTu+8Xdu3fjH//4Bz7xiU/gv//9b3dIO8B7of/93/8d1jGcTifWrl2LlJQU7N27t5cDc9u2baP+HMLoESdPAKGLPA0NDb3eq6urw9NPP4309PTuRPTU1FQkJSUN2MlBF1WefvppNDY2etSLx8TEoK6uDk888QQSEhK62+kBQGdnJ/bs2dMtKOlcdtll+Pjjj0f8GUc6npGQn58PTdOwYMGCUe1HEAKFsRCKAeDf//43AOCqq67qd51169ahtrYWy5cv7+5sNxj9ib7R0dG9RN/777+/XxE6NzcXv/71r5Gamorf//73Qzq2t8dz5ZVX4rHHHsPnPve5Pm96e6KL0PrPQBACifH8o9RY/P4NBX+Zn0Y7/8pcIwjBgdvtxs9+9jMAwPXXX9/9elFREQDg6quv9hB4AGDnzp1ob2/vtS89v8flcvV6r6amBg0NDVi5cmUvgaelpaV7/hd8i4g8AYTePv1vf/sbWltbu19vaWnB9ddfj9raWjzxxBMICwsDwL80rV69GjU1Nd2/4D2Ji4tDfX09nnjiCVx55ZXIy8vrfs9ut2PHjh145513cNddd3mE+dXU1MDlcmHSpEke+5s0aRLOnDnT6zi33HLLkCzSIx3PSNi+fTsAYM2aNaPajyAECmMhFLvdbrz99tsA+r/JKCoqwkMPPQSbzTasmxgRoUWEFgITf/qj1FjhD/OTN+ZfmWsEIfA5e/YsvvSlL2Hjxo3IyMjA97///e73dLF948aNvbb55je/2ef+9JLaviI4kpOTERERgT179nRnfQEsv7333nul9NNPEJEnQHA6nTh06FB3neO8efNw//3346677kJOTg7effddPP744x52YAC47rrrAKDf1HRdVKmoqOjV9SE2NhaHDx+GzWbDt771rVGN3+12A0AvBdlX4wGA9957DxaLpfsiUxCCnbEQirdt24ba2lqkpqZi8eLFvd5/7733cOGFF6KpqQmvvvoqFi5cOOTxiggtIrQQmPjTH6WGy1jPB8OdnwbCG/OvzDWCEFisXbsWa9euxcMPP4x77rkHl1xyCdLT0/H6669j2bJl+OCDD5CYmNi9/tKlS7Fq1Sq88cYbWLlyJb73ve/h5ptvxpw5cxAREdEdi2HkvPPLine truncated
|
||||
"text/plain": [
|
||||
"<Figure size 2000x400 with 10 Axes>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"plot(I=0)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"id": "a1fbd00b-e7f2-4eff-90db-e1d44d104fba",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"image/png": "iVBORw0KGgoAAAANSUhEUgAABHkAAAGGCAYAAADvkF2zAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAAEAAElEQVR4nOydd3gc1fX+39kuraRVl9UlF7n3XgDTiUMNJCR0AiQEAqYkfCEFTEggyS8FnJBCLykkJBBIQug2GOPeu2TLkmVZfdWlXW2Z3x/Hs/eupFVdaYvO53n20d3ZKXdmd65m3jnnPYqqqioYhmEYhmEYhmEYhmGYiEYX6g4wDMMwDMMwDMMwDMMww4dFHoZhGIZhGIZhGIZhmCiARR6GYRiGYRiGYRiGYZgogEUehmEYhmEYhmEYhmGYKIBFHoZhGIZhGIZhGIZhmCiARR6GYRiGYRiGYRiGYZgogEUehmEYhmEYhmEYhmGYKIBFHoZhGIZhGIZhGIZhmCiARR6GYRiGYRiGYRiGYZgogEUehmEYhmEYhmEYhmGYKIBFHoZhGIZhGIZhGIZhmCiARZ4xwo033oj09HS0t7eHuishZ8eOHVAUBc8991you8IwDMMwDMMwDMMwQYNFnjHAtm3b8Oqrr+LBBx+E1WoNdXdCzvz583H55Zfjhz/8Idra2kLdHYaJCEZKKF6zZg0URUFZWVlQ1xupsAjNjHX4oZQ/IzVG8ljDMJGFoih+L7PZjLS0NMybNw+33nor/ve//8Hj8QRlWy+99BIURcFLL70UlPUxow+LPGOA73//+0hISMC3vvWtUHclbHjooYdQXV2NtWvXhrorDBP2sFA8erAIzYxleKwZPXisYZjI5JFHHsEjjzyCBx54AF/96leRmJiIV199FatWrcKSJUtQXFwc6i4yYYAh1B1gRpbi4mJ8+OGHuPXWWxETExPq7oQNixYtwpQpU/DHP/4RDz74IHQ61jsZJhB9CcWVlZV49tln8cknn+Do0aNoaGiA0+nE+PHjER8fj87OTmRnZ2Px4sX46le/ipkzZ4ZgDyKLhx56CIsXL8batWvxve99L9TdYZhRgx9KDY7hjr881jBM5LFmzZoe02pqanDXXXfh9ddfx3nnnYft27cjPT199DvHhA18ZxthbNq0CYqi4K677sJf//pXnHHGGbDZbDCbzVi0aBHWrVvnN/8LL7wAVVVx9dVX91jXRx99BEVR8PDDD/tN37Jliy8UsHt48HXXXQedTocjR44Efd9Guz9f/epXceLECXzwwQfD6TbDRDWaUPyVr3zFTyhWVRW/+MUv8POf/xwrV67EU089BavVis7OTlx55ZUoKSnBzp07sW/fPnznO9/Bm2++iVmzZuH+++8P4d5EBrII7fV6Q90dhhkVAo01TE+CNf7yWMMw0UFGRgZee+01rFy5EhUVFXj88cf9Pt+xYwdWr16N2bNnIzk5GRaLBZMmTcL999+PxsZGv3lXrlyJm2++GQBw8803+6WIafdhp06dwo9+9CMsX74c48aNg8lkQlZWFq655hocPHhwVPaZ6RsWeSKMHTt2AAA+/PBDXH/99UhOTsbtt9+OM888E9u2bcOqVatw4sQJ3/wffvgh9Ho9lixZ0mNdSUlJAIDW1la/6T/72c98bbvd7mufOnUKf//733HJJZdg8uTJQd2vUPRn+fLlAMAiDzOmCJZQXF1djQsvvBBPPfUUVq5ciVmzZsFgoODQ6667zjefwWDARRddhHXr1sFms+FXv/rVgM45FqFZhGYiG34oNXL9Ceb4y2MNw0QHOp0OP/jBDwAAf/3rX6Gqqu+zZ599Fq+99homT56Mm2++Gd/61reQmZmJX/3qV1i+fLnfvddNN92Eyy67DABw2WWX+dLDHnnkESQmJgIAPv30U/z0pz9FYmIirrzyStx7771YsmQJ/vGPf2DRokXYs2fP6O040zsqE1HcfPPNKgA1ISFB3bBhg99nd911lwpAvf/++1VVVdW2tjZVr9erM2bM6HVdpaWlKgD1lltu8U07cuSIqtPp1Msvv1wFoH744Ye+z773ve+pAHpsN1iMdn+amppUAOrChQuH3XeGiRR+85vfqADUKVOmqHq9Xr300kvVBx54QD3vvPNUAKrFYlHLy8t988+fP1/V6/VqW1tbn+s9duyYb/n29vZe57nllltUAOptt93mm/bII4+oANTjx4/7zbtjxw4VgHrPPff4Tb/iiitUACoAdceOHb7plZWVqtFoVC+99NKBHopBMdr9+eCDD/zGc4aJNII51ozl8SDQGCkz1PFXVXmsYZhIQRtb+sLhcKgGg0EFoJaWlvqml5WVqW63u8f8zz33nApA/elPf+o3/cUXX1QBqC+++GKv26mpqVFbWlp6TN+9e7dqtVrViy66aAB7xIwkHMkTYezcuRMA8Jvf/AYrVqzw++zWW28FAF+YXGVlJTweDzIzM3tdV2+RM7/4xS8QHx+PBx54AICInHE4HHjmmWewePHiHtsNFqPdH5vNBovF4hf5xDDRjjaGnDp1CuvXr8dbb72Fn/3sZ/jggw9w1113weFw+AzJ29vbsXv3bkydOrVfE9S3334bAHDuueciNja213nGjx8PAKioqOi3n72NB8XFxXjrrbdw+eWXA4BfiPHTTz8Nl8uF7373u/2ueyiMdn8WLlwIgJ6WMUwkEsyxZqyPB/0xnPGXxxqGiR7MZjNSUlIAAHV1db7p+fn50Ov1Peb/+te/joSEBLz33nuD2k56ejri4+N7TJ89ezbOOeccrFu3Di6Xa5C9Z4IJizwRhNPpxMGDB5Gbm+sXjquhndTaSdXQ0ABAXIx0x2azQafT+S5Sampq8Morr+Cb3/wmcnNzAQhR5dVXX0V9fT2+853v+K3jd7/7HQoLC2GxWDB//nxs2LBhyPs33P58+umnuPTSS5GdnT3gsn/Jycmor68fcp8ZJtIIplAs8+9//xsAcPHFFwecR6vgYjKZ+l0fi9AsQjORDT+UGr3+DGf85bGGYaIL9XSalqIovmkulwu//e1vsWLFCiQnJ0Ov10NRFOh0OrS0tKCysnLQ2/nvf/+LSy65BJmZmTAajb5U1X//+99wOp18fxViWOSJIPbt2weXy4ULL7yw12pQWv53Xl4eAPiMCx0OR6/rUxQFNpvNd5Hy1FNPQVVVrF69GjabDYB4EvXUU09hwoQJ+NKXvuRb/m9/+xtWr16N733ve9i1axeWLVuGL3zhC0O+UBhuf9ra2jBjxgw89dRTAzZt7OzsZINHZswQbKFYo6WlxSfw9nWTUV5eDgCYPn16v31lEZpFaCZyCbeHUkM5//oinManYIy/PNYwTHTgcDh8Y09aWppv+tVXX4277roLVVVVuOyyy/DAAw/4fHZsNhucTuegtvPUU0/h4osvxsaNG3HmmWfinnvuwcMPP4xHHnkEs2fPBoBBr5MJLlxCPYLQnooVFBT0+vl///tfAMAFF1wAAL7SedrFU28kJSWhtbUVbW1t+P3vf49rrrkGWVlZAAC9Xg+73Y73338fBw4cwG9/+1s/celXv/oVbrrpJtx2220A6Gndu+++i9///vd44oknhrSPw+nPqlWrsGrVKgBkGtYfXq8XTU1NKCwsHFJfGSbSCLZQrPHuu+/C5XJh7ty5yMnJ6XUej8fjCwf+4he/2G9f+xJ9tRDhgYjQv/vd77BixQr87ne/wxe+8AUcPHjQt3+DYbj90UToG264ATfccMOAtskiNBOpjOZDqZE6//oinManYIy/PNYwTHTw2Wefwe12IyMjw3e/uH37drz55ps477zz8L///c9n0g7QvdDPf/7zQW3D7XZjzZo1GDduHHbu3NkjAnPTpk3D3g9m+HAkTwShiTxNTU09PrPb7XjmmWeQm5vrc0TPzMxEWlpan5UcNFHlmWeeQXNzs1++eEJCAux2O5588kmkpKT4yukBQFdXF3bs2OETlDQuuOACfP7550Pex6H2ZygcOXIEqqpizpw5w1oPw0QKIyEUA8B//vMfAMAll1wScJ4XX3wRDQ0NWLx4sa+yXX8EEn3j4+N7iL733ntvQBF66tSp+M1vfoPMzEz8/ve/H9C2g92fVatW4fHHH8dVV13V601vdzQRWvsOGCaSGM2HUiNx/g2EcBmfhjv+8ljDMNGB1+vFT37yEwDANddc45t+9OhRAMCll17qJ/AAwNatW9HZ2dljXZp/j8fj6fFZfX09mpqasGzZsh4CT1tbm2/8Z0ILizwRhFY+/R//+Afa29t909va2nDNNdegoaEBTz75JCwWCwB60nTmmWeivr7ed4J3JykpCY2NjXjyySexatUqTJs2zfeZzWbDli1b8O677+KOO+7wM/Orr6+Hx+NBRkaG3/oyMjJQXV3dYzs33XTTgEKkh9qfobB582YAwNlnnz2s9TBMpDASQrHX68U777wDIPBNxtGjR/HQQw/BbDYP6iaGRWgWoZnIJJweSo0U4TA+BWP85bGGYSKf2tpafPWrX8X69euRl5eH733ve77PNLF9/fr1PZa58847e12fllLbmwVHeno6YmNjsWPHDp/XF0Dpt6tXr+bUzzCBRZ4Iwe12Y9++fb48x1mzZuHee+/FHXfcgcmTJ+O9997DE0884RcODABXXnklAAR0TddElYqKih5VHxITE7F//36YzWZ8+9vfHlb/vV4vAPRQkEPVHwB4//33odfrfReZDBPtjIRQvGnTJjQ0NCAzMxPz58/v8fn777+Ps846Cy0tLfjrX/+KuXPnDri/LEKzCM1EJuH0UGqwjPR4MNjxqS+CMf7yWMMwkcWaNWuwZs0aPPzww7j77rtx7rnnIjc3F6+//joWLVqEjz76CKmpqb75Fy5ciOXLl+ONN97AsmXL8MADD+DGG2/EjBkzEBsb67PFkFm6dCliY2Px5JNP4tvf/jZ+/OMf48c//jGam5uh0+lw9913o7y8HDNnzsQLine truncated
|
||||
"text/plain": [
|
||||
"<Figure size 2000x400 with 10 Axes>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"plot(I=1)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 9,
|
||||
"id": "ae81e3fe-0e5d-4d56-86e6-ed5f8e6ec66f",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"image/png": "iVBORw0KGgoAAAANSUhEUgAABHkAAAGGCAYAAADvkF2zAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAAEAAElEQVR4nOydd3gc5fW2n9muVe/NkiUXyZZ7L7jTHVqoCaEGE6oxJeGDJGADCST5hQSbFsBUk0BC6CVgMLZx7zbukqxiFav3ttoy3x/Ho3dW0kpaaaXdlc59XXtpdnZ25p3Z2aOZZ895jiTLsgyGYRiGYRiGYRiGYRjGr9F4ewAMwzAMwzAMwzAMwzBM32GRh2EYhmEYhmEYhmEYZhDAIg/DMAzDMAzDMAzDMMwggEUehmEYhmEYhmEYhmGYQQCLPAzDMAzDMAzDMAzDMIMAFnkYhmEYhmEYhmEYhmEGASzyMAzDMAzDMAzDMAzDDAJY5GEYhmEYhmEYhmEYhhkEsMjDMAzDMAzDMAzDMAwzCGCRh2EYhmEYhmEYhmEYZhDAIg/DMAzDMAzDMAzDMMwggEWeIcLNN9+MmJgYNDY2ensoXmffvn2QJAlr16719lAYhmEYhmEYhmEYxmOwyDME2LNnD9atW4dHHnkEgYGB3h6O15k2bRquuOIKPPbYY2hoaPD2cBjGL+gvoXjVqlWQJAl5eXkeXa+/wiI0M9ThH6Wc6a8YybGGYfwLSZKcHkajEdHR0Zg6dSqWLVuG//3vf7Db7R7Z1ltvvQVJkvDWW295ZH3MwMMizxDgd7/7HUJCQnDXXXd5eyg+w6OPPoqSkhKsWbPG20NhGJ+HheKBg0VoZijDsWbg4FjDMP7JypUrsXLlSjz88MP42c9+hrCwMKxbtw5Lly7F7NmzkZmZ6e0hMj6AztsDYPqXzMxMfPfdd1i2bBkCAgK8PRyfYebMmRgzZgxeeeUVPPLII9BoWO9kGFd0JRQXFRXhtddew+bNm5GdnY3KykpYLBaMGDECwcHBaG5uRmJiImbNmoWf/exnmDBhghf2wL949NFHMWvWLKxZswa//e1vvT0chhkw+Ecp9+hr/OVYwzD+x6pVqzrMKy0txfLly/HBBx/gvPPOw969exETEzPwg2N8Br6z9TN27NgBSZKwfPlyvPfee5g/fz5CQ0NhNBoxc+ZMbNy40Wn5N954A7Is47rrruuwrg0bNkCSJDz++ONO83ft2tWWCtg+PfiGG26ARqPByZMnPb5vAz2en/3sZzh9+jS+/fbbvgybYQY1ilB87bXXOgnFsizjr3/9K/7yl79g0aJFWL16NQIDA9Hc3IyrrroKWVlZ2L9/Pw4fPoxf//rX+PjjjzFx4kQ89NBDXtwb/0AtQjscDm8Ph2EGBFexhumIp+IvxxqGGRzExsbi/fffx6JFi1BQUICnn37a6fV9+/ZhxYoVmDRpEiIiImAymTB69Gg89NBDqK6udlp20aJFuPXWWwEAt956q1OJmHIfVlxcjCeffBLnnHMO4uLiYDAYkJCQgOuvvx7Hjh0bkH1muoZFHj9j3759AIDvvvsON954IyIiInDnnXdiwYIF2LNnD5YuXYrTp0+3Lf/dd99Bq9Vi9uzZHdYVHh4OAKivr3ea/+c//7ltuqqqqm26uLgY//nPf3DppZciPT3do/vljfGcc845AMAiDzOk8JRQXFJSggsvvBCrV6/GokWLMHHiROh0lBx6ww03tC2n0+lw0UUXYePGjQgNDcXf/va3Hn3nWIRmEZrxb/hHqf4bjyfjL8cahhkcaDQa/P73vwcAvPfee5Blue211157De+//z7S09Nx66234q677kJ8fDz+9re/4ZxzznG697rllltw+eWXAwAuv/zytvKwlStXIiwsDADwww8/4E9/+hPCwsJw1VVX4YEHHsDs2bPx3//+FzNnzsShQ4cGbseZzpEZv+LWW2+VAcghISHyli1bnF5bvny5DEB+6KGHZFmW5YaGBlmr1crjx4/vdF05OTkyAPm2225rm3fy5ElZo9HIV1xxhQxA/u6779pe++1vfysD6LBdTzHQ46mpqZEByDNmzOjz2BnGX3j++edlAPKYMWNkrVYrX3bZZfLDDz8sn3feeTIA2WQyyfn5+W3LT5s2TdZqtXJDQ0OX6z116lTb+xsbGztd5rbbbpMByLfffnvbvJUrV8oA5NzcXKdl9+3bJwOQ77//fqf5P/3pT2UAMgB53759bfOLiopkvV4vX3bZZT09FG4x0OP59ttvneI5w/gbnow1QzkeuIqRanobf2WZYw3D+AtKbOmKlpYWWafTyQDknJyctvl5eXmyzWbrsPzatWtlAPKf/vQnp/lvvvmmDEB+8803O91OaWmpXFdX12H+wYMH5cDAQPmiiy7qwR4x/Qln8vgZ+/fvBwA8//zzmDdvntNry5YtA4C2NLmioiLY7XbEx8d3uq7OMmf++te/Ijg4GA8//DAAkTnT0tKCV199FbNmzeqwXU8x0OMJDQ2FyWRyynximMGOEkOKi4uxadMmfPrpp/jzn/+Mb7/9FsuXL0dLS0ubIXljYyMOHjyIsWPHdmuC+tlnnwEAzj33XJjN5k6XGTFiBACgoKCg23F2Fg8yMzPx6aef4oorrgAApxTjF198EVarFb/5zW+6XXdvGOjxzJgxAwD9WsYw/ognY81Qjwfd0Zf4y7GGYQYPRqMRkZGRAIDy8vK2+cOHD4dWq+2w/C9/+UuEhITgm2++cWs7MTExCA4O7jB/0qRJWLJkCTZu3Air1erm6BlPwiKPH2GxWHDs2DEkJSU5peMqKF9q5UtVWVkJQFyMtCc0NBQajabtIqW0tBTvvPMO7rjjDiQlJQEQosq6detQUVGBX//6107reOmll5CamgqTyYRp06Zhy5Ytvd6/vo7nhx9+wGWXXYbExMQet/2LiIhARUVFr8fMMP6GJ4ViNZ9//jkA4JJLLnG5jNLBxWAwdLs+FqFZhGb8G/5RauDG05f4y7GGYQYX8tkyLUmS2uZZrVa88MILmDdvHiIiIqDVaiFJEjQaDerq6lBUVOT2dr788ktceumliI+Ph16vbytV/fzzz2GxWPj+ysuwyONHHD58GFarFRdeeGGn3aCU+u/k5GQAaDMubGlp6XR9kiQhNDS07SJl9erVkGUZK1asQGhoKADxS9Tq1asxcuRIXHnllW3v//e//40VK1bgt7/9LQ4cOIC5c+fi4osv7vWFQl/H09DQgPHjx2P16tU9Nm1sbm5mg0dmyOBpoVihrq6uTeDt6iYjPz8fADBu3Lhux8oiNIvQjP/iaz9K9eb71xW+FJ88EX851jDM4KClpaUt9kRHR7fNv+6667B8+XKcOXMGl19+OR5++OE2n53Q0FBYLBa3trN69Wpccskl2LZtGxYsWID7778fjz/+OFauXIlJkyYBgNvrZDwLt1D3I5RfxVJSUjp9/csvvwQAXHDBBQDQ1jpPuXjqjPDwcNTX16OhoQEvv/wyrr/+eiQkJAAAtFotqqqqsH79ehw9ehQvvPCCk7j0t7/9Dbfccgtuv/12APRr3ddff42XX34ZzzzzTK/2sS/jWbp0KZYuXQqATMO6w+FwoKamBqmpqb0aK8P4G54WihW+/vprWK1WTJkyBcOGDet0Gbvd3pYO/JOf/KTbsXYl+iopwj0RoV966SXMmzcPL730Ei6++GIcO3asbf/coa/jUUTom266CTfddFOPtskiNOOvDOSPUv31/esKX4pPnoi/HGsYZnCwdetW2Gw2xMbGtt0v7t27Fx9//DHOO+88/O9//2szaQfoXugvf/mLW9uw2WxYtWoV4uLisH///g4ZmDt27OjzfjB9hzN5/AhF5KmpqenwWlVVFV599VUkJSW1OaLHx8cjOjq6y04Oiqjy6quvora21qlePCQkBFVVVXjuuecQGRnZ1k4PAFpbW7Fv3742QUnhggsuwPbt23u9j70dT284efIkZFnG5MmT+7QehvEX+kMoBoAvvvgCAHDppZe6XObNN99EZWUlZs2a1dbZrjtcib7BwcEdRN8HHnjApQg9duxYPP/884iPj8fLL7/co217ejxLly7F008/jauvvrrTm972KCK08hkwjD8xkD9K9cf3ryf4Snzqa/zlWMMwgwOHw4E//vGPAIDrr7++bX52djYA4LLLLnMSeABg9+7daG5u7rAuxb/Hbrd3eK2iogI1NTWYO3duB4GnoaGhLf4z3oVFHj9CaZ/+3//+F42NjW3zGxoacP3116OyshLPPfccTCYTAPqlacGCBaioqGj7grcnPDwc1dXVeO6557B06VJkZGS0vRYaGopdu3bh66+/xt133+1k5ldRUQG73Y7Y2Fin9cXGxqKkpKTDdm655ZYepUj3djy9YefOnQCAxYsX92k9DOMv9IdQ7HA48NVXXwFwfZORnZ2NRx99FEaj0a2bGBahWYRm/BNf+lGqv/CF+OSJ+MuxhmH8n7KyMvzsZz/Dpk2bkJycjN/+9rdtryli+6ZNmzq855577ul0fUpJbWcWHDExMTCbzdi3b1+b1xdA5bcrVqzg0k8fgUUeP8Fms+Hw4cNtdY4TJ07EAw88gLvvvhvp6en45ptv8MwzzzilAwPAVVddBQAuXdMVUaWgoKBD14ewsDAcOXIERqMR9957b5/G73A4AKCDguyt8QDA+vXrodVq2y4yGWaw0x9C8Y4dO1BZWYn4+HhMmzatw+vr16/HwoULUVdXh/feew9Tpkzp8XhZhGYRmvFPfOlHKXfp73jgbnzqCk/EX441DONfrFq1CqtWrcLjjz+O++67D+eeey6SkpLwwQcfYObMmdiwYQOioqLalp8xYwbOOeccfPTRR5g7dy4efvhh3HzzzRg/fjzMZnObLYaaOXPmwGw247nnnsO9996LP/zhD/jDH/6A2tpaaDQa3HfffcjPz8eECRNw//334+6778bEiRPxxRdfcCzLine truncated
|
||||
"text/plain": [
|
||||
"<Figure size 2000x400 with 10 Axes>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"plot(I=2)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"id": "8149698a-81e9-4a28-b6eb-39c5a47a8f13",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"image/png": "iVBORw0KGgoAAAANSUhEUgAABHkAAAGGCAYAAADvkF2zAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAAEAAElEQVR4nOydd3gc5fW2n9muuuq92bItudvYlo0bxoDp1bRQTYDQAqZ/kIJNSCDJLwUcSmIgNoYEiAMJJKEYjA3GvVfZkiWr996273x/HI9mVtpVl7bo3Ne1l96dnXJmpH218+w5zxFEURTBMAzDMAzDMAzDMAzD+DUqbwfAMAzDMAzDMAzDMAzDDB4WeRiGYRiGYRiGYRiGYQIAFnkYhmEYhmEYhmEYhmECABZ5GIZhGIZhGIZhGIZhAgAWeRiGYRiGYRiGYRiGYQIAFnkYhmEYhmEYhmEYhmECABZ5GIZhGIZhGIZhGIZhAgAWeRiGYRiGYRiGYRiGYQIAFnkYhmEYhmEYhmEYhmECABZ5GIZhGIZhGIZhGIZhAgAWeRiGYRiGYRiGYRiGYQIAFnlGCXfeeSfi4uLQ3t7u7VC8zv79+yEIAt566y1vh8IwDMMwDMMwDMMwQwaLPKOAvXv34t1338UzzzyDkJAQb4fjdWbNmoVrrrkGP//5z9HW1ubtcBjGLxguoXj16tUQBAFFRUVDul9/hUVoZrTDX0q5MlxzJM81DONfCILg8tDr9YiNjcU555yDe+65B59//jkcDseQHGv9+vUQBAHr168fkv0xIw+LPKOAn/70pwgPD8cDDzzg7VB8hmeffRZVVVVYs2aNt0NhGJ+HheKRg0VoZjTDc83IwXMNw/gnq1atwqpVq/D000/j5ptvRkREBN59911cdtllmDdvHvLy8rwdIuMDaLwdADO85OXl4euvv8Y999yDoKAgb4fjM+Tk5CA7Oxt/+ctf8Mwzz0ClYr2TYTzRk1BcXl6ON998E99++y1Onz6N+vp6WCwWjB07FmFhYTCZTEhOTsbcuXNx8803Y+rUqV44A//i2Wefxdy5c7FmzRr85Cc/8XY4DDNi8JdS/WOw8y/PNQzjf6xevbrbsurqajz88MPYuHEjLrzwQuzbtw9xcXEjHxzjM/CdrZ+xc+dOCIKAhx9+GO+//z4WLVoEo9EIvV6PnJwcbNmyxWX9v/71rxBFETfddFO3fW3evBmCIOC5555zWb579+7OVMCu6cG33XYbVCoVTp06NeTnNtLx3HzzzSgpKcFXX301mLAZJqCRhOIbb7zRRSgWRRG/+93v8Nvf/hZLlizBK6+8gpCQEJhMJixfvhz5+fk4cOAAjh49iieffBL/+te/MG3aNDzxxBNePBv/QClCO51Ob4fDMCOCp7mG6c5Qzb881zBMYBAfH48PPvgAS5YsQWlpKV588UWX1/fv34+VK1di+vTpiIqKgsFgwPjx4/HEE0+gsbHRZd0lS5bgrrvuAgDcddddLiVi0n1YRUUFfvGLX2DBggVISEiATqdDUlISbrnlFpw4cWJEzpnpGRZ5/Iz9+/cDAL7++mvcfvvtiIqKwv3334/Fixdj7969uOyyy1BSUtK5/tdffw21Wo158+Z121dkZCQAoLW11WX5b37zm85xQ0ND57iiogL/+Mc/cOWVVyIrK2tIz8sb8SxYsAAAWORhRhVDJRRXVVXh4osvxiuvvIIlS5Zg2rRp0GgoOfS2227rXE+j0eCSSy7Bli1bYDQa8Yc//KFP7zkWoVmEZvwb/lJq+OIZyvmX5xqGCQxUKhV+9rOfAQDef/99iKLY+dqbb76JDz74AFlZWbjrrrvwwAMPIDExEX/4wx+wYMECl3uvFStW4OqrrwYAXH311Z3lYatWrUJERAQA4LvvvsOvf/1rREREYPny5Xjssccwb948/POf/0ROTg4OHz48cifOuEdk/Iq77rpLBCCGh4eL27Ztc3nt4YcfFgGITzzxhCiKotjW1iaq1WpxypQpbvdVWFgoAhDvvvvuzmWnTp0SVSqVeM0114gAxK+//rrztZ/85CcigG7HHSpGOp6mpiYRgDhnzpxBx84w/sKf/vQnEYCYnZ0tqtVq8aqrrhKffvpp8cILLxQBiAaDQSwuLu5cf9asWaJarRbb2tp63G9BQUHn9u3t7W7Xufvuu0UA4r333tu5bNWqVSIA8cyZMy7r7t+/XwQgPvrooy7Lr732WhGACEDcv39/5/Ly8nJRq9WKV111VV8vRb8Y6Xi++uorl/mcYfyNoZxrRvN84GmOVDLQ+VcUea5hGH9Bmlt6wmw2ixqNRgQgFhYWdi4vKioS7XZ7t/XfeustEYD461//2mX5unXrRADiunXr3B6nurpabGlp6bb80KFDYkhIiHjJJZf04YyY4YQzefyMAwcOAAD+9Kc/YeHChS6v3XPPPQDQmSZXXl4Oh8OBxMREt/tylznzu9/9DmFhYXj66acByJkzZrMZa9euxdy5c7sdd6gY6XiMRiMMBoNL5hPDBDrSHFJRUYGtW7fik08+wW9+8xt89dVXePjhh2E2mzsNydvb23Ho0CFMnDixVxPUTz/9FABwwQUXIDg42O06Y8eOBQCUlpb2Gqe7+SAvLw+ffPIJrrnmGgBwSTF+7bXXYLPZ8NRTT/W674Ew0vHMmTMHAH1bxjD+yFDONaN9PuiNwcy/PNcwTOCg1+sRHR0NAKitre1cnp6eDrVa3W39H/7whwgPD8eXX37Zr+PExcUhLCys2/Lp06dj6dKl2LJlC2w2Wz+jZ4YSFnn8CIvFghMnTiA1NdUlHVdCelNLb6r6+noA8oeRrhiNRqhUqs4PKdXV1diwYQPuu+8+pKamApBFlXfffRd1dXV48sknXfbx+uuvY8yYMTAYDJg1axa2bds24PMbbDzfffcdrrrqKiQnJ/e57V9UVBTq6uoGHDPD+BtDKRQr+c9//gMAuOKKKzyuI3Vw0el0ve6PRWgWoRn/hr+UGrl4BjP/8lzDMIGFeLZMSxCEzmU2mw2vvvoqFi5ciKioKKjVagiCAJVKhZaWFpSXl/f7OP/73/9w5ZVXIjExEVqttrNU9T//+Q8sFgvfX3kZFnn8iKNHj8Jms+Hiiy922w1Kqv9OS0sDgE7jQrPZ7HZ/giDAaDR2fkh55ZVXIIoiVq5cCaPRCED+JuqVV15BZmYmrrvuus7tP/zwQ6xcuRI/+clPcPDgQcyfPx+XXnrpgD8oDDaetrY2TJkyBa+88kqfTRtNJhMbPDKjhqEWiiVaWlo6Bd6ebjKKi4sBAJMnT+41VhahWYRm/Bdf+1JqIO+/nvCl+Wko5l+eaxgmMDCbzZ1zT2xsbOfym266CQ8//DAqKytx9dVX4+mnn+702TEajbBYLP06ziuvvIIrrrgC27dvx+LFi/Hoo4/iueeew6pVqzB9+nQA6Pc+maGFW6j7EdK3YhkZGW5f/9///gcAWLZsGQB0ts6TPjy5IzIyEq2trWhra8Mbb7yBW265BUlJSQAAtVqNhoYGbNq0CcePH8err77qIi794Q9/wIoVK3DvvfcCoG/rvvjiC7zxxht46aWXBnSOg4nnsssuw2WXXQaATMN6w+l0oqmpCWPGjBlQrAzjbwy1UCzxxRdfwGazYebMmUhJSXG7jsPh6EwHvvzyy3uNtSfRV0oR7osI/frrr2PhwoV4/fXXcemll+LEiROd59cfBhuPJELfcccduOOOO/p0TBahGX9lJL+UGq73X0/40vw0FPMvzzUMExh8//33sNvtiI+P77xf3LdvH/71r3/hwgsvxOeff95p0g7QvdBvf/vbfh3Dbrdj9erVSEhIwIEDB7plYO7cuXPQ58EMHs7k8SMkkaepqanbaw0NDVi7di1SU1M7HdETExMRGxvbYycHSVRZu3YtmpubXerFw8PD0dDQgJdffhnR0dGd7fQAwGq1Yv/+/Z2CksSyZcuwY8eOAZ/jQOMZCKdOnYIoipgxY8ag9sMw/sJwCMUA8N///hcAcOWVV3pcZ926daivr8fcuXM7O9v1hifRNywsrJvo+9hjj3kUoSdOnIg//elPSExMxBtvvNGnYw91PJdddhlefPFFXH/99W5versiidDS74Bh/ImR/FJqON5/fcFX5qfBzr881zBMYOB0OvGrX/0KAHDLLbd0Lj99+jQA4KqrrnIReABgz549MJlM3fYl+fc4HI5ur9XV1aGpqQnz58/vJvC0tbV1zv+Md2GRx4+Q2qf/85//RHt7e+fytrY23HLLLaivr8fLL78Mg8EAgL5pWrx4Merq6jrf4F2JjIxEY2MjXn75ZVx22WWYNGlS52tGoxG7d+/GF198gQcffNDFzK+urg4OhwPx8fEu+4uPj0dVVVW346xYsaJPKdIDjWcg7Nq1CwBw/vnnD2o/DOMvDIdQ7HQ68dlnnwHwfJNx+vRpPPvss9Dr9f26iWERmkVoxj/xpS+lhgtfmJ+GYv7luYZh/J+amhrcfPPN2Lp1K9LS0vCTn/yk8zVJbN+6dWu3bR566CG3+5NKat1ZcMTFxSE4OBj79+/v9PoCqPx25cqVXPrpI7DI4yfY7XYcPXq0s85x2rRpeOyxx/Dggw8iKysLX375JV566SWXdGAAWL58OQB4dE2XRJXS0tJuXR8iIiJw7Ngx6PV6/PjHPx5U/E6nEwC6KcjeigcANm3aBLVa3fkhk2ECneEQinfu3In6+nokJiZi1qxZ3V7ftGkTzjvvPLS0tOD999/HzJkz+xwvi9AsQjP+iS99KdVfhns+6O/81BNDMf/yXMMw/sXq1auxevVqPPfcc3jkkUdwwQUXIDU1FRs3bkROTg42b96MmJiYzvXnzJmDBQsW4OOPP8b8+fPx9NNP484778SUKVMQHBzcaYuh5Nxzz0VwcDBefvll/PjHP8Yvf/lL/PKXv0RzczNUKhUeeeQRFBcXY+rUqXj00Ufx4IMPYtq0afjvf//Lc4mPwJ4Line truncated
|
||||
"text/plain": [
|
||||
"<Figure size 2000x400 with 10 Axes>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"plot(I=3)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "ea1a699d-8e0c-4652-ac5e-0e68dcc6251a",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "05f0627c-aa65-47e0-8f15-770daa7051e8",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Bayesian fits in practice\n",
|
||||
"\n",
|
||||
"The plots shown previously look very nice, but in practice this is extremely slow: we are effectively trying all lines that may fit the points and excluding the ones that do not. We can do better than this in the simple case of a line fit.\n",
|
||||
"\n",
|
||||
"Imagine we are trying to find the weights $w$ for a function $y = f(x) = w_M x_M + \\ldots + w_2 x_2 + w_1 x_1 + w_0$. We assume that any mismatch between data and the prediction is caused by some random noise $\\epsilon$ that is Gaussian and has standard deviation $1/\\sqrt{\\beta}$. In this case, we can assume that the probability that data $\\mathcal{D}$ matches our prediction for a certain $x$ is given by the likelihood $p(\\mathcal{D}|w) = \\mathcal{N}(y=f(x), \\frac{1}{\\beta})$.\n",
|
||||
"\n",
|
||||
"For a line fit, we can assume that the prior probability of the weights is given by $p(w) = \\mathcal{N}(m_0, S_0)$ (ie: a Gaussian distribution, assuming we expect the weights to be close to $m_0$ and with some covariance $S_0$) and use the data likelihood $p(\\mathcal{D}|w)$ to get the posterior probability $p(w|\\mathcal{D})$.\n",
|
||||
"\n",
|
||||
"One can use Bayes' theorem to obtain the posterior, which follows exactly a Gaussian distribution for the weights $w$ with mean $m$ and covariance $S$ given by:\n",
|
||||
"\n",
|
||||
"$m = S \\, (S_0^{-1} m_0 + \\beta \\, \\Phi^T t)$,\n",
|
||||
"\n",
|
||||
"$S = S_0^{-1} + \\beta \\, \\Phi^T \\Phi$.\n",
|
||||
"\n",
|
||||
"This has been found by calculating the derivatives of the posterior distribution and setting it to zero to find its maximum.\n",
|
||||
"In those equations, $t$ is a vector with the value of $y$ for each point in the dataset, and $\\Phi$ (called the design matrix), includes the values of the vector $x$ for each point in the dataset.\n",
|
||||
"\n",
|
||||
"$\\Phi = \\begin{pmatrix}\n",
|
||||
"x_{M1} & \\ldots & x_{21} & x_{11} \\\\\n",
|
||||
"x_{M2} & \\ldots & x_{22} & x_{12} \\\\\n",
|
||||
"\\vdots & \\ddots & \\vdots & \\vdots \\\\\n",
|
||||
"x_{MN} & \\ldots & x_{2N} & x_{1N}\n",
|
||||
"\\end{pmatrix}\n",
|
||||
"$\n",
|
||||
"\n",
|
||||
"For more information, check Chapter 3 of C. Bishop's \"Pattern Recognition and Machine Learning\"."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "a52832a1-a1a5-4937-81ea-11ee35041fbf",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"The issue with this method is that one needs to make a prior assumption about how strongly we believe the weights $w$ to be close to our prior assumption $m_0$. That is: what are $m_0$ and $S_0$? These are very important to regularize the fit, which prevents the fit from trying to fit only outliers.\n",
|
||||
"\n",
|
||||
"To avoid this, hierarchical Bayesian fits are often used. In this case, we assume that $m_0$ is zero and $S_0$ depends of a result $\\lambda$ of a different probability distribution. That is, if we imagine how data would be produce in a purely theoretical way, we have a cascade of probability distribution: we pick $\\lambda$ from some distribution $p(\\lambda)$, then we pick $w$ from $p(w|\\lambda)$ and that allows us to produce data using $p(\\mathcal{D}|w, \\lambda)$.\n",
|
||||
"\n",
|
||||
"Several textbooks are dedicated to this method and the choice of such distributions. One common example is the usage of a $\\Gamma$ probability distribution for the variable $\\lambda$. This is done in the `BayesianRidge` method implemented in scikit-learn. You can find more information on this here: \n",
|
||||
"\n",
|
||||
"https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.BayesianRidge.html#sklearn.linear_model.BayesianRidge"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "b95b2b78-60ab-4d7b-91e9-3468c2c3d3d9",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Let's try using this method to fit the data in the variable `data`:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 11,
|
||||
"id": "46b4cee0-db3c-4352-886f-da9867f7ef53",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/html": [
|
||||
"<style>#sk-container-id-1 {color: black;}#sk-container-id-1 pre{padding: 0;}#sk-container-id-1 div.sk-toggleable {background-color: white;}#sk-container-id-1 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-1 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-1 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-1 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-1 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-1 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-1 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-1 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-1 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-1 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-1 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-1 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-1 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-1 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-1 div.sk-item {position: relative;z-index: 1;}#sk-container-id-1 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-1 div.sk-item::before, #sk-container-id-1 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-1 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-1 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-1 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-1 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-1 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-1 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-1 div.sk-label-container {text-align: center;}#sk-container-id-1 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-1 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-1\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>BayesianRidge()</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-1\" type=\"checkbox\" checked><label for=\"sk-estimator-id-1\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">BayesianRidge</label><div class=\"sk-toggleable__content\"><pre>BayesianRidge()</pre></div></div></div></div></div>"
|
||||
],
|
||||
"text/plain": [
|
||||
"BayesianRidge()"
|
||||
]
|
||||
},
|
||||
"execution_count": 11,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from sklearn.linear_model import BayesianRidge\n",
|
||||
"model = BayesianRidge()\n",
|
||||
"model.fit(data[:,[0]], data[:,1])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 12,
|
||||
"id": "6b6be7c2-3470-4621-b0dc-1eb90e6ac266",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<matplotlib.legend.Legend at 0x7fcc18691d60>"
|
||||
]
|
||||
},
|
||||
"execution_count": 12,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"image/png": "iVBORw0KGgoAAAANSUhEUgAABMUAAAGxCAYAAABiJQERAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAACqfElEQVR4nOzdeXhTZdoG8DtN1yTdF1poWQURRFqa4opA3UcRRcd9wd3xc8V9FgVHR0dFxW1wmREVd2VGHDeUTdybsgiIyqigINCFtknTNm2S8/3xcHqSNmnTNmt7/66rF01ymr5ND5Fz+zzPq1MURQEREREREREREdEAEhfpBRAREREREREREYUbQzEiIiIiIiIiIhpwGIoREREREREREdGAw1CMiIiIiIiIiIgGHIZiREREREREREQ04DAUIyIiIiIiIiKiAYehGBERERERERERDTjxkV5AX7ndbvz2229ITU2FTqeL9HKIiIiIiIiIiCiCFEWBzWbD4MGDERfnvx4s5kOx3377DUVFRZFeBhERERERERERRZFff/0VhYWFfh+P+VAsNTUVgPygaWlpEV4NERERERERERFFktVqRVFRUXtm5E/Mh2Jqy2RaWhpDMSIiIiIiIiIiAoBux2xx0D4REREREREREQ04DMWIiIiIiIiIiGjAYShGREREREREREQDDkMxIiIiIiIiIiIacBiKERERERERERHRgMNQjIiIiIiIiIiIBhyGYkRERERERERENOAwFCMiIiIiIiIiogGHoRgREREREREREQ04DMWIiIiIiIiIiGjAYShGREREREREREQDDkMxIiIiIiIiIiIacBiKERERERERERHRgMNQjIiIiIiIiIiIBhyGYkREREREREREA5GiAHY70NAQ6ZVERHykF0BERERERERERGHicABWqwRh1dXyeUoKcPjhgF4f6dWFFSvFyKdt27ZBp9Nh9uzZkV4KBWDVqlXQ6XSYO3dupJdCRERERERE0cTpBOrqgO3bAYsFWLMG+PxzYNMmCcYUBWhrkz8HGIZiA5BOp+vyY9GiRX6/dvbs2dDpdNi2bVvY1huLFi1a1O1r2RMMKYmIiIiIiCggbrdUf+3cCWzcCHzyCfDpp8DatcCePUBCAlBQABQVAbm5QHJypFccMWyfHMDuvPNOn/cXFxdjyJAh2LJlC9LT08O8KuqNyZMnY8uWLcjJyYn0UoiIiIiIiCjcmpq0lsg9e4DGRmmT1OsBo1HCr4SESK8y6jAUG8C6a7UbO3ZseBZCfWYwGPj7IiIiIiIiGigcDsBm0+aCNTQAzc2ATifzwdLTB3QFWKAi2j65fPlynHrqqcjPz0dSUhIGDx6M4447Du+9914kl0Xw3a6n0+nw/PPPAwBGjBjR3m45fPjwgJ/3lVdewfTp05GRkYHk5GQccMABuPvuu+FwONqPqaurw/Dhw5GUlITKykqvr3e73Zg+fTp0Oh1efPHF9vvnzp0LnU6HVatW4fnnn0dJSQlSUlKQl5eHiy++GLt37/a5nr179+L222/HAQccgJSUFKSnp+Ooo47CsmXL/P4Mr732Go466ihkZWUhOTkZw4cPx9lnnw2LxQIAmDZtGi666CIAwEUXXeTVmqq2nf7222+46667cPjhhyM/Px+JiYkYPHgwzjnnHHz77bde32/u3LkYMWIEAOD555/32era1UyxrVu34oILLsCQIUPav88FF1yArVu3djrW83V88803MXnyZBgMBmRlZeGss87Czp07/b4uREREREREFCLqXLBffpE2yDVrgM8+A775BqivBwwGoLBQPrKzGYgFKGKVYrfccgseeOABFBYW4uSTT0ZOTg6qq6tRWVmJVatW4Xe/+12klkZ+3HnnnfjPf/6DDRs24LrrrkNGRgYAtP/ZnYsvvhjPPfccCgsLcdpppyEjIwNffvkl/vKXv2D58uX46KOPEB8fj8zMTLzyyis48sgjceaZZ2LdunVITU0FAMybNw+rVq3C7Nmzcf7553f6Hg8//DCWLVuGM888E8cffzw+/fRTPPfcc1i1ahW++uor5Obmth+7fft2TJs2Ddu2bcOUKVNw/PHHw26347///S+OP/54PPXUU7jsssvaj1cUBRdddBGef/555OTkYNasWcjNzcWOHTuwcuVK7L///jCbzZg9ezYyMjLw9ttvY+bMmSguLm5/DvW1+uSTT3Dfffdh+vTpOO2002AymbB161a8+eabWLp0KT777DNMnDgRgIRs9fX1WLBgASZOnIhTTjml/fk8n9uXiooKHH300bDZbDj55JMxbtw4fPfdd1i8eDHefvttfPzxxygrK+v0dU8++SSWLl2Kk08+GVOnTsVXX32F1157DRs2bMD69euRlJTU3a+biIiIiIiIesvtBux2aYncu1eqwZqaZCB+YqK0RBYUDLjdIoMtIqHYM888gwceeAAXXnghnn76aSQmJno93tbWFollAQ89JB/dmTQJWLrU+76TT5a0tjtz5siHymYDDjggsPW9/TZQWhrYsQHwVVU0fPhwv8Pc586di23btmHDhg24/vrre1QhtmjRIjz33HM49dRT8dJLLyElJcXreefNm4cnnngC1113HQDg0EMPxT333INbb70Vl19+OV555RWsXLkSd999Nw444AA88cQTPr/P+++/j6+++golJSXt991www145JFHcNttt+Gf//xn+/0XXnghtm/fjldeeQVnnXVW+/319fWYNm0arr32Wpx88skYNGgQADlvn3/+eZSVleGjjz7ymrfmcrlQVVUFAO2v39tvv41TTjnF5+tZXl6OPXv2tId9qg0bNuDwww/Hbbfdhvfffx+AhGLDhw/HggULUFxcHPAOk4qi4IILLoDVasXixYtx7rnntj/22muv4ayzzsL555+Pb7/9FnFx3kWjH3zwASoqKjBhwoT2+8455xy88sorePvtt3HGGWcEtAYiIiIiIiIKUFOTZAT19UBVlXzuORcsJ4dzwYIs7KGYw+HAn/70JwwdOtRnIAYACZH6Jau7M3SnqKjzfdXVgX2t1ep9W1EC+zoAaG0N7LgAzZs3r9N9U6dODckOhwsWLEB8fDz+9a9/eQViAPCXv/wFjz/+OF566aX2UAwAbr75ZqxcuRKvvvoqDjroIDz22GNISkrCa6+9BoPB4PP7nH/++V6BGCCh23PPPYeXX34ZTz75JJKSkrBhwwasXr0ap59+ulcgBkg117x583DKKafgrbfewlVXXQUAeOyxxwAATz31VKcNCPR6PQoKCgJ+PfLy8nzeP3HiRJSXl2PZsmVoa2vr09+Fzz//HN999x0OPfRQr0AMAM4880w8/vjj+PTTT/Hpp5/iyCOP9Hr82muv9QrEAOCyyy7DK6+8gq+//pqhGBERERERUV+1tmrD8T3nggHSDsm5YCEX9lDso48+QnV1Na6//nrExcXh3XffxaZNm5CcnIzJkyfj0EMPDfeSNGlpwJAh3R/n0YLndV8gX5uW5n1bpwvs6wApkQwiRVGC+nz+NDU1YcOGDcjJycEjjzzi85ikpCRs2bLF6z6dTocXXngBxcXF+OMf/whAAqmOYY2nqVOndrovPT0dxcXFWL16NbZs2YLi4mJ88cUXAICGhgaflVfV1dUA0L4mu92OTZs2YdCgQZ1Ct9569913sXDhQlgsFtTU1MDpdHo9XlNT06OgraO1+yoXy8vLfT5eXl6OTz/9FOvWresUipnN5k7HF+0Lg+vq6nq9JiIiIiIiogHL5ZLqL6sVqKmRtki7XVolk5OlGiw7W3ICCouwh2IVFRUAgOTkZJSUlGDTpk1ejx955JF48803vWY/eXI4HF5D2a0dK6/6omNrY090bKcMVGoqsGNH7742RtTV1UFRFFRXV/usTutKbm4ujjzySLz66qvIzs72OUfMk9rq2FF+fj4ACcEAoLa2FoCEtB999JHf52tsbAQgLZUAMCTQALMbCxYswPXXX4/MzEwcc8wxGDp0KAwGA3Q6XfvcNs/zvDfUn9VfsKber/5snnzNiYuPl7cLl8vVp3URERERERENCIoCNDZKCFZXJ9VgdrtUiCUlcS5YFAh7KKbOXXrggQcwbtw4rFmzBsXFxfj5559x0003YdmyZfj973+PVatW+fz6e++9t8fBCkWW2mpYUlLSXr0UqFdffRWvvvoqcnJyUFNTg2uvvRbPPPOM3+P37Nnj835190l1LeqfCxYswLXXXtvtOtSQKBi7LzqdTsydOxf5+flYu3Ztp9BKrWLrK/Vn9Lfz5q5du7yOIyIiIiIioj5qbpYQTJ0L1tgItLRwLliUiuv+kOByu90ApOpk6dKlOOKII2AymTBhwgT8+9//RmFhIVavXu03GLj99tvR0NDQ/vHrr7+Gc/kDnn5fgt2TaiGTyYTx48dj8+bN2Lt3b8Bf97///Q+XX345cnNz21v8nn32Wbz66qt+v2b16tWd7mtoaMD69euRnJyMA/ZtanDIIYcAANasWRPQWoxGIw488EDs2bMH69at6/b4rl6nmpoa1NfX47DDDusUiDU2NvoMDnvzuqttnv4C5pUrVwIAJk2aFPBzEhERERERkYfWVmmF/Okn4MsvgU8+Ab74AtiyRQKytDRg6FAZm5SRwUAsyoQ9FFMrbkpKSjrtXmgwGHDccccBAL7++mufX5+UlIS0tDSvDwqf7OxsAMAvv/zSo6+bM2cOWltbcfHFF/ts16urq/MKg1pbW3HWWWehsbERzz//PAoLC/Hyyy8jOzsbV1xxBX788Uef3+fFF1/sFFrNnTsXDQ0NOPvss5GUlARAZmZNmTIFS5Yswb/+9S+fz7Vx48b2ykYA7RVlV1xxRXtrosrtdrdXXgFdv055eXkwGAyorKxsb88EZNfV6667DjU1NZ2+JjMzEzqdrkev++GHH479998fn376Kd58802vx958802sWbMGY8aMwRFHHBHwcxIREREREQ1oLpdUgf3Line truncated
|
||||
"text/plain": [
|
||||
"<Figure size 1500x500 with 1 Axes>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"# let's create x values to predict y for\n",
|
||||
"x_axis = np.linspace(-5, 10, 100)\n",
|
||||
"# predict it\n",
|
||||
"y_prediction, y_uncertainty = model.predict(x_axis[:, np.newaxis], return_std=True)\n",
|
||||
"\n",
|
||||
"# now plot it:\n",
|
||||
"plt.scatter(data[:,0], data[:,1], c='blue')\n",
|
||||
"plt.plot(x_axis, y_prediction, c='red', lw=2, ls='--', label=\"Fit expectation\")\n",
|
||||
"plt.fill_between(x_axis, y_prediction + 2*y_uncertainty, y_prediction - 2*y_uncertainty, color=\"red\", alpha=0.2, label=\"95% CL uncertainty\")\n",
|
||||
"plt.legend(frameon=False)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "8f30a61e-bcad-425d-ac9b-191c93734c2a",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Outliers"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "29563a43-67d0-4679-a4ea-16e4f883f9de",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"In the previous notebook, we discussed that outliers are a big problem there, since there is no regularization. Since the method above includes a prior probability on the weights that is Gaussian, this restricts the variation of the weights due to outliers. This is not perfect and the degree of regularization needs to be tuned in case of large outliers, but it is more resilient than the unregularized method.\n",
|
||||
"\n",
|
||||
"Feel free to try adding outliers to the data and checking what happens to the parameters of the fit."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "dc429c63-6255-4c28-ba31-4f2831d039ee",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "b6c0ced7-ce4b-420a-8c50-8b71c6c10e59",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Is this the only way?"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "13aebeb5-99b4-44e6-9106-2c0ac7c0e874",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"There are *a lot* of other line fitting methods! They make different assumptions on the priors and likelihoods. We could make the assumption that $\\epsilon$ follows a different distribution, changing the fit entirely: this is helpful to avoid sensitivity to outliers in some cases. Another change is that the priors could also follow a different distribuion. Using the Laplace distribution instead of the Gaussian often makes those parameters much more robust to outliers (this is called the *LASSO* estimator). Other line fitting methods could even assume that the prior probabilities depend on another probability distribution, called the *hyper-prior*, which then would also be included in Bayes' Theorem.\n",
|
||||
"\n",
|
||||
"One example is called Automatic Relevance Determination, which is a variation of this method, using a slightly different setup for the probability distribution of $\\lambda$. This other method has the advantage of being very resilient to noisy data in multi-dimensional cases and setting several weights $w$ to zero if they are smaller than the uncertainty.\n",
|
||||
"\n",
|
||||
"More information on an implementation of ARD can be found here: https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.ARDRegression.html#sklearn.linear_model.ARDRegression\n",
|
||||
"\n",
|
||||
"Why not try it in this simple example?"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "acfac850-d650-4f25-ae1e-6108498bb743",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "b6c3c7b6-8a2f-4d4e-9ae6-c2ee18232c06",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.10.12"
|
||||
},
|
||||
"widgets": {
|
||||
"application/vnd.jupyter.widget-state+json": {
|
||||
"state": {},
|
||||
"version_major": 2,
|
||||
"version_minor": 0
|
||||
}
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
@@ -0,0 +1,492 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "1bba0128",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Non-linear fits as extensions of line fits\n",
|
||||
"\n",
|
||||
"Line fits are really useful and they can be easily modified to perform non-linear fits too.\n",
|
||||
"\n",
|
||||
"One way of modifying them is to simply \"create\" new variables. For example, if one would like to fit a parabola $y = f(x) = a x^2 + b x + c$, one could define a new variable $u = x^2$ and fit the \"plane\" $y = f(x, u) = a u + b x + c$. This fit may be performed with the methods described previously. There are even tools in standard libraries to aid fitting using such procedures.\n",
|
||||
"\n",
|
||||
"An alternative is to perform a non-linear transformation on $x$, in such a way that the important non-linear effects are captured.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"id": "896d5976-fe56-4a2b-8093-c01fc53b0068",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from typing import Tuple\n",
|
||||
"\n",
|
||||
"import pandas as pd\n",
|
||||
"import numpy as np\n",
|
||||
"import matplotlib.pyplot as plt\n",
|
||||
"\n",
|
||||
"import matplotlib.pylab as pylab\n",
|
||||
"params = {'legend.fontsize': 'x-large',\n",
|
||||
" 'figure.figsize': (15, 5),\n",
|
||||
" 'axes.labelsize': 'x-large',\n",
|
||||
" 'axes.titlesize':'x-large',\n",
|
||||
" 'xtick.labelsize':'x-large',\n",
|
||||
" 'ytick.labelsize':'x-large'}\n",
|
||||
"pylab.rcParams.update(params)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "843e1f00-4846-4e15-9861-6e9e866a6d62",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Fitting polynomials\n",
|
||||
"\n",
|
||||
"scikit-learn offers the possibility of easily creating new variables corresponding to the polynomial expansions of $x$.\n",
|
||||
"This can be done using: https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.PolynomialFeatures.html#sklearn.preprocessing.PolynomialFeatures\n",
|
||||
"\n",
|
||||
"Let's look at an example of how this could be done:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "bb1286f0",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"The simulated example data will be $f(x) = 0.01 x^2 + 0.5 x + 0.3 + \\epsilon$, where $\\epsilon \\sim \\mathcal{N}(\\mu=0, \\sigma=0.5)$.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"id": "aef54435-78af-4b13-aee2-2bc576c773cd",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def generate_data(N: int) -> np.ndarray:\n",
|
||||
" x = 2*np.random.randn(N, 1)\n",
|
||||
" epsilon = 0.5*np.random.randn(N, 1)\n",
|
||||
" z = 0.1*x**2 + 0.5*x + 0.3 + epsilon\n",
|
||||
" return np.concatenate((x, z), axis=1).astype(np.float32)\n",
|
||||
"\n",
|
||||
"data = generate_data(N=100)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "00e098e8-0082-4451-927a-073cf242787e",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Let's plot it:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"id": "08d9daaf-25c5-4d06-b4a6-6cecaaa35651",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<matplotlib.collections.PathCollection at 0x7f1a8832b4f0>"
|
||||
]
|
||||
},
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"image/png": "iVBORw0KGgoAAAANSUhEUgAABMkAAAGxCAYAAAB4GeGfAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAABHjElEQVR4nO3df3TcdZ0v/tckQMOPZqDFmgDF1iJoTnWhnm+/VCs/qpWKp4s/jqwuxR8rKL0rVFn3AoezlMBd2O7KRTgqXsFvUfP9cr14VeiV21u3LfWCxV4tZbdbF7AGbhdSK+2S1LqJkMz3j9wpTZM0M5OZ+Xxm5vE4p39k5pPMK5NJmzz7er3emVwulwsAAAAAaGBNSRcAAAAAAEkTkgEAAADQ8IRkAAAAADQ8IRkAAAAADU9IBgAAAEDDE5IBAAAA0PCEZAAAAAA0vKOSLmCyhoaG4sUXX4ypU6dGJpNJuhwAAAAAEpTL5WL//v1xyimnRFNT4f1hNR+SvfjiizFz5sykywAAAAAgRXbt2hWnnXZawdfXfEg2derUiBj+xFtbWxOuBgAAAIAk9fX1xcyZMw9mRoWq+ZAsP2LZ2toqJAMAAAAgIqLotVwW9wMAAADQ8IRkAAAAADQ8IRkAAAAADU9IBgAAAEDDE5IBAAAA0PCEZAAAAAA0PCEZAAAAAA1PSAYAAABAwxOSAQAAANDwhGQAAAAANLyjki4AAAAAIO0Gh3KxpXtf7NnfHzOmtsT82dOiuSmTdFmUkZAMAAAA4AjWbu+JzjU7oqe3/+Bt7dmWWLm0I5bMbU+wMsrJuCUAAADAONZu74nlXVtHBGQREbt7+2N519ZYu70nocooNyEZAAAAwBgGh3LRuWZH5Ma4L39b55odMTg01hXUGiEZAAAAwBi2dO8b1UF2qFxE9PT2x5bufdUrqowGh3KxeefeeGjbC7F5596GD/vsJAMAAAAYw5794wdkpVyXJvasjaaTDAAAAGAMM6a2lPW6tLBnbWyJhGSzZs2KTCYz5p+2trYkSgIAAAAYYf7sadGebYnMOPdnYrj7av7sadUsa1LsWRtfYuOW2Ww2Pv/5z4+6/YQTTqh+MQAAAACHaW7KxMqlHbG8a2tkIkYES/ngbOXSjmhuGi9GS59i9qwtmDO9eoWlQGIh2Yknnhg333xzUg8PAAAAMKElc9vjnmXzRu3vaqvR/V31vGdtsizuBwAAAIo2OJSLLd37Ys/+/pgxdXjksJY6qoqxZG57LO5oq4vPt173rJVDYiHZwMBAdHV1xf/+3/87jj/++Hjb294W5513XjQ3NydVEgAAAFCARjwZsbkpUxfjh/k9a7t7+8fcS5aJ4S65WtqzVi6ZXC5X9U1ss2bNiueff37U7bNnz47Vq1fH+eefP+77DgwMxMDAwMG3+/r6YubMmdHb2xutra0VqRcAAAAYlj8Z8fAwId9Tdc+yeXUblNWL/NcwYuw9a7X+Nezr64tsNlt0VpTI6Zaf+tSnYv369bF79+44cOBA/OM//mN89rOfjeeeey7e9773xVNPPTXu+95+++2RzWYP/pk5c2YVKwcAAIDG5WTE+pDfs9aWHTlS2ZZtqfmAbDIS6SQbzxe/+MW444474gMf+ED84Ac/GPManWQAAACQjM0798bH7n1iwuseuPLcuhhNrHf1uleu1E6yVC3uv+qqq+KOO+6In/zkJ+NeM2XKlJgyZUoVqwIAAAAinIxYb+plz1q5JDJuOZ7Xve51ERFx4MCBhCsBAAAADudkROpZqkKyJ54Ybtl84xvfmHAlAAAAwOHyJyOON5CXieFTLhvxZERqX9VDsl/+8pdjdoo999xz8bnPfS4iIpYtW1btsgAAAIAJNDdlYuXSjoiIUUFZ/u2VSzvqYq8VjafqO8m++93vxh133BHnnXdevOENb4ipU6fGzp0740c/+lH09/fHxRdfHF/84herXRYAAABQgPzJiJ1rdkRP72u7x9qyLbFyaUfDnoxI7at6SHbhhRfG008/HU8++WQ8/vjjceDAgTjxxBNj4cKFcfnll8fll18emYzEGQAAANJqydz2WNzRVpcnI9K4MrlcLpd0EZNR6rGeAAAAANSfUrOiVC3uBwAAAIAkCMkAAAAAaHhCMgAAAAAanpAMAAAAgIYnJAMAAACg4QnJAAAAAGh4QjIAAAAAGp6QDAAAAICGJyQDAAAAoOEJyQAAAABoeEIyAAAAABqekAwAAACAhickAwAAAKDhCckAAAAAaHhCMgAAAAAanpAMAAAAgIYnJAMAAACg4QnJAAAAAGh4QjIAAAAAGp6QDAAAAICGJyQDAAAAoOEJyQAAAABoeEIyAAAAABqekAwAAACAhickAwAAAKDhCckAAAAAaHhCMgAAAAAanpAMAAAAgIYnJAMAAACg4QnJAAAAAGh4RyVdAAAAAJAOg0O52NK9L/bs748ZU1ti/uxp0dyUSbosqAohGQAAABBrt/dE55od0dPbf/C29mxLrFzaEUvmtidYGVSHcUsAAACoksGhXGzeuTce2vZCbN65NwaHckmXFBHDAdnyrq0jArKIiN29/bG8a2us3d6TUGVQPTrJAAAAoArS2qk1OJSLzjU7Yqy4LhcRmYjoXLMjFne0Gb2krukkAwAAgApLc6fWlu59o+o6VC4ienr7Y0v3vuoVBQlITUjW1dUVmUwmMplM3HfffUmXAwAAAGUxUadWxHCnVlKjl3v2jx+QlXId1KpUhGS7du2Kz33uc3HCCSckXQoAAACUVdo7tWZMbSnrdVCrEg/JcrlcfOpTn4rp06fHVVddlXQ5AAAAUFZp79SaP3tatGdbYrxtY5kY3p02f/a0apYFVZd4SHb33XfHhg0bYvXq1XH88ccnXQ4AAACUVdo7tZqbMrFyaUdExKigLP/2yqUdlvZT9xINyX75y1/G9ddfHytWrIjzzjsvyVIAAACgImqhU2vJ3Pa4Z9m8aMuODOrasi1xz7J5iZ6+CdVyVFIP/Oqrr8bll18ep59+etx2220Fv9/AwEAMDAwcfLuvr68S5QEAAEBZ5Du1lndtjUzEiAX+aerUWjK3PRZ3tMWW7n2xZ39/zJg6HNwlXRdUS2KdZLfccks8+eSTcf/998exxx5b8Pvdfvvtkc1mD/6ZOXNmBasEAACAyauVTq3mpkwsmDM9Ljn71FgwZ7qAjIaSSCfZz372s7jtttviL/7iL2LBggVFve8NN9wQ11577cG3+/r6BGUAAACknk4tSLeqh2SvvvpqfPzjH48zzzwzbr311qLff8qUKTFlypQKVAYAAACVle/UAtInk8vlchNfVj4vv/xynHTSSQVdu2LFivjyl798xGv6+voim81Gb29vtLa2lqFCAAAAAGpVqVlR1TvJpkyZEp/+9KfHvG/r1q3x5JNPxsKFC+Oss84qehQTAAAAAEpR9ZDs2GOPjfvuu2/M+26++eZ48skn4xOf+ERcccUVVa4MAAAAgEaV2OmWAAAAAJAWQjIAAAAAGl7VF/eXm8X9AAAAAOSVmhXpJAMAAACg4QnJAAAAAGh4QjIAAAAAGp6QDAAAAICGJyQDAAAAoOEJyQAAAABoeEIyAAAAABreUUkXAAAAAIUYHMrFlu59sWd/f8yY2hLzZ0+L5qZM0mUBdUJIBgAAQOqt3d4TnWt2RE9v/8Hb2rMtsXJpRyyZ255gZbVDyAhHJiQDAAAg1dZu74nlXVsjd9jtu3v7Y3nX1rhn2TxB2QSEjDAxO8kAAABIrcGhXHSu2TEqIIuIg7d1rtkRg0NjXUHEayHjoQFZxGsh49rtPQlVBukiJAMAACC1tnTvGxXuHCoXET29/bGle1/1iqohQkYonJAMAACA1Nqzf/yArJTrGo2QEQonJAMAACC1ZkxtKet1jUbICIUTkgEAAJBa82dPi/ZsS4x3BmMmhhfQz589rZpl1QwhIxROSAYAAEBqNTdlYuXSjoiIUUFZ/u2VSzuiuWm8GK2xCRmhcEIyAAAAUm3J3Pa4Z9m8aMuO7HZqy7bEPcvmxZK57QlVln5CRihcJpfL1fQRFn19fZHNZqO3tzdaW1uTLgcAAIAKGRzKxZbufbFnf3/MmDrc/STcKcza7T3RuWbHiCX+7dmWWLm0Q8hI3Sk1KxKSAQAAQAMQMtIoSs2KjqpgTQAAAEBKNDdlYsGc6UmXAallJxkAAAAADU8nGQAAQI0zRgcweUIyAACAGmYhO0B5GLcEAACoUWu398Tyrq0jArKIiN29/bG8a2us3d6TUGUAtUdIBgAAUIMGh3LRuWZH5Ma4L39b55odMTg01hUAHE5IBgAAUIO2dO8b1UF2qFxE9PT2x5bufdUrKgGDQ7nYvHNvPLTthdi8c69QECiZnWQAAAA1aM/+8QOyUq6rRfaxAeWkkwwAAKAGzZjaUtbrao19bEC5CckAAABq0PzZ06I92xKZce7PxHBX1fzZ06pZVlXYxwZUgpAMAACgBjU3ZWLl0o6IiFFBWf7tlUs7orlpvBitdtnHBlSCkAwAAKBGLZnbHvcsmxdt2ZEjlW3Zlrhn2by63ctlHxtQCRb3AwAA1LAlc9tjcUdbbOneF3v298eMqcMjlvXYQZbX6PvYgMoQkgEAANS45qZMLJgzPekyqia/j213b/+Ye8kyMdxNV4/72IDKMW4JAABATWnkfWxA5SQSkl133XXx7ne/O2bOnBnHHntsTJs2Lc4555zo7OyMvXv3JlESAAAANaRR97EBlZPJ5XJVPxP3mGOOiXnz5kVHR0fMmDEjDhw4EE888UT8/Oc/j1NOOSWeeOKJmDlzZkEfq6+vL7LZbPT29kZra2uFKwcAACBNBodyDbWPDZhYqVlRIjvJ+vr6oqVl9ALFG2+8MW677ba4/fbb42tf+1oClQEAAFBLGm0fG1A5iYxbjhWQRURceumlERHx7LPPVrMcAAAAABpcqhb3r1mzJiIi3va2tyVcCQAAAACNJJFxy7wvfelLine truncated
|
||||
"text/plain": [
|
||||
"<Figure size 1500x500 with 1 Axes>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"plt.scatter(data[:,0], data[:,1])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "e9932b25-ffa6-4690-965e-6822643a8edd",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Now, let's fit it with a line:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"id": "50578b88-f210-4a35-bada-71a26e45e689",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/html": [
|
||||
"<style>#sk-container-id-1 {color: black;}#sk-container-id-1 pre{padding: 0;}#sk-container-id-1 div.sk-toggleable {background-color: white;}#sk-container-id-1 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-1 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-1 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-1 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-1 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-1 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-1 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-1 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-1 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-1 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-1 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-1 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-1 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-1 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-1 div.sk-item {position: relative;z-index: 1;}#sk-container-id-1 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-1 div.sk-item::before, #sk-container-id-1 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-1 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-1 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-1 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-1 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-1 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-1 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-1 div.sk-label-container {text-align: center;}#sk-container-id-1 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-1 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-1\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>BayesianRidge()</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-1\" type=\"checkbox\" checked><label for=\"sk-estimator-id-1\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">BayesianRidge</label><div class=\"sk-toggleable__content\"><pre>BayesianRidge()</pre></div></div></div></div></div>"
|
||||
],
|
||||
"text/plain": [
|
||||
"BayesianRidge()"
|
||||
]
|
||||
},
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from sklearn.linear_model import BayesianRidge\n",
|
||||
"\n",
|
||||
"model = BayesianRidge()\n",
|
||||
"model.fit(data[:, [0]], data[:, 1])\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"id": "9e317638-ed75-418f-af18-da8dd585265a",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<matplotlib.legend.Legend at 0x7f1a7ee36640>"
|
||||
]
|
||||
},
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"image/png": "iVBORw0KGgoAAAANSUhEUgAABMkAAAGxCAYAAAB4GeGfAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAACjZklEQVR4nOzdeXicZfn+/3OSZpnJ1jRp0i0llKUL0LRQEYXSBeEDIoKAigiyKeCCoCiLIBRFRX6CwNcFRKUKIoiWTXYpRVaxWNKFnUIppVvSJmn2zPL74+LJM5NMkkkymSXzfh1HjjbPPJm5k2Y6M+dc13V7QqFQSAAAAAAAAEAGy0r2AgAAAAAAAIBkIyQDAAAAAABAxiMkAwAAAAAAQMYjJAMAAAAAAEDGIyQDAAAAAABAxiMkAwAAAAAAQMYjJAMAAAAAAEDGG5PsBQxXMBjUhx9+qKKiInk8nmQvBwAAAAAAAEkUCoW0a9cuTZo0SVlZsdeHpX1I9uGHH6qqqirZywAAAAAAAEAK2bhxo6ZMmRLz+WkfkhUVFUmyb7y4uDjJqwEAAAAAAEAyNTU1qaqqqjszilXah2ROi2VxcTEhGQAAAAAAACRp0GO5GNwPAAAAAACAjEdIBgAAAAAAgIxHSAYAAAAAAICMR0gGAAAAAACAjEdIBgAAAAAAgIxHSAYAAAAAAICMR0gGAAAAAACAjEdIBgAAAAAAgIxHSAYAAAAAAICMR0gGAAAAAACAjDcm2QsAAAAAAABIdYGA9Mwz0ubN0sSJ0vz5UnZ2sleFeCIkAwAAAAAA6MeyZdL550sffOAemzJFuvFG6fjjk7cuxBftlgAAAAAAAH1Ytkw68cTIgEySNm2y48uWJWddiD9CMgAAAAAAgCgCAasgC4V6X+Ycu+ACOy8dBQLSihXSX/9qf6br9xEvhGQAAAAAAABRPPNM7wqycKGQtHGjnZduli2TqqulRYukk0+2P6urM7syjpAMAAAAAAAgis2b43teqqCFNDpCMgAAAAAAgCgmTozvealgtLeQDgchWQbyeDwRH3l5eRo/frz2339/ffWrX9UjjzyiQJzuDUuXLpXH49HSpUvjcn0AAAAAACTK/Pm2i6XHE/1yj0eqqrLz0sVobiEdrjHJXgCS58orr5QkBQIBNTQ0aN26dbr99tv1hz/8QfPmzdNf/vIX7b333kleJQAAAAAgVQUCFqZs3mzVVPPnS9nZyV5V/GRnSzfeaC2IHo+UFfKrUM0qVLN8atX7od10ww15afU9j9YW0nggJMtgS5Ys6XVs69atOu+883TPPffoU5/6lFauXKmKiorELw4AAAAAkNKWLbO2vfCqpClTLFQ6/vjkrSuuOjp0/IJmPfSbZt145Q51btshr9qUq06NLc/RddeN1zHH5yV7lYMyGltI48UTCkXrQk0fTU1NKikpUWNjo4qLi5O9HEmpn6R7PqoT7eufPhgM6rDDDtOKFSt0/vnn64Ybbui+7OWXX9af//xnrVixQhs3blRra6uqqqr02c9+VpdffrlKS0u7z124cKGefvrpqLfx7rvvqrq6Wh9++KF+//vf67HHHtM777yjHTt2qLy8XAsXLtTll1+uWbNmxe8bBwAAAADEhTP4vefLSqct8e9/T8OgLBSS2tqk5mb7qKuTGhul1lYpGFQgO1er3/Zq2y6fyiqyNHfSVmUfeogU9jo4HQQCtovlpk3R55J5PBZ2vvtuamUZgzHUrIiQLM7SIUkfKCSTpCeffFKf+tSnVFFRoS1btnR/zbnnnqt7771XCxYsUFVVlYLBoF5++WU988wzmjlzpv7zn/+oqKhIks0ju++++3T//ffr2GOP1Zw5c7qv/4ILLtDYsWN111136cwzz9SiRYtUXV2twsJCvfXWW/rnP/+p3NxcPffcc6qpqRm5HwYAAAAAYFCckKWvuVZpE7IEg1JLiwViu3ZZKLZrl9TebulRXp7k80leb+9vJBCQtm6VDkm/kExyQ04pMihL65AzDCFZCoRk6ZKkxxKSdXR0qLCwUH6/X+vXr9fuu+8uSdqwYYOmTJmi7B7/QfzhD3/QV7/6VV1zzTW6+OKLu48vXbpUZ5xxhm677TadfvrpvW5n27Zt8nq93cGao7a2VgcffLDmz5+vRx55ZKjfKgAAAAAgzlaskBYtGvi8p56SFi4c6dUMgt/vhmINDRaKtbZKHR32wj0/30Kx/Hwpa4B9DtM8JJOiF/lUVUk33JAa2cVwDDUrYnfLOBltW6jm5eWprKxMkrR9+/bu47vttluvgEySzjzzTBUXF+uxxx4b1O1UVFT0CsgkqaamRosXL9ZTTz2lrq6uQa4eAAAAADBS0mbwe2entGOH9P77Um2t9O9/S88+K730kvT223Z5cbGVvU2ZIpWXW0g2UEA2Shx/vPTeexZm3nmn/fnuu+kfkA0Hg/vjZDBbqKZUkt4Pp9LME7bXbVdXl2655RbdddddevXVV9XY2KhgMNh9+aZNmwZ9Ow899JBuvvlmrVy5UnV1dfL7/RGX19XVaWImTgwEAAAAgBSUsoPf+5onFghIOTnWNjlunJSbm+CFpa7s7PTJKBKBkCxO0iZJj1F7e7t27NghSRo/fnz38S9+8Yu69957NW3aNB177LGaMGGC8vJsJ48bbrhBHR0dg7qdG2+8URdccIFKS0t1+OGHa+rUqfL5fPJ4PLrvvvtUW1s76OsEAAAAAIyc+fOt8Gqgwe/z54/gIkKh3vPEmprceWK5uVYVVlkpjSH6QGz4TYmTlE3Sh+jZZ5+V3+9XZWWlqqurJUkrV67Uvffeq0996lN65JFHNCbsP5pgMKhrr712ULfh9/u1ZMkSTZgwQf/73/96VYu98MILw/4+AAAAAADxlZ1tm9OdeKIFYtEGv99wQ5yH9gcCbpVYY6M7T6y93Z0n5lSKZUi7JOKPkCxOUiJJj5NgMKif/OQnkqSTTz65+/jbb78tSfrsZz8bEZBJ0ksvvaS2trZe1+XMLwtEGcZWV1enhoYGHX/88b0CsubmZv3vf/8b3jcCAAAAABgRxx9vm9P1HPw+ZUqcBr93dVmFWHOztHOnzRZrbbXjWVkWiBUV2RyxsBFBwHAQksVJUpL0EbBt2zZ961vf0ooVKzR16lT94Ac/6L7MqShbsWKFzjvvvIiv+eY3vxn1+pzh/++//36vyyoqKuTz+fTyyy+rublZhYWFkmzu2fnnn6+6urp4fVsAAAAAgDg7/njp2GNt9vbmzdY5NX/+EF/3trf3PU9szBjmiSEhCMniaMST9DhbsmSJJKsca2ho0Lp16/Tss8+qs7NTBx54oP7yl7+ovLy8+/yPfexjOvjgg7Vs2TJ98pOf1CGHHKKtW7fqkUce0fTp0zVp0qRet/GJT3xCPp9PN9xwg+rr6zVhwgRJ0nnnnaeSkhJ9+9vf1jXXXKP99ttPxx57rDo7O/XUU09px44dWrRokZ566qmE/CwAAAAAAIM3pMHvoZAFYD3nibW1ScGgBWEFBVJFBfPEkFCeUChac2D6aGpqUklJiRobG1VcXJzs5UiyoDsuSfoI8fQoRc3NzVVRUZF222037b///jrhhBN0xBFHKCtKH/eOHTt0+eWX6+GHH9aWLVs0efJkffGLX9Tll1+uWbNmSZLee++9iK959NFHddVVV2nNmjVqaWmRJL377ruqrq6W3+/XTTfdpN///vd69913VVJSosMPP1xXX321rrzySv3pT3/qPhcAAAAAkIYCAXfIflOTtH27fd7ebpc788R8vvSZJxYISFu3SoccIpWWJns16GGoWREhGQAAAAAAiJ+uLrd1cudOqb7eKsc6O915Yj6fhWPpOk+MkCylDTUrom4RAAAAAAAMXfg8sfp6qaHBQjG/39olfT4LkvLykr1SoF+EZAAAAAAAIDb9zRMLhaScHAvFmCeGNMRvLAAAAAAAiK7nPLG6Ovu8rc0ud+aJjR2bWsO4gSEgJAMAAAAAACZ8nlhDg7Rjh4VinZ02P8zrtZ0ny8rSd54Y0AdCMgAAAAAAMlX4PLEdO2zQflubzRPLzrbWybFjmSeGjEBIBgAAAABAJog2T2zXLjsWPk9s/HjmiSEj8VsPAAAAAMBoFAy6VWLh88Ta2y0Uc+aJlZQwTwwQIRkAAAAAAKMD88SAYSEkAwAAAAAgHTFPDIgrQjIAAAAAAFId88SAEcc9BwAAAACAVMM8MSDhCMkAAAAAAEg25okBSUdIBgAAAABAojFPDEg5WcleABDO4/Fo4cKFEceWLFkij8ejFStWjMhtvvfee/J4PDr99NNH5PoBAAAAZLhQyKrCtm6V3n5bevFF6ZlnpOeek155Rdq8WcrKsnliVVXSpEkEZEASUEmWgTw9SnOzsrJUWlqq2bNn66tf/apOPvnkJK1s5Hg8Hi1YsGDEgjYAAAAA6BYIWCjGPDEgrRCSZbArr7xSktTV1aXXX39d999/v5566imtXLlS119/fZJX5/rWt76lk046SVOnTh2R6588ebJee+01lZSUjMj1AwAAABjles4Tq69354llZTFPDEgThGQZbMmSJRGfP/nkkzr88MN1ww036Nvf/raqq6uTsq6eysvLVV5ePmLXn5OToxkzZozY9QMAAAAYZcLnidXXWzDW2mrzxMaMsXlipaW0SwJphplk6HbYYYdpxowZCoVC+u9//yspch7YnXfeqY9//OMqLCyMCNBaW1v1s5/9THPmzFFBQYEKCwv1iU98Qn/961+j3k5nZ6d+/OMfa4899lBeXp523313XX755ero6Ih6fn8zyV5//XWdeeaZqq6uVl5enioqKjR//nz99re/lSQtXbq0u7306aeflsfj6f5wQsL+ZpJt3rxZ3/zmN1VdXa3c3FyNHz9exx9/vF5++eVe5zq3tXTpUj311FNauHChioqKVFxcrKOPPlqvvfZaXz96AAAAAKkq2jyxf//b5omtWiVt2WLVYhUV0tSpzBMLine truncated
|
||||
"text/plain": [
|
||||
"<Figure size 1500x500 with 1 Axes>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"x_axis = np.linspace(-5, 5, 100)\n",
|
||||
"y_p, y_u = model.predict(x_axis[:, np.newaxis], return_std=True)\n",
|
||||
"\n",
|
||||
"plt.scatter(data[:, 0], data[:, 1], c='b', label=\"Data\")\n",
|
||||
"plt.plot(x_axis, y_p, c='r', lw=2, ls='--', label=\"Prediction\")\n",
|
||||
"plt.fill_between(x_axis, y_p-2*y_u, y_p+2*y_u, color='red', alpha=0.2, label=\"95% CL uncertainty\")\n",
|
||||
"plt.legend(frameon=False)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "446bb34d-7ccc-481a-8514-c1ee20fcc5e0",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"This has a large uncertainty and it really does not capture the quadratic effect! Let's try to fit it adding one extra variable $u=x^2$. We could just calculate this variable and add it to the dataset, but when working with many dimensions and with higher powers (not only second-order polynomials), it is easier to use the auxiliary object `PolynomialFeatures`."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"id": "53a6c63b-0144-424d-b191-3dd8b6a090d4",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/html": [
|
||||
"<style>#sk-container-id-3 {color: black;}#sk-container-id-3 pre{padding: 0;}#sk-container-id-3 div.sk-toggleable {background-color: white;}#sk-container-id-3 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-3 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-3 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-3 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-3 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-3 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-3 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-3 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-3 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-3 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-3 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-3 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-3 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-3 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-3 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-3 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-3 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-3 div.sk-item {position: relative;z-index: 1;}#sk-container-id-3 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-3 div.sk-item::before, #sk-container-id-3 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-3 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-3 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-3 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-3 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-3 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-3 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-3 div.sk-label-container {text-align: center;}#sk-container-id-3 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-3 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-3\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>BayesianRidge()</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-3\" type=\"checkbox\" checked><label for=\"sk-estimator-id-3\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">BayesianRidge</label><div class=\"sk-toggleable__content\"><pre>BayesianRidge()</pre></div></div></div></div></div>"
|
||||
],
|
||||
"text/plain": [
|
||||
"BayesianRidge()"
|
||||
]
|
||||
},
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from sklearn.linear_model import BayesianRidge\n",
|
||||
"from sklearn.preprocessing import PolynomialFeatures\n",
|
||||
"\n",
|
||||
"feature_extractor = PolynomialFeatures(2)\n",
|
||||
"data_preprocessed = feature_extractor.fit_transform(data[:, [0]])\n",
|
||||
"\n",
|
||||
"model = BayesianRidge()\n",
|
||||
"model.fit(data_preprocessed, data[:, 1])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"id": "c9fd10d6-1c1b-4707-ad9f-bfed56e3ab9e",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<matplotlib.legend.Legend at 0x7f1a803ee970>"
|
||||
]
|
||||
},
|
||||
"execution_count": 8,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"image/png": "iVBORw0KGgoAAAANSUhEUgAABMkAAAGxCAYAAAB4GeGfAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAACpjElEQVR4nOzdeXiTZdYG8DtN9yXdN9pCAdlBKCJuIFAURUUYxXEZF9wdR0XFYfTTEXAXRdFxFLcRFXEBRVFRQAqCiiIIqODKXpaWQpvuaZO83x+Ht1nbpmn23L/rytU2eZM8bdO0uXvOeTSKoiggIiIiIiIiIiIKYxH+XgAREREREREREZG/MSQjIiIiIiIiIqKwx5CMiIiIiIiIiIjCHkMyIiIiIiIiIiIKewzJiIiIiIiIiIgo7DEkIyIiIiIiIiKisMeQjIiIiIiIiIiIwl6kvxfQWWazGQcOHEBSUhI0Go2/l0NERERERERERH6kKApqamrQpUsXRES4Xh8W9CHZgQMHUFBQ4O9lEBERERERERFRANm3bx/y8/NdPj7oQ7KkpCQA8onrdDo/r4aIiIiIiIiIiPypuroaBQUFLZmRq4I+JFNbLHU6HUMyIiIiIiIiIiICgA6P5eLgfiIiIiIiIiIiCnsMyYiIiIiIiIiIKOwxJCMiIiIiIiIiorDHkIyIiIiIiIiIiMIeQzIiIiIiIiIiIgp7DMmIiIiIiIiIiCjsMSQjIiIiIiIiIqKwx5CMiIiIiIiIiIjCHkMyIiIiIiIiIiIKewzJiIiIiIiIiIgo7DEkIyIiIiIiIiKisMeQjIiIiIiIiIiIwh5DMiIiIiIiIiIiCnsMyYiIiIiIiIiIKOwxJCMiIiIiIiIi6ghFAfR6eUshgyEZEREREREREZGr6uqAn34CfvhBgjIKGZH+XgARERERERERUcAzmYDSUuDPP4GqKiA6mpVkIYaVZGFIo9HYnGJiYpCZmYmhQ4fiuuuuw2effQaTyeSR+5o/fz40Gg3mz5/vkdsjIiIiIiIi8rmjR4GNG4HNmwGzGcjP9/eKyAtYSRbGZsyYAQAwmUyoqqrCtm3b8Oabb+LVV1/FsGHD8NZbb6F3795+XiURERERERGRnxgMwO7dwM6dQHMzkJsLREZKVRmFHIZkYWzmzJkO55WVleHWW2/FokWLcMYZZ2Djxo3Iysry/eKIiIiIiIiI/EVRgEOHgN9/B44cAdLTgcREf6+KvIztll5gMgFr1gBvvy1vgylgzs7OxjvvvIPRo0dj3759eOSRR2wu37RpE6ZOnYrBgwcjLS0NsbGx6NWrF6ZNm4bKykqbY0ePHo2rr74aAHD11VfbtHju3r0bAHDgwAE88MADOO2005CTk4Po6Gh06dIFl112GbZv3+6Tz5mIiIiIiIioRW0tsHUr8P33QH09UFDAgCxMaBQluKfMVVdXIzk5GXq9Hjqdzt/LwQcfAFOnyiw/VX4+8MwzwAUX+G9d1jQaDQCgrW/9qlWrcMYZZyArKwuHDh1quc5NN92EJUuWYNSoUSgoKIDZbMamTZuwbt069OvXD9999x2SkpIAyDyyDz/8EB999BEmTpyIIUOGtNz+7bffjpSUFLzzzju45pprMGbMGBQWFiIxMRF//PEHPvnkE0RHR+Prr7/G4MGDvffFICIiIiIiIgIAo9EymL+mBsjKAmJjnR9rMgFlZcCIEUBqqm/XSe1yNytiSOZBH3wATJ7suLnFsXwJixcHRlDmSkhmMBiQmJgIo9GInTt3onv37gCAPXv2ID8/H1qt1ub4V199Fddddx0ee+wx/Otf/2o5f/78+bj66qvx2muvYcqUKQ73U15ejri4uJZgTbV161acdtppGDlyJD777DN3P1UiIiIiIiKi9lVUAH/8ARw8COh07QdfDMkCmrtZEdstPcRkkgoyZ7mTet7ttwdP62VMTAzS09MBAIcPH245v1u3bg4BGQBcc8010Ol0WL58eYfuJysryyEgA4DBgwejuLgYq1evRnNzcwdXT0REREREROSChgZg2zbgu+8kKMvLY+gVxhiSeci6dbYtlvYUBdi3T44LFmqlmVp5BgDNzc147rnnMGLECKSlpUGr1UKj0SAiIgLV1dXYv39/h+/n008/xYQJE5Cbm4uoqKiWuWUff/wxDAYDKioqPPY5EREREREREcFkkhfx334L/PabVI916SI7V1LY4nffQw4e9Oxx/tbY2IijR48CADIzM1vOv/jii7FkyRL06NEDEydORE5ODmJiYgAAc+fOhcFg6ND9PPPMM7j99tuRmpqKM888E127dkV8fDw0Gg0+/PBDbN26tcO3SURERERERNSqykqZO7Z/PxAXB3TtapmTRGGNIZmH5OZ69jh/++qrr2A0GpGdnY3CwkIAwMaNG7FkyRKcccYZ+OyzzxBplbCbzWbMnj27Q/dhNBoxc+ZM5OTk4IcffkCu3Rdn/fr1nf48iIiIiIiIiAAABgOwezewa5e8n5MDREX5e1UUQNhu6SEjR8oulq2FzxqN7Bo7cqRv1+UOs9mMhx9+GABw2WWXtZz/559/AgDOP/98m4AMADZs2ICGhgaH21Lnl5mcDGOrqKhAVVUVTj31VIeArLa2Fj/88EPnPhEiIiIiIiIis1naujZskPljsbHyAp4BGdlhSOYhWi3wzDPyvn1Qpn48d64cF8jKy8txySWXYM2aNejatSv+7//+r+UytaJszZo1Dtf5xz/+4fT21OH/e/fudbgsKysL8fHx2LRpE2pra1vOb25uxtSpUzmLjIiIiIiIiDpHrwe2bpWArLZWqlecbB5HBPi53XLVqlV47rnnsH79elRWViI9PR2DBg3C1KlTcc455/hzaW654AJg8WLZ5dJ6iH9+vgRkF1zgt6U5NXPmTABSOVZVVYVt27bhq6++QlNTE4YPH4633noLGRkZLcefeOKJOO200/DBBx/g1FNPxYgRI1BWVobPPvsMffr0QZcuXRzu45RTTkF8fDzmzp2LI0eOICcnBwBw6623Ijk5Gbfddhsee+wxDBo0CBMnTkRTUxNWr16No0ePYsyYMVi9erVPvhZEREREREQUQpqagL17gR07ZAfLrCzg2DxtotZoFHULQx+bPn06nnjiCeTn52P8+PHIyMjA4cOHsWnTJpxxxhkuz7eqrq5GcnIy9Ho9dDqdl1ftGpNJdrE8eFBmkI0cGVgVZBq7Urfo6GgkJSWhW7duGDp0KC688EKMGzcOERGOhYZHjx7Ffffdh2XLluHQoUPIy8vDxRdfjPvuuw/9+/cHAOzevdvmOp9//jlmzZqFn376CXV1dQCAXbt2obCwEEajEc8++yxeeeUV7Nq1C8nJyTjzzDPx0EMPYcaMGXj99ddbjiUiIiIiIiJqk6IAZWUymL+8HEhNlZ0rPc1kkvsZMULugwKKu1mRX0Kyl19+GTfccAOuuuoqvPTSS4iOjra5vLm5GVEu9gYHYkhGRERERERERD5WXQ3s3CkVZJGRQEaG9ypWGJIFNHezIp+3WxoMBtx7773o2rWr04AMgMsBGRERERERERGFueZmS2tlfT2QmSnD+Yk6yOch2cqVK3H48GHcfvvtiIiIwKeffoqff/4ZsbGxGD58OE455RRfL4mIiIiIiIiIgo2iSEul2lqZnCyD+Ync5POQ7PvvvwcAxMbGoqioCD///LPN5aeffjoWL16MzMxMp9c3GAwwGAwtH1dXV3tvsUREREREREQUeGpqpHJs715pqczLC6xh4BSUHCeze1l5eTkA4IknnoBGo8G6detQU1ODH3/8EePGjcPatWtx0UUXtXr9Rx99FMnJyS2nAqbEREREREREROGhqUnCsfXrgd27gfR0ICeHARl5hM9DMrPZDACIjIzE0qVLMWLECCQmJmLQoEFYsmQJ8vPz8eWXX2L9+vVOr3/PPfdAr9e3nPbt2+fL5RMRERERERGRrykKcOgQsGEDsHWrDOYvKODsMfIon4dkKSkpAICioiIUFhbaXBYfH4+zzjoLALBhwwan14+JiYFOp7M5EREREREREVGI0uuBLVskIKuuBvLzZf4YkYf5fCZZnz59AFjCMnupx7ZObWho8NWSiIiIiIiIiCjQGAzAvn3SXtnQAGRlATEx/l4VhTCfh2Rjx46FRqPB9u3bYTabERFhW8ymDvLv3r27r5dGRERERERERP5mNktr5Y4dwOHDQGoqkJHh71VRGPB5u2W3bt0wYcIE7N27F88884zNZStWrMDy5cuRkpKCs88+29dLIyIiIiIiIiJ/qqwENm8Gvv8eqK2VuWMcs0Q+4vNKMgD473//i82bN+POO+/Ep59+iqKiIuzatQsffvghtFotXnnlFSSzv5iIiIiIiIgoPDQ0AHv2ALt2yQ6WWVlAdLS/V0Vhxi8hWX5+PjZt2oQHHngAS5cuxdq1a6HT6TBhwgTcc889GD58uD+WRURERERERES+ZDIBBw4Af/4pVWTp6RKQEfmBRlEUxd+L6Izq6mokJydDr9dzp0siIiIiIiKiYKAowJEjMnfswAEgIQFISwM0Gn+vzDUmE1BWBowYITPTKKC4mxX5pZKMiIiIiIiIiMJUba20Ve7dK2FZly5AJOMJ8j+fD+4naotGo8Ho0aNtzps5cyY0Gg3WrFnjlfvcvXs3NBoNpkyZ4pXbJyIiIiIiIgDNzcDOncC330p7ZXIykJvLgIwCBkOyMKTRaGxOWq0WGRkZKC4uxsKFC/29PK9wFr4RERERERGRD5jNwMGDwHffAVu2SEtl165AXJy/V0Zkg3FtGJsxYwYAoLm5Gb/++is++ugjrF69Ghs3bsRTTz3l59VZ3HLLLbjkkkvQtWtXr9x+Xl4efvnlF+6oSkRERERE5GmVlVI9VloKREUB+fmAVuvvVRE5xZAsjM2cOdPm41WrVuHMM8/E3Llzcdttt6GwsNAv67KXkZGBjIwMr91+VFQU+vbt67XbJyIiIiILine truncated
|
||||
"text/plain": [
|
||||
"<Figure size 1500x500 with 1 Axes>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"preprocessed_x_axis = feature_extractor.fit_transform(x_axis[:, np.newaxis])\n",
|
||||
"y_p, y_u = model.predict(preprocessed_x_axis, return_std=True)\n",
|
||||
"\n",
|
||||
"plt.scatter(data[:, 0], data[:, 1], c='b', label=\"Data\")\n",
|
||||
"plt.plot(x_axis, y_p, c='r', lw=2, ls='--', label=\"Prediction\")\n",
|
||||
"plt.fill_between(x_axis, y_p-2*y_u, y_p+2*y_u, color='red', alpha=0.2, label=\"95% CL uncertainty\")\n",
|
||||
"plt.legend(frameon=False)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "ff9c4999-5ef9-42da-a748-6b1384f5651f",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"This looks much better! We can also do this in a more streamlined way using the concept of `Pipeline` in scikit-learn, which does this transformation for us in a more transparent way."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 9,
|
||||
"id": "9cc91240-2ad2-4912-b4a3-96e0d12f119a",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/html": [
|
||||
"<style>#sk-container-id-4 {color: black;}#sk-container-id-4 pre{padding: 0;}#sk-container-id-4 div.sk-toggleable {background-color: white;}#sk-container-id-4 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-4 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-4 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-4 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-4 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-4 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-4 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-4 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-4 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-4 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-4 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-4 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-4 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-4 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-4 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-4 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-4 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-4 div.sk-item {position: relative;z-index: 1;}#sk-container-id-4 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-4 div.sk-item::before, #sk-container-id-4 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-4 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-4 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-4 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-4 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-4 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-4 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-4 div.sk-label-container {text-align: center;}#sk-container-id-4 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-4 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-4\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>Pipeline(steps=[('feature_extractor', PolynomialFeatures()),\n",
|
||||
" ('fit', BayesianRidge())])</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-4\" type=\"checkbox\" ><label for=\"sk-estimator-id-4\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">Pipeline</label><div class=\"sk-toggleable__content\"><pre>Pipeline(steps=[('feature_extractor', PolynomialFeatures()),\n",
|
||||
" ('fit', BayesianRidge())])</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-5\" type=\"checkbox\" ><label for=\"sk-estimator-id-5\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">PolynomialFeatures</label><div class=\"sk-toggleable__content\"><pre>PolynomialFeatures()</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-6\" type=\"checkbox\" ><label for=\"sk-estimator-id-6\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">BayesianRidge</label><div class=\"sk-toggleable__content\"><pre>BayesianRidge()</pre></div></div></div></div></div></div></div>"
|
||||
],
|
||||
"text/plain": [
|
||||
"Pipeline(steps=[('feature_extractor', PolynomialFeatures()),\n",
|
||||
" ('fit', BayesianRidge())])"
|
||||
]
|
||||
},
|
||||
"execution_count": 9,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from sklearn.linear_model import BayesianRidge\n",
|
||||
"from sklearn.preprocessing import PolynomialFeatures\n",
|
||||
"from sklearn.pipeline import Pipeline\n",
|
||||
"\n",
|
||||
"model = Pipeline([('feature_extractor', PolynomialFeatures(2)),\n",
|
||||
" ('fit', BayesianRidge())\n",
|
||||
" ])\n",
|
||||
"model.fit(data[:,[0]], data[:,1])\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"id": "a16075c0-d671-4d06-9cfe-ab191cddd24d",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<matplotlib.legend.Legend at 0x7f1a7ee748e0>"
|
||||
]
|
||||
},
|
||||
"execution_count": 10,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"image/png": "iVBORw0KGgoAAAANSUhEUgAABMkAAAGxCAYAAAB4GeGfAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAACpjElEQVR4nOzdeXiTZdYG8DtN9yXdN9pCAdlBKCJuIFAURUUYxXEZF9wdR0XFYfTTEXAXRdFxFLcRFXEBRVFRQAqCiiIIqODKXpaWQpvuaZO83x+Ht1nbpmn23L/rytU2eZM8bdO0uXvOeTSKoiggIiIiIiIiIiIKYxH+XgAREREREREREZG/MSQjIiIiIiIiIqKwx5CMiIiIiIiIiIjCHkMyIiIiIiIiIiIKewzJiIiIiIiIiIgo7DEkIyIiIiIiIiKisMeQjIiIiIiIiIiIwl6kvxfQWWazGQcOHEBSUhI0Go2/l0NERERERERERH6kKApqamrQpUsXRES4Xh8W9CHZgQMHUFBQ4O9lEBERERERERFRANm3bx/y8/NdPj7oQ7KkpCQA8onrdDo/r4aIiIiIiIiIiPypuroaBQUFLZmRq4I+JFNbLHU6HUMyIiIiIiIiIiICgA6P5eLgfiIiIiIiIiIiCnsMyYiIiIiIiIiIKOwxJCMiIiIiIiIiorDHkIyIiIiIiIiIiMIeQzIiIiIiIiIiIgp7DMmIiIiIiIiIiCjsMSQjIiIiIiIiIqKwx5CMiIiIiIiIiIjCHkMyIiIiIiIiIiIKewzJiIiIiIiIiIgo7DEkIyIiIiIiIiKisMeQjIiIiIiIiIiIwh5DMiIiIiIiIiIiCnsMyYiIiIiIiIiIKOwxJCMiIiIiIiIi6ghFAfR6eUshgyEZEREREREREZGr6uqAn34CfvhBgjIKGZH+XgARERERERERUcAzmYDSUuDPP4GqKiA6mpVkIYaVZGFIo9HYnGJiYpCZmYmhQ4fiuuuuw2effQaTyeSR+5o/fz40Gg3mz5/vkdsjIiIiIiIi8rmjR4GNG4HNmwGzGcjP9/eKyAtYSRbGZsyYAQAwmUyoqqrCtm3b8Oabb+LVV1/FsGHD8NZbb6F3795+XiURERERERGRnxgMwO7dwM6dQHMzkJsLREZKVRmFHIZkYWzmzJkO55WVleHWW2/FokWLcMYZZ2Djxo3Iysry/eKIiIiIiIiI/EVRgEOHgN9/B44cAdLTgcREf6+KvIztll5gMgFr1gBvvy1vgylgzs7OxjvvvIPRo0dj3759eOSRR2wu37RpE6ZOnYrBgwcjLS0NsbGx6NWrF6ZNm4bKykqbY0ePHo2rr74aAHD11VfbtHju3r0bAHDgwAE88MADOO2005CTk4Po6Gh06dIFl112GbZv3+6Tz5mIiIiIiIioRW0tsHUr8P33QH09UFDAgCxMaBQluKfMVVdXIzk5GXq9Hjqdzt/LwQcfAFOnyiw/VX4+8MwzwAUX+G9d1jQaDQCgrW/9qlWrcMYZZyArKwuHDh1quc5NN92EJUuWYNSoUSgoKIDZbMamTZuwbt069OvXD9999x2SkpIAyDyyDz/8EB999BEmTpyIIUOGtNz+7bffjpSUFLzzzju45pprMGbMGBQWFiIxMRF//PEHPvnkE0RHR+Prr7/G4MGDvffFICIiIiIiIgIAo9EymL+mBsjKAmJjnR9rMgFlZcCIEUBqqm/XSe1yNytiSOZBH3wATJ7suLnFsXwJixcHRlDmSkhmMBiQmJgIo9GInTt3onv37gCAPXv2ID8/H1qt1ub4V199Fddddx0ee+wx/Otf/2o5f/78+bj66qvx2muvYcqUKQ73U15ejri4uJZgTbV161acdtppGDlyJD777DN3P1UiIiIiIiKi9lVUAH/8ARw8COh07QdfDMkCmrtZEdstPcRkkgoyZ7mTet7ttwdP62VMTAzS09MBAIcPH245v1u3bg4BGQBcc8010Ol0WL58eYfuJysryyEgA4DBgwejuLgYq1evRnNzcwdXT0REREREROSChgZg2zbgu+8kKMvLY+gVxhiSeci6dbYtlvYUBdi3T44LFmqlmVp5BgDNzc147rnnMGLECKSlpUGr1UKj0SAiIgLV1dXYv39/h+/n008/xYQJE5Cbm4uoqKiWuWUff/wxDAYDKioqPPY5EREREREREcFkkhfx334L/PabVI916SI7V1LY4nffQw4e9Oxx/tbY2IijR48CADIzM1vOv/jii7FkyRL06NEDEydORE5ODmJiYgAAc+fOhcFg6ND9PPPMM7j99tuRmpqKM888E127dkV8fDw0Gg0+/PBDbN26tcO3SURERERERNSqykqZO7Z/PxAXB3TtapmTRGGNIZmH5OZ69jh/++qrr2A0GpGdnY3CwkIAwMaNG7FkyRKcccYZ+OyzzxBplbCbzWbMnj27Q/dhNBoxc+ZM5OTk4IcffkCu3Rdn/fr1nf48iIiIiIiIiAAABgOwezewa5e8n5MDREX5e1UUQNhu6SEjR8oulq2FzxqN7Bo7cqRv1+UOs9mMhx9+GABw2WWXtZz/559/AgDOP/98m4AMADZs2ICGhgaH21Lnl5mcDGOrqKhAVVUVTj31VIeArLa2Fj/88EPnPhEiIiIiIiIis1naujZskPljsbHyAp4BGdlhSOYhWi3wzDPyvn1Qpn48d64cF8jKy8txySWXYM2aNejatSv+7//+r+UytaJszZo1Dtf5xz/+4fT21OH/e/fudbgsKysL8fHx2LRpE2pra1vOb25uxtSpUzmLjIiIiIiIiDpHrwe2bpWArLZWqlecbB5HBPi53XLVqlV47rnnsH79elRWViI9PR2DBg3C1KlTcc455/hzaW654AJg8WLZ5dJ6iH9+vgRkF1zgt6U5NXPmTABSOVZVVYVt27bhq6++QlNTE4YPH4633noLGRkZLcefeOKJOO200/DBBx/g1FNPxYgRI1BWVobPPvsMffr0QZcuXRzu45RTTkF8fDzmzp2LI0eOICcnBwBw6623Ijk5Gbfddhsee+wxDBo0CBMnTkRTUxNWr16No0ePYsyYMVi9erVPvhZEREREREQUQpqagL17gR07ZAfLrCzg2DxtotZoFHULQx+bPn06nnjiCeTn52P8+PHIyMjA4cOHsWnTJpxxxhkuz7eqrq5GcnIy9Ho9dDqdl1ftGpNJdrE8eFBmkI0cGVgVZBq7Urfo6GgkJSWhW7duGDp0KC688EKMGzcOERGOhYZHjx7Ffffdh2XLluHQoUPIy8vDxRdfjPvuuw/9+/cHAOzevdvmOp9//jlmzZqFn376CXV1dQCAXbt2obCwEEajEc8++yxeeeUV7Nq1C8nJyTjzzDPx0EMPYcaMGXj99ddbjiUiIiIiIiJqk6IAZWUymL+8HEhNlZ0rPc1kkvsZMULugwKKu1mRX0Kyl19+GTfccAOuuuoqvPTSS4iOjra5vLm5GVEu9gYHYkhGRERERERERD5WXQ3s3CkVZJGRQEaG9ypWGJIFNHezIp+3WxoMBtx7773o2rWr04AMgMsBGRERERERERGFueZmS2tlfT2QmSnD+Yk6yOch2cqVK3H48GHcfvvtiIiIwKeffoqff/4ZsbGxGD58OE455RRfL4mIiIiIiIiIgo2iSEul2lqZnCyD+Ync5POQ7PvvvwcAxMbGoqioCD///LPN5aeffjoWL16MzMxMp9c3GAwwGAwtH1dXV3tvsUREREREREQUeGpqpHJs715pqczLC6xh4BSUHCeze1l5eTkA4IknnoBGo8G6detQU1ODH3/8EePGjcPatWtx0UUXtXr9Rx99FMnJyS2nAqbEREREREREROGhqUnCsfXrgd27gfR0ICeHARl5hM9DMrPZDACIjIzE0qVLMWLECCQmJmLQoEFYsmQJ8vPz8eWXX2L9+vVOr3/PPfdAr9e3nPbt2+fL5RMRERERERGRrykKcOgQsGEDsHWrDOYvKODsMfIon4dkKSkpAICioiIUFhbaXBYfH4+zzjoLALBhwwan14+JiYFOp7M5EREREREREVGI0uuBLVskIKuuBvLzZf4YkYf5fCZZnz59AFjCMnupx7ZObWho8NWSiIiIiIiIiCjQGAzAvn3SXtnQAGRlATEx/l4VhTCfh2Rjx46FRqPB9u3bYTabERFhW8ymDvLv3r27r5dGRERERERERP5mNktr5Y4dwOHDQGoqkJHh71VRGPB5u2W3bt0wYcIE7N27F88884zNZStWrMDy5cuRkpKCs88+29dLIyIiIiIiIiJ/qqwENm8Gvv8eqK2VuWMcs0Q+4vNKMgD473//i82bN+POO+/Ep59+iqKiIuzatQsffvghtFotXnnlFSSzv5iIiIiIiIgoPDQ0AHv2ALt2yQ6WWVlAdLS/V0Vhxi8hWX5+PjZt2oQHHngAS5cuxdq1a6HT6TBhwgTcc889GD58uD+WRURERERERES+ZDIBBw4Af/4pVWTp6RKQEfmBRlEUxd+L6Izq6mokJydDr9dzp0siIiIiIiKiYKAowJEjMnfswAEgIQFISwM0Gn+vzDUmE1BWBowYITPTKKC4mxX5pZKMiIiIiIiIiMJUba20Ve7dK2FZly5AJOMJ8j+fD+4naotGo8Ho0aNtzps5cyY0Gg3WrFnjlfvcvXs3NBoNpkyZ4pXbJyIiIiIiIgDNzcDOncC330p7ZXIykJvLgIwCBkOyMKTRaGxOWq0WGRkZKC4uxsKFC/29PK9wFr4RERERERGRD5jNwMGDwHffAVu2SEtl165AXJy/V0Zkg3FtGJsxYwYAoLm5Gb/++is++ugjrF69Ghs3bsRTTz3l59VZ3HLLLbjkkkvQtWtXr9x+Xl4efvnlF+6oSkRERERE5GmVlVI9VloKREUB+fmAVuvvVRE5xZAsjM2cOdPm41WrVuHMM8/E3Llzcdttt6GwsNAv67KXkZGBjIwMr91+VFQU+vbt67XbJyIiIiILine truncated
|
||||
"text/plain": [
|
||||
"<Figure size 1500x500 with 1 Axes>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"y_p, y_u = model.predict(x_axis[:, np.newaxis], return_std=True)\n",
|
||||
"\n",
|
||||
"plt.scatter(data[:, 0], data[:, 1], c='b', label=\"Data\")\n",
|
||||
"plt.plot(x_axis, y_p, c='r', lw=2, ls='--', label=\"Prediction\")\n",
|
||||
"plt.fill_between(x_axis, y_p-2*y_u, y_p+2*y_u, color='red', alpha=0.2, label=\"95% CL uncertainty\")\n",
|
||||
"plt.legend(frameon=False)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "b0a02e5f-ce76-4149-b51a-e114dccccf3c",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Kernel methods\n",
|
||||
"\n",
|
||||
"Another simple extension of linear fits is to apply a non-linear transformation on the input $x$ values indirectly. The logic here is the following: when looking at the derivation of the equations for linear fit results, we see that the results don't really depend on the $x$ values of each point directly. They only depend on the dot-product between $x$ values between different points, that is, on $\\vec{x}_1 \\cdot \\vec{x}_2 = \\sum_i x_{1i} x_{2i}$. If we substituted the *definition of dot product* in our space with something else, we are effectively changing how we measure distances in the space of points. This change in the curvature of the space may detect non-linear relationships more easily.\n",
|
||||
"\n",
|
||||
"When using this method we are effectively performing the fit:\n",
|
||||
"\n",
|
||||
"$y = f(x) = \\sum_i w_i \\phi_i(x_i) + w_0$\n",
|
||||
"\n",
|
||||
"The only change here is that the only thing that enters the method are the pre-transformed versions of the input $x_i$ components.\n",
|
||||
"We never really have to calculate $\\phi_i$, since the only thing that enters our linear fit equations for $\\phi_i(x_i)$ is actually $\\phi_i(x)_i \\phi_j(x_j)$ for all combinations of $i$ and $j$. We define this quantity kernel $K(x, x^\\prime) = \\phi(x) \\cdot \\phi(x^\\prime)$ and it is conceptualized as a measure of relationship between the different $x$ values."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "396e37cf-5597-468c-9272-30cb80c71e61",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Let's look at an example of this using the kernel function $K(x, x^\\prime) = (\\gamma x \\cdot x^\\prime + 1)^3$. This is called the *polynomial* kernel, but you can define your own kernel."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 11,
|
||||
"id": "f294b5e9-cd9a-4c99-b9aa-08c4b0f8bfab",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/html": [
|
||||
"<style>#sk-container-id-5 {color: black;}#sk-container-id-5 pre{padding: 0;}#sk-container-id-5 div.sk-toggleable {background-color: white;}#sk-container-id-5 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-5 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-5 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-5 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-5 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-5 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-5 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-5 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-5 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-5 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-5 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-5 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-5 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-5 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-5 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-5 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-5 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-5 div.sk-item {position: relative;z-index: 1;}#sk-container-id-5 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-5 div.sk-item::before, #sk-container-id-5 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-5 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-5 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-5 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-5 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-5 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-5 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-5 div.sk-label-container {text-align: center;}#sk-container-id-5 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-5 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-5\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>KernelRidge(kernel='polynomial')</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-7\" type=\"checkbox\" checked><label for=\"sk-estimator-id-7\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">KernelRidge</label><div class=\"sk-toggleable__content\"><pre>KernelRidge(kernel='polynomial')</pre></div></div></div></div></div>"
|
||||
],
|
||||
"text/plain": [
|
||||
"KernelRidge(kernel='polynomial')"
|
||||
]
|
||||
},
|
||||
"execution_count": 11,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from sklearn.kernel_ridge import KernelRidge\n",
|
||||
"\n",
|
||||
"model = KernelRidge(kernel=\"polynomial\")\n",
|
||||
"model.fit(data[:,[0]], data[:,1])\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 12,
|
||||
"id": "33a9a495-db13-44d8-95d9-6d0ca1401164",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<matplotlib.legend.Legend at 0x7f1a7d88e3d0>"
|
||||
]
|
||||
},
|
||||
"execution_count": 12,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"image/png": "iVBORw0KGgoAAAANSUhEUgAABMkAAAGxCAYAAAB4GeGfAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAAB5LklEQVR4nO3deXiTVfrG8TsttKytLAKFFopsgqiAiAoUCorgilZcQHBlBMcF1J8L6ggoiijIMo67gsqADlhBBRXFgqiggogbKEvBAgUEpQWUAml+fxzTNG3aJmmSN2m+n+vKlfa8b5LTkJbm7nOeY3M4HA4BAAAAAAAAUSzG6gkAAAAAAAAAViMkAwAAAAAAQNQjJAMAAAAAAEDUIyQDAAAAAABA1CMkAwAAAAAAQNQjJAMAAAAAAEDUIyQDAAAAAABA1Ktm9QQqq7CwUDt37lTdunVls9msng4AAAAAAAAs5HA4dODAATVt2lQxMd7Xh0V8SLZz506lpKRYPQ0AAAAAAACEkZycHCUnJ3t9fsSHZHXr1pVkvvCEhASLZwMAAAAAAAAr5efnKyUlpSgz8lbEh2TOJZYJCQmEZAAAAAAAAJAkn9ty0bgfAAAAAAAAUY+QDAAAAAAAAFGPkAwAAAAAAABRj5AMAAAAAAAAUY+QDAAAAAAAAFGPkAwAAAAAAABRj5AMAAAAAAAAUY+QDAAAAAAAAFGPkAwAAAAAAABRj5AMAAAAAAAAUa+a1RMAAAAAAAAId3a7tGKFlJsrJSVJaWlSbKzVs0IgEZIBAAAAAACUIzNTGjVK2r7dNZacLE2fLmVkWDcvBBbLLQEAAAAAAMqQmSkNGuQekEnSjh1mPDPTmnkh8AjJAAAAAAAAPLDbTQWZw1H6mHNs9GhzHiIfIRkAAAAAAIAHK1aUriArzuGQcnLMeZHIbpeWLZPmzjXX0R720ZMMAAAAAADAg9zcwJ4XTuizVhqVZAAAAAAAAB4kJQX2vHBBnzXPLAnJUlNTZbPZPF6aNGlixZSiSsnnPD4+Xscff7y6dOmi4cOH6/3335c9QDWWs2bNks1m06xZswJyfwAAAAAAhEpamqmustk8H7fZpJQUc16koM9a2SxbbpmYmKjRo0eXGq9Tp07oJxOlxo4dK0my2+3av3+/fvzxR73++ut6+eWX1bVrV/33v/9V27ZtLZ4lAAAAAADWiI01yw8HDTKBWPFgyRmcTZtmzosUvvRZS08P2bTCgmUh2XHHHadx48ZZ9fCQPD7/u3fv1m233aZ58+bpnHPO0erVq9WoUaPQTw4AAAAAgDCQkSHNn++5f9e0aZHXv6sq91mrLHqSBUEk7w7RuHFjvfHGG0pPT1dOTo4ee+wxt+Nr1qzRqFGjdOqpp6p+/fqqUaOG2rRpo7vuukt//PGH27np6em6/vrrJUnXX3+92xLPrVu3SpJ27typhx9+WD169FCTJk0UFxenpk2basiQIfrpp59C8jUDAAAAAPwTye9/fZGRIW3dKmVlSXPmmOvs7MgLyKSq22ctEGwOh6dVqMGVmpqqgoICPfnkk/r1119Vu3ZtnXLKKerVq5difaxRzM/PV2JiovLy8pSQkBCkGXsvEnaHsP1dE1reP/3SpUt1zjnnqFGjRtq1a1fRbUaOHKm3335bvXv3VkpKigoLC7VmzRqtWLFC7du315dffqm6detKMv3IFixYoIULF2rgwIHq1KlT0f2PHj1axx13nN544w3dcMMN6tOnj1JTU1WnTh1t3LhR7733nuLi4vT555/r1FNPDd6TAQAAAADwSyS8/0VpdruUmmqa9HuKBWw28++YnR1Zy0iL8zcrsiwk27ZtW6nxli1baubMmerdu3eZty0oKFBBQUHR5/n5+UpJSQmLkMy5O0TJZ9S5Tnn+/PD4QeFNSFZQUKA6dero2LFj2rJli1q2bClJ2rZtm5KTk0uFmS+//LKGDx+uxx9/XPfee2/R+KxZs3T99ddr5syZuu6660o9zp49e1SzZs2iYM1p3bp16tGjh9LS0vT+++/7+6UCAAAAAIIgUt7/wjPnv5/kuc9apP/7+RuSWbLc8vrrr9fSpUu1a9cuHTp0SN9//71GjBihrVu36rzzztO6devKvO3EiROVmJhYdElJSQnhzMtW1XaHiI+PV4MGDSRJv/32W9F4ixYtPFb73XDDDUpISNCHH37o0+M0atSoVEAmSaeeeqr69u2rrKwsHT161MfZAwAAAACCpaq9/41Gzj5rzZq5jycnR35AVhmWhGRjx45V37591bhxY9WqVUsdO3bUc889pzvvvFN//fVXuQ39x4wZo7y8vKJLTk5O6CZeDl92h4gUzkozW7G9bo8ePaqnn35aPXv2VP369RUbGyubzaaYmBjl5+drx44dPj/OokWLdNFFFykpKUnVq1cv6lv27rvvqqCgQHv37g3Y1wQAAAAAqJyq+P43GlWlPmuBYtnulp6MHDlSU6ZM0aefflrmOfHx8YqPjw/hrLxT1XaHOHz4sH7//XdJ0vHHH180fuWVV+rtt9/WCSecoIEDB6pJkyZF/x7Tpk1zWwrrjenTp2v06NGqV6+e+vXrp+bNm6tWrVqy2WxasGCB1q1b5/N9AgAAAACCp6q9/41msbFSerrVswgfYRWSOcOYQ4cOWTwT31W13SE+++wzHTt2TI0bN1ZqaqokafXq1Xr77bd1zjnn6P3331e1aq6XT2FhoZ544gmfHuPYsWMaN26cmjRpom+++UZJJZ6clStXVvrrAAAAAAAEVlV7/ws4WbLcsiyrVq2SJJ1wwgkWz8R3aWlm7W6xlYlubDYpJcWcF+4KCwv16KOPSpKGDBlSNL5p0yZJ0sUXX+wWkEnSV199pb/++qvUfTn7l9k9LEbfu3ev9u/fr+7du5cKyA4ePKhvvvmmcl8IAAAAACDgqtL7X7/s2yd99JHVs0AQhDwkW79+vcdKsa1bt+rWW2+VJA0dOjTU06q02Fizza1U+geF8/Np08J/+9Q9e/boqquu0rJly9S8eXPdf//9RcecFWXLli0rdZtbbrnF4/05m///+uuvpY41atRItWrV0po1a3Tw4MGi8aNHj2rUqFH0IgMAAACAMFRV3v/67OhR84W3bi1ddpm0a5fVM0KA2RwOT/tRBM+4ceM0ZcoU9erVSy1atFDdunW1efNmLVq0SIcPH9b555+vt99+W3FxcV7dn7/begZLZqbZ5aN4E8OUFPMDIlya3zkb8Y8dO1aSqRzbv3+/fvzxR3322Wc6cuSIunXrpv/+979q3bp10e3sdrt69+6tzz//XGeddZZ69uyp3bt36/3331e7du20ZcsWVa9eXVu3bi26zR9//KHk5GRVq1ZNw4YNU5MmTSRJt912mxITEzVmzBg9/vjjSk1N1cCBA3XkyBFlZWXp999/10knnaSsrCxlZ2cXBXQAAAAAgPAQCe9/A8LhkBYvlu66S/r5Z9f4yJHSs89aNy+Uyd+sKOQh2fLly/Xcc89p7dq12rVrlw4dOqTjjjtOnTp10rBhwzRs2DC33RQrEm4hmWS2uV2xwjQpTEoyJabhlKCXfH7j4uJUt25dtWjRQl26dNFll12mc889VzExpQsNf//9dz344INavHixdu3apWbNmunKK6/Ugw8+qA4dOkiSW0gmSR988IHGjx+v77//vqiK0Bl8HTt2TDNmzNBLL72k7OxsJSYmql+/fpowYYLGjh2rV199lZAMAAAAAMJUuL//rbQff5TuvFNassR9/Nprpccek5o2tWZeKFfEhGSBFo4hGQAAAAAAiGC//SaNHSs9/7xUWOga79HDlMp17WrZ1FAxf7OisGrcDwAAAAAAYLlXXzVLKZ0BWYsW0ptvmrI5ArIqi5AMAAAAAACguFtvlVq2lOrUMcsq16+Xrrii7C09USVUs3oCAAAAAAAAllm3Tlq1ShoxwjVWo4b0xhtmJ4KkJOvmhpCikgwAAAAAAESf3bulf/xD6tzZVI4V37lSkrp1IyCLMoRkAAAAAAAgehw+LD3+uNSmjfTSS5LDIR07Jk2ebPXMYDGWWwIAAAAAgKrP4ZDmz5fuuUfautU1Xreu9K9/SbffbtnUEB4IyQAAAAAAQNW2erV0xx3SZ5+5xmJipOHDpYcflho3tm5uCBuEZAAAAAAAoOr6+GOpXz/3sXPOkZ56Sjr5ZGvmhLBETzIAAAAAAFB1padL7dubj9u2ld59V1qyhIAMpVBJBgAAAAAAqobCQunLL6WzznKNVasmzZgh/fCD9M9/SnFx1s0PYY1KMgAAAAAAEPk+/1w64wypRw9p7Vr3Y+ecI40eTUCGchGSAQAAAACAyJWdLV1xhdSzp2nQ73BId95prgEfsNwSAAAAAABEnvx8aeJEaepUqaDANd6xo3T//ZLNZt3cEJGoJENYsdlsSk9PdxsbN26cbDabli1bFpTH3Lp1q2w2m6677rqg3D8AAAAAIICOHZNeeEFq00Z6/HFXQHb88dJzz5mlliV3swS8QEgWhWw2m9slNjZWDRs2VN++fTVnzhyrpxcUnsI3AAAAAECE2blT6txZGjFC2rPHjMXFSffcI23caMarsWgO/uGVE8XGjh0rSTp69Kg2bNighQsXKisrS6tXr9ZTTz1l8excbr31Vl111VVq3rx5UO6/WbNmWr9+vRITE4Ny/wAAAACAAGnSxL35/uWXm2qyE06wbk6oMmwOR2R3ssvPz1diYqLy8vKUkJBg9XQigu3vddkl/+mXLl2qfn+XpG7ZskWpqamhnppsNpt69+4d8KWVwbpfAAAAAEAQHTwo1anjPvbpp9Ldd0tPPWV2sgRK8DcrYrklipx99tk68cQT5XA49PXXX0ty7wc2Z84cnXHGGapTp45bgPbnn39q4sSJ6tSpk2rXrq06derorLPO0ty5cz0+zpEjR/TII4+oVatWio+PV8uWLfXggw+qoHijxWLK60m2YcMG3XDDDUpNTVV8fLwLine truncated
|
||||
"text/plain": [
|
||||
"<Figure size 1500x500 with 1 Axes>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"y_p = model.predict(x_axis[:, np.newaxis])\n",
|
||||
"\n",
|
||||
"plt.scatter(data[:, 0], data[:, 1], c='b', label=\"Data\")\n",
|
||||
"plt.plot(x_axis, y_p, c='r', lw=2, ls='--', label=\"Prediction\")\n",
|
||||
"plt.legend(frameon=False)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "6d78f2e4-c687-4fbc-b5ab-3ad2b1175c03",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"This particular model implementation does not define uncertainties in the result (more on that later), but notice that the non-linearity is correctly captured by such a method."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "cc211612-29fc-45f4-bdd4-897778dac430",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"A different way of looking at this is that the kernel encodes a transformation to a space of functions that could fit this data. That is, there are infinite functions that fit the data, but the choice of the kernel selects some of them.\n",
|
||||
"\n",
|
||||
"The disadvantage here is that the parabolic nature of the original function is not so obvious, but the entire structure of the non-linearity is shifted to the choice of the kernel function. Choosing a different kernel leads to a very different result.\n",
|
||||
"\n",
|
||||
"The underlying code used here is actually very similar to the linear fit done previously: the key difference is that the *design* matrix is changed to depend on $K$ instead of the $x$ values directly. What we are doing here is still, behind the scenes, a linear fit!\n",
|
||||
"\n",
|
||||
"This method is called often the *kernel trick*, since it allows very complex fits with a small change in the fit procedure."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "9b3ab83e-4be8-41c4-ade8-4ef19c638fbd",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.10.12"
|
||||
},
|
||||
"widgets": {
|
||||
"application/vnd.jupyter.widget-state+json": {
|
||||
"state": {},
|
||||
"version_major": 2,
|
||||
"version_minor": 0
|
||||
}
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
@@ -0,0 +1,655 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "41989e0a-dac9-4b54-9774-0dfbd3ca76c4",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Classification\n",
|
||||
"\n",
|
||||
"So far we have looking at how to fit functions to data. This is a regression problem. Now we are going to change pace a bit and focus on a classification task.\n",
|
||||
"The problem here is not to discover a function that fit the data points. Here we assume that there are two classes of data, labelled as class $1$ or class $-1$. We would like to separate points from those two classes.\n",
|
||||
"\n",
|
||||
"The idea is to fit a function $f(x)$, which is positive to class 1 and negative for class -1.\n",
|
||||
"\n",
|
||||
"We are going to take a more heuristic approach here, because the objective is to show one example on which we can apply the kernel trick. We will start first with a function $f(x)$ that is a line (or hyper-plane in $n$-dimensional space). After we establish how that can be done, we will use the kernel trick to obtain a non-linear separation between the two classes.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {
|
||||
"SVM_margin.png": {
|
||||
"image/png": "iVBORw0KGgoAAAANSUhEUgAACGsAAAguCAYAAACmBkjxAAAAAXNSR0IArs4c6QAAAAlwSFlzAAA9hAAAPYQB1ayvdAAAAgtpVFh0WE1MOmNvbS5hZG9iZS54bXAAAAAAADx4OnhtcG1ldGEgeG1sbnM6eD0iYWRvYmU6bnM6bWV0YS8iIHg6eG1wdGs9IlhNUCBDb3JlIDUuNC4wIj4KICAgPHJkZjpSREYgeG1sbnM6cmRmPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5LzAyLzIyLXJkZi1zeW50YXgtbnMjIj4KICAgICAgPHJkZjpEZXNjcmlwdGlvbiByZGY6YWJvdXQ9IiIKICAgICAgICAgICAgeG1sbnM6dGlmZj0iaHR0cDovL25zLmFkb2JlLmNvbS90aWZmLzEuMC8iPgogICAgICAgICA8dGlmZjpSZXNvbHV0aW9uVW5pdD4yPC90aWZmOlJlc29sdXRpb25Vbml0PgogICAgICAgICA8dGlmZjpDb21wcmVzc2lvbj41PC90aWZmOkNvbXByZXNzaW9uPgogICAgICAgICA8dGlmZjpPcmllbnRhdGlvbj4xPC90aWZmOk9yaWVudGF0aW9uPgogICAgICAgICA8dGlmZjpQaG90b21ldHJpY0ludGVycHJldGF0aW9uPjI8L3RpZmY6UGhvdG9tZXRyaWNJbnRlcnByZXRhdGlvbj4KICAgICAgPC9yZGY6RGVzY3JpcHRpb24+CiAgIDwvcmRmOlJERj4KPC94OnhtcG1ldGE+CobSriQAAEAASURBVHgB7N0J2CRVdTDg6pkBZmQdERCYAUXCRFwj7qgxKoqKxrgkUUE/NG5Z/I1GjaAR4xb3BUmMGsE9JgombtEo4DoSEZcIBhdwZVOGTYZtZuo/F6aZ/vqr7q7eq6vf+zzHrq66de+5b/U4Sh2qskwjQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQGBGBO4beabQCBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQGCEAitGONYsD3WLSP5eEQdF7B5xq62fy+Lz6ogrI86N+FbE9yM2RUyr7RgT3y1i7dZYE5+3jNgYkfK8IuK8iJTn/0VcG6ERGERgYetJXx/kZOcQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAIF2gbvGjjdG/E/EDRF5ybgm+p0RcXzE/SIm0VbGJI+N+LeIVDxSNtdUVHJWxHERab0agbICq6Lj5VsjbWsECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQGAggfQEjedEnBlRtuChV7/0xI2/itg+YtRtuxjw+RGXRfTKo8zxn8Q4z4pI42oEugk8KQ42f1NpWyNAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAn0JLI/eT4/4VUTzBvSoP38cYz8hYlTtUTHQDyNGnWca7/yIoyMaERqBIoHPx87mby9tawQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAoLTAvaLn2RHNG8/j/vxozLVz6eyWdkxP6HhnxLjzTOOfEjFMrnG6VkOBNbGmzRHN32DaTvs0AgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIEBiBQN2frPDsMHpbRJlXlFwX/dZHnBVxccQVEbeOODDi9hGHRJRt6dUoj4tIRSL9tDTfxyPu2+OkDXH8SxHfi/h1RMp1VcTaiP0j7h1xUESZ9oPo9OCIC8t01mcuBI6JVb66baXHxvfXtO3zlQABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQI3CyQChdOjGg+GaDT58bok55icVhEOqdbWxcH3xWxJaLTeK37fxP9Do4o2w6Ijr+IaB2jffvMOP7QiGURvdq+0eGpEd+IaB+n/fvXos92ERqBJJCKjdp/I2mfRoAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECgVuG3vT0zHabza3fk9PkUhPCtg9ot/2oDghnd86XqftX0W/VITRq+0XHX4a0WmcH8WxP4kY5EkoqbDjZV3Gbs759uijEUhPdWn+Jto/ez3xhR4BAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIzKFAeupEekVI+03m5vdNcez1ESsjhml3iZMvj2iO2+3zJ9Fv1y6TpVef/LjLWK+LY4M89SIVdjwr4qIuY7fmfUP0S0/j0OZbID09pvV30bqdjmkECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQOBmgfvH1jURrTeXW7e/H8fueXPv4Tce3mWu1nnT9ns7TJeKML7SZZyXdjiv1+6DosOXuozbnl/z+yt7Dex4rQXSq4C6FSGlY71eF1RrIIsjQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAgW0Cd4jNyyKaRQftn2+OY9tv6z6yrf/qMmd7Do8smPVtXc5/fkH/MrvuHJ263XBvz6v1+0/LTKBPbQWeFCtr/T0UbT+xtqu3MAIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAoLbAmev4ioujGctr3wtIj9d/xbl3mbc/nV9F3p5YpHtPl3Be19Otnc7/onOZpn7vs9y1x7g79TKhvrQQ+F6vp9VtJfTQCBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQmGOB1bH29HqTohvMm2L/0yZg88MO8xfl1Hxaxi3jnAs7nPeF2N+I6LctixO+G1E0bz/70itUtPkTWBtL3hzR67eS+qS+GgECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAjMoUB6AsSXI4puLl8X+x87IZN3dMihKK/0BJDtIk7qcE56lcugN8LLvMKiKKf2fQ+OHLT5E3hpLLn9t9Dpe+qrESBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgMAcCrw+1tzpZvKfTdDjD7vkUZTfP0b/9LqRomNPHjDv9FSNczqMWTRPt333GzAHp822QD9PiEl9NQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBCYM4H7xHo7vbLhPRO2uEPM1634oeyxLwyR971HlEPK9YAh8nDqbAqkAp2yv9NmP0U9s3mtZU2AAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAIGBBFbFWedGNG8at35+M/an16NMsu0Vk7XmMOj2oUMk/fQR5ZByT77afAmkAqd+f7fpHI0AAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIE5kTgLbHOohvLv4n9+0/BYEXM2em1JkV5Fu377yHzflOcXzRuv/tSsYs2XwKpOOeKiH5/K+kchT3z9VuxWgIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAAQIE5lTgAbHuToURj5yiycaYu9+b3a39h3mqRlr2B4ecv5nLi9Ng2lwJHBmrbV7/fj/TuRoBAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQI1FhgeaztBxFFN5Q/M8V1L4u5OxWQFOXavm/9CHJ/QweX9rm6fb86xth3BLkYYrYEvhDpdvtddDuWztUIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAoMYCT4m1Fd043hT7D57iunfrkFdRrkX7njmC3BeGzCHldcwI8jDEbAmsjXQ3RxT9LsvsS+emMTQCBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQqKHAdrGm8yKKbiD/45TXe5cOeRXl2r4vvT5llxHkv3OMcdUQeXwlzt1hBHkYYrYEXhrptv8m+/2extAIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAoIYCz4o1Fd1EviL27zHl9T6uQ25F+bbv++AIc3/JgHl8O85LTwfR5k/gh7Hk9t9kv9/TGBoBAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQI1ExgZaznlxFFN5FfVIG1vrhDbkX5tu971Ajzb8RY/9ZHLumpHi+L8ESNQJjDdr9Yc/vvcdDvaSyNAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBGok8PRYS9FN5F/H/ioUGpzUIb+inFv3pWKJVRGjbKlg4zkRV0a0ztW+/Z9x/LYR2vwKvCeW3v67GPR7GksjQIAAAQIECBAgQIAAAQIECBAgQIAAAQIECBAgQIAAgZIC6eZ+1dtXI8FDC5J8c+x7QcH+Se/6bkx45wEm/XScc8QA55U5JRWBHB7xexF7R2wfcXbEWRHptSeXRtSlbRcL2TNi94hNEZdvjVQMoxULpN/HRRG7FB/ue28qDrp1xDV9n+kEAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIEKicwEGRLine truncated
|
||||
}
|
||||
},
|
||||
"cell_type": "markdown",
|
||||
"id": "71a8e7d5",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Support Vector Machines\n",
|
||||
"\n",
|
||||
"Support Vector Machines (SVM) are supervised classification and regression methods which are extremely robust to outliers.\n",
|
||||
"\n",
|
||||
"Let us start with a linear classification: in this case, we try to choose a line or hyper-plane that separates the data in two classes. The SVM model chooses this hyperplane based only on the data points close to the decision boundary and any point in time and not based on the entire dataset. Because the method focus only on points close to the decision boundary, any outlier away from the decision boundary does not have a strong influence on the choice of the boundary itself.\n",
|
||||
"\n",
|
||||
"Let us assume we have some data points $x_1, x_2, x_3, \\ldots, x_N$ and each of them belong to either class $-1$ or class $1$. Their true class is informed by the known target classes $t_1, t_2, t_3, \\ldots, t_N$. We want to choose the parameters $w$ and $b$ of a line (or hyperplane in more than 2 dimensions) such that we can easily estimate the unknown class of a new data point $x^\\prime$. For that, we define the decision boundary as:\n",
|
||||
"\n",
|
||||
"$y(x) = w^T x + b$,\n",
|
||||
"\n",
|
||||
"and we say that if $y(x^\\prime) > 0$, the data point belongs to class $1$, otherwise it belongs to class $-1$. The equation of the decision boundary is then $y(x) = 0$. See the figure below (from Wikipedia).\n",
|
||||
"\n",
|
||||
"\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {
|
||||
"linear_classifier.png": {
|
||||
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAg0AAAHBCAYAAADq2ojGAAAABHNCSVQICAgIfAhkiAAAABl0RVh0U29mdHdhcmUAZ25vbWUtc2NyZWVuc2hvdO8Dvz4AACAASURBVHic7N15nFt1vf/x1wDSIMvEKjYCtakoTfFKg1ujqA24EFxoUKFhkaZKISxKWBsQJKhIitCGPRSEFC8QuCqn/FSioISraKoIAYSmQCWVLVwEThVpBqT5/XE6M5npzHSWZE6W95NH72XOSTJf2jr55PP5fD/frmq1WkVERERkK7axewEiIiLSGhQ0iIiIyKgoaBAREZFRUdAgIiIio6KgQUREREZFQYOIiIiMioIGERERGRUFDSIiIjIqChpERERkVBQ0iIiIyKgoaBAREZFRUdAgIiIio6KgQUREREZFQYOIiIiMioIGERERGRUFDSIiIjIqChpERERkVBQ0SMv44x//yDvf+U5+8IMf2L0UEZGO1FWtVqt2L0JkND70oQ/xwAMP4HQ6eeWVV+xejohIx1GmQVqCaZoUi8W+f8/lcvYuSESkAylokJZgGAavvfZa39fpdNq+xYiIdCiVJ6QluN1u1q9fP+DaK6+8gtPptGlFIiKdR5kGaXq5XG6LgAEgmUzasBoRkc6loEGa3nCliJUrV07uQkREOpzKE9LUSqUSM2fOHPb+7bffTjAYnMQViYh0LmUapKkZhjGh+yIiUj/b2b0AkZGEw2FM0wSgUCiwatUqPvjBD/KlL30JQFkGEZFJpPKEtIx0Os2iRYtYvHgxK1assHs5IiIdR+UJERERGRWVJxqhnCMRT1N2e3GbJQqmh2gigrcTRgpUimSSBqbDJJ+vEEom8ZVSpHIVKBUglCTm74TfCBGR9qOgod4qOSK+CE6jQNLrAMDMhfH6TbKFGB6bl9dYZTKxNK54gpATQik/7nCYWChGLAYpf5zjkwFi/pDdCxURkXFQ0FBnxWSEa1xRnt8cMAA4/XEizCSSDpMLu2xcXWNVsgnygQTJzYkEh9PBhl+VcGU8QIGK08+SaKDv8WYhTSJdBEyKxRKuUIJk2ItjyFcXERG7dXbQUEoRCGbxxxNEgh4mnjQvk82sZZrfy8DQwI3XM4WzMjkq4VDbvilW3BFinv7/unwuD3MTBJ0AXqK12yNNg3jGTSIZtn4/zAxBt5+Qs4QRVPlCRKQZdXYjpDtCNhvDmY3g8waJZQqYE3m9ShbjIXA6tnzTc7qckDPITeT1m5zT46kJlgoY2Q3MCQWGDsbyGdLpDPm+J4eIBCqsSmWpTMJaRURk7Fo7aChnSaVSJCIhYllz0K0YoUR+mCfWcPmIpHIUszE8+Rg+b2D8wUO5TBkrLT+Yw+GAnjLlCUUlLaSYwVg/g0DA3X+tVKTU++/+GKlkGG/fTZNyuQcctG0mRkSk1bVw0FAklSgTjEQIeYosTWRq3uhNjMRSssUxfGZ1+QgnsxRzCSt48ASIpvNjCx7MMibDBA04ABOzbT9GlzCiQcKZEgCFdIb13X4CfZ2fJumE0Z9FcHgJhXz9WQgzSybfzcKIhjWJiDSr1g0acmlKwTAuTHLGQ3S73P1vQJUc2Tz4A76xv67TawUP+QS+Yhy/x08knac8weVWKpvfLts1aCgbxC/NY+KASpZ02c+8miCpnIlTDEWH2T1iko3EMeM5UgHlGUREmlXrBg3eKDE/UM6QurebYLi/K598llzPXAL+CbwBOb2EElkK+RT+YoKAx0/EKG3lOS4rcBkiMDArJuDE2a49fq4gseMCeMoGyWSZcCpNJu3GiCZJpRJkHFESQ/55VCgkQiS9aXIx7ZwQEWlmrbt7YvMbdNlIs7o7SKImZihmc2yYEcRfx92NldGkCJwuXEC5YsKg/ROmuYG2DhpwE0qlGTCBIZgkvZVqQzEdIelKkQ27gQpZI48/6FfwICLShFo307BZPruaKf4g/r4rJrncWrr9gYkNUjILZGJBPL4IOXeUXDFPKuge+TnOAME5UC5tWcyomMC82nVKORMmVgwR81UoFosUCmkyeTVCiog0q9bNNABQolgEd7gmPKjkyK4Gf2wc/QxgBQvxGPFsBX8sQa7oY/QJCxfB0BxOMQqU8Q/Yfpgvwtyovw6zINpEIYbv8JWsZyWrlvZfnnv18/atSURERtTimQYXbg9Uyr2lgwqFeJxVzCXgG+PnVbNAOhrA44uR98bJF3OkwmMJGCzuaJKF5RSJQn85w8wmyThOJhV1j/HV2pg3QalapTroVz7SvhMzRURaXYtnGhyEkreTiySIJn24K1AqlGFWePT9DOU86UScRA4CsQT5pHdi2QCHn3Q+SSIWIerx4qZIvujFyEdrZhKIiIi0nhYPGgB3kFS2t9uuRCJzCrPC4dH1M2weI+2tR7BQyxUglg5s/XEiIiItpIXLEyZG2I0rmOnb11DJJUiWDiQeGWULpDtCtmCQCNUxYBAREWlTLZxpKGDkXASTm7fnmTmi0RwhI09IEYCIiEjdtXDQ4CeWCJI1s6SSZYpFk4BRIOjWhj0REZFGaOGgATyh2MRmMYiIiMiotXBPg4iIiEwmBQ0iIiIyKgoaREREZFQUNIiIiMioKGgQERGRUVHQICIiIqPS0lsuG6KcIxFPU3Z7cZslCqaHaCKCVwOjRESkwyloqFXJEfFFcBoFkl5rSJSZC+P1m2QLmgkhIiKdTeWJGsVkhGtcUaLe/qmSTn+cCGcRSZdtXJmIiIj9FDT0KZPNrGWaz8vAU7XdeD1TuDeT6zsYS0REpBMpaOhVyWI8BE7Hls0LTpcTcga5yV+ViIhI07AvaChnSaVSJCIhYllz0K0YoUR+ktdTpgw4nFseeOVwOKCnTNnc8mkiIiKdwqagoUgqUSYYiRDyFFmayND/fmxiJJaSLU5yMcAsYzJM0IADMDFVnxARkQ5mz+6JXJpSMIELk7TxEN0uN31FgUqObB78Ed8IL2BihP3EC2P5pi4imSyRcWyBqFQqgAM1NYiISCezJ2jwRok5gXKG1L3dBO8M9N/LZ8n1zCXh3/ITfz8nwXSBYD3X5HRZgcsQgYFZMQE3zqFmNTz+OAQC8Pa3w7Jl8MlP1nNVIiIiTcOe8sTmN+iykWZ1d5BwTcxQzObYMMOP3zXssxu2Jhe9AcJAprkBcA4dNFx/PTz1FNx/P8ybB9/8Jmzc2OjVioiITDpbhzvls6uZ4o/h77tiksutpdsf2MogJRMjEiCeH0u9YCvlCWeA4ByIl8ow6LtXTGBesGadNQ45BK65BkwTqlW44gr4xS9g5UplHUREpK3YGDSUKBbBHa55g67kyK4Gf2ykfgYAJ8FUvr7lCVwEQ3M4xShQxl8zq6FAvghzo36GnCQ9dy4UixAOQzZrXXvqKSvrcOKJcNFFsMMOdV2piIiIHWyc0+DC7YFKuTdbUKEQj7OKuQR8I/UzNI47mmRhOUWi0J/BMLNJMo6TSUXdwz9x2jS4804ru/C2t1nXerMOs2fDffc1duEiIiKTwMZMg4NQ8nZykQTRpA93BUqFMswKT34/Q9+S/KTzSRKxCFGPFzdF8kUvRj6KdzTPP/poOPDAgVmH9eutMoWyDiIi0uK6qtVq1e5FWEokvDNJh9ZQjLXB0VA33gjRKLzySv+1GTPgpptgv/3sW1cLS6fTLFq0iMWLF7NixQq7lyMi0nFsKk+YGGE3rmCmb4djJZcgWTqQ+HgGKTSjo4+GNWus7Zi9erMO3/qWdliIiEjLsSloKGDkXATDfhwAZo5oNEfIyBAastuwRQ3X63D55ep1EBGRlmNT0OAnlgjiMbOkkgmisSwBo0DS304RQw1lHUREpA3Y1gjpCcW2MouhzfRmHWp7HXqzDnfcoV4HERFpejoae7Ip6yAiIi1KQYMdansdemdTq9dBRESanIIGOx19tDVNsumzDmXy6RjhgA+Py4nT6cDR1UVXVxddXQ4cThcer59gNEW+bPdaRUSkURQ02K3Jsw5mPkk4GCOHn3AiQcC5gQ2uKLnnn+f5p55izYM5jFQUn7NE9tLj+Zg7QGbLM79ERKQNKGhoFk2Ydajko/gCOYLpNLFwAL/XQcWEKV4fPpcLl9uNx+sjEIqRzhUwFkyDnhzZwqQvVUREJoGChmbSVFmHCjmjRNhIE+zdCVspUnwBPN6hhmo78fk9gBdfR22LERHpHAoamlFTZB0cBBIGsdrZGYUCRcDjdQ/x+Aq5XIHuBTFCdp0dIiIiDdVeQUM5RyISJppIkoxFCUdSFFq1vt5UWQeLWSzwArPwDpFJMHNR4sUQRio49BHiIiLS8mw85bLOKjkivghOo0DSax2tbebCeP0m2UILD5Ia6eTMk06CpUsn7eTMfDYP08L43b1XKpSLeXJGilTBTzofwWvPqeYiIjIJ2ibTUExGuMYVJVrzruX0x4lwFpF0i+8DbIqsQ4Fsvocp/gC+SoFU2I/bsQPvCqYhkCKXGSlgqFAu5kiFPQTTrZr6ERGRpg4aKqUcqbAfXzS3lUeWyWbWMs3nZWA53Y3XM4V7M7m+0zRbmp29DuUcufUQCAbA4cLrdVDpAcomDvdIBYkKhUwSo1ChVFyL2RZ/ECIinakpg4ZKKUsy7MMbSlEJpckl/Vt5QhbjIXA6tnzzcrqckDPINWSlNujNOqTTk5p1KBsGDzGPYMABuPBFM6Tmd8OGVUQTI+2xdOANxYiE/HjU7CAi0tLGFzSUs6RSKRKRELGsOehWjFAiP66XrZSyJEM+vKEMhDIU8hmiATdbLZOXy5QBh3PLRzocDugpU263rPjChZOYdShjpO+FA8P92y9xEkzGmAOsT8Yx2u33V0REtjCOoKFIKlEmGIkQ8hRZmsjQ/35hYiSWki2OLQddKRokQl4rWIhkKOTTowsW+r5tGZNhggYcgNmeafHJyjqUM6RWT2F+ODRwZ4Q7Snz+FOhZRSxR7L9eKZAMBkgWB7+QiIi0srEHDbk0pWAYFyY54yG6Xe7+Line truncated
|
||||
}
|
||||
},
|
||||
"cell_type": "markdown",
|
||||
"id": "6a764d92",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n",
|
||||
"Because of our choice of $t_i \\in \\{+1, -1\\}$, we can also write for all data points for which we know the true class: $t_i y(x_i) > 0$, since $y(x_i) < 0$ if $t_i = -1$ and $y (x_i) > 0$ if $t_i > 0$. The idea in SVMs is choose $w$ and $b$ to maximize the distance between the boundary points and the line. What is this distance between a given point and the decision boundary? This can be seen from simple geometrical calculations in the figure below (from Bishop (2006)). Given a known decision boundary, this distance for a point $x$ shall be $\\frac{|y(x)|}{|w|}$.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"As we want that $t_i y(x_i) > 0$ be true for all points close to the boundary, we can use $t_i y(x_i)$ instead of $|y(x_i)|$, using $t_i$ to guarantee that the numerator is always positive, instead of relying on the absolute value. To focus only on the points close to the decision boundary, we try to find $w$ and $b$ which *maximize* the *minimum distance* between the decision boundary and the data points. As the minimum distance is determined only by the points close to the boundary, this makes the method insensitive to what happens away from the decision boundary. Our objective is then to find $w$ and $b$ that satisfy:\n",
|
||||
"\n",
|
||||
"$\\text{argmax}_{w, b} \\left[\\frac{1}{|w|} \\min_{i} t_i \\left(w^T x + b\\right) \\right]$.\n",
|
||||
"\n",
|
||||
"This is a very hard problem, but notice that we only care about the distance between the points and the boundary, and if $w$ and $b$ are both multiplied by the same number, this distance does not change. We can therefore choose a normalisation so that $t_i (w^T x_i + b) = 1$ for the point closest to the boundary. This implies that $t_k (w^T x_k + b) \\geq 1$ for all points in the dataset. Since there is always one point closest, the condition is to maximize $\\frac{1}{|w|}$ (that is, the remaining factor) for that point, which is equivalent to minimize $|w|^2$ for the closest point while miximizing the condition above.\n",
|
||||
"\n",
|
||||
"This can be summarized in simpler form using the Lagrange multiplier formalism, which translates this complex constrained optimization into a simpler unconstrained optimization problem. In that formalism we need to minimize a cost function $\\mathcal{L}$, which introduces the Lagrange multipliers $a_i$:\n",
|
||||
"\n",
|
||||
"$\\mathcal{L}(w, b, a) = \\frac{1}{2} |w|^2 - \\sum_i a_i \\left[ t_i \\left(w^T x_i + b\\right) - 1\\right]$\n",
|
||||
"\n",
|
||||
"One can set the derivative of $\\mathcal{L}$ to zero with respect to $w$, $b$ and $a$ to obtain a simpler version of this optimization problem. Further details can be consulted in Bishop (2006) or in the Wikipedia description.\n",
|
||||
"\n",
|
||||
"It is interesting to note that by using the Kernel trick, one can choose the function $y(x) = w^T \\phi(x) + b$ and with an appropriate choise of $\\phi$, we can apply a linear classification after a non-linear transformation which allows this method to be expanded into non-linear classification. It can be shown following the procedure above that the final algorithm does not depend on $\\phi(x)$, but only on $k(x, x^\\prime) = \\phi(x)\\phi(x^\\prime)$, so one needs only to specify the kernel function $k$.\n",
|
||||
"\n",
|
||||
"Similar logic can be used for regression, on which only the *worse* prediction of the feature becomes relevant.\n",
|
||||
"\n",
|
||||
"Fortunately the details of the algorithm are not needed to implement this for classification. In the following example, we show how the algorithm can be easily used from the scikit-learn modules."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "fce4d8e8",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We start by loading the necessary Python modules. If you have not yet installed them, run the following cell to install them with pip:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"id": "300cf8d3",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import matplotlib.pyplot as plt\n",
|
||||
"\n",
|
||||
"import pandas as pd\n",
|
||||
"import numpy as np\n",
|
||||
"from sklearn import svm"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "0ecd6a69",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Let's generate the fake data now to have something to cluster."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"id": "4959a292",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def generate_data(N: int) ->np.ndarray:\n",
|
||||
" assert N > 1\n",
|
||||
" loc = [np.array([2.0, 2.0]), np.array([-1.0, 1.0])]\n",
|
||||
" scale = [np.array([0.5, 0.5]), np.array([0.5, 0.2])]\n",
|
||||
" data = np.concatenate([np.random.default_rng().laplace(loc=l, scale=s, size=(N, 2))\n",
|
||||
" for l, s in zip(loc, scale)], axis=0)\n",
|
||||
" source = np.concatenate([k*np.ones([N, 1]) for k in range(len(loc))], axis=0)\n",
|
||||
" return np.concatenate([data, source], axis=1)\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"id": "82929490",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"data = generate_data(N=1000)\n",
|
||||
"data = pd.DataFrame(data, columns=[\"x\", \"y\", \"source\"])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "d8295e8a",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Let's print out the dataset read first."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"id": "024fb65a",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/html": [
|
||||
"<div>\n",
|
||||
"<style scoped>\n",
|
||||
" .dataframe tbody tr th:only-of-type {\n",
|
||||
" vertical-align: middle;\n",
|
||||
" }\n",
|
||||
"\n",
|
||||
" .dataframe tbody tr th {\n",
|
||||
" vertical-align: top;\n",
|
||||
" }\n",
|
||||
"\n",
|
||||
" .dataframe thead th {\n",
|
||||
" text-align: right;\n",
|
||||
" }\n",
|
||||
"</style>\n",
|
||||
"<table border=\"1\" class=\"dataframe\">\n",
|
||||
" <thead>\n",
|
||||
" <tr style=\"text-align: right;\">\n",
|
||||
" <th></th>\n",
|
||||
" <th>x</th>\n",
|
||||
" <th>y</th>\n",
|
||||
" <th>source</th>\n",
|
||||
" </tr>\n",
|
||||
" </thead>\n",
|
||||
" <tbody>\n",
|
||||
" <tr>\n",
|
||||
" <th>0</th>\n",
|
||||
" <td>2.696133</td>\n",
|
||||
" <td>2.569388</td>\n",
|
||||
" <td>0.0</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>1</th>\n",
|
||||
" <td>2.156646</td>\n",
|
||||
" <td>1.907207</td>\n",
|
||||
" <td>0.0</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>2</th>\n",
|
||||
" <td>3.219006</td>\n",
|
||||
" <td>2.012231</td>\n",
|
||||
" <td>0.0</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>3</th>\n",
|
||||
" <td>2.075721</td>\n",
|
||||
" <td>1.932324</td>\n",
|
||||
" <td>0.0</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>4</th>\n",
|
||||
" <td>1.698718</td>\n",
|
||||
" <td>2.143439</td>\n",
|
||||
" <td>0.0</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>...</th>\n",
|
||||
" <td>...</td>\n",
|
||||
" <td>...</td>\n",
|
||||
" <td>...</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>1995</th>\n",
|
||||
" <td>-2.023710</td>\n",
|
||||
" <td>1.375677</td>\n",
|
||||
" <td>1.0</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>1996</th>\n",
|
||||
" <td>1.190933</td>\n",
|
||||
" <td>0.369058</td>\n",
|
||||
" <td>1.0</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>1997</th>\n",
|
||||
" <td>-1.525767</td>\n",
|
||||
" <td>1.248843</td>\n",
|
||||
" <td>1.0</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>1998</th>\n",
|
||||
" <td>-1.191291</td>\n",
|
||||
" <td>0.644813</td>\n",
|
||||
" <td>1.0</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>1999</th>\n",
|
||||
" <td>1.103239</td>\n",
|
||||
" <td>0.973839</td>\n",
|
||||
" <td>1.0</td>\n",
|
||||
" </tr>\n",
|
||||
" </tbody>\n",
|
||||
"</table>\n",
|
||||
"<p>2000 rows × 3 columns</p>\n",
|
||||
"</div>"
|
||||
],
|
||||
"text/plain": [
|
||||
" x y source\n",
|
||||
"0 2.696133 2.569388 0.0\n",
|
||||
"1 2.156646 1.907207 0.0\n",
|
||||
"2 3.219006 2.012231 0.0\n",
|
||||
"3 2.075721 1.932324 0.0\n",
|
||||
"4 1.698718 2.143439 0.0\n",
|
||||
"... ... ... ...\n",
|
||||
"1995 -2.023710 1.375677 1.0\n",
|
||||
"1996 1.190933 0.369058 1.0\n",
|
||||
"1997 -1.525767 1.248843 1.0\n",
|
||||
"1998 -1.191291 0.644813 1.0\n",
|
||||
"1999 1.103239 0.973839 1.0\n",
|
||||
"\n",
|
||||
"[2000 rows x 3 columns]"
|
||||
]
|
||||
},
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"data"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "1c178424",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We can plot this fairly easily using Matplotlib."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"id": "e63b38c5",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAqYAAAKsCAYAAAA+4H/5AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAAEAAElEQVR4nOzddXgT2RoH4N9MUndqlFKKFIprkeLu7i53gYXF2cXdbdHFFnbRxd0p7lIo7l6BUndJm5m5f4QGQqRJaZOUfu/z5LmbkXPOtL3ky5HvMIIgCCCEEEIIIcTAWEM3gBBCCCGEEIACU0IIIYQQYiQoMCWEEEIIIUaBAlNCCCGEEGIUKDAlhBBCCCFGgQJTQgghhBBiFCgwJYQQQgghRoECU0IIIYQQYhQoMCWEEEIIIUaBAlNCCCGEEGIUKDAlhBBCCPkJXLlyBW3atEGBAgXAMAwOHz6c6T2XLl1C5cqVYWZmBi8vL2zZsiXH26kJBaaEEEIIIT+BpKQkVKhQAWvWrNHq+vfv36NVq1Zo0KABHjx4gNGjR2PgwIHw8/PL4ZaqxwiCIBisdkIIIYQQku0YhsGhQ4fQvn17tddMmDABJ06cwJMnT+THunfvjtjYWJw+fVoPrVQmNkitBsLzPD59+gQbGxswDGPo5hBCCCFEC4IgICEhAQUKFADL6n+wNzU1FWlpaXqvF5A9+/cxi5mZGczMzH647Js3b6Jx48YKx5o1a4bRo0f/cNlZlacC00+fPsHDw8PQzSCEEEJIFgQHB6NgwYJ6rTM1NRVFPK3xOZzTa70ZrK2tkZiYqHBsxowZmDlz5g+X/fnzZ7i6uiocc3V1RXx8PFJSUmBhYfHDdegqVwWmHz9+xIQJE3Dq1CkkJyfDy8sLmzdvho+Pj1b329jYAJD9Ydva2uZkUwkhhBCSTeLj4+Hh4SH/HNentLQ0fA7nEBhQGLY2+u2tjU/g4Vnlg1Lckh29pcYq1wSmMTExqFWrFho0aIBTp07B2dkZr1+/hoODg9ZlZHSF29raUmBKCCGE5DKGnIZnbcPA2ka/9fPI2bglf/78CAsLUzgWFhYGW1tbg/SWArkoMF20aBE8PDywefNm+bEiRYpovEcikUAikcjfx8fH51j7CCGEEEJyE19fX5w8eVLh2NmzZ+Hr62ugFuWidFFHjx6Fj48PunTpAhcXF1SqVAkbN27UeM+CBQtgZ2cnf9H8UkIIIYT8rBITE/HgwQM8ePAAgCwd1IMHDxAUFAQAmDRpEvr27Su/fsiQIXj37h3Gjx+PFy9eYO3atdi7dy/GjBljiOYDyEWB6bt377Bu3ToUL14cfn5+GDp0KEaOHImtW7eqvWfSpEmIi4uTv4KDg/XYYkIIIYT8LDiBN8hLF3fv3kWlSpVQqVIlAMDYsWNRqVIlTJ8+HQAQGhoqD1IB2cjziRMncPbsWVSoUAFLly7FP//8g2bNmmXfD05HuSaPqampKXx8fHDjxg35sZEjR+LOnTu4efOmVmXEx8fDzs4OcXFxNMeUEEIIySUM+fmdUXf4S0+DLH5y8Q7MU3FLrukxdXNzQ+nSpRWOlSpVSiHyJ4QQQgjJCTwEg7zymlwTmNaqVQsvX75UOPbq1St4enoaqEWEEEIIISQ75ZpV+WPGjEHNmjUxf/58dO3aFf7+/tiwYQM2bNhg6KYRQggh5CfHg4duMz6zp868Jtf0mFatWhWHDh3Crl27ULZsWcyZMwcrVqxAr169DN00QgghhBCSDXJNjykAtG7dGq1btzZ0MwghhBBCSA7IVYEpIYQQQoghcIIATs+JjPRdnzHINUP5hBBCCCHk50Y9poQQQgghmTBE+iZKF0UIIYQQQoiBUGBKCCGEEEKMAg3lE0IIIYRkgocAjobycxz1mBJCCCGEEKNAPaaEEEIIIZmgxU/6QT2mhBBCCCHEKFCPKSGEEEJIJijBvn5QjykhhBBCCDEKFJgSQggxetJ0KRJiEiHkwR4kQvISGsonhBBitCQpEmyavAsnNp6DJFkCp4KO6DWlE1oNbgyGYQzdPJKH8F9e+q4zr6HAlBBCiNGa3WUZ7py+D4GX9ZRGhkRh5dANSJeko8PIlgZuHSEku9FQPiGEEKP0+t47+J+8Jw9Kv7V99j6kp6UboFUkr+K+JNjX9yuvocCUEEKIUXp+6zXUjdYnRCfi8/tw/TaIEJLjKDAlhBBilGwdraF2rRMD2OSz1mt7CCE5j+aYEkIIMUo12vjAyt4SyfEpCsP5rIiFT/OKsHe2M2DrSF7DCbKXvuvMa6jHlBBCiFEytzTDrIPjYWZhCjCAyEQEACjglR9jNwwxcOsIITmBekwJIYQYrQr1y2BX8N+4uPs6oj5Fo1jFIvBtUwViE/r4IvpF6aL0g/6fTQghxKhZ21uhzZCmhm4GIUQPKDAlhBBCCMkEDwYc9LupA6/n+owBzTElhBBCCCFGgQJTQgghhBBiFGgonxBCCCEkE7wge+m7zryGekwJIYQQQohRoB5TQgghhJBMcAZY/KTv+owB9ZgSQgghhBCjQIEpIYQQQggxCjSUTwghhBCSCRrK1w/qMSWEEEIIIUaBekwJIYQQQjLBCwx4Qc87P+m5PmNAPaaEEEIIIcQoUI8pIYQQQkgmaI6pflCPKSGEEEIIMQoUmBJCCCGEEKNAQ/mEEEIIIZngwILTc38ep9fajAP1mBJCCCGEEKNAPaaEEEIIIZkQDJAuSqB0UYQQQgghhBgGBaaEEEIIIcQo0FA+IYQQQkgmKI+pflCPKSGEEEIIMQrUY0oIIYQQkglOYMEJek4XJei1OqNAPaaEEEIIIcQoUI8pIYQQQkgmeDDg9dyfxyPvdZlSjykhhBBCCDEKFJgSQgghhBCjQEP5hBBCCCGZoHRR+kE9poQQQgghxChQjykhhBBCSCYMky6KFj8RQgghhBBiEBSYEkIIIYQQo0BD+YQQQgghmZDlMdXvYiR912cMqMeUEEIIIYQYBeoxJYQQQgjJBA8WHO38lOOox5QQQgghhBgFCkwJIYQQQohRoKF8QgghhJBMUB5T/aAeU0IIIYQQYhSox5QQQgghJBM8WPC0+CnHUY8pIYQQQggxCtRjSgghhBCSCU5gwAn6TXiv7/qMAfWYEkIIIYQQo0CBKSGEEEIIMQo0lE8IIYQQkgnOADs/cbT4yXjNnDkTDMMovEqWLGnoZhFCCCGEkGySq3pMy5Qpg3Pnzsnfi8W5qvmEEEIIyaV4gQWv5wT7fB5MsJ+rIjuxWIz8+fMbuhmEEEIIISQH5JqhfAB4/fo1ChQogKJFi6JXr14ICgrSeL1EIkF8fLzCixBCCCGEGKdcE5hWr14dW7ZswenTp7Fu3Tq8f/8ederUQUJCgtp7FixYADs7O/nLw8NDjy0mhJC8RxAEvPB/jSv7byLwWbChm0NItslY/KTvV17DCELunMAQGxsLT09PLFu2DL/88ovKayQSCSQSifx9fHw8PDw8EBcXB1tbW301lRBC8oTwoAhMb78Ybx98kB/zaV4RU3eNhpWdleEaRnK9+Ph42NnZGeTzO6PujfeqwNJGpNe6kxM4DKockKfillwbitvb26NEiRJ48+aN2mvMzMxga2ur8CKEEJL9BEHA1DYL8eGJ4hSre2cfYdng9QZqFSHZh8fX3Z/09eIN/dAGkGsD08TERLx9+xZubm6GbgohhOR5T2+8xPvHQeCkih+lPMfj6v7biAqNMVDLCCG5Sa4JTP/44w9cvnwZHz58wI0bN9ChQweIRCL06NHD0E0jhJA8L/RdmNpzgiAgIjhSj60hJPvxYA3yymtyTbqokJAQ9OjRA1FRUXB2dkbt2rVx69YtODs7G7pphBCS5xUqVVDtOZGYhVtRVz22hhCSW+WawHT37t2GbgIhhBA1SlQpijI1vfH89mvw3NfhfIZl0KRvfdg50Rx/Qkjm8l4fMSGEkGzHMAxmHR6Pyo3LfT3GMmjSpx5GrFadOYWQ3IQTWIO88ppc02NKCCHEuNk52WLBqakIfR+G8KBIeHgXQL78DoZuFiEkF6HAlBBCSLZyK+IKtyI0p5T8XHgw4MHovc68Ju/1ERNCCCGEEKNEgSkhhBBCCDEKNJRPCCGEEJIJQyxGyouLn/LeExNCCCGEEKNEPaaEEEIIIZngwILTc3+evuszBnnviQkhhBBCiFGiHlNCCCGEkEzwAgNe0HO6KD3XZwwoMCWEEPJTiAiJwq3jARB4AdVaVkL+wi6GbhIhREcUmBJCCMn1di86jE1TdkIQBDAAhBFAr8md0G92NzBM3ut1IiS3osCUEEJIrnbv3CP8O2mH/L3w5X93zDsAr8pFULtDdcM0jPxUeAMsfuLz4FKgvPfEhBBCfionNp4FK1L+OGNFLI7/fdYALSKEZBX1mBJCCMnVIoKjwXO80nGe4xH5MdoALSI/I15gwes54b2+6zMGee+JCSGE/FS8qxYDK1b+OBOJWZSs5mWAFhFCsooCU0IIIbla+xEtYGIiBst+XeTEsAwYlkWnMa0N2DJCiK4oMCWEEJKruXu5YcmFmShaobD8mGfpgljoNxVFyhYyXMPIT4UDY5BXXkNzTAkhhOR6paoXx7qAxYj8GAVBAJzc81GaKEJyIQpMCSGE/DSc3B0N3QTyk6LFT/qR956YEEIIIYQYJeoxJYQQQgjJBAfofc4np9fajAP1mBJCCCGEEKNAgSkhhBBCCDEKNJRPCCGEEJIJWvykH3nviQkhhBBCiFGiHlNCCCGEkExwAgtOzz2Y+q7PGOS9JyaEEEIIIUaJAlNCCCGEEGIUaCifEEIIISQTAhjwes5jKui5PmNAPaaEEEJyhfCgCAScfYiQV58M3RRCjNaaNWtQuHBhmJubo3r16vD399d4/YoVK+Dt7Q0LCwt4eHhgzJgxSE1N1VNrlVGPKSGEEIMKfvkRd/0ewsTMBDXb+SBffgeF8+8efcDc7isQ/OKj/FjFBmUxeddoOLjY6buLine truncated
|
||||
"text/plain": [
|
||||
"<Figure size 800x800 with 2 Axes>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"fig, ax = plt.subplots(figsize=(8, 8))\n",
|
||||
"data.plot.scatter(x=\"x\", y=\"y\", c=\"source\", colormap='viridis', ax=ax)\n",
|
||||
"ax.set(xlabel=\"x\", ylabel=r\"y\", title=\"\")\n",
|
||||
"plt.show()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "a376636d",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"The usage of the scikit-learn interface is very standard, and one does not even need to know the details of how the SVM algorithm operates. It is however important to understand the basics, to understand how it operates.\n",
|
||||
"\n",
|
||||
"The kernel choice is effectively changing the choice for $\\phi(x)$ in the explanation above."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"id": "0837b3ff",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"clf = svm.SVC(kernel=\"linear\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"id": "8798f857",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/html": [
|
||||
"<style>#sk-container-id-1 {color: black;}#sk-container-id-1 pre{padding: 0;}#sk-container-id-1 div.sk-toggleable {background-color: white;}#sk-container-id-1 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-1 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-1 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-1 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-1 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-1 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-1 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-1 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-1 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-1 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-1 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-1 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-1 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-1 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-1 div.sk-item {position: relative;z-index: 1;}#sk-container-id-1 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-1 div.sk-item::before, #sk-container-id-1 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-1 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-1 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-1 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-1 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-1 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-1 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-1 div.sk-label-container {text-align: center;}#sk-container-id-1 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-1 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-1\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>SVC(kernel='linear')</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-1\" type=\"checkbox\" checked><label for=\"sk-estimator-id-1\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SVC</label><div class=\"sk-toggleable__content\"><pre>SVC(kernel='linear')</pre></div></div></div></div></div>"
|
||||
],
|
||||
"text/plain": [
|
||||
"SVC(kernel='linear')"
|
||||
]
|
||||
},
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"clf.fit(data.loc[:, [\"x\", \"y\"]], data.loc[:, \"source\"])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "e928f498",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Only a few vectors are needed to choose the decision boundary, since only they contribute to the minimum distance shown before. Those vectors are called the support vectors and give the name to the method. They can be accessed using the following attribute:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"id": "fb5796e5",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"array([[ 0.41056889, 1.78218835],\n",
|
||||
" [ 0.30256648, 2.3182121 ],\n",
|
||||
" [ 2.14295088, -0.34278388],\n",
|
||||
" [ 0.58553232, 1.48987089],\n",
|
||||
" [ 0.28752702, 2.23120296],\n",
|
||||
" [ 0.15072118, 2.1413319 ],\n",
|
||||
" [ 1.43461398, 0.03317411],\n",
|
||||
" [-0.3891846 , 2.26540759],\n",
|
||||
" [ 1.83334223, 0.12445596],\n",
|
||||
" [ 1.29007474, 0.97506863],\n",
|
||||
" [ 0.75711383, 0.90276538],\n",
|
||||
" [ 0.09565665, 2.07091918],\n",
|
||||
" [ 1.34500228, 0.88297939],\n",
|
||||
" [ 0.61904021, 1.83612709],\n",
|
||||
" [-0.14119261, 2.5988676 ],\n",
|
||||
" [ 0.77180767, 0.86440934],\n",
|
||||
" [ 0.2669691 , 2.0392128 ],\n",
|
||||
" [ 1.68926936, 0.01840854],\n",
|
||||
" [ 0.50613241, 1.16810632],\n",
|
||||
" [ 0.01945822, 2.25873483],\n",
|
||||
" [ 0.82867552, 1.58547561],\n",
|
||||
" [ 1.85308781, -0.14917069],\n",
|
||||
" [ 0.17348833, 2.27440134],\n",
|
||||
" [-0.75842541, 2.11432881],\n",
|
||||
" [ 0.25062787, 1.52335682],\n",
|
||||
" [ 1.56146592, -1.51366547],\n",
|
||||
" [ 0.44701481, 1.21964164],\n",
|
||||
" [ 2.19568687, -0.31415198],\n",
|
||||
" [ 1.4528239 , 0.73307476],\n",
|
||||
" [ 0.79153547, 1.23148589],\n",
|
||||
" [ 1.57443318, 0.08585887],\n",
|
||||
" [ 1.93792403, -0.5280567 ],\n",
|
||||
" [ 0.68552079, 1.52893339],\n",
|
||||
" [-0.03819845, 0.82612821],\n",
|
||||
" [ 0.59291621, 0.85866499],\n",
|
||||
" [ 0.33683056, 2.16828835],\n",
|
||||
" [ 0.98648254, 1.29485525],\n",
|
||||
" [ 0.53769412, 1.37441508],\n",
|
||||
" [-0.03288985, 2.06483046],\n",
|
||||
" [-0.32436547, 2.13636717],\n",
|
||||
" [ 1.21624257, 0.45929627],\n",
|
||||
" [ 1.07143935, 1.20320275],\n",
|
||||
" [ 1.71590049, -0.04618048],\n",
|
||||
" [ 0.91647289, 1.01049619],\n",
|
||||
" [ 0.60226815, 1.71458321],\n",
|
||||
" [ 0.58849745, 1.57218807],\n",
|
||||
" [ 0.69651967, 1.50309205],\n",
|
||||
" [ 0.44647927, 0.93266835],\n",
|
||||
" [ 0.51224545, 1.54683233],\n",
|
||||
" [ 1.07088746, 0.67769166],\n",
|
||||
" [ 0.12893323, 2.94243204],\n",
|
||||
" [ 1.1462404 , 1.32702971],\n",
|
||||
" [ 0.45149753, 1.48930132],\n",
|
||||
" [ 0.64129091, 1.31913684],\n",
|
||||
" [ 0.39026459, 1.08642014],\n",
|
||||
" [ 1.36879445, 1.09708582],\n",
|
||||
" [ 0.39157111, 1.03846165],\n",
|
||||
" [ 0.75509546, 1.01569198],\n",
|
||||
" [ 0.58964822, 0.82917338],\n",
|
||||
" [ 1.22083799, 0.5120612 ],\n",
|
||||
" [ 0.38112858, 1.01779536],\n",
|
||||
" [ 1.55761155, 0.88953808],\n",
|
||||
" [ 1.71111786, 0.96239078],\n",
|
||||
" [ 1.25401074, 1.02403181],\n",
|
||||
" [ 0.75523897, 1.13468049],\n",
|
||||
" [ 0.37188489, 1.03140049],\n",
|
||||
" [ 0.57774317, 0.9754727 ],\n",
|
||||
" [ 0.2353235 , 1.48854786],\n",
|
||||
" [ 0.44508102, 1.01534577],\n",
|
||||
" [ 0.28534178, 1.51117762],\n",
|
||||
" [ 0.34122272, 1.34914158],\n",
|
||||
" [ 0.28452689, 1.21300192],\n",
|
||||
" [ 0.21199647, 1.48375117],\n",
|
||||
" [ 0.85917257, 0.88256397],\n",
|
||||
" [ 0.55795284, 0.92718673],\n",
|
||||
" [ 1.08655858, 1.36719775],\n",
|
||||
" [ 0.14497746, 1.31181893],\n",
|
||||
" [ 0.31228955, 1.34121278],\n",
|
||||
" [ 1.03246285, 1.23435219],\n",
|
||||
" [ 0.66432187, 0.88589108],\n",
|
||||
" [ 0.679391 , 1.2631045 ],\n",
|
||||
" [ 1.40086613, 1.14475732],\n",
|
||||
" [-0.00687548, 1.53140058],\n",
|
||||
" [ 0.78257947, 0.69636807],\n",
|
||||
" [ 0.24026626, 1.28957287],\n",
|
||||
" [ 1.78549799, 0.77823661],\n",
|
||||
" [ 0.6066357 , 1.09370956],\n",
|
||||
" [ 2.10168948, 0.99111199],\n",
|
||||
" [ 0.45349731, 0.94162822],\n",
|
||||
" [ 0.6041748 , 0.879707 ],\n",
|
||||
" [ 0.29106853, 1.60620007],\n",
|
||||
" [ 0.06685012, 1.61460892],\n",
|
||||
" [ 1.19093279, 0.36905819],\n",
|
||||
" [ 1.10323939, 0.97383906]])"
|
||||
]
|
||||
},
|
||||
"execution_count": 8,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"clf.support_vectors_"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "b54001fc",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We can now predict to which class a new data point belongs to using `clf.predict(new_data_samples)`. It is however interesting to visualize the decision boundary itself.\n",
|
||||
"\n",
|
||||
"(Taken from https://scikit-learn.org/stable/auto_examples/svm/plot_separating_hyperplane.html#sphx-glr-auto-examples-svm-plot-separating-hyperplane-py -- take a look there for more resources and more examples)\n",
|
||||
"\n",
|
||||
"The code below also highlights the support vectors."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 9,
|
||||
"id": "cc8fc1f1",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"/home/daniloefl/workspace/ml-lecture-oct-2023/env/lib/python3.9/site-packages/sklearn/base.py:465: UserWarning: X does not have valid feature names, but SVC was fitted with feature names\n",
|
||||
" warnings.warn(\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAp4AAAKUCAYAAACt7Tj1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAAEAAElEQVR4nOz9d3Sc93Un/r+f6b2i904ABHvvYqdk2Za865IojpO4yI7Wm9ibk/16zya/OHsSJ+uzide2WtyiTdwVS5ZkEWDvnQRIgui9YzCD6b08vz+GMwQIYGZQngftvs6BLXLKfQaFuPP53M+9DMuyLAghhBBCCOGYYLEvgBBCCCGErA6UeBJCCCGEEF5Q4kkIIYQQQnhBiSchhBBCCOEFJZ6EEEIIIYQXlHgSQgghhBBeUOJJCCGEEEJ4QYknIYQQQgjhhWixLyCRSCSCoaEhqNVqMAyz2JdDCCGEEEKewrIsnE4ncnJyIBAkXtNc0onn0NAQ8vPzF/syCCGEEEJIEv39/cjLy0t4H84Tz8HBQfz3//7fcfLkSXg8HpSVleEnP/kJtm7dmvSxarUaQPSFaDQari+VLBCXy4U333wTgUAAzz33HDZs2MB5zF/+8pfo6upCaWkpPvWpT3Eer6GhASdPnoREIsHLL78MlUrFeUxCCCFkKXI4HMjPz4/nbYlwmnharVbs2bMHBw8exMmTJ5Geno729nbo9fqUHh/bXtdoNJR4LiMajQbHjx/HqVOncPPmTWzfvh1SqZTTmC+++CJef/11DAwMYHR0FOXl5ZzG27dvH1paWjA0NIQ7d+7gYx/7GKfxCCGEkKUulbJITg8X/eM//iPy8/Pxk5/8BNu3b0dxcTGOHTuG0tJSLsOSJWDHjh0wGo1wuVy4ePEi5/HS0tKwY8cOAEBtbS3C4TCn8RiGwbPPPgsAqK+vx9DQEKfxCCGEkJWA08Tzvffew9atW/HJT34SGRkZ2LRpE37wgx/MeH+/3w+HwzHpgyxPQqEQJ06cAADcvHkTZrOZ85gHDhyASqWCxWLBzZs3OY+Xn5+P9evXg2VZnDx5EizLch6TEEIIWc44TTy7urrw+uuvo7y8HHV1dfjKV76C//pf/yveeuutae//rW99C1qtNv5BB4uWt/LyclRUVCAcDqOuro7zeFKpFIcPHwYAXLx4ES6Xi/OYR48ehUQiQX9/Px4+fMh5PEIIIWQ54zTxjEQi2Lx5M/7+7/8emzZtwpe+9CV88YtfxBtvvDHt/b/xjW/AbrfHP/r7+7m8PMKD48ePQygUwm63w+v1ch5v48aNyM3Nhd/vx9mzZzmPp1arsW/fPgDA8PAw5/EIIYSQ5YzTw0XZ2dmorq6e9HdVVVX4j//4j2nvL5VKOT+EQvhlNBrxuc99Dnl5eUl7ey2EWO3lD3/4Q9TX12Pr1q3Izc3lNOauXbtQXFyctIUEIYQQstpxmgns2bMHra2tk/6ura0NhYWFXIYlS0xBQQEvSWdMXl5evIUTH7WXIpGIkk5CCCEkBZxmA1/72tdw48YN/P3f/z06Ojrws5/9DP/yL/+CV155hcuwZIkKhUK4evUqnE4n57GOHDkCiUSCgYEBPHjwgPN4MTabDVevXuUtHiGEELKccJp4btu2De+88w5+/vOfo6amBv/rf/0vfOc738FLL73EZViyRL377rs4ffo0b7WX+/fvBwCcOXMGfr+f85g+nw+vv/46Tp8+jc7OTs7jEUIIIcsN5/ufzz//PB4+fAifz4fm5mZ88Ytf5DokWaJ27twJIDr1Z2BggJd4BoMBTqcTly9f5jyeTCbDpk2bAPDTS5QQQghZbvgrvCOrXl5eHjZu3AiAv9rL48ePAwCuX7+O8fFxTuMBwDPPPAOFQoGxsTHcvn2b83iEEELIckKJJ+HV4cOHIZFIMDg4iPv373Mer6KiAmVlZbz1EpXJZPFeohcuXIDb7eY8JiGEELJcUOJJeKVWq3HgwAEA/NReMgyDEydOQCAQoLW1FR0dHZzGA4BNmzYhOzsbPp8P586d4zweIYQQslxQ4kl4N3GO+6VLlziPx/ccd4FAEJ/jfu/ePWosTwghhDxGiSfhXaz2sqysLH4Yh2sHDhyAUqmE2WzGrVu3OI9XUFCATZs2Yf/+/UhLS+M8HiGEELIcMCzXJzzmweFwQKvVwm63Q6PRLPblkAXGsiwYhuEt3r179/Dee+9BKpXiq1/9KlQqFafx+H59hBBCyGKYTb5GK55k0UxMyoLBIOfxNm3ahJycHPj9fl5qLye+PpZlEQqFOI9JCCGELGWUeJJF5fV68f777+PNN9/kvPYyNscdAOrr6zE0NMRpvJjBwUH84Ac/wMWLF3mJRwghhCxVlHiSRcUwDFpaWmA2m3Hz5k3O4+Xn52P9+vVgWZaXXqJAdAtiaGgI169fh9Vq5TweIYQQslRR4kkWlUwmw5EjRwAAFy9ehMvl4jxmbI57f38/Hj58yHm8yspKlJSUIBQK8dJLlBBCCFmqKPEki27jxo3Izc2F3+/nZY67RqPBvn37AACnT59GIBDgNN7EXqItLS3o6uriNB4hhBCyVFHiSRbd07WXg4ODnMfctWsX9Ho9b3PcMzIysG3bNgDRcaE0x50QspDCERbjngCsngAiS7dZDSGUeJKlIS8vDxs2bADA/xz3a9eu8T7H/c6dO5zHI4SsfCzL4tGIA+82DqGu1YTaVhN+2ziMtjEXLzXshMwWJZ5kyYjVXlosFl4SwTVr1qC0tJS3Oe5yuRyHDh0CALS3t9MvBULIvN0btOHBsAOB8JN/T3yhCO4O2NA06lzEKyNkeqLFvgBCYtRqNT796U8jJycHcrmc83ix2svXX38dra2t6OzsRGlpKacxN2/eDIVCgaqqKmouTwiZF5c/hLYx94y3PxxxoCxNCalIyONVEZIYrXiSJaW0tJSXpDMmPT0d27dvB8DfHPfq6mpKOgkh89Zn8yDRvyQsCwzYfbxdDyGpoMSTLEksy6K5uZn32svbt29zHi8mEAjgzp07tOVOCJmTYDj5vx3BcISHKyEkdZR4kiXp/Pnz+OUvf8lL7aVMJsPhw4cBABcuXIDbPfPW1UKJRCJ488038cEHH+DRo0ecxyOErDw6uRjJUk+dTMzLtRCSKko8yZK0fv16CAQCtLa2oqOjg/N4mzZtQnZ2Nnw+Hy+9RAUCAdavXw8AOHXqFOe9RAkhK0+eVg6paPpf4wwAlVSITLWU34siJAlKPMmSlJaWhh07dgDgr/aS7znuu3fvhk6ng8PhwJUrVziPRwhZWYQCBvtLjBAJplZ6SoQC7C9Oo3pysuRQ4kmWrAMHDkCpVMJsNuPWrVucxysoKMC6devAsixqa2s5r70Ui8WTeonSHHdCyGylKaV4vjoL67I1SFdKkKGSYGOOFh+pzoJWTtvsZOmhxJMsWU/XXvIxx/3o0aMQi8Xo6+tDY2Mj5/EqKytRXFyMUCiEU6dOcR6PELLyyMVC1GRpcKQiA4fLM1CVqZ5xC56QxUbfmWRJ27RpE3Jyclb0HPdnn30WAoEAzc3NNMedEELIikaJJ1nSYolZeno61q5dy0vM3bt3Q6/Xw+Fw8DrHvaamBkajkfN4hBBCyGJh2CXcRNDhcECr1cJut0Oj0Sz25ZBFxLIsr0Xyzc3N+OUvfwmRSIRXXnkFer2e03h8vz5CCCFkocwmX6MVT7IsTEzKIhHuGyJXVlaipKQEoVCIl16iTyedfLxGQgghhG+UeJJlIxwO49q1a3j99dd5qb08ceIEBAIBWlpaeKu9tNvtePvtt3Hy5Ele4hFCCCF8osSTLBssy+LOnTsYGxvjtfYSAE6ePMl5L1EAsNlsaGxsxJ07dzAyMsJ5PEIIIYRPlHiSZUMkEk3qe8n3HPc7d+5wHq+wsBA1NTVgWRYnT56kOe6EEEJWFEo8ybJSUVGB0tJShMNhXmov5XI5Dh06BCA6P56POe6xXqK9vb1oamriPB4hhBDCF0o8ybIysfaytbUVnZ2dnMfcvHkzsrKy4PP5cO7cOc7jabVa7N27F0B0jnswGOQ8JiGEEMIHSjzJspOeno7t27cD4H+O+7179zA8PMxpPODJHHe73U5z3AkhhKwYlHiSZemZZ56BUqmExWJBf38/5/H4rr0Ui8U4duwYAODhw4e8HGwihBBCuCZa7AsgZC5kMhleeOEF6HQ6pKen8xLz6NGjaG1tRV9fHx49eoSamhpO41VVVeGjH/0o1q1bB6FQyGksQgghhA+04kmWrfLyct6STmBq7SUfvUS3bNkCiUTCaRxCCCGEL5R4khVhdHQUJpOJ8zix2kuHw4GrV69yHi+GZVk0NTXRRCNCCCHLGiWeZNm7f/8+3njjDXzwwQe81F7GeolevXoVVquV03gxP//5z/GrX/2Kl16ihBBCCFco8STLXnFxMUQiEfr6+tDY2Mh5vMrKShQXFyMUCuHUqVOcxwOiZQVAtJeox+PhJSYhhBCy0CjxJMueRqPBvn37AACnT5/mpfby2WefhUAgQHNzM7q7uzmNBwBbtmxBZmYmvF4vzp8/z3k8QgghhAuUeJIVYffu3dDr9XA4HLz0vczIyMDWrVsBROe4c117ObGX6J07dzA6OsppPEIIIYQLlHiSFUEkEsX7Xl67do2X2suDBw9CoVDAZDLxUntZVFSE6upqmuNOCCFk2aLEk6wYlZWVKCkpQSgUWpQ57nzUXh47dgwikQg9PT00x50QQsiyQ4knWTFic9yVSiWKi4t5WRGMzXHnq/ZSp9Nh7969yMnJgVar5TweIYQQspAYdgnv1zkcDmi1Wtjtdmg0msW+HLJMhEIhiET8DeXq7e3FT37yEzAMg5dffhlZWVmcxguHwxAIBGAYhtM4hBBCSCpmk6/RiidZcSYmnXy8ryosLMTatWt5q70UCoWTks4l/N6REEIImYQST7IisSyL1tZW/OAHP4Db7eY83rFjxyAWi9Hb28tb7WULine truncated
|
||||
"text/plain": [
|
||||
"<Figure size 800x800 with 1 Axes>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"fig, ax = plt.subplots(figsize=(8, 8))\n",
|
||||
"plt.scatter(data.loc[:, \"x\"], data.loc[:, \"y\"], c=data.loc[:, \"source\"], s=30, cmap=plt.cm.Paired)\n",
|
||||
"\n",
|
||||
"# plot the decision function\n",
|
||||
"ax = plt.gca()\n",
|
||||
"xlim = ax.get_xlim()\n",
|
||||
"ylim = ax.get_ylim()\n",
|
||||
"\n",
|
||||
"# create grid to evaluate model\n",
|
||||
"xx = np.linspace(xlim[0], xlim[1], 30)\n",
|
||||
"yy = np.linspace(ylim[0], ylim[1], 30)\n",
|
||||
"YY, XX = np.meshgrid(yy, xx)\n",
|
||||
"xy = np.vstack([XX.ravel(), YY.ravel()]).T\n",
|
||||
"Z = clf.decision_function(xy).reshape(XX.shape)\n",
|
||||
"\n",
|
||||
"# plot decision boundary and margins\n",
|
||||
"ax.contour(\n",
|
||||
" XX, YY, Z, colors=\"k\", levels=[-1, 0, 1], alpha=0.5, linestyles=[\"--\", \"-\", \"--\"]\n",
|
||||
")\n",
|
||||
"# plot support vectors\n",
|
||||
"ax.scatter(\n",
|
||||
" clf.support_vectors_[:, 0],\n",
|
||||
" clf.support_vectors_[:, 1],\n",
|
||||
" s=100,\n",
|
||||
" linewidth=1,\n",
|
||||
" facecolors=\"none\",\n",
|
||||
" edgecolors=\"k\",\n",
|
||||
")\n",
|
||||
"plt.show()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "0ee97208",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Try the same using another kernel to see how the decision boundary changes:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"id": "d98ececa",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"/home/daniloefl/workspace/ml-lecture-oct-2023/env/lib/python3.9/site-packages/sklearn/base.py:465: UserWarning: X does not have valid feature names, but SVC was fitted with feature names\n",
|
||||
" warnings.warn(\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAp4AAAKUCAYAAACt7Tj1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAAEAAElEQVR4nOzdd3jb93Xo//cXG+AAwE2KQ5SovfewZcm25BWP1ImdWHbs2G6cpGna9P5un7a393bc3t40zXPT+t42aZM4HvFQYscjduIl27Jly9qStUVSHOKeIPbG9/cHRJgUSZCUSILjvJ5HlgWAwAEXDj6fzzlHUVVVRQghhBBCiHGmSXUAQgghhBBiZpDEUwghhBBCTAhJPIUQQgghxISQxFMIIYQQQkwISTyFEEIIIcSEkMRTCCGEEEJMCEk8hRBCCCHEhJDEUwghhBBCTAhdqgNIJhaL0dzcTEZGBoqipDocIYQQQghxGVVVcbvdFBUVodEkX9Oc1Ilnc3MzJSUlqQ5DCCGEEEIMo6GhgeLi4qS3GffEs6mpib/4i7/gzTffxOfzUVFRwZNPPsnatWuH/diMjAwg/kQyMzPHO1QhhBBCCDFKLpeLkpKSRN6WzLgmng6Hg2uuuYbrr7+eN998k9zcXKqqqrDb7SP6+N7t9czMTEk8hRBCCCEmsZEcixzXxPMHP/gBJSUlPPnkk4nLysvLx/MhhRBCCCHEJDWuVe2//e1vWbt2Lffccw95eXmsWrWKn/3sZ0PePhgM4nK5+v0RQgghhBDTw7gmnjU1NfzkJz9h3rx5vP3223z729/mT/7kT3j66acHvf33v/99rFZr4o8UFgkhhBBCTB+KqqrqeN25wWBg7dq17Nu3L3HZn/zJn3Do0CE+/fTTAbcPBoMEg8HEv3sPqzqdTjnjKYQQQggxCblcLqxW64jytXFd8SwsLGTx4sX9Llu0aBEXL14c9PZGozFRSCQFRUIIIYQQ08u4Jp7XXHMN58+f73dZZWUlZWVl4/mwQgghhBBiEhrXxPPP/uzP2L9/P//7f/9vqquref755/npT3/Kd77znfF8WCGEEEIIMQmNa+K5bt06XnnlFV544QWWLl3KP/zDP/Cv//qv3H///eP5sEIIIYQQYhIa1+KiqzWaw6pCCCGEEGLiTZriIiGEEEIIIXpJ4imEEEIIISaEJJ5CCCGEEGJCSOIphBBCCCEmhCSeQgghhBBiQkjiKYQQQgghJoQknkIIIYQQYkJI4imEEEIIISaELtUBCCGEEOLKRaNRPv30U5qbm1EUhdLSUtavX4+iKKkOTYgBJPEUQgghpiCHw8FPf/pT/uM//oO6urp+1y1cuJBvf/vbPPLII6Snp6cmQCEGIVvtQgghxBRTWVnJmjVr+Nu//Vu2bt3Kvn37cDgcdHd388EHH7B8+XL+v//v/2PTpk00NDSkOlwhEmRWuxBCCDGFNDU1sWHDBjIzM/n973/P7NmzicZUnIEwCmA169EoCmfOnOHWW2/FbDazb98+srKyUh26mKZGk6/JVrsQQggxhfz5n/85qqqye/duCgsLOd3q4ly7m1A0vo5k0mlYUpDJokWL2L17N+vXr+fv//7vefzxx1McuRCy1S6EEEJMGW1tbbz00kv81//6XykqKuJoUw8nWlyJpBMgEIlxpLGHM21u5s2bx7e//W2eeuopvF5vCiMXIk4STyGEEGKKeOqpp9DpdHz961/HE4xQ2TF0Mnmy1UUwEuWb3/wmbrebXbt2TWCkQgxOEk8hhBBiijh9+jSrV6/GbrdzscdHsoZJqgqNzgBlZWXMnz+fM2fOTFicQgxFEk8hhBBiiggGg5jNZgDC0eFrg8PRGAAmk4lAIDCusQkxEpJ4CiGEEFOE3W6nsbERVVWxmfVcnnqqqkooGMDd46CzpYn2+gscOHCA+vp66Q4jJgWpahdCCCEmsVgshtfrxev1snbtWv7zP/+TX/ziFxSXlIJ9DuiMAJw5/Cnnjx8kGomiAHqdQmdOOlVVVfT09HDdddcl7vPo0aMEg0E2btwoE47EhJLEUwghhJik3nvvPfbu3Zv4dywWw2q18vjjj3P33Xdzz4PzOeNWiMRUNBoN0UgUAKPRwLLSPHJsmbz66qvMnTuX9evXA9DZ2ckbb7xBLBajs7OT2267Da1Wm5LnJ2YeSTyFEEKIScpojK9mKoqCxWIhPT2du+66i2eeeYYbbriBInsmc0oyudDlJWPtGlavXE55fjYLCmwYdRp+8IMfcOLECX7961+TnZ0NQHZ2Njt27OCdd97hyJEjOBwO7rnnnsTZUSHGk0wuEkIIISaJlpYWurq6WLp0KQCBQIBoNIrZbEajiZdlqKrKd77zHX7yk5/w53/+5/zZn/0ZhYWF/e6nvr6eH/zgB/zkJz/hf/yP/8H//J//c8BjnT9/nt/85jeEQiFycnLYuXOnTDcSV2Q0+ZoknkIIIcQkcObMGV555RVisRhf//rXKSkpGfK2qqryj//4j3z/+98nFApx1113sXjxYlRV5fjx4/z+978nPT2d//W//hff/e53h7yf1tZWnn/+eVwuFxaLha985SuUlZWNx9MT05gknkIIIcQUoaoqH330ER988AEAFRUVfPnLX8ZkMg37sT09PTzzzDM8++yztLS0oCgKJSUlPPTQQ+zcuZP09PRh78PtdvPCCy/Q3NzMjTfeyJYtW676OYmZRRJPIYQQYgoIh8O8+uqrnD59GoCNGzdy0003JbbVJzKO48ePs3btWqlyF6M2mnxN+ngKIYQQKeByuXjyySc5ffo0Wq2WO++8k1tuuWXCk04AvV7PunXrEklnKBRi9+7dhMPhCY9FTG9S1S6EEEKkwKlTp2hubp6UZytfe+01Tp8+TU1NDffddx8ZGRmpDklME5J4CiGEECmwadMm/H5/Yvb6ZLJ+/Xpqa2tpbm7mZz/7GTt37qSgoCDVYYlpQLbahRBCiAmgqioHDhwgFAoB8d6cN95446RLOgHKysr4wz/8Q3JycnC5XPziF7/g/PnzqQ5LTAOSeAohhBDjLBQK8atf/Yo333yTV199lUlc15uQlZXFo48+ypw5cwiFQuzatYt9+/ZNidjF5CWJpxBCCDGOamtr+clPfsK5c+fQarUsWLBgylSOm81m7r//ftauXYuqquzduxeXy5XqsMQUJmc8hRBCiHEQDAZ59913OXz4MABWq5Uvf/nLSRvDT0ZarZYvfOEL5OXlUV5ejtVqTXVIYgqTxFMIIYQYYy0tLezatQun0wnAunXr2L59e2L2+lSjKArr16/vd1kkEkGnkzRCjI58xwghhBBjzGazEY1Gsdvt3HnnnZSXl6c6pDFVW1vLK6+8ItXuYtTkjKcQQggxBhobGxOFN2azmQceeIBvf/vb0y7pVFWVTz75BJfLxXPPPUdPT0+qQxJTiCSeQgghxFUIh8O88sor/PznP+fkyZOJywsKCjAYDCmMbHwoisKXv/xl8vLycLvdPPfcc/j9/lSHJaYISTyFEEKIKxQMBnnuuef47LPPUBRlxqz+mUwm7r//fjIzM+no6GDXrl1EIpFUhyWmAEk8hRBCiCsQCAT45S9/SV1dHUajkYceeojrrrsu1WFNGKvVyv3334/RaKS+vp7XXnst1SGJKUASTyGEEGKUfD4fTz/9NI2NjZjNZh588EFmz56d6rAmXH5+Pl/96lfRaDScPHmSpqamVIckJjmpahdCCCFGIRwO89RTT9He3k5aWhoPPvgg+fn5qQ4rZcrLy1m7di06nY7MzMxUhyMmOUk8hRBCiFHQ6/UsX76cgwcP8uCDD5KTk5PqkFLutttuS3UIYoqQxFMIIYQYpWuvvZY1a9ZgNptTHYoQU4qc8RRCCCGG0Vu5HQwGE5dJ0tmfqqrU1dWxa9cuvF5vqsMRk5QknkIIIUQSra2tPPnkk5w7d4533nkn1eFMau+++y7nzp3j4MGDqQ5FTFKSeAohhBBDaGxs5KmnnsLn81FUVMSNN96Y6pAmLUVRuOaaawA4ePAgoVAoxRGJyUgSTyGEEGIQdXV1PPPMMwQCAUpKSnjwwQexWCypDmtSW7hwIVlZWfj9fo4dO5bqcMQkJImnEEIIcZnq6mqee+45QqEQ5eXlfO1rX8NkMqU6rElPo9GwadMmAD799FNisViKIxKTjSSeQgghRB+RSIQ33niDcDjMvHnz2Llz57ScuT5eVq5cSVpaGj09PZw+fTrV4YhJRhJPIYQQog+dTsfOnTtZvXo1X/3qV9Hr9akOaUrR6/WsX78egE8++QRVVVMckZhMJPEUQgghAI/Hk/j/vLw87rzzTrRabQojmrrWrVtHQUEBa9eulcRT9CMN5IUQQsx4hw4d4p133uH++++fkTPXx5rFYuFb3/pWqsMQk5AknkIIIWYsl8vF+++/z/HjxwG4cOGCJJ5CjCNJPIUQQsw4wWCQffv2sW/fPsLhMADXXXcd119/fYojm15CoRAnT57E7/dz7bXXpjocMQlI4imEEGJGOX36NG+++WbiTGdpaSk33XQTxcXFKY5s+mlpaeH1119Hr9ezdu1aaUklJPEUQggxsyiKgsfjISsrix07drBw4UIURUl1WNNSaWkpubm5dHR0cOLEiUS1u5i5pKpdCCHEtBYKhWhubk78e9GiRXzpS1/iO9/5DosWLZKkcxwpisLatWsBOHz4sFS4C0k8hRBCTF+VlZX8+7//O8899xx+vx+IJ0PLli2TVkkTZMWKFej1etrb27l48WKqwxEpJomnEEKIacftdvPiiy/y/PPP43Q60ev1OJ3OVIc1I5lMJpYtWwbEVz3FzCZnPIUQQkwbqqpy5MgRdu/eTSAQQKPRsHHjRrZt2yZjL1No7dq1HD16lDNnznDLLbeQlpaW6pBEikjiKYQQYlqIRCI888wzie3coqIi7rzzTgoKClIcmSgqKqK4uBiz2UwgEJDEcwaTxFMIIcS0oNPpyMnLine truncated
|
||||
"text/plain": [
|
||||
"<Figure size 800x800 with 1 Axes>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"clf_kernel = svm.SVC(kernel=\"rbf\")\n",
|
||||
"clf_kernel.fit(data.loc[:, [\"x\", \"y\"]], data.loc[:, \"source\"])\n",
|
||||
"\n",
|
||||
"fig, ax = plt.subplots(figsize=(8, 8))\n",
|
||||
"plt.scatter(data.loc[:, \"x\"], data.loc[:, \"y\"], c=data.loc[:, \"source\"], s=30, cmap=plt.cm.Paired)\n",
|
||||
"\n",
|
||||
"# plot the decision function\n",
|
||||
"ax = plt.gca()\n",
|
||||
"xlim = ax.get_xlim()\n",
|
||||
"ylim = ax.get_ylim()\n",
|
||||
"\n",
|
||||
"# create grid to evaluate model\n",
|
||||
"xx = np.linspace(xlim[0], xlim[1], 30)\n",
|
||||
"yy = np.linspace(ylim[0], ylim[1], 30)\n",
|
||||
"YY, XX = np.meshgrid(yy, xx)\n",
|
||||
"xy = np.vstack([XX.ravel(), YY.ravel()]).T\n",
|
||||
"Z = clf_kernel.decision_function(xy).reshape(XX.shape)\n",
|
||||
"\n",
|
||||
"# plot decision boundary and margins\n",
|
||||
"ax.contour(\n",
|
||||
" XX, YY, Z, colors=\"k\", levels=[-1, 0, 1], alpha=0.5, linestyles=[\"--\", \"-\", \"--\"]\n",
|
||||
")\n",
|
||||
"# plot support vectors\n",
|
||||
"ax.scatter(\n",
|
||||
" clf_kernel.support_vectors_[:, 0],\n",
|
||||
" clf_kernel.support_vectors_[:, 1],\n",
|
||||
" s=100,\n",
|
||||
" linewidth=1,\n",
|
||||
" facecolors=\"none\",\n",
|
||||
" edgecolors=\"k\",\n",
|
||||
")\n",
|
||||
"plt.show()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "f0099362-0145-44f6-a777-50e09eba20c3",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Other kernels\n",
|
||||
"\n",
|
||||
"What if you try other kernels? How does this separation curve change?"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "ce1a3211-4119-42b2-9341-e93e2dbf584b",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.10.12"
|
||||
},
|
||||
"widgets": {
|
||||
"application/vnd.jupyter.widget-state+json": {
|
||||
"state": {},
|
||||
"version_major": 2,
|
||||
"version_minor": 0
|
||||
}
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
@@ -0,0 +1,777 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "6a764d92",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Gaussian Processes\n",
|
||||
"\n",
|
||||
"If one is ready to assume the data available is such that the variable being fit is Gaussian for any pair of samples taken, one can use fit the data and obtain a reliable uncertainty with a very small set of assumptions.\n",
|
||||
"\n",
|
||||
"For this example, we will use the dataset collected in Mauna Loa showing the amount of CO2 present in the atmosphere. This data may be downloaded from https://gml.noaa.gov/ccgg/trends/data.html and it should already be available in this repository.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"id": "bea9d764-7713-4d25-91bd-b2322960a30e",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"#!wget https://raw.githubusercontent.com/European-XFEL/ml-lecture-oct-2023/main/co2_weekly_mlo.txt"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "c8131588",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We can take a quick look at the file to see what is inside:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"id": "07ecc0d3",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"# --------------------------------------------------------------------\n",
|
||||
"# USE OF NOAA GML DATA\n",
|
||||
"# \n",
|
||||
"# These data are made freely available to the public and the\n",
|
||||
"# scientific community in the belief that their wide dissemination\n",
|
||||
"# will lead to greater understanding and new scientific insights.\n",
|
||||
"# The availability of these data does not constitute publication\n",
|
||||
"# of the data. NOAA relies on the ethics and integrity of the user to\n",
|
||||
"# ensure that GML receives fair credit for their work. If the data \n",
|
||||
"# are obtained for potential use in a publication or presentation, \n",
|
||||
"# GML should be informed at the outset of the nature of this work. \n",
|
||||
"# If the GML data are essential to the work, or if an important \n",
|
||||
"# result or conclusion depends on the GML data, co-authorship\n",
|
||||
"# may be appropriate. This should be discussed at an early stage in\n",
|
||||
"# the work. Manuscripts using the GML data should be sent to GML\n",
|
||||
"# for review before they are submitted for publication so we can\n",
|
||||
"# ensure that the quality and limitations of the data are accurately\n",
|
||||
"# represented.\n",
|
||||
"# \n",
|
||||
"# Contact: Pieter Tans (303 497 6678; pieter.tans@noaa.gov)\n",
|
||||
"# \n",
|
||||
"# File Creation: Fri Jan 21 05:00:12 2022\n",
|
||||
"# \n",
|
||||
"# RECIPROCITY\n",
|
||||
"# \n",
|
||||
"# Use of these data implies an agreement to reciprocate.\n",
|
||||
"# Laboratories making similar measurements agree to make their\n",
|
||||
"# own data available to the general public and to the scientific\n",
|
||||
"# community in an equally complete and easily accessible form.\n",
|
||||
"# Modelers are encouraged to make available to the community,\n",
|
||||
"# upon request, their own tools used in the interpretation\n",
|
||||
"# of the GML data, namely well documented model code, transport\n",
|
||||
"# fields, and additional information necessary for other\n",
|
||||
"# scientists to repeat the work and to run modified versions.\n",
|
||||
"# Model availability includes collaborative support for new\n",
|
||||
"# users of the models.\n",
|
||||
"# --------------------------------------------------------------------\n",
|
||||
"# \n",
|
||||
"# \n",
|
||||
"# See www.esrl.noaa.gov/gmd/ccgg/trends/ for additional details.\n",
|
||||
"# \n",
|
||||
"# NOTE: DATA FOR THE LAST SEVERAL MONTHS ARE PRELIMINARY, ARE STILL SUBJECT\n",
|
||||
"# TO QUALITY CONTROL PROCEDURES.\n",
|
||||
"# NOTE: The week \"1 yr ago\" is exactly 365 days ago, and thus does not run from\n",
|
||||
"# Sunday through Saturday. 365 also ignores the possibility of a leap year.\n",
|
||||
"# The week \"10 yr ago\" is exactly 10*365 days +3 days (for leap years) ago.\n",
|
||||
"# \n",
|
||||
"# Start of week CO2 molfrac (-999.99 = no data) increase\n",
|
||||
"# (yr, mon, day, decimal) (ppm) #days 1 yr ago 10 yr ago since 1800\n",
|
||||
" 1974 5 19 1974.3795 333.37 5 -999.99 -999.99 50.40\n",
|
||||
" 1974 5 26 1974.3986 332.95 6 -999.99 -999.99 50.06\n",
|
||||
" 1974 6 2 1974.4178 332.35 5 -999.99 -999.99 49.60\n",
|
||||
" 1974 6 9 1974.4370 332.20 7 -999.99 -999.99 49.65\n",
|
||||
" 1974 6 16 1974.4562 332.37 7 -999.99 -999.99 50.06\n",
|
||||
" 1974 6 23 1974.4753 331.73 5 -999.99 -999.99 49.72\n",
|
||||
" 1974 6 30 1974.4945 331.68 6 -999.99 -999.99 50.02\n",
|
||||
" 1974 7 7 1974.5137 331.46 6 -999.99 -999.99 50.20\n",
|
||||
" 1974 7 14 1974.5329 330.83 5 -999.99 -999.99 50.01\n",
|
||||
" 1974 7 21 1974.5521 330.76 7 -999.99 -999.99 50.41\n",
|
||||
" 1974 7 28 1974.5712 329.80 4 -999.99 -999.99 49.97\n",
|
||||
" 1974 8 4 1974.5904 329.85 5 -999.99 -999.99 50.54\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"with open(\"co2_weekly_mlo.txt\", \"r\") as f:\n",
|
||||
" for i, line in enumerate(f.readlines()):\n",
|
||||
" if i > 60: # let's print only the first 60 lines only\n",
|
||||
" break\n",
|
||||
" print(line[:-1])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "fce4d8e8",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We start by loading the necessary Python modules."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"id": "300cf8d3",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import matplotlib.pyplot as plt\n",
|
||||
"\n",
|
||||
"import pandas as pd\n",
|
||||
"import numpy as np\n",
|
||||
"from sklearn.gaussian_process import GaussianProcessRegressor\n",
|
||||
"from sklearn.gaussian_process.kernels import RBF, ExpSineSquared, WhiteKernel, RationalQuadratic"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "0ecd6a69",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Let us now load the data downloaded using pandas. If you are not familiar with Pandas, take a look at the documentation for a quick introduction at https://pandas.pydata.org/docs/user_guide/10min.html\n",
|
||||
"\n",
|
||||
"We can easily parse the data file using Pandas' read_csv function. It would require some coding to do it manually. As Pandas does not detect the column names, we have to specify it manually in the names parameter below."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"id": "4959a292",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"data = pd.read_csv(\"co2_weekly_mlo.txt\",\n",
|
||||
" delim_whitespace=True,\n",
|
||||
" comment='#',\n",
|
||||
" header=None,\n",
|
||||
" names=[\"year\", \"month\", \"day\", \"decimal\",\n",
|
||||
" \"co2\",\n",
|
||||
" \"ndays\",\n",
|
||||
" \"last_year\", \"last_decade\",\n",
|
||||
" \"increase\"])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "d8295e8a",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Let's print out the dataset read first."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"id": "024fb65a",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/html": [
|
||||
"<div>\n",
|
||||
"<style scoped>\n",
|
||||
" .dataframe tbody tr th:only-of-type {\n",
|
||||
" vertical-align: middle;\n",
|
||||
" }\n",
|
||||
"\n",
|
||||
" .dataframe tbody tr th {\n",
|
||||
" vertical-align: top;\n",
|
||||
" }\n",
|
||||
"\n",
|
||||
" .dataframe thead th {\n",
|
||||
" text-align: right;\n",
|
||||
" }\n",
|
||||
"</style>\n",
|
||||
"<table border=\"1\" class=\"dataframe\">\n",
|
||||
" <thead>\n",
|
||||
" <tr style=\"text-align: right;\">\n",
|
||||
" <th></th>\n",
|
||||
" <th>year</th>\n",
|
||||
" <th>month</th>\n",
|
||||
" <th>day</th>\n",
|
||||
" <th>decimal</th>\n",
|
||||
" <th>co2</th>\n",
|
||||
" <th>ndays</th>\n",
|
||||
" <th>last_year</th>\n",
|
||||
" <th>last_decade</th>\n",
|
||||
" <th>increase</th>\n",
|
||||
" </tr>\n",
|
||||
" </thead>\n",
|
||||
" <tbody>\n",
|
||||
" <tr>\n",
|
||||
" <th>0</th>\n",
|
||||
" <td>1974</td>\n",
|
||||
" <td>5</td>\n",
|
||||
" <td>19</td>\n",
|
||||
" <td>1974.3795</td>\n",
|
||||
" <td>333.37</td>\n",
|
||||
" <td>5</td>\n",
|
||||
" <td>-999.99</td>\n",
|
||||
" <td>-999.99</td>\n",
|
||||
" <td>50.40</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>1</th>\n",
|
||||
" <td>1974</td>\n",
|
||||
" <td>5</td>\n",
|
||||
" <td>26</td>\n",
|
||||
" <td>1974.3986</td>\n",
|
||||
" <td>332.95</td>\n",
|
||||
" <td>6</td>\n",
|
||||
" <td>-999.99</td>\n",
|
||||
" <td>-999.99</td>\n",
|
||||
" <td>50.06</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>2</th>\n",
|
||||
" <td>1974</td>\n",
|
||||
" <td>6</td>\n",
|
||||
" <td>2</td>\n",
|
||||
" <td>1974.4178</td>\n",
|
||||
" <td>332.35</td>\n",
|
||||
" <td>5</td>\n",
|
||||
" <td>-999.99</td>\n",
|
||||
" <td>-999.99</td>\n",
|
||||
" <td>49.60</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>3</th>\n",
|
||||
" <td>1974</td>\n",
|
||||
" <td>6</td>\n",
|
||||
" <td>9</td>\n",
|
||||
" <td>1974.4370</td>\n",
|
||||
" <td>332.20</td>\n",
|
||||
" <td>7</td>\n",
|
||||
" <td>-999.99</td>\n",
|
||||
" <td>-999.99</td>\n",
|
||||
" <td>49.65</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>4</th>\n",
|
||||
" <td>1974</td>\n",
|
||||
" <td>6</td>\n",
|
||||
" <td>16</td>\n",
|
||||
" <td>1974.4562</td>\n",
|
||||
" <td>332.37</td>\n",
|
||||
" <td>7</td>\n",
|
||||
" <td>-999.99</td>\n",
|
||||
" <td>-999.99</td>\n",
|
||||
" <td>50.06</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>...</th>\n",
|
||||
" <td>...</td>\n",
|
||||
" <td>...</td>\n",
|
||||
" <td>...</td>\n",
|
||||
" <td>...</td>\n",
|
||||
" <td>...</td>\n",
|
||||
" <td>...</td>\n",
|
||||
" <td>...</td>\n",
|
||||
" <td>...</td>\n",
|
||||
" <td>...</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>2482</th>\n",
|
||||
" <td>2021</td>\n",
|
||||
" <td>12</td>\n",
|
||||
" <td>12</td>\n",
|
||||
" <td>2021.9466</td>\n",
|
||||
" <td>416.45</td>\n",
|
||||
" <td>7</td>\n",
|
||||
" <td>414.00</td>\n",
|
||||
" <td>392.19</td>\n",
|
||||
" <td>137.16</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>2483</th>\n",
|
||||
" <td>2021</td>\n",
|
||||
" <td>12</td>\n",
|
||||
" <td>19</td>\n",
|
||||
" <td>2021.9658</td>\n",
|
||||
" <td>417.49</td>\n",
|
||||
" <td>6</td>\n",
|
||||
" <td>414.86</td>\n",
|
||||
" <td>392.54</td>\n",
|
||||
" <td>137.93</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>2484</th>\n",
|
||||
" <td>2021</td>\n",
|
||||
" <td>12</td>\n",
|
||||
" <td>26</td>\n",
|
||||
" <td>2021.9849</td>\n",
|
||||
" <td>417.46</td>\n",
|
||||
" <td>7</td>\n",
|
||||
" <td>415.32</td>\n",
|
||||
" <td>392.43</td>\n",
|
||||
" <td>137.66</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>2485</th>\n",
|
||||
" <td>2022</td>\n",
|
||||
" <td>1</td>\n",
|
||||
" <td>2</td>\n",
|
||||
" <td>2022.0041</td>\n",
|
||||
" <td>417.43</td>\n",
|
||||
" <td>6</td>\n",
|
||||
" <td>415.44</td>\n",
|
||||
" <td>393.18</td>\n",
|
||||
" <td>137.40</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>2486</th>\n",
|
||||
" <td>2022</td>\n",
|
||||
" <td>1</td>\n",
|
||||
" <td>9</td>\n",
|
||||
" <td>2022.0233</td>\n",
|
||||
" <td>418.33</td>\n",
|
||||
" <td>7</td>\n",
|
||||
" <td>415.06</td>\n",
|
||||
" <td>393.99</td>\n",
|
||||
" <td>138.10</td>\n",
|
||||
" </tr>\n",
|
||||
" </tbody>\n",
|
||||
"</table>\n",
|
||||
"<p>2487 rows × 9 columns</p>\n",
|
||||
"</div>"
|
||||
],
|
||||
"text/plain": [
|
||||
" year month day decimal co2 ndays last_year last_decade \\\n",
|
||||
"0 1974 5 19 1974.3795 333.37 5 -999.99 -999.99 \n",
|
||||
"1 1974 5 26 1974.3986 332.95 6 -999.99 -999.99 \n",
|
||||
"2 1974 6 2 1974.4178 332.35 5 -999.99 -999.99 \n",
|
||||
"3 1974 6 9 1974.4370 332.20 7 -999.99 -999.99 \n",
|
||||
"4 1974 6 16 1974.4562 332.37 7 -999.99 -999.99 \n",
|
||||
"... ... ... ... ... ... ... ... ... \n",
|
||||
"2482 2021 12 12 2021.9466 416.45 7 414.00 392.19 \n",
|
||||
"2483 2021 12 19 2021.9658 417.49 6 414.86 392.54 \n",
|
||||
"2484 2021 12 26 2021.9849 417.46 7 415.32 392.43 \n",
|
||||
"2485 2022 1 2 2022.0041 417.43 6 415.44 393.18 \n",
|
||||
"2486 2022 1 9 2022.0233 418.33 7 415.06 393.99 \n",
|
||||
"\n",
|
||||
" increase \n",
|
||||
"0 50.40 \n",
|
||||
"1 50.06 \n",
|
||||
"2 49.60 \n",
|
||||
"3 49.65 \n",
|
||||
"4 50.06 \n",
|
||||
"... ... \n",
|
||||
"2482 137.16 \n",
|
||||
"2483 137.93 \n",
|
||||
"2484 137.66 \n",
|
||||
"2485 137.40 \n",
|
||||
"2486 138.10 \n",
|
||||
"\n",
|
||||
"[2487 rows x 9 columns]"
|
||||
]
|
||||
},
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"data"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "1c178424",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We can plot this fairly easily using Matplotlib."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"id": "e63b38c5",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAtAAAAK/CAYAAABA9foTAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAAByMklEQVR4nO3dd3gU1eLG8XfTE1IIpFFC6L2DQkAUFBO9gKKCikoTERBUwIvIVQH1KioqogK2K3AVRFHxWhAITcFEei+hF4GEnkCA1Pn9kV/WrJsNGVI2JN/P8+QhO3N25syeZPNy9sw5FsMwDAEAAAAoEBdnVwAAAAC4nhCgAQAAABMI0AAAAIAJBGgAAADABAI0AAAAYAIBGgAAADCBAA0AAACYQIAGAAAATCBAAwAAACYQoAEAAAATCNAAAACACQRoAEVi4sSJslgsOn36dImcb9asWbJYLDp06FCJnA9lkzN/jvgZBq5fBGigmEyfPl0Wi0Xt2rXLc39sbKwmTpyo8+fPl2zF4ND+/fs1ZMgQ1a5dW15eXvL391fHjh01depUXb582absjh079Mgjj6hatWry9PRU1apV9fDDD2vHjh1Oqv31rbh/H8rL71tOKLdYLFq9erXdfsMwFB4eLovFou7duzuhhkUv55rXr1/v7KqgHCFAA8Vkzpw5qlmzptauXat9+/bZ7Y+NjdVLL71U5v+gF5e+ffvq8uXLioiIKJLj/fzzz2rWrJm+/vpr9ejRQ++//74mTZqkGjVqaMyYMXr66aetZb/77ju1bt1ay5Yt08CBAzV9+nQNGjRIK1asUOvWrbVgwYIiqVN5Uty/D46OX9Q/R2YU57m9vLw0d+5cu+2//vqr/vzzT3l6ehb5OYHyxM3ZFQDKooMHDyo2NlbfffedhgwZojlz5mjChAnOrlaZkJKSogoVKsjV1VWurq5FcsyDBw/qwQcfVEREhJYvX64qVapY9w0fPlz79u3Tzz//LCm7l7pv376qXbu2fvvtNwUHB1vLPv300+rUqZP69u2rrVu3qnbt2kVSP9jL+TkorKL8OSpN5/7HP/6h+fPn67333pOb219/6ufOnas2bdqU2FAroKyiBxooBnPmzFFgYKC6deumXr16ac6cOTb7J06cqDFjxkiSatWqZf3I9dChQ9axxHv27NEjjzyigIAABQcH68UXX5RhGDp69Kjuvvtu+fv7KywsTG+//Xaeddi0aZPuvPNO+fv7y9fXV7fddpv++OMPmzIXLlzQyJEjVbNmTXl6eiokJES33367Nm7caK2nxWLR7t27df/998vf31+VK1fW008/rStXruR53vPnz2vAgAGqWLGiAgICNHDgQF26dMmu3LFjx/Too48qNDRUnp6eatKkiT777DO718lisWjnzp166KGHFBgYqJtuuklS3uNHjx07pkGDBqlq1ary9PRUrVq1NGzYMKWlpeXTWtKbb76pixcv6j//+Y9NeM5Rt25daw/05MmTdenSJX388cc24VmSgoKC9NFHHyklJUVvvvlmvucsqIJcU0HaOue13LdvX4Hb52rnNdOG+Z03v9+H3MfI6+fg8OHDeuKJJ9SgQQN5e3urcuXK6t27t83PRX7HdzQOuThe07/L69yFPWaOPn366MyZM4qJibFuS0tL0zfffKOHHnrIrnxBXkdJGjBggGrWrGn3/Jx657XtatdS0HMXVkHatKTqgusfPdBAMZgzZ47uvfdeeXh4qE+fPpoxY4bWrVunG264QZJ07733as+ePfryyy81ZcoUBQUFSZJNIHvggQfUqFEjvf766/r555/173//W5UqVdJHH32kW2+9VW+88YbmzJmjf/7zn7rhhht08803W5+7Y8cOderUSf7+/nr22Wfl7u6ujz76SJ07d9avv/5qHZc9dOhQffPNNxoxYoQaN26sM2fOaPXq1dq1a5dat25tPd7999+vmjVratKkSfrjjz/03nvv6dy5c/rvf/9rd+3333+/atWqpUmTJmnjxo369NNPFRISojfeeMNaJjExUe3bt5fFYtGIESMUHBysX375RYMGDVJycrJGjhxpc8zevXurXr16eu2112QYRp6v+fHjx3XjjTfq/Pnzevzxx9WwYUMdO3ZM33zzjS5duiQPDw+H7fXjjz+qdu3a6tChg8MyucvWrFlTnTp1ynP/zTffrJo1a1p7rAujINdU0LbOUZD2Kch5zbZhfuctyO+DlPfPwbp16xQbG6sHH3xQ1atX16FDhzRjxgx17txZO3fulI+PT4GPn6M4XlOzCnvMmjVrKjIyUl9++aXuvPNOSdIvv/yipKQkPfjgg3rvvfdsyhfkdSyuaynOc+coaJuWRF1QRhgAitT69esNSUZMTIxhGIaRlZVlVK9e3Xj66adtyk2ePNmQZBw8eNBm+4QJEwxJxuOPP27dlpGRYVSvXt2wWCzG66+/bt1+7tw5w9vb2+jfv7/NMXr27Gl4eHgY+/fvt247fvy44efnZ9x8883WbQEBAcbw4cMdXktOXe666y6b7U888YQhydiyZYtd2UcffdSm7D333GNUrlzZZtugQYOMKlWqGKdPn7bZ/uCDDxoBAQHGpUuXbI7Zp08fu7rNnDnT5vXr16+f4eLiYqxbt86ubFZWlsNrTEpKMiQZd999t8MyOc6fP1+gsnfddZchyUhOTr7qMfNTkGsqaFubaZ+CnNdsG17tvI5+H3IfI6+fg5zz5BYXF2dIMv773/9e9fh//zkyjOJ5TfOS17mL6pjr1q0zPvjgA8PPz8/6GvXu3dvo0qWLYRiGERERYXTr1s36vIK+jv379zciIiLsyubUO69tV7uWgp67INfsSEHbtLB1QfnBEA6giM2ZM0ehoaHq0qWLJMliseiBBx7QvHnzlJmZWeDjPPbYY9bvXV1d1bZtWxmGoUGDBlm3V6xYUQ0aNNCBAwes2zIzM7VkyRL17NnTZgxulSpV9NBDD2n16tVKTk62Pn/NmjU6fvx4vnUZPny4zeMnn3xSkrRw4UK7skOHDrV53KlTJ505c8Z6TsMw9O2336pHjx4yDEOnT5+2fkVHRyspKck6hMTRMf8uKytL33//vXr06KG2bdva7f/7R8u55dTLz88v33NI2UNeClI2Z3/Osa9FQa7JTFvnuFr7FOS8RdGGfz9vQeT1c+Dt7W39Pj09XWfOnFHdunVVsWJFuzoURHG8pteiKI55//336/Lly/rpp5904cIF/fTTT3kO35CK/nXM7WrXUpznlsy1aXHXBWUHARooQpmZmZo3b566dOmigwcPat++fdq3b5/atWunxMRELVu2rMDHqlGjhs3jgIAAeXl5WT9+zr393Llz1senTp3SpUuX1KBBA7tjNmrUSFlZWTp69Kik7LG/27dvV3h4uG688UZNnDjRJoznqFevns3jOnXqyMXFJc9xgX+vd2BgoCRZ63jq1CmdP3/eOoY499fAgQMlSSdPnrQ5Rq1atezOk9upU6eUnJyspk2b5lsuL/7+/pL+Csf5yQnGVyubO2inpqbq0UcfVY0aNeTv76/27dsrLi7uqucqyDWZaescBWmfgpzXbBte7bwFkdfPweXLlzV+/HiFh4fL09NTQUFBCg4O1vnz55WUlFTgY+cojtf0WhTFMYODg9W1a1fNnTtX3333nTIzM9WrV688yxb165jb1a6lOM8tmWvT4q4Lyg7GQANFaPny5Tpx4oTmzZunefPm2e2fM2eOoqKiCnSsvO7Od3THvuFgXPDV3H///erUqZMWLFigJUuWaPLkyXrjjTf03XffWcdN5iW/Ht2r1TErK0uS9Mgjj6h///55lm3evLnN49y9QkXN399fVatW1fbt269aNiAgQFWqVNHWrVvzLbd161ZVq1ZN/v7+SklJUc2aNbV69WpVr17dOk3eoUOH5OvrW1SXUWBF8TN0LW1YFOfN6+fgySef1MyZMzVy5EhFRkYqICBAFotFDz74oLWexa2ofy+L8pgPPfSQBg8erISEBN15552qWLFinuUK+jo6+t3P79O1q11LaWjDHKWpLijdCNBAEZozZ45CQkI0bdo0u33fffedFixYoA8//FDe3t75htDCCA4Olo+Pj+Lj4+327d69Wy4uLgoPD7duq1Klip544gk98cQTOnnypFq3bq1XX33VJkDv3bvXpvdv3759ysrKyvNu/ILUz8/PT5mZmeratavp5zs6pr+/f4FCcF66d++ujz/+WHFxcYqMjLxq2U8++USrV6+2zgSR26pVq3To0CENGTJEklShQgWNHz/euv/BBx/U6NGjFR8frzZt2hTqmsy2dUEU9LxF3YbX+vvwzTffqH///jaz0Vy5csVuvueCHr84XlNnuueeezRkyBD98ccf+uqrrxyWK+jrGBgYmOdc3YcPH77mOhb03NfKTJsWd11QdjCEAygily9f1nfffafu3burV69edl8jRozQhQsX9MMPP0iSdQ7bon5jdnV1VVRUlP73v//ZDLFITEzU3LlzddNNN8nf31+ZmZl2H0mGhISoatWqSk1Ntdn+9/8QvP/++5KUby91fvW777779O233+YZ0k6dOmX6mC4uLurZs6d+/PHHPFcju1qv3bPPPqsKFSroscceU2Jiot3+/fv3a+rUqZKkMWPGyNvbW0OGDNGZM2dsyp09e1ZDhw6Vj4+Pddq0v9u7d6/Onj2runXrFvqaCtrWZhT0vEXdhtf6++Dq6mrXvu+//75dj2hBj18cr6kz+fr6asaMGZo4caJ69OjhsFxBX8c6deooKSnJ5lOYEydOFGrxoIKeuzDHL2ibFnddUHbQAw0UkR9++EEXLlzQXXfdlef+9u3bKzg4WHPmzNEDDzxg7X18/vnn9eCDD8rd3T3fP3Bm/Pvf/1ZMTIxuuukmPfHEE3Jzc9NHH32k1NRU6/zEFy5cUPXq1dWrVy+1aNFCvr6+Wrp0qdatW2c3t/TBgwd111136Y477lBcXJyLine truncated
|
||||
"text/plain": [
|
||||
"<Figure size 800x800 with 1 Axes>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"fig, ax = plt.subplots(figsize=(8, 8))\n",
|
||||
"data.plot(x=\"decimal\", y=\"co2\", legend=False, ax=ax)\n",
|
||||
"ax.set(xlabel=\"Time [years]\", ylabel=r\"CO$_2$ [ppm]\", title=\"Atmospheric CO$_2$ concentration in Mauna Loa\")\n",
|
||||
"plt.show()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "efd249fe",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"The spikes are related to missing data. We can clean those up and plot again."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"id": "0837b3ff",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"data = data[data.co2 > -999]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"id": "333581b5",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"image/png": "iVBORw0KGgoAAAANSUhEUgAAArsAAAK/CAYAAABp8gWUAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAADHGUlEQVR4nOzdd3gc5fU24Gf7SiutenGRe8cFbINx6JhuSggldEgI1RDKF5LwI4WQAiEJCb0kBFIMJBAgQOi9uGBswL13q9eVtpf5/ph5Z2d2Z1YrW7JX0nNfly9mdt+dGVkuh+PznmORJEkCEREREdEAZD3QD0BERERE1FcY7BIRERHRgMVgl4iIiIgGLAa7RERERDRgMdglIiIiogGLwS4RERERDVgMdomIiIhowGKwS0REREQDFoNdIiIiIhqwGOwSERER0YDFYJeIiIiIBiwGu0SD1J133gmLxYLm5ub9cr+nn34aFosF27dv3y/3o4HpQP464q9hov6JwS6R4pFHHoHFYsGcOXMM31+0aBHuvPNOtLe3798HI1NbtmzBNddcgzFjxsDtdsPr9eKII47A/fffj2AwqFu7Zs0aXHLJJRg2bBhcLheGDh2Kiy++GGvWrDlAT9+/9fXvh8Hy+00E0BaLBZ9++mna+5IkoaamBhaLBaeffvoBeMLeJ77mL7744kA/Cg0SDHaJFAsXLsSoUaPw+eefY/PmzWnvL1q0CL/4xS8G/F++feXSSy9FMBjEyJEje+V6//vf/zBt2jT8+9//xhlnnIEHH3wQd999N0aMGIHbbrsNN910k7r2xRdfxMyZM/Hee+/hO9/5Dh555BFceeWV+OCDDzBz5ky89NJLvfJMg0lf/34wu35v/zrqib68t9vtxjPPPJP2+kcffYTdu3fD5XL1+j2JBgv7gX4Aolywbds2LFq0CC+++CKuueYaLFy4ED//+c8P9GMNCH6/Hx6PBzabDTabrVeuuW3bNlxwwQUYOXIk3n//fQwZMkR9b8GCBdi8eTP+97//AZCzv5deeinGjBmDjz/+GBUVFeram266CUcddRQuvfRSrFy5EmPGjOmV56N04tfBvurNX0e5dO/TTjsNzz//PB544AHY7cm/mp955hnMmjVrv5UbEQ1EzOwSQc7qlpSUYP78+Tj33HOxcOFC3ft33nknbrvtNgDA6NGj1X923L59u1r7unHjRlxyySUoKipCRUUFfvrTn0KSJOzatQtnnXUWvF4vqqur8Yc//MHwGb788kuceuqp8Hq9KCgowLx587BkyRLdms7OTtx8880YNWoUXC4XKisrceKJJ2LFihXqc1osFqxfvx7nn38+vF4vysrKcNNNNyEUChnet729HVdccQWKi4tRVFSE73znOwgEAmnr9uzZg+9+97uoqqqCy+XCQQcdhL/+9a9pP08WiwVr167FRRddhJKSEhx55JEAjOsd9+zZgyuvvBJDhw6Fy+XC6NGjcd111yESiWT4bgH33nsvurq68OSTT+oCXWHcuHFqZvd3v/sdAoEAnnjiCV2gCwDl5eV4/PHH4ff7ce+992a8Z7ay+Zqy+V6Ln8vNmzdn/f3p7r49+R5mum+m3w/aaxj9OtixYweuv/56TJw4EXl5eSgrK8N5552n+3WR6fpmdbN98XOayuje+3pN4cILL0RLSwveeecd9bVIJIIXXngBF110Udr6bH4eAeCKK67AqFGj0j4vntvote6+lmzvva+y+Z7ur2eh/o2ZXSLIwe63vvUtOJ1OXHjhhXj00UexbNkyHHrooQCAb33rW9i4cSOeffZZ/PGPf0R5eTkA6IKnb3/725g8eTLuuece/O9//8OvfvUrlJaW4vHHH8fxxx+P3/72t1i4cCF+8IMf4NBDD8XRRx+tfnbNmjU46qij4PV68cMf/hAOhwOPP/44jj32WHz00UdqHfG1116LF154ATfccAOmTJmClpYWfPrpp1i3bh1mzpypXu/888/HqFGjcPfdd2PJkiV44IEH0NbWhr///e9pX/v555+P0aNH4+6778aKFSvwl7/8BZWVlfjtb3+rrmloaMDhhx8Oi8WCG264ARUVFXjjjTdw5ZVXwufz4eabb9Zd87zzzsP48ePxm9/8BpIkGf6c19bW4rDDDkN7ezuuvvpqTJo0CXv27MELL7yAQCAAp9Np+v169dVXMWbMGHzjG98wXaNdO2rUKBx11FGG7x999NEYNWqUmgneF9l8Tdl+r4Vsvj/Z3Len38NM983m9wNg/Otg2bJlWLRoES644AIMHz4c27dvx6OPPopjjz0Wa9euRX5+ftbXF/ri57Sn9vWao0aNwty5c/Hss8/i1FNPBQC88cYb6OjowAUXXIAHHnhAtz6bn8e++lr68t5Ctt/T/fEsNABIRIPcF198IQGQ3nnnHUmSJCmRSEjDhw+XbrrpJt263/3udxIAadu2bbrXf/7zn0sApKuvvlp9LRaLScOHD5csFot0zz33qK+3tbVJeXl50uWXX667xje/+U3J6XRKW7ZsUV+rra2VCgsLpaOPPlp9raioSFqwYIHp1yKe5cwzz9S9fv3110sApK+//jpt7Xe/+13d2rPPPlsqKyvTvXbllVdKQ4YMkZqbm3WvX3DBBVJRUZEUCAR017zwwgvTnu2pp57S/fxddtllktVqlZYtW5a2NpFImH6NHR0dEgDprLPOMl0jtLe3Z7X2zDPPlABIPp+v22tmks3XlO33uiffn2zu29PvYXf3Nfv9oL2G0a8DcR+txYsXSwCkv//9791eP/XXkST1zc+pEaN799Y1ly1bJj300ENSYWGh+nN03nnnSccdd5wkSZI0cuRIaf78+ernsv15vPzyy6WRI0emrRXPbfRad19LtvfO5ms2k+33dF+fhQYHljHQoLdw4UJUVVXhuOOOAwBYLBZ8+9vfxnPPPYd4PJ71db73ve+pxzabDbNnz4YkSbjyyivV14uLizFx4kRs3bpVfS0ej+Ptt9/GN7/5TV3N6JAhQ3DRRRfh008/hc/nUz+/dOlS1NbWZnyWBQsW6M5vvPFGAMDrr7+etvbaa6/VnR911FFoaWlR7ylJEv7zn//gjDPOgCRJaG5uVn+cfPLJ6OjoUMsozK6ZKpFI4OWXX8YZZ5yB2bNnp72f+s+rWuK5CgsLM94DkMs+slkr3hfX3hvZfE09+V4L3X1/srlvb3wPU++bDaNfB3l5eepxNBpFS0sLxo0bh+Li4rRnyEZf/Jzujd645vnnn49gMIjXXnsNnZ2deO211wxLGIDe/3nU6u5r6ct7Az37nvb1s9DAwGCXBrV4PI7nnnsOxx13HLZt24bNmzdj8+bNmDNnDhoaGvDee+9lfa0RI0bozouKiuB2u9V/gtW+3tbWpp43NTUhEAhg4sSJadecPHkyEokEdu3aBUCuVV29ejVqampw2GGH4c4779QFzsL48eN152PHjoXVajWsY0t97pKSEgBQn7GpqQnt7e1qzav2x3e+8x0AQGNjo+4ao0ePTruPVlNTE3w+H6ZOnZpxnRGv1wsgGchmIoLY7tZqg+JwOIzvfve7GDFiBLxeLw4//HAsXry423tl8zX15HstZPP9yea+Pf0ednffbBj9OggGg/jZz36GmpoauFwulJeXo6KiAu3t7ejo6Mj62kJf/Jzujd64ZkVFBU444QQ888wzePHFFxGPx3Huuecaru3tn0et7r6Wvrw30LPvaV8/Cw0MrNmlQe39999HXV0dnnvuOTz33HNp7y9cuBAnnXRSVtcy2qVttnNbMqlj7c7555+Po446Ci+99BLefvtt/O53v8Nvf/tbvPjii2qdn5FMmdLunjGRSAAALrnkElx++eWGa6dPn64712ZbepvX68XQoUOxevXqbtcWFRVhyJAhWLlyZcZ1K1euxLBhw+D1euH3+zFq1Ch8+umnGD58uNrabPv27SgoKOitLyNrvfFraG++h71xX6NfBzfeeCOeeuop3HzzzZg7dy6KiopgsVhwwQUXqM/Z13r792VvXvOiiy7CVVddhfr6epx66qkoLi42XJftz6PZ7/1M/2rV3deSC99DIZeehXIXg10a1BYuXIjKyko8/PDDae+9+OKLeOmll/DYY48hLy8vY8C4LyoqKpCfn48NGzakvbd+/XpYrVbU1NSorw0ZMgTXX389rr/+ejQ2NmLmzJn49a9/rQt2N23apMuqbd68GYlEwnBXdjbPV1hYiHg8jhNOOKHHnze7ptfrzSpgNXL66afjiSeewOLFizF37txu1/75z3/Gp59+qnYE0Prkk0+wfft2XHPNNQAAj8eDn/3sZ+r7F1xwAW699VZs2LABs2bN2qevqaff62xke9/e/h7u7e+HF154AZdffrmuK0koFErrp5vt9fvi5/RAOvvss3HNNddgyZIl+Ne//mW6Ltufx5KSEsNeyDt27NjrZ8z23nurJ9/Tvn4WGhhYxkCDVjAYxIsvvojTTz8d5557btqPG264AZ2dnXjllVcAQO0R2tt/iNpsNpx00kn473//qyszaGhowDPPPIMjjzwSXq8X8Xg87Z/lKisrMXToUITDYd3rqcH7gw8+CAAZs7+Znu+cc87Bf/7zH8OAqqmpqcfXtFqt+OY3v4lXX33VcIpSd9mwH/7wh/B4PPje976HhoaGtPe3bNmC+++/HwBw2223IS8vD9dccw1aWlp061pbW3HttdciPz9fbXWVatOmTWhtbcW4ceP2+WvK9nvdE9net7e/h3v7+8Fms6V9fx988MG0TGO21++Ln9MDqaCgAI8++ijuvPNOnHHGGabrsv15HDt2LDo6OnT/ulFXV7dPg1Syvfe+XD/b72lfPwsNDMzs0qD1yiuvoLOzE2eeeabh+4cffjgqKiqwcOFCfPvb31azenfccQcuuOACOByOjH8Z9cSvfvUrvPPOOzjyyCNx/fXXw2634/HHH0c4HFb7v3Z2dmL48OE499xzMWPGDBQLine truncated
|
||||
"text/plain": [
|
||||
"<Figure size 800x800 with 1 Axes>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"fig, ax = plt.subplots(figsize=(8, 8))\n",
|
||||
"data.plot(x=\"decimal\", y=\"co2\", legend=False, ax=ax)\n",
|
||||
"ax.set(xlabel=\"Time [years]\", ylabel=r\"CO$_2$ [ppm]\", title=\"Atmospheric CO$_2$ concentration in Mauna Loa\")\n",
|
||||
"plt.show()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "c62fd7f3",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"That's better.\n",
|
||||
"\n",
|
||||
"We can now try to model this behaviour with Gaussian Processes. The key parameter we must choose is the covariance matrix analytic form, which establishes how correlated two points in the time axis are.\n",
|
||||
"\n",
|
||||
"One simple assumption would be that two values of the CO2 concentration are highly correlated they happened at close by times. We use a Gaussian-based model (named here \"Radial Basis Function\", as this does not refer to a probability distribution) to establish that if two points are one year apart they are likely to be highly correlated.\n",
|
||||
"\n",
|
||||
"Additionally, we assume that there is a correlation between points within certain period, since we know that data from the same season in an year is expected to be correlated to data if that same season in the year (ie: CO$_2$ emissions have a rough periodicity of one year due to the season changes).\n",
|
||||
"\n",
|
||||
"Another term of the covariance matrix must be related to the random noise appearing in the data.\n",
|
||||
"\n",
|
||||
"All those assumptions can be motivated by an analysis of the covariance matrix of the data. That is, we can take the covariance matrix of all the data points and analyse it to establish whether our assumptions here are valid or not. A very long and detailed explanation on this point can be found in Bishop (2006), which has a very pedagogical chapter on Gaussian Processes. Online resources can be found on: http://www.gaussianprocess.org/\n",
|
||||
"\n",
|
||||
"Note that all of those are very strong assumptions, but they are made directly on the covariance matrix of the fit function. The actual numbers used below are taken from an example, but they will be optimized in the fit below (unless explicitly asked not to).\n",
|
||||
"\n",
|
||||
"Note: this notebook is a simplified example with extra explanations, with lots of material taken from https://scikit-learn.org/stable/auto_examples/gaussian_process/plot_gpr_co2.html -- take a look at the original example for more details.\n",
|
||||
"\n",
|
||||
"We also assume the data within an year from each other is highly correlated and should show a periodicity of 1 year."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"id": "a31d7eca",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Kernel with parameters given in GPML book\n",
|
||||
"\n",
|
||||
"# these hyper parameters are optimised in the GP fit\n",
|
||||
"# we provide here only a starting value\n",
|
||||
"\n",
|
||||
"# Kernel with optimized initial parameters\n",
|
||||
"k1 = 50.0 ** 2 * RBF(length_scale=50.0) # long term smooth rising trend\n",
|
||||
"k2 = (\n",
|
||||
" 2.0 ** 2\n",
|
||||
" * RBF(length_scale=100.0)\n",
|
||||
" * ExpSineSquared(length_scale=1.0, periodicity=1.0, periodicity_bounds=\"fixed\")\n",
|
||||
") # seasonal component\n",
|
||||
"# medium term irregularities\n",
|
||||
"k3 = 0.5 ** 2 * RationalQuadratic(length_scale=1.0, alpha=1.0)\n",
|
||||
"k4 = 0.1 ** 2 * RBF(length_scale=0.1) + WhiteKernel(\n",
|
||||
" noise_level=0.1 ** 2, noise_level_bounds=(1e-5, np.inf)\n",
|
||||
") # noise terms\n",
|
||||
"kernel = k1 + k2 + k3 + k4\n",
|
||||
"\n",
|
||||
"# simpler kernel: RBF kernel with white noise on likelihood\n",
|
||||
"# what happens if you use this instead?\n",
|
||||
"#kernel = (10.0 ** 2 * RBF(length_scale=10.0)\n",
|
||||
"# + WhiteKernel(noise_level=1**2, noise_level_bounds=(1e-5, np.inf))\n",
|
||||
"# )"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "0ad8ddf0",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"The kernel hyper-parameters are:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 9,
|
||||
"id": "032ada0d",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"50**2 * RBF(length_scale=50) + 2**2 * RBF(length_scale=100) * ExpSineSquared(length_scale=1, periodicity=1) + 0.5**2 * RationalQuadratic(alpha=1, length_scale=1) + 0.1**2 * RBF(length_scale=0.1) + WhiteKernel(noise_level=0.01)\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"print(kernel)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "1956bd05-4658-4c41-931c-8cb003534eb4",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We can now fit the data that we have. But let us not fit the full data: we can fit data using only information until 2014 to use the period between 2014 and now as an estimate of the fit quality."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"id": "22046fb8-135c-4048-a0c5-ceb1e8190049",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"fit_data = data.loc[data.year <= 2014, :]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 11,
|
||||
"id": "4412a9a7",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"gp = GaussianProcessRegressor(kernel=kernel, normalize_y=True)\n",
|
||||
"time = fit_data.decimal.to_numpy()[:, np.newaxis]\n",
|
||||
"co2 = fit_data.co2.to_numpy()[:, np.newaxis]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 14,
|
||||
"id": "88bcd2c1-c9ae-4f89-ba9a-96d731907138",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"gp = gp.fit(time, co2)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "eb69bb80",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"These are the hyper-parameters after the fit and the log-likelihood given the data:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 15,
|
||||
"id": "667db862",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"GPML kernel: 3.27**2 * RBF(length_scale=37.8) + 0.175**2 * RBF(length_scale=290) * ExpSineSquared(length_scale=1.59, periodicity=1) + 0.322**2 * RationalQuadratic(alpha=0.00131, length_scale=6.52) + 0.0121**2 * RBF(length_scale=0.02) + WhiteKernel(noise_level=0.000209)\n",
|
||||
"Log-marginal-likelihood: 5275.122\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"print(\"GPML kernel: %s\" % gp.kernel_)\n",
|
||||
"print(\"Log-marginal-likelihood: %.3f\" % gp.log_marginal_likelihood(gp.kernel_.theta))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "4968439e",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Now we can use the fit model to predict how the CO2 concentration will evolve at other times."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 16,
|
||||
"id": "ef6b7762",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"new_time = np.linspace(time.min(), time.max()+20, 1000)[:, np.newaxis]\n",
|
||||
"co2_pred, co2_std = gp.predict(new_time, return_std=True)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "7c471fa1-8221-4d0a-bcb8-bc2515487c7b",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"First let's plot the prediction with uncertainty only:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 17,
|
||||
"id": "262c8e85-6f9c-447f-9e50-a356267e0597",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"image/png": "iVBORw0KGgoAAAANSUhEUgAAArsAAAK/CAYAAABp8gWUAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAADap0lEQVR4nOzdd5hV1dk28PtMr2d6oRfpVRFRbLGCiiWW2FDQ8FoSNGq+mLy+aWpiMCbGxBo1xhQxJLbE2FFjC4hILwKCwAxM772f74+HNWutvfcZZmCGOTNz/65rrrPLKfswUe88PPtZvkAgEAARERERUT8U1tsXQERERETUUxh2iYiIiKjfYtglIiIion6LYZeIiIiI+i2GXSIiIiLqtxh2iYiIiKjfYtglIiIion6LYZeIiIiI+i2GXSIiIiLqtxh2iYiIiKjfYtglIiIion6LYZdogLr77rvh8/lQUlJyRD7vT3/6E3w+H/bs2XNEPo/6p9783xH/N0zUNzHsEh3w+OOPw+fz4fjjj/c8v2LFCtx9992oqKg4shdGQe3atQs33XQTRo8ejZiYGPj9fpx00kn43e9+h/r6euu5W7ZswTXXXIMhQ4YgOjoagwcPxvz587Fly5Zeuvq+raf/eRgo/7ypAO3z+fDJJ5+4zgcCAQwbNgw+nw/nn39+L1xh91Pf+fPPP+/tS6EBgmGX6IClS5di5MiR+Oyzz7Bz507X+RUrVuCee+7p9//x7SnXXnst6uvrMWLEiG55v9dffx1Tp07FP/7xD1xwwQV45JFHsGTJEgwfPhx33nknbrvttvbnvvzyy5gxYwbee+89XH/99Xj88cexaNEi/Oc//8GMGTPwyiuvdMs1DSQ9/c9DsPfv7v8ddUVPfnZMTAyef/551/EPP/wQ+/btQ3R0dLd/JtFAEdHbF0AUCnbv3o0VK1bg5Zdfxk033YSlS5fipz/9aW9fVr9QW1uL+Ph4hIeHIzw8vFvec/fu3bjyyisxYsQIvP/++xg0aFD7ucWLF2Pnzp14/fXXAUj199prr8Xo0aPx0UcfISMjo/25t912G0455RRce+212LhxI0aPHt0t10du6n8Hh6s7/3cUSp993nnn4YUXXsDDDz+MiAj9n+bnn38exx577BFrNyLqj1jZJYJUdVNSUjBv3jxcdtllWLp0qXX+7rvvxp133gkAGDVqVPtfO+7Zs6e993XHjh245pprkJSUhIyMDPz4xz9GIBBAbm4uLrroIvj9fmRnZ+PBBx/0vIZ169bh3HPPhd/vR0JCAs4880x8+umn1nOqq6tx++23Y+TIkYiOjkZmZibOPvtsrF27tv06fT4ftm3bhssvvxx+vx9paWm47bbb0NDQ4Pm5FRUVuO6665CcnIykpCRcf/31qKurcz1v//79+OY3v4msrCxER0dj8uTJ+OMf/+j6c/L5fNi6dSuuvvpqpKSk4OSTTwbg3e+4f/9+LFq0CIMHD0Z0dDRGjRqFb33rW2hqaurgtwU88MADqKmpwTPPPGMFXWXMmDHtld1f/epXqKurw1NPPWUFXQBIT0/Hk08+idraWjzwwAMdfmZndeY7deZ3rf4sd+7c2enfz8E+tyu/w44+t6N/Hsz38Prfwd69e/Htb38b48ePR2xsLNLS0vCNb3zD+t9FR+8frG+2J/5Mnbw++3DfU7nqqqtQWlqK5cuXtx9ramrCiy++iKuvvtr1/M78OQLAddddh5EjR7per67b69jBvktnP/twdeZ3eqSuhfo2VnaJIGH3kksuQVRUFK666io88cQTWL16NY477jgAwCWXXIIdO3bgb3/7Gx566CGkp6cDgBWerrjiCkycOBH3338/Xn/9dfz85z9HamoqnnzySZxxxhn45S9/iaVLl+J73/sejjvuOJx66qntr92yZQtOOeUU+P1+fP/730dkZCSefPJJnHbaafjwww/b+4hvvvlmvPjii7jlllswadIklJaW4pNPPsEXX3yBGTNmtL/f5ZdfjpEjR2LJkiX49NNP8fDDD6O8vBx/+ctfXN/98ssvx6hRo7BkyRKsXbsWf/jDH5CZmYlf/vKX7c8pLCzECSecAJ/Ph1tuuQUZGRl48803sWjRIlRVVeH222+33vMb3/gGxo4di1/84hcIBAKef+Z5eXmYNWsWKioqcOONN2LChAnYv38/XnzxRdTV1SEqKiro7+vf//43Ro8ejRNPPDHoc8znjhw5Eqeccorn+VNPPRUjR45srwQfjs58p87+rpXO/H4687ld/R129Lmd+ecB8P7fwerVq7FixQpceeWVGDp0KPbs2YMnnngCp512GrZu3Yq4uLhOv7/SE3+mXXW47zly5EjMnj0bf/vb33DuuecCAN58801UVlbiyiuvxMMPP2w9vzN/jj31XXrys5XO/k6PxLVQPxAgGuA+//zzAIDA8uXLA4FAINDW1hYYOnRo4LbbbrOe96tf/SoAILB7927r+E9/+tMAgMCNN97YfqylpSUwdOjQgM/nC9x///3tx8vLywOxsbGBhQsXWu/x9a9/PRAVFRXYtWtX+7G8vLxAYmJi4NRTT20/lpSUFFi8eHHQ76Ku5cILL7SOf/vb3w4ACGzYsMH13G9+85vWcy+++OJAWlqadWzRokWBQYMGBUpKSqzjV155ZSApKSlQV1dnvedVV13lurZnn33W+vNbsGBBICwsLLB69WrXc9va2oJ+x8rKygCAwEUXXRT0OUpFRUWnnnvhhRcGAASqqqoO+p4d6cx36uzvuiu/n858bld/hwf73GD/PJjv4fW/A/U5ppUrVwYABP7yl78c9P2d/zsKBHrmz9SL12d313uuXr068OijjwYSExPb/4y+8Y1vBE4//fRAIBAIjBgxIjBv3rz213X2z3HhwoWBESNGuJ6rrtvr2MG+S2c/uzPfOZjO/k4P91poYGAbAw14S5cuRVZWFk4//XQAgM/nwxVXXIFly5ahtbW10+/zP//zP+3b4eHhmDlzJgKBABYtWtR+PDk5GePHj8dXX33Vfqy1tRXvvPMOvv71r1s9o4MGDcLVV1+NTz75BFVVVe2vX7VqFfLy8jq8lsWLF1v7t956KwDgjTfecD335ptvtvZPOeUUlJaWtn9mIBDASy+9hAsuuACBQAAlJSXtP3PnzkVlZWV7G0Ww93Rqa2vDP//5T1xwwQWYOXOm67zzr1dN6roSExM7/AxA2j4681x1Xr33oejMd+rK71o52O+nM5/bHb9D5+d2htf/DmJjY9u3m5ubUVpaijFjxiA5Odl1DZ3RE3+mh6I73vPyyy9HfX09XnvtNVRXV+O1117zbGEAuv/P0XSw79KTnw107Xfa09dC/QPDLg1ora2tWLZsGU4//XTs3r0bO3fuxM6dO3H88cejsLAQ7733Xqffa/jw4dZ+UlISYmJi2v8K1jxeXl7evl9cXIy6ujqMHz/e9Z4TJ05EW1sbcnNzAUiv6ubNmzFs2DDMmjULd999txWclbFjx1r7Rx11FMLCwjz72JzXnZKSAgDt11hcXIyKior2nlfz5/rrrwcAFBUVWe8xatQo1+eYiouLUVVVhSlTpnT4PC9+vx+ADrIdUSH2YM81Q3FjYyO++c1vYvjw4fD7/TjhhBOwcuXKg35WZ75TV37XSmd+P5353K7+Dg/2uZ3h9b+D+vp6/OQnP8GwYcMQHR2N9PR0ZGRkoKKiApWVlZ1+b6Un/kwPRXe8Z0ZGBs466yw8//zzePnll9Ha2orLLrvM87nd/edoOth36cnPBrr2O+3pa6H+gT27NKC9//77yM/Px7Jly7Bs2TLX+aVLl2LOnDmdei+vu7SD3bkdCNLHejCXX345TjnlFLzyyit455138Ktf/Qq//OUv8fLLL7f3+XnpqFJ6sGtsa2sDAFxzzTVYuHCh53OnTZtm7ZvVlu7m9/sxePBgbN68+aDPTUpKwqBBg7Bx48YOn7dx40YMGTIEfr8ftbW1GDlyJD755BMMHTq0fbTZnj17kJCQ0F1fo9O6439Dh/I77I7P9frfwa233opnn30Wt99+O2bPno2kpCT4fD5ceeWV7dfZ07r7n8vufM+rr74aN9xwAwoKCnDuueciOTnZ83md/XMM9s9+R39rdbDvEgq/QyWUroVCF8MuDWhLly5FZmYmHnvsMde5l19+Ga+88gp+//vfIzY2tsPAeDgyMjIQFxeH7du3u85t27YNYWFhGDZsWPuxQYMG4dvf/ja+/e1vo6ioCDNmzMB9991nhd0vv/zSqqrt3LkTbW1tnndld+b6EhMT0drairPOOqvLrw/2nn6/v1OB1cv555+Pp556CitXrsTs2bMP+tynn34an3zySftEANPHH3+MPXv24KabbgIAxMfH4yc/+Un7+SuvvBLf/e53sX37dhx77LGH9Z26+rvujM5+bnf/Dg/1n4cXX3wRCxcutKaSNDQ0uObpdvb9e+LPtDddfPHFuOmmm/Dpp5/i73//e9DndfbPMSUlxXMW8t69ew/5Gjv72YeqK7/Tnr4W6h/YxkADVn19PV5++WWcf/75uOyyy1w/t9xyC6qrq/Hqq68CQPuM0O7+l2h4eDjmzJmDf/3rX1abQWFhIZ5//nmcfPLJ8Pv9aG1tdf21XGZmJgYPHozGxkbruDO8P/LIIwDQYfW3o+u79NJL8dJLL3kGquLi4i6/Z1hYGL7+9a/j3//+t+cqSgerhn3/+99HfHw8/ud//geFhYWu87t27cLvfvc7AMCdd96J2NhY3HTTTSgtLbWeV1ZWhptvvhlxcXHto66cvvzyS5SVlWHMmDGH/Z06+7vuis5+bnf/Dg/1n4fw8HDX7/eRRx5xVRo7+/498WfamxISEvDEE0/g7rvvxgUXXBD0eZ39czzqqKNQWVlp/e1Gfn7+YS2k0tnPPpz37+zvtKevhfoHVnZpwHr11VdRXV2NCy+80PP8CSecgIyMDCxduhRXXHFFe1Xvhz/8Ia688kpLine truncated
|
||||
"text/plain": [
|
||||
"<Figure size 800x800 with 1 Axes>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"fig, ax = plt.subplots(figsize=(8, 8))\n",
|
||||
"ax.plot(new_time[:,0],\n",
|
||||
" co2_pred,\n",
|
||||
" color=\"b\",\n",
|
||||
" lw=2,\n",
|
||||
" label=\"Mean expectation\")\n",
|
||||
"ax.fill_between(new_time[:,0],\n",
|
||||
" co2_pred - co2_std,\n",
|
||||
" co2_pred + co2_std,\n",
|
||||
" color=\"b\",\n",
|
||||
" alpha=0.4,\n",
|
||||
" label=\"68% Confidence Level uncertainty\")\n",
|
||||
"ax.set(xlabel=\"Time [years]\", ylabel=r\"CO$_2$ [ppm]\", title=\"Atmospheric CO$_2$ concentration in Mauna Loa\")\n",
|
||||
"ax.legend(frameon=False, loc=\"lower right\")\n",
|
||||
"plt.show()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "3397756c-ea3a-4cf8-a4cc-643837baca33",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Now let's plot the actual dat above it, to check how far off we are. Let's also plot the training data and full data separately, so we can see where data not used in the fit starts: this is where our uncertainty should start becoming bigger if everything worked out correctly."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 18,
|
||||
"id": "5b263358",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"image/png": "iVBORw0KGgoAAAANSUhEUgAAArsAAAK/CAYAAABp8gWUAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAAEAAElEQVR4nOzdeXzcdbX/8dfMZGayTrbuTRegZSmrrBZEERDEUkQEZBFQEVAKgl7h6k+9et1Q3BfkAnIFFURBUKheBUQQLGhlkR3a0tKmbZo9k2X2+f7+ON/vLMmkTdukTdP300cek/lu851JS48n53OOz3EcBxERERGRCci/s29ARERERGSsKNgVERERkQlLwa6IiIiITFgKdkVERERkwlKwKyIiIiITloJdEREREZmwFOyKiIiIyISlYFdEREREJiwFuyIiIiIyYSnYFREREZEJS8GuiIiIiExYCnZFdlNf+tKX8Pl8tLe375DXu+222/D5fKxZs2aHvJ5MTDvzz5H+DIvsmhTsirh+8pOf4PP5OOqoo0ruX7ZsGV/60pfo7u7esTcmw1q1ahWXXXYZe+65J+Xl5UQiEY455hh+8IMfEIvFio596aWX+OAHP8jMmTMJh8PMmDGD888/n5deemkn3f2ubaz/Puwuf9+8ANrn8/HEE08M2e84DrNmzcLn83HqqafuhDscfd57/te//rWzb0V2Ewp2RVx33HEHc+fO5Z///CcrV64csn/ZsmX893//94T/x3esXHDBBcRiMebMmTMq1/vDH/7AgQceyG9+8xsWL17Mj370I6677jpmz57NNddcw1VXXZU79t577+XQQw/lL3/5Cx/+8If5yU9+wsUXX8xf//pXDj30UO67775RuafdyVj/fRju+qP952hrjOVrl5eXc+eddw7Z/thjj9Hc3Ew4HB711xTZXZTt7BsQGQ9Wr17NsmXLuPfee7nsssu44447+OIXv7izb2tC6O/vp6qqikAgQCAQGJVrrl69mnPOOYc5c+bwyCOPMH369Ny+JUuWsHLlSv7whz8Alv294IIL2HPPPfnb3/7G5MmTc8deddVVHHvssVxwwQU8//zz7LnnnqNyfzKU9+dge43mn6Px9Nrvec97uPvuu/nhD39IWVn+n+Y777yTww47bIeVG4lMRMrsimBZ3fr6ehYtWsSZZ57JHXfcUbT/S1/6Etdccw0Ae+yxR+7XjmvWrMnVvr7++ut88IMfpLa2lsmTJ/OFL3wBx3FYt24d733ve4lEIkybNo3vfOc7Je/h2Wef5ZRTTiESiVBdXc0JJ5zAU089VXRMb28vV199NXPnziUcDjNlyhTe9a538cwzz+Tu0+fz8eqrr3L22WcTiURobGzkqquuIh6Pl3zd7u5uPvShD1FXV0dtbS0f/vCHGRgYGHLc+vXr+chHPsLUqVMJh8Psv//+/O///u+Qz8nn8/Hyyy9z3nnnUV9fz9ve9jagdL3j+vXrufjii5kxYwbhcJg99tiDj3/84ySTyc38tOD666+nr6+PW2+9tSjQ9cybNy+X2f3Wt77FwMAAN998c1GgCzBp0iRuuukm+vv7uf766zf7miM1kvc0kp+191muXLlyxD+fLb3u1vwMN/e6m/v7UHiNUn8O3nzzTS6//HL22WcfKioqaGxs5Kyzzir6c7G56w9XNzsWn+lgpV57e6/pOffcc+no6OChhx7KbUsmk9xzzz2cd955Q44fyecI8KEPfYi5c+cOOd+771LbtvReRvra22skP9MddS+ya1NmVwQLds844wxCoRDnnnsuN954I8uXL+eII44A4IwzzuD111/nV7/6Fd/73veYNGkSQFHw9IEPfID99tuPb3zjG/zhD3/gq1/9Kg0NDdx0000cf/zxfPOb3+SOO+7g05/+NEcccQRvf/vbc+e+9NJLHHvssUQiEa699lqCwSA33XQTxx13HI899liujvhjH/sY99xzD1dccQULFiygo6ODJ554gldeeYVDDz00d72zzz6buXPnct111/HUU0/xwx/+kK6uLn7+858Pee9nn302e+yxB9dddx3PPPMMP/3pT5kyZQrf/OY3c8ds2rSJt771rfh8Pq644gomT57M//3f/3HxxRcTjUa5+uqri6551llnMX/+fL7+9a/jOE7Jz3zDhg0ceeSRdHd3c+mll7Lvvvuyfv167rnnHgYGBgiFQsP+vB544AH23HNPjj766GGPKTx27ty5HHvssSX3v/3tb2fu3Lm5TPD2GMl7GunP2jOSn89IXndrf4abe92R/H2A0n8Oli9fzrJlyzjnnHNoampizZo13HjjjRx33HG8/PLLVFZWjvj6nrH4TLfW9l5z7ty5LFy4kF/96leccsopAPzf//0fPT09nHPOOfzwhz8sOn4kn+NYvZexfG3PSH+mO+JeZAJwRHZz//rXvxzAeeihhxzHcZxsNus0NTU5V111VdFx3/rWtxzAWb16ddH2L37xiw7gXHrppblt6XTaaWpqcnw+n/ONb3wjt72rq8upqKhwLrrooqJrnH766U4oFHJWrVqV27ZhwwanpqbGefvb357bVltb6yxZsmTY9+Ldy2mnnVa0/fLLL3cA59///veQYz/ykY8UHfu+973PaWxsLNp28cUXO9OnT3fa29uLtp9zzjlObW2tMzAwUHTNc889d8i9/exnPyv6/C688ELH7/c7y5cvH3JsNpsd9j329PQ4gPPe97532GM83d3dIzr2tNNOcwAnGo1u8ZqbM5L3NNKf9db8fEbyulv7M9zS6w7396HwGqX+HHivU+jJJ590AOfnP//5Fq8/+M+R44zNZ1pKqdcerWsuX77c+fGPf+zU1NTkPqOzzjrLeec73+k4juPMmTPHWbRoUe68kX6OF110kTNnzpwhx3r3XWrblt7LSF97JO95OCP9mW7vvcjuQWUMstu74447mDp1Ku985zsB8Pl8fOADH+Cuu+4ik8mM+Dof/ehHc98HAgEOP/xwHMfh4osvzm2vq6tjn3324Y033shty2QyPPjgg5x++ulFNaPTp0/nvPPO44knniAajebO/8c//sGGDRs2ey9Lliwpen7llVcC8Mc//nHIsR/72MeKnh977LF0dHTkXtNxHH7729+yePFiHMehvb0993XyySfT09OTK6MY7pqDZbNZfve737F48WIOP/zwIfsH/3q1kHdfNTU1m30NsLKPkRzr7feuvS1G8p625mft2dLPZySvOxo/w8GvOxKl/hxUVFTkvk+lUnR0dDBv3jzq6uqG3MNIjMVnui1G45pnn302sViMpUuX0tvby9KlS0uWMMDof46FtvRexvK1Yet+pmN9LzIxKNiV3Vomk+Guu+7ine98J6tXr2blypWsXLmSo446ik2bNvGXv/xlxNeaPXt20fPa2lrKy8tzv4It3N7V1ZV73tbWxsDAAPvss8+Qa+63335ks1nWrVsHWK3qiy++yKxZszjyyCP50pe+VBQ4e+bPn1/0fK+99sLv95esYxt83/X19QC5e2xra6O7uztX81r49eEPfxiA1tbWomvsscceQ16nUFtbG9FolAMOOGCzx5USiUSAfCC7OV4Qu6VjC4PiRCLBRz7yEWbPnk0kEuGtb30rTz755BZfayTvaWt+1p6R/HxG8rpb+zPc0uuORKk/B7FYjP/6r/9i1qxZhMNhJk2axOTJk+nu7qanp2fE1/aMxWe6LUbjmpMnT+bEE0/kzjvv5N577yWTyXDmmWeWPHa0P8dCW3ovY/nasHU/07G+F5kYVLMru7VHHnmEjRs3ctddd3HXXXcN2X/HHXdw0kknjehapVZpD7dy2xmmjnVLzj77bI499ljuu+8+HnzwQb71rW/xzW9+k3vvvTdX51fK5jKlW7rHbDYLwAc/+EEuuuiikscedNBBRc8Lsy2jLRKJMGPGDF588cUtHltbW8v06dN5/vnnN3vc888/z8yZM4lEIvT39zN37lyeeOIJmpqacq3N1qxZQ3V19Wi9jREbjT9D2/IzHI3XLfXn4Morr+RnP/sZV199NQsXLqS2thafz8c555yTu8+xNtp/L0fzmueddx6XXHIJLS0tnHLKKdTV1ZU8bqSf43B/9zf3W6stvZfx8DP0jKd7kfFLwa7s1u644w6mTJnCDTfcMGTfvffey3333cf//M//UFFRsdmAcXtMnjyZyspKXnvttSH7Xn31Vfx+P7Nmzcptmz59OpdffjmXX345ra2tHHrooXzta18rCnZXrFhRlFVbuXIl2Wy25KrskdxfTU0NmUyGE088cavPH+6akUhkRAFrKaeeeio333wzTz75JAsXLtzisbfccgtPPPFEriNAoccff5w1a9Zw2WWXAVBVVcV//dd/5fafc845fOpTn+K1117jsMMO2673tLU/65EY6euO9s9wW/8+3HPPPVx00UVFXUni8fiQfrojvf5YfKY70/ve9z4uu+wynnrqKX79618Pe9xIP8f6+vqSvZDffPPNbb7Hkb72ttqan+lY34tMDCpjkN1WLBbj3nvv5dRTT+XMM88c8nXFFVfQ29vL/fffD5DrETra/xENBAKcdNJJ/P73vy8qM9i0aRN33nknb3vb24hEImQymSG/lpsyZQozZswgkUgUbR8cvP/oRz8C2Gz2d3P39/73v5/f/va3JQOqtra2rb6m3+/n9NNP54EHHig5RWlL2bBrr72WqqoqPvrRj7Jp06Yh+1etWsUPfvADAK655hoqKiq47LLL6OjoKDqus7OTj33sY1RWVuZaXQ22YsUKOjs7mTdv3na/p5H+rLfGSF93tH+G2/r3IRAIDPn5/uhHPxqSaRzp9cfiM92ZqqurufHGG/nSl77E4sWLhz1upJ/jXnvtRU9PT9FvNzZu3Lhdg1RG+trbc/2R/kzH+l5kYlBmV3Zb999/P729vZx22mkl97/1rW9l8uTJ3HHHHXzgAx/IZfU+97nPcc455xAMBjf7j9HLine truncated
|
||||
"text/plain": [
|
||||
"<Figure size 800x800 with 1 Axes>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"fig, ax = plt.subplots(figsize=(8, 8))\n",
|
||||
"ax.scatter(time, co2, color='red', marker='o', label=\"Fit data\")\n",
|
||||
"ax.scatter(data.decimal, data.co2, color='green', marker='.', label=\"All data\")\n",
|
||||
"ax.fill_between(new_time[:,0],\n",
|
||||
" co2_pred - co2_std,\n",
|
||||
" co2_pred + co2_std,\n",
|
||||
" color=\"b\",\n",
|
||||
" alpha=0.4,\n",
|
||||
" label=\"Predictions\")\n",
|
||||
"ax.set(xlabel=\"Time [years]\", ylabel=r\"CO$_2$ [ppm]\", title=\"Atmospheric CO$_2$ concentration in Mauna Loa\")\n",
|
||||
"ax.legend(frameon=False, loc=\"lower right\")\n",
|
||||
"plt.show()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "4beb5578-8437-4696-994a-7cde29266f15",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Try changing the kernel above and check what happens. Notice how large an effect the kernel definition has in the results!\n",
|
||||
"\n",
|
||||
"The kernel is one of our key assumptions here!"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "b625b695",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Many more details on the kernel selection as well as further details in this example can be seen in the following references:\n",
|
||||
"\n",
|
||||
"https://scikit-learn.org/stable/auto_examples/gaussian_process/plot_gpr_co2.html\n",
|
||||
"\n",
|
||||
"https://scikit-learn.org/stable/auto_examples/gaussian_process/plot_gpr_noisy_targets.html#sphx-glr-auto-examples-gaussian-process-plot-gpr-noisy-targets-py\n",
|
||||
"\n",
|
||||
"To read more on Gaussian Processes:\n",
|
||||
"\n",
|
||||
"http://www.gaussianprocess.org/gpml/chapters/RW.pdf"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "663ff599-7b8d-4f13-a02f-3b9a7808814e",
|
||||
"metadata": {},
|
||||
"source": []
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.10.12"
|
||||
},
|
||||
"widgets": {
|
||||
"application/vnd.jupyter.widget-state+json": {
|
||||
"state": {},
|
||||
"version_major": 2,
|
||||
"version_minor": 0
|
||||
}
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
@@ -0,0 +1,638 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "6a764d92",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Bayesian Optimization\n",
|
||||
"\n",
|
||||
"We can take advantage of the previous method by using it to optimize controls in an experiment. This procedure is called Bayesian Optimization: we collect data and calculate a metric indicating how good the current setup is. We then fit a function using a Gaussian Process to identify how this quality metric behaves as a function of the parameters we actually control. From this fit we may find the optimum of the quality function and shift our control parameters there.\n",
|
||||
"\n",
|
||||
"When collecting data points with several inputs in a system is very time-consuming, this methodology is particularly powerful, since it allows one to reach the optimum very fast.\n",
|
||||
"\n",
|
||||
"Here we will create an example function that we want to maximize, but we assume that collecting the value of the function in each point is very time-consuming or resource-intensive and therefore we would like to find the maximum of that function with very few attempts. The same logic would work for finding the minimum of a function.\n",
|
||||
"\n",
|
||||
"Let us start by importing some modules and creating an example function for us to maximize.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"id": "8933f30f-4862-4b36-b19a-a3511726d77f",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Needed in Google Colab, since it does not have gpytorch by default\n",
|
||||
"#!pip install gpytorch torch botorch"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"id": "3ee0feee-7453-4982-ac31-c661ce692449",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"/home/daniloefl/workspace/ml-lecture-oct-2023/env/lib/python3.9/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
|
||||
" from .autonotebook import tqdm as notebook_tqdm\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from typing import List, Tuple, Optional\n",
|
||||
"\n",
|
||||
"import numpy as np\n",
|
||||
"\n",
|
||||
"from numpy.random import MT19937\n",
|
||||
"from numpy.random import RandomState, SeedSequence\n",
|
||||
"import matplotlib.pyplot as plt\n",
|
||||
"\n",
|
||||
"import torch\n",
|
||||
"import gpytorch\n",
|
||||
"\n",
|
||||
"from botorch.acquisition import UpperConfidenceBound\n",
|
||||
"from botorch.fit import fit_gpytorch_model\n",
|
||||
"from botorch.optim import optimize_acqf\n",
|
||||
"from botorch.acquisition.objective import ScalarizedPosteriorTransform\n",
|
||||
"from botorch.models import SingleTaskGP\n",
|
||||
"from torch import Tensor\n",
|
||||
"from gpytorch.kernels.scale_kernel import ScaleKernel\n",
|
||||
"from gpytorch.kernels.rbf_kernel import RBFKernel\n",
|
||||
"from gpytorch.mlls import ExactMarginalLogLikelihood"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"id": "fec4cda5-6574-4f34-8b29-02061aab35c8",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def true_f(x: Tensor) -> Tensor:\n",
|
||||
" \"\"\"Function that we would like to optimize. Usually we do not know this!\"\"\"\n",
|
||||
" return ((torch.sin(2*np.pi*x) + torch.sin(2*np.pi*2*x))*torch.exp(-x**2/2.0/0.2)).sum(1, keepdims=True)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "9bdd019f-5bae-401b-9c73-1e7a58fe2a24",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We can now plot this function to see how it looks like. Usually, this is very hard to do, since we have many parameter dimensions. In this example, we have a single parameter dimension to makes things easy to visualize."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"id": "c9a7d52c-588c-49dc-a05e-7f3bcc78e7a3",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"[<matplotlib.lines.Line2D at 0x7f49a4ebc8e0>]"
|
||||
]
|
||||
},
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAjcAAAGfCAYAAACjj/OwAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAABZhUlEQVR4nO3deXhTZdoG8PskbZKu6b5BoWWnLAUKlCIuaLWgg+CKqIAdxU/UcanLDDMKKo6oowzqMKIIgo6CooioWNFqRXYolLUspTtturfpmjbJ+f5IE6xQaEuTk+X+XVeumaYnp08O0tyc93nfVxBFUQQRERGRk5BJXQARERFRT2K4ISIiIqfCcENEREROheGGiIiInArDDRERETkVhhsiIiJyKgw3RERE5FQYboiIiMipMNwQERGRU2G4ISIiIqfiZs2Tb9u2Df/617+QkZGBkpISfPXVV5gxY0aHx993331Yu3btec/HxMTg2LFjAIAXXngBL774YrvvDx48GCdOnOh0XUajEcXFxfDx8YEgCJ1+HREREUlHFEXU1dUhIiICMlnH92esGm4aGhoQGxuLP//5z7j11lsvefxbb72FV1991fK1Xq9HbGws7rjjjnbHDRs2DD/99JPlaze3rr2N4uJiREZGduk1REREZB8KCwvRu3fvDr9v1XAzdepUTJ06tdPHq9VqqNVqy9ebNm1CdXU1kpOT2x3n5uaGsLCwbtfl4+MDwHRxfH19u30eIiIish2tVovIyEjL53hHrBpuLteqVauQmJiIvn37tnv+9OnTiIiIgEqlQkJCApYsWYI+ffp0eB6dTgedTmf5uq6uDgDg6+vLcENERORgLtVSYrcNxcXFxfj+++/xwAMPtHs+Pj4ea9asQWpqKt59913k5ubiyiuvtASWC1myZInlrpBareaQFBERkRMTRFEUbfKDBOGSDcW/t2TJErz55psoLi6GQqHo8Liamhr07dsXS5cuxf3333/BY/5458Z8W6u2tpZ3boiIiByEVquFWq2+5Oe3XQ5LiaKI1atXY/bs2RcNNgDg5+eHQYMGITs7u8NjlEollEplT5dJREREdsguh6V+/fVXZGdnd3gn5vfq6+tx5swZhIeH26AyIiIisndWDTf19fXIzMxEZmYmACA3NxeZmZkoKCgAACxYsABz5sw573WrVq1CfHw8hg8fft73nn76afz666/Iy8vDzp07ccstt0Aul2PWrFnWfCtERETkIKw6LLV//35MnjzZ8nVKSgoAYO7cuVizZg1KSkosQcestrYWX375Jd56660LnrOoqAizZs1CZWUlgoODMWnSJOzevRvBwcHWeyNERETkMGzWUGxPOtuQRERERPajs5/fdtlzQ0RERNRdDDdERETkVBhuiIiIyKkw3BAREZFTYbghIiIip2KXKxSTYzEaRRwr1mLb6XLUNevh5+mO3v4eSBwaCpW7XOryiIjIxTDcULeJooiNB87itdQTKKvTnfd9P0933Dk2EvOv7g9/r4tvo0FERNRTGG6oW4prmvDsF4exPbsCAOCtdENC/0D08vNAbVMr9uZW4WxNE97floNvDhXjP3ePRlzfAImrJiIiV8BwQ11WWNWIu97fjbM1TVC6yfBE4iDcPykaCrdzLVwGo4hfTpThlS1ZyKlowMz3duOFm4fh3gl9JayciIhcAcMNdUlBZSNmrTQFm+ggL3x43zhEBXmdd5xcJiAxJhQT+gdiwcYj+OZQMZ7bdBQe7nLcFtdbgsqJiMhVcLYUdVpdcyvmrN6DszVN6BfkhfUPTrhgsPk9b6Ub3r5rFB6YFA0AePbLw/jxeKktyiUiIhfFcEOdIooi/v7VUeRVNqKXnwfWPzgBob6qTr1WEAT8/cahuG1MbxiMIh799ABOldZZuWIiInJVDDfUKZ/vL8Q3h4ohlwl4e9YohHQy2JjJZAJeu20ErhwYBJ3eiMfWHYROb7BStURE5MoYbuiSCqsasWjzMQDAUzcM6vasJze5DG/eEYsALwVOaOrweurJniyTiIgIAMMNdcIrW7LQ3GpEfHQAHrqq/2WdK8RXhX/dPhIAsGp7LnbnVPZEiURERBYMN3RRO89U4PujGsgE4MXpwyCTCZd9zuuGhmLW+D4AgBc2H4PeYLzscxIREZkx3FCH9AYjXtx8HABw74S+GBLm22Pn/uuUwfDzdMcJTR0+3VvQY+clIiJiuKEObcgowsnSOvh5uiPl+kE9em4/TwWeumEwAODNradQ3dDSo+cnIiLXxXBDF6Q3GPFu+hkAwKOTB8DPs+f3hrp7fB8MDfdFbVMr3ko73ePnJyIi18RwQxf0zeFiFFQ1IsBLgbvj+1jlZ8hlAp67aSgA4NO9BdDUNlvl5xARkWthuKHzGI0i/vuL6a7N/ZOi4amw3i4dE/sHYnxUAFr0Rqz49YzVfg4REbkOhhs6z9bjGpwuq4ePyg2zE6y70aUgCHg8cSAA092bUi3v3hAR0eVhuKHzrPwtFwAwNyEKvip3q/+8if0DMS7KHy36c30+RERE3cVwQ+1klWiRkV8NN5mAOVa+a2MmCAIev840G2v9vgLUNHLmFBERdR/DDbXzv935AIAbhoV2ef+oy3HFgEDEhPuiudXIdW+IiOiyMNyQRb1Oj00HzwIA7o23zV0bM0EQ8OdJ0QCAj3bmo5WrFhMRUTcx3JDFVwfPoqHFgH7BXkjoH2jznz8tNhxB3gpotM34/qjG5j+fiIicA8MNAQBEUcQnbUNS98T3hSBc/h5SXaV0k+PeCaY7Rqu359r85xMRkXNguCEAQFZJHU5o6qCQy3D7mN6S1XFPfF8o5DJkFtbgcFGNZHUQEZHjYrghAMDXmaZem+uGhkDtaf3p3x0J9lFiyvAwAMD6fYWS1UFERI6L4YZgMIr4OrMYADB9VC+JqwHuGh8JANicWYzGFr3E1RARkaOxarjZtm0bpk2bhoiICAiCgE2bNl30+PT0dAiCcN5Do2nfXLp8+XJERUVBpVIhPj4ee/futeK7cH57ciqh0TbDV+WGyUOCpS4HE6ID0TfQE/U6Pb49XCJ1OURE5GCsGm4aGhoQGxuL5cuXd+l1J0+eRElJieUREhJi+d5nn32GlJQULFq0CAcOHEBsbCySkpJQVlbW0+W7jE1tQ1I3jYyA0k0ucTWATCbgzrGmuzefcWiKiIi6yKrhZurUqXj55Zdxyy23dOl1ISEhCAsLszxksnNlLl26FPPmzUNycjJiYmKwYsUKeHp6YvXq1T1dvktobjXg+yOmO2O3jJZ+SMrsjrjekMsEZORX41RpndTlEBGRA7HLnptRo0YhPDwc119/PXbs2GF5vqWlBRkZGUhMTLQ8J5PJkJiYiF27dnV4Pp1OB61W2+5BJr+eKkedTo9efh4Y29df6nIsQnxVuHaI6Y7dlweKJK6GiIgciV2Fm/DwcKxYsQJffvklvvzyS0RGRuKaa67BgQMHAAAVFRUwGAwIDQ1t97rQ0NDz+nJ+b8mSJVCr1ZZHZGSkVd+HI/nhmOm6JQ0Lg0xm+7VtLubWtjtJ32QWw2gUJa6GiIgchZvUBfze4MGDMXjwYMvXEydOxJkzZ/Dvf/8bH3/8cbfPu2DBAqSkpFi+1mq1DDgAWg1G/HS8FAAs06/tyeQhIfBRuaG4thl786owoZ/tV00mIiLHY1d3bi5k/PjxyM7OBgAEBQVBLpejtLS03TGlpaUIC+v4w1mpVMLX17fdg4A9OVXQNusR6KVAnB0NSZmp3OW4cXg4AFj2vCIiIroUuw83mZmZCA83fcApFArExcUhLS3N8n2j0Yi0tDQkJCRIVaLDSj1mmmZ9fUwo5HY2JGU2fXQEAGDLkRLo9AaJqyEiIkdg1WGp+vp6y10XAMjNzUVmZiYCAgLQp08fLFiwAGfPnsVHH30EAFi2bBmio6MxbNgwNDc344MPPsDPP/+MrVu3Ws6RkpKCuXPnYuzYsRg/fjyWLVuGhoYGJCcnW/OtOB2jUcTWY6Y7YEl2OCRlNiE6EOFqFUpqm/HLiXK7HD4jIiL7YtVws3//fkyePNnytbnvZe7cuVizZg1KSkpQUFBg+X5LSwueeuopnD17Fp6enhg5ciR++umndueYOXMmysvLsXDhQmg0GowaNQqpqannNRnTxR0srEFZnQ7eSjdMlGAH8M6SyQTcHBuB97blYPOhsww3RER0SYIoii43DUWr1UKtVqO2ttZl+29eSz2Bd9PPYFpsBN6ZNVrqci7q6Nla/Omd7fBwlyPj+UR4KuyqD56IiGyks5/fdt9zQ9bxywnTis7XDQm5xJHSGxbhi8gADzS1GvDryXKpyyEiIjvHcOOCNLXNOKGpgyAAVw2Sfi+pSxEEwTJr6rsj3GuKiIgujuHGBaWfNN21ie3thwAvhcTVdM7UEaZw8/OJMjS3ctYUERF1jOHGBaW3De1cM9j+79qYxfZWo5efBxpbDPj1FIemiIioYww3LqbVYMT27AoAwDWD7b/fxkwQBExtmyn1PYemiIjoIhhuXMz+vGrU60yrEo/spZa6nC4xD039lFWGFr1R4mqIiMheMdy4mPRTpn6bqwYF291GmZcyOtIPwT5K1Ov02J1TKXU5RERkpxhuXMy2U+YhKcfptzGTyQQkDjUNpf2UVXqJo4mIyFUx3LiQynodskq0AICJ/YMkrqZ7ro8xrUT90/FSuOD6k0RE1AkMNy5kV9tQzuBQHwT7KCWupnsm9g+Ch7scxbXNOFaslbocIiKyQww3LmRHtincTBxgv3tJXYrKXY6rBpnuOv14nENTRER0PoYbF7LzjKnf5goHHZIyuz7GNCWc4YaIiC6E4cZFFFU3Ir+yEXKZgPh+AVKXc1muHRICmQAcL9HibE2T1OUQEZGdYbhxETvbhqRie6vho3KXuJrLE+ClQFxffwDnNgAlIiIyY7hxETvMQ1IDHHtIysy8urJ5nywiIiIzhhsXIIoidp5payZ28H4bs8lLine truncated
|
||||
"text/plain": [
|
||||
"<Figure size 640x480 with 1 Axes>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"x_axis = torch.from_numpy(np.linspace(0.0, 1.0, 200)).unsqueeze(1)\n",
|
||||
"y_axis = true_f(x_axis).detach()\n",
|
||||
"plt.plot(x_axis.numpy()[:,0], y_axis.numpy())"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "bee56813-6465-465a-9e93-232eada9ba46",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We are now going to pretend we can only take a few points of this curve, because each time we \"measure\" the value of the function for a given x-axis value, it takes a lot of our resources.\n",
|
||||
"\n",
|
||||
"To this end, we start by taking 5 points at random from the curve to make an initial fit."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"id": "442d8fc1-e45c-4162-b34c-f33ce56afd4a",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def initial_data(n: int=5) -> Tensor:\n",
|
||||
" \"\"\"Randomly select some points to test f on.\"\"\"\n",
|
||||
" rng = RandomState(MT19937(SeedSequence(123456789)))\n",
|
||||
" return 0.2+torch.from_numpy(rng.rand(n, 1)*0.3).double()\n",
|
||||
" \n",
|
||||
"x0 = initial_data(n=5)\n",
|
||||
"y0 = true_f(x0)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "af6e1101-80bf-4e36-a5bf-508d47f73614",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Let's now plot it to visualize it, but let's make the plot a bit more detailed than before:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"id": "d0825829-f78b-4443-84ca-922c0ef3c32a",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def plot_simple(x: Tensor, y: Tensor):\n",
|
||||
" \"\"\"\n",
|
||||
" Plot the observed x, y and the true function.\n",
|
||||
" \"\"\"\n",
|
||||
" # get a set of x-axis points from 0 to 1 for plotting\n",
|
||||
" x_axis = torch.from_numpy(np.linspace(0.0, 1.0, 200)).double().unsqueeze(1)\n",
|
||||
" \n",
|
||||
" # get the true function value\n",
|
||||
" true_y = true_f(x_axis)\n",
|
||||
"\n",
|
||||
" f, ax = plt.subplots(1, 1, figsize=(12, 4))\n",
|
||||
"\n",
|
||||
" # first the true function so we can compare things with it\n",
|
||||
" ax.plot(x_axis, true_y, 'g--', label=r\"True $f$\", lw=2)\n",
|
||||
"\n",
|
||||
" # plot the actual points we collected\n",
|
||||
" ax.plot(x.cpu().numpy()[:,0], y.cpu().numpy(), 'k^', label=\"Observation\", markersize=8)\n",
|
||||
"\n",
|
||||
" # make the plot look pretty\n",
|
||||
" ax.set(xlabel=\"$x$\", ylabel=r\"$f(x)$\", title=\"\", xlim=(0.0, 1.0))\n",
|
||||
" ax.legend(loc='upper left', frameon=False, bbox_to_anchor=(1.0,1), borderaxespad=0)\n",
|
||||
"\n",
|
||||
" # prettify\n",
|
||||
" plt.tight_layout()\n",
|
||||
" plt.subplots_adjust(right=0.80,\n",
|
||||
" top=0.99)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"id": "0876f631-e6a3-4104-839d-522244708982",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"image/png": "iVBORw0KGgoAAAANSUhEUgAABEsAAAGTCAYAAADQjZyeAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAAB+s0lEQVR4nOzdd3gU1dvG8XuzqZSElhACgdB7LyECCgoERIoVEARpKgoi5aeg0mzYQFBQEESwASJFEAWk906kSG8JJaEnEELK7rx/8Lqy0hJSJuX7ua693DlzZvbeOBsyz545YzEMwxAAAAAAAAAkSS5mBwAAAAAAAMhMKJYAAAAAAADchGIJAAAAAADATSiWAAAAAAAA3IRiCQAAAAAAwE0olgAAAAAAANyEYgkAAAAAAMBNKJYAAAAAAADchGIJAAAAAADATSiWAAAAAAAA3MTV7AD3Y82aNfrkk0+0fft2nTlzRvPmzVO7du3u2P/555/X9OnTb2mvVKmS9u7dK0kaMWKERo4c6bS+fPny2r9/f7Jz2e12nT59Wnnz5pXFYkn2dgAAAACAtGMYhq5cuaKAgAC5uDBGACmXJYslsbGxql69urp3764nnnjinv3HjRunDz/80LGclJSk6tWr6+mnn3bqV7lyZS1btsyx7Oqash/P6dOnFRgYmKJtAAAAAADpIyIiQsWKFTM7BrKgLFksadmypVq2bJns/j4+PvLx8XEsz58/X5cuXVK3bt2c+rm6usrf3/++c+XNm1fSjQ+kt7f3fe8HAAAAAHD/YmJiFBgY6DhHA1IqSxZLUuubb75R06ZNVaJECaf2Q4cOKSAgQJ6engoJCdGoUaNUvHjxO+4nPj5e8fHxjuUrV65Ikry9vSmWAAAAAIDJmB4B9yvHXbx1+vRp/fHHH+rZs6dTe3BwsKZNm6bFixfrq6++0rFjx9SoUSNHAeR2Ro0a5Ri14uPjwyU4AAAAAABkAxbDMAyzQ6SGxWK55wSvNxs1apRGjx6t06dPy93d/Y79Ll++rBIlSmjMmDHq0aPHbfv8d2TJP0O9oqOjGVkCAAAAACaJiYmRj48P52a4bznqMhzDMDR16lQ999xzdy2USFK+fPlUrlw5HT58+I59PDw85OHhkdYxAQAAAACAiXLUZTirV6/W4cOH7zhS5GZXr17VkSNHVKRIkQxIBgAAAAAAMossWSy5evWqwsLCFBYWJkk6duyYwsLCFB4eLkkaMmSIunTpcst233zzjYKDg1WlSpVb1g0aNEirV6/W8ePHtWHDBj3++OOyWq3q2LFjur4XAAAAAAAysy+++EIlSpSQq6urBg0aZHacDJElL8PZtm2bmjRp4lgeMGCAJKlr166aNm2azpw54yic/CM6Olpz5szRuHHjbrvPkydPqmPHjrpw4YJ8fX3VsGFDbdq0Sb6+vun3RgAAAAAA+H/3unvP8OHDNWLEiIwJ8//++usvDRgwQL/++qtq1qwpHx+fDH19s2T5CV4zEyYRAgAAAADzZdVzs8jISMfzWbNmadiwYTpw4ICjLU+ePMqTJ4/TNgkJCfeckzM13n//ff3+++9av359ur1GZpQlL8MBAAAAACC78ff3dzx8fHxksVic2vLkyaPGjRurT58+eu2111SoUCGFhoZKkoKCgjR27Fin/dWoUcMxEsVut2vUqFEqWbKkvLy8VL16df3yyy93zVOmTBm9/fbb2rBhgywWy22nu8iusuRlOAAAAAAA5FTTp09X7969UzTaY9SoUfrhhx80ceJElS1bVmvWrFHnzp3l6+urhx566LbbbNiwQSEhIerdu7c6d+58y6iW7IxiCQAAAAAgxxizcYzGbBxzz361itTSgo4LnNrazGijHWd23HPbASEDNCBkwH1nvJeyZcvq448/Tnb/+Ph4ffDBB1q2bJlCQkIkSaVKldK6des0adKkOxZL8uTJo+PHj6thw4by9/dPk+xZBcUSIIUMw1CCLUHuVvd7TsAEAAAAIHOJiY/RqSun7tkv0CfwlrZz184la9uY+Jj7ypZctWvXTlH/w4cP69q1a2rWrJlTe0JCgmrWrHnH7Xbt2iVJqlq1aspDZnEUS4C7OHThkHK751ZA3gBH26Xrl1Tw44KyyCJPV095uXkpt1tuVShUQbWL1FatIrVUO6C2SuYrSTEFAAAAyGS8PbxVNG/Re/bzzXXrnVF9c/kma1tvj/SdVDZ37ty3tLm4uOi/929JTEyUJF29elWStGjRIhUt6pzfw8Pjjq8TFhamMmXK3Pb1sjuKJcBNbHab1kes18IDC7Xw4EIduHBAwx8arhGNRzj65PfML28Pb8XExyguKU5xSXG6GHdRETER+vPon45+RfMW1ZjQMXqm8jMmvBMAAAAAt5OaS2T+e1lOZuLr66szZ844lmNiYnTs2DFJUqVKleTh4aHw8PA7XnJzO2FhYapevXqaZ80KKJYAkq4nXde3O7/VJxs+0bHLx5zWLTy40KlYYrFY9ETFJ/T3ub8Vl/hvseRi3EWn7U5dOaUCXgUyIj4AAACAHO7hhx/WtGnT1Lp1a+XLl0/Dhg2T1WqVJOXNm1eDBg1S//79Zbfb1bBhQ0VHR2v9+vXy9vZW165db7vPsLAwtWnTJiPfRqZBsQQ5WvT1aH217SuN3TRWUbFRTutcLC5qENhAbcq3kWEYTpfUfNv2W6e+hmHo1JVT2nFmh7af3q7NpzZr//n9ahLUxKnf+vD1WnpkqQaEDJCPp0/6vTEAAAAAOcqQIUN07NgxPfbYY/Lx8dG7777rGFkiSe+++658fX01atQoHT16VPny5VOtWrX05ptv3nZ/drtdu3fv1tChQzPqLWQqFuO/FzXhvsXExMjHx0fR0dHy9k7fa9SQen8e+VPtf2mvS9cvObU3K9VMXap3UcsyLVUwV8H73v8/k8D+wzAMNZneRKtPrFZ+z/wa9MAgvRr8qvK455zbbwEAAAAZgXMzpJaL2QEAs1T0ragke5IkySKLnq70tLa/sF1Ln1uqztU6p6pQIsmpUCJJBy8c1PqIG/dBv3T9kt5a8ZYqTqioPw79karXAQAAAACkLYolyLGKeRfTh00/VMcqHXWgzwH9/PTPqlWkVrq9XvlC5XWgzwF1rd5VLpYbH72TMSf16E+Pqvuv3XX5+uV0e20AAAAAQPJxGU4aYqhX5nUu9pzeWf2OPm72sbzcvBzt/52LJKMcOH9Aff/o63T3nIC8Afr6sa/VqlyrDM8DAAAAZCecmyG1GFmCbC8sMkx1J9fV+K3j9cJvLzjde9yMQol0Y5TJks5LNLn1ZMc92E9fOa3HZjymd1a/Y0omAAAAAMANFEuQrc3eO1sNpjbQiegTkqRlR5fp9JXTJqe6wWKxqGetntrTe49alGlxo00W1Staz+RkAAAAAJCzcetgZEuGYWjEqhF6Z82/ozTqFa2nuc/MVVHvoiYmu1WgT6B+f/Z3Tdg6QdeTrjsKJwAAAAAAc1AsQbZjGIYGLBmgsZvHOtq6VO+iSY9Nkqerp3nB7sJisahPvT63XXf44mGVKVAmgxMBAAAAQM7FZTjIVgzD0Kt/vOoolFhk0ejmozWt7bRMWyi5my82f6GKEypqxu4ZZkcBAAAAgByDkSXINuyGXa8sekUTt0+UdKNQ8k2bb9StZjeTk92fVcdX6dXFr0qSOs3tpKsJV9Wrdi+TUwEAAABA9sfIEmQb8Unx2nNujyTJxeKi6e2mZ9lCiSQ9WOJBvVj7RUmSIUMv/PaCxmwcY3IqAAAAIGdZtmyZKlWqpGXLlpkdxUlQUJDGjh1rdow0s2rVKlksFl2+fNnsKJIoliAb8XLz0u/P/q5GxRvp+8e/13PVnzM7Uqq4WFz0VauvNDBkoKNt4NKBGrlqpImpAAAAgJzDMAy9+eab2rdvn958800ZhpEhrxsREaHu3bsrICBA7u7uKlGihPr166cLFy5kyOunt8aNG+u1115zanvggQd05swZ+fj4mBPqPyiWIFvJ65FXq55fpWerPmt2lDRhsVj0SbNPNLLxvwWSEatH6JP1n5iYCgAAAMgZli5dqq1bt0qStm7dqqVLl6b7ax49elR16tTRoUOHNGPGDB0+fFgTJ07U8uXLFRISoosXL6Z7htux2Wyy2+3ptn93d3f5+/vLYrGk22ukBMUSZGlrT6xV9PVopzYXS/Y6rC0Wi4Y9NEyjm492tL2+7HV9s+MbE1MBAAAA2ZthGBo6dKisVqskyWq1aujQoek+uuSVV16Ru7u7li5dqoceekjFixdXy5YttWzZMp06dUpvvfWWo++VK1fUsWNH5c6dW0WLFtWECROc8o8YMULFixeXh4eHAgIC9OqrrzrWx8fHa9CgQSpatKhy586t4OBgrVq1yrF+2rRpypcvnxYsWKBKlSrJw8NDU6ZMkaen5y2XyvTr108PP/ywJOnChQvq2LGjihYtqly5cqlq1aqaMePfG1Y8//zzWr16tcaNGyeLxSKLxaLjx4/f9jKcOXPmqHLlyvLw8FBQUJBGjx7t9LpBQUH64IMP1L17d+XNm1fFixfX119/nZofv0P2OqtEjrL11FaF/hCqRt82UkR0hNlx0t2AkAF6r8l7juUXfntBc/fNNTERAAAAkH39M6rEZrNJujGyIr1Hl1y8eFFLlizRyy+/LC8vL6d1/v7+6tSpk2bNmuUo2HzyySeqXr26du7cqcGDB6tfv376888/Jd0oNHz22WeaNGmSDh06pPnz56tq1aqO/fXp00cbN27UzJkztWvXLj399NNq0aKFDh065Ohz7do1ffTRR5oyZYr27t2rTp06KV++fJozZ46jj81m06xZs9SpUydJ0vXr11W7dm0tWrRIe/bs0QsvvKDnnntOW7ZskSSNGzdOISEh6tWrl86cOaMzZ84oMDDwlp/F9u3b9cwzz6hDhw7avXu3RowYoaFDh2ratGlO/UaPHq06depo586devnll9W7d28dOHAgFf8Line truncated
|
||||
"text/plain": [
|
||||
"<Figure size 1200x400 with 1 Axes>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"plot_simple(x0, y0)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "a0d87ed3-af4c-4379-8e81-0cf61610de0d",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"So, now we have some initial points. We are assuming the true function is not known, so how can we fit those points to make this curve?\n",
|
||||
"Let us create a fit model using a Gaussian Process. We create an object to do some preprocessing and hold the most important fit elements in the `FitModel` class.\n",
|
||||
"Check the `botorch` documentation for more information."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"id": "5dd8089c-b3ed-4d67-b67e-9d34fc7ed77f",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"class FitModel(SingleTaskGP):\n",
|
||||
" \"\"\"Model to fit the function using the few points we know.\n",
|
||||
"\n",
|
||||
" Args:\n",
|
||||
" x_train: The parameters we control. Assumed of shape (B, K), where B is the number of points and K is the dimension of the data.\n",
|
||||
" y_train: The function value. Assumed of shape (B, 1), containing a single value per point.\n",
|
||||
" \"\"\"\n",
|
||||
" def __init__(self, x_train: Tensor, y_train: Tensor):\n",
|
||||
" assert len(x_train.shape) == 2\n",
|
||||
" assert len(y_train.shape) == 2\n",
|
||||
"\n",
|
||||
" # calculate the mean and root-mean-squared error to standardize the input data\n",
|
||||
" self.x = x_train.detach()\n",
|
||||
" self.y = y_train.detach()\n",
|
||||
" self.mx = self.x.mean(dim=-2, keepdim=True)\n",
|
||||
" self.sx = torch.clamp(self.x.std(dim=-2, keepdim=True), min=1e-6)\n",
|
||||
" self.my = self.y.mean(dim=-2, keepdim=True)\n",
|
||||
" self.sy = torch.clamp(self.y.std(dim=-2, keepdim=True), min=1e-6)\n",
|
||||
"\n",
|
||||
" # the input data after some preprocessing\n",
|
||||
" self.x_tr = self.transform_x(self.x)\n",
|
||||
" self.y_tr = self.transform_y(self.y)\n",
|
||||
"\n",
|
||||
" # representation of the covariance model\n",
|
||||
" # this is the core of the model and it encodes a covariance function C(x1, x2) = S*exp(-|x2 - x1|^2/L^2)\n",
|
||||
" # S encoded using the `ScaleKernel` object\n",
|
||||
" # L is encoded using the `RBFKernel` object and it is called the lengthscale\n",
|
||||
" covar_module = ScaleKernel(RBFKernel())\n",
|
||||
"\n",
|
||||
" super(FitModel, self).__init__(self.x_tr, self.y_tr, covar_module=covar_module)\n",
|
||||
" self.best_y_tr = torch.max(self.y_tr)\n",
|
||||
"\n",
|
||||
" def transform_x(self, x: Tensor) -> Tensor:\n",
|
||||
" \"\"\"Transform inputs.\"\"\"\n",
|
||||
" x_tr = (x - self.mx)/self.sx\n",
|
||||
" return x_tr\n",
|
||||
"\n",
|
||||
" def transform_y(self, y: Tensor) -> Tensor:\n",
|
||||
" \"\"\"Transform outputs.\"\"\"\n",
|
||||
" y_tr = (y - self.my)/self.sy\n",
|
||||
" return y_tr\n",
|
||||
"\n",
|
||||
" def untransform_x(self, x: Tensor, no_constant: bool=False) -> Tensor:\n",
|
||||
" \"\"\"Undo transform of inputs.\"\"\"\n",
|
||||
" x_tr = x*self.sx\n",
|
||||
" if not no_constant:\n",
|
||||
" x_tr += self.mx\n",
|
||||
" return x_tr\n",
|
||||
"\n",
|
||||
" def untransform_y(self, y: Tensor, no_constant: bool=False) -> Tensor:\n",
|
||||
" \"\"\"Undo transform of outputs.\"\"\"\n",
|
||||
" y_tr = y*self.sy\n",
|
||||
" if not no_constant:\n",
|
||||
" y_tr += self.my\n",
|
||||
" return y_tr\n",
|
||||
"\n",
|
||||
" def predict(self, x: Tensor) -> Tuple[Tensor, Tensor]:\n",
|
||||
" \"\"\"Predict the posterior mean and uncertainty.\n",
|
||||
"\n",
|
||||
" Args:\n",
|
||||
" x: Input parameters\n",
|
||||
"\n",
|
||||
" Returns: The mean and uncertainty prediction.\n",
|
||||
" \"\"\"\n",
|
||||
" x_tr = self.transform_x(x)\n",
|
||||
" pred = self.likelihood(self(x_tr))\n",
|
||||
" m = self.untransform_y(pred.mean)\n",
|
||||
" s = self.untransform_y(torch.sqrt(pred.variance), no_constant=True)\n",
|
||||
" return m, s\n",
|
||||
"\n",
|
||||
" def get_acquisition(self):\n",
|
||||
" \"\"\"Get the acquisition function.\"\"\"\n",
|
||||
" posterior_transform = ScalarizedPosteriorTransform(weights=self.sy.view(-1), offset=self.my.item())\n",
|
||||
" UCB = UpperConfidenceBound(model=self, beta=0.1, posterior_transform=posterior_transform)\n",
|
||||
" return UCB"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "310d19f5-187c-4887-b66c-dbf268709901",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Let's also write down a function to make the fit using the initial data and from the fit, extract what is the next candidate to be probed:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 9,
|
||||
"id": "c53800fb-65f9-44d5-bff5-0308b4afc452",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def update_params(x: Tensor, y: Tensor) -> Tuple[Tensor, FitModel]:\n",
|
||||
" \"\"\"\n",
|
||||
" Given some input past parameters and the value of the function, get me a new value to probe.\n",
|
||||
" \"\"\"\n",
|
||||
" # create a fit object\n",
|
||||
" model = FitModel(x, y)\n",
|
||||
" # get the log-likelihood for it\n",
|
||||
" mll = ExactMarginalLogLikelihood(model.likelihood, model)\n",
|
||||
" # make the fit of the model to the observed data\n",
|
||||
" opt_mll = fit_gpytorch_model(mll)\n",
|
||||
"\n",
|
||||
" # get the acquisition function\n",
|
||||
" acq = model.get_acquisition()\n",
|
||||
"\n",
|
||||
" # only focus in the bounded region [0, 1]\n",
|
||||
" d_min = torch.zeros((1, x.shape[-1]))\n",
|
||||
" d_max = torch.ones((1, x.shape[-1]))\n",
|
||||
" # transform it\n",
|
||||
" d_min_tr = model.transform_x(d_min)\n",
|
||||
" d_max_tr = model.transform_x(d_max)\n",
|
||||
" bounds = torch.cat([d_min_tr, d_max_tr], dim=0)\n",
|
||||
"\n",
|
||||
" # find the maximum of the acquisition function\n",
|
||||
" next_parameter, _ = optimize_acqf(\n",
|
||||
" acq_function=acq,\n",
|
||||
" bounds=bounds,\n",
|
||||
" q=1,\n",
|
||||
" num_restarts=10,\n",
|
||||
" raw_samples=512,\n",
|
||||
" )\n",
|
||||
" # transform the proposed result back to the original space\n",
|
||||
" next_parameter = model.untransform_x(next_parameter)\n",
|
||||
" # return the parameter and the fit result, in case we want to plot it\n",
|
||||
" return next_parameter, model\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "edb4bacf-34d5-4c66-b66c-9910d63ef434",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Finally, let's make an auxiliary plot function to show what is happening:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"id": "5eee5e53-6382-49b1-9be2-09544abdefb7",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def plot(x: Tensor, y: Tensor, model: FitModel):\n",
|
||||
" \"\"\"\n",
|
||||
" Plot the observed x and y, and the fit result.\n",
|
||||
" \"\"\"\n",
|
||||
" # get a set of x-axis points from 0 to 1 for plotting\n",
|
||||
" x_axis = torch.from_numpy(np.linspace(0.0, 1.0, 200)).double().unsqueeze(1)\n",
|
||||
" \n",
|
||||
" # get the true function value\n",
|
||||
" true_y = true_f(x_axis)\n",
|
||||
" \n",
|
||||
" # get the prediction from the fit\n",
|
||||
" m, s = model.predict(x_axis)\n",
|
||||
" \n",
|
||||
" # get the value of the acquisition function:\n",
|
||||
" ei = model.get_acquisition()(model.transform_x(x_axis).unsqueeze(1))\n",
|
||||
" \n",
|
||||
" # transform the results into numpy objects and drop the extra dimension if needed\n",
|
||||
" m = m.detach().cpu().numpy()[0,:]\n",
|
||||
" s = s.detach().cpu().numpy()[0,:]\n",
|
||||
" true_y = true_y.detach().cpu().numpy()\n",
|
||||
" ei = ei.detach().cpu().numpy()\n",
|
||||
" x_axis = x_axis.cpu().numpy()[:,0]\n",
|
||||
" \n",
|
||||
" # plot it\n",
|
||||
" f, ax = plt.subplots(2, 1, figsize=(12, 8))\n",
|
||||
"\n",
|
||||
" # first the true function so we can compare things with it\n",
|
||||
" ax[0].plot(x_axis, true_y, 'g--', label=r\"True $f$\", lw=2)\n",
|
||||
" # now the fit result\n",
|
||||
" ax[0].plot(x_axis, m, 'b-', label=\"Mean\", lw=2)\n",
|
||||
" # now the uncertainty of the fit\n",
|
||||
" ax[0].fill_between(x_axis, (m - 1*s), (m + 1*s), alpha=0.3, label=\"68% CL\")\n",
|
||||
"\n",
|
||||
" # plot the actual points we collected\n",
|
||||
" ax[0].plot(x.cpu().numpy()[:,0], y.cpu().numpy(), 'k^', label=\"Observation\", markersize=8)\n",
|
||||
"\n",
|
||||
" # highlight the last point we collected\n",
|
||||
" ax[0].plot(x.cpu().numpy()[-1:, 0], y.cpu().numpy()[-1:], 'ro', label=\"Last observation\", markersize=10)\n",
|
||||
"\n",
|
||||
" # make the plot look pretty\n",
|
||||
" ax[0].set(xlabel=\"\", ylabel=r\"$f(x)$\", title=\"\", xlim=(0.0, 1.0))\n",
|
||||
" ax[0].set_xticks([])\n",
|
||||
" ax[0].legend(loc='upper left', frameon=False, bbox_to_anchor=(1.0,1), borderaxespad=0)\n",
|
||||
"\n",
|
||||
" # now show the acquision function in the bottom plot\n",
|
||||
" ax[1].plot(x_axis, ei, 'k-', lw=2, label=r\"$\\mathcal{A}$\")\n",
|
||||
" ax[1].set(xlabel=r\"$x$\", ylabel=r\"Acquisition function\", title=\"\", xlim=(0.0, 1.0))\n",
|
||||
"\n",
|
||||
" # find the maximum of the acquision function and show some dotted lines to indicate it in both plots\n",
|
||||
" idx_next = np.argmax(ei)\n",
|
||||
" x_next = x_axis[idx_next]\n",
|
||||
" ax[1].axvline(x_next, 0, 1, c='r', ls=':', lw=2, label=r\"Next $\\vec{x}$\")\n",
|
||||
" ax[0].axvline(x_next, 0, 1, c='r', ls=':', lw=2)\n",
|
||||
" ax[1].legend(loc='upper left', frameon=False, bbox_to_anchor=(1.0,1), borderaxespad=0)\n",
|
||||
"\n",
|
||||
" # prettify\n",
|
||||
" plt.tight_layout()\n",
|
||||
" plt.subplots_adjust(\n",
|
||||
" #left=0.05,\n",
|
||||
" #bottom=0.05,\n",
|
||||
" right=0.80,\n",
|
||||
" top=0.99,\n",
|
||||
" #wspace=0.4,\n",
|
||||
" hspace=0\n",
|
||||
" )"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "c7db2a9b-2d9e-4182-b558-3c1d35be17d4",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Now, let's use the initial x and y values to make the fit itself:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 11,
|
||||
"id": "8f7ebf8a-7503-4b2c-a544-691f1f13271f",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"/home/daniloefl/workspace/ml-lecture-oct-2023/env/lib/python3.9/site-packages/botorch/models/utils/assorted.py:173: InputDataWarning: Input data is not contained to the unit cube. Please consider min-max scaling the input data.\n",
|
||||
" warnings.warn(msg, InputDataWarning)\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"next_parameter, model = update_params(x0, y0)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 12,
|
||||
"id": "edec7a8e-ec9f-48cb-a725-44a7d56f1456",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"image/png": "iVBORw0KGgoAAAANSUhEUgAABGoAAAMcCAYAAAAIa0NUAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAAEAAElEQVR4nOzdd3iTZfvG8e+TtEn3YlPKEmTvDSqoKCgO9BUVUcSBCjgQt68bBRe4t6/gnigOfspQQRmyBNl7lNEyOtKZ/fz+CBQqG9ombc/PceRI8uQZV4NIcva+r9swTdNERERERERERESCzhLsAkREREREREREJEBBjYiIiIiIiIhIiFBQIyIiIiIiIiISIhTUiIiIiIiIiIiECAU1IiIiIiIiIiIhQkGNiIiIiIiIiEiIUFAjIiIiIiIiIhIiwoJdQLD4/X527txJbGwshmEEuxwREREREREpIaZpkpubS+3atbFYND5BypdKG9Ts3LmTlJSUYJchIiIiIiIipWTbtm3UqVMn2GWInJBKG9TExsYCgb+4cXFxQa5GRERERERESkpOTg4pKSlF3/tEypNKG9Tsn+4UFxenoEZERERERKQCUpsLKY80WU9EREREREREJEQoqKloGjYEuz1wLyIiIiIiIiLlioKaisbtPnATERERERERkXKl0vaoqbBatoTq1QM3ERERERERESlXFNRUNL/8EuwKREREREREROQkaeqTiIiIiIiIiBzRa6+9Rr169QgLC+Pee+8NdjkVnoIaERERERERkRBgGMZRb0888USZ1/TPP/8watQo3nrrLbZt28ZTTz1V5jVUNpr6JCIiIiIiIhIC0tLSih5/+eWXPPbYY6xdu7ZoW0xMzCHHuN1ubDZbqdX0008/0blzZy688MJSu4YUpxE1Fc1998HNNwfuRUREREREpNyoWbNm0S0+Ph7DMIpti4mJoVevXtx+++2MHDmSqlWr0qdPHwDq16/Pyy+/XOx8bdu2LRqF4/f7GTt2LA0aNCAyMpI2bdrwzTffHLWeRo0a8cgjjzB37lwMw2Dw4MGl8WPLv2hETUXz+eewYwckJ8MLLwS7GhERERERkZAyft54xs8bf8z92tdqzw8Dfyi27ZLPL+HvtL+PeeyobqMY1W3USdd4LB9++CHDhg1jzpw5x33M2LFj+eSTT3j77bdp3Lgxf/zxB9deey3VqlWjZ8+ehz1m7ty5dOvWjWHDhnHttdcedkSPlDwFNSIiIiIiIlJp5Lhy2JG745j7pcSnHLJtT8Ge4zo2x5VzUrUdr8aNG/P8888f9/4ul4sxY8YwY8YMunXrBkDDhg2ZPXs277zzzhGDmpiYGLZs2cIZZ5xBzZo1S6R2OTYFNRXNr7+C1wth+qMVERERERH5tzh7HMmxycfcr1pUtcNuO55j4+xxJ1Xb8erQocMJ7b9hwwYKCgo477zzim13u920a9fuiMctW7YMgFatWp14kXLS9G2+omnSJNgViIiIiIiIhKxTmZb076lQwRIdHX3INovFgmmaxbZ5PB4A8vLyAJgyZQrJycWDJrvdfsTrLF26lEaNGh32elJ6FNScAI/Pj9dn4vX78fvBZ5r4/OYhfxkgsKxamMXAYjGwWgKPbVYLFosRhMpFRERERESkIqtWrVqxVaNycnLYvHkzAM2bN8dut5OamnrEaU6Hs3TpUtq0aVPitcrRKagBTNOk0OOjwO3D6fHh9Phxeny4vH7cXj8eX+B2mDzmhFmtgcDGFmYhIsxKRLiFiHArkTYrUTYrkeFWDENhjoiIiIiIiBy/c845h4kTJ3LxxReTkJDAY489htVqBSA2NpZ7772Xu+++G7/fzxlnnIHD4WDOnDnExcVx/fXXH/acS5cu5ZJLLinLH0NQUMP8jRlYIgpLJIQ5Hj6fSaHPR6HbhwPPIa9bLQZRNivR9jBiI8KIiwgnLjIc6/GOxJk3D1wusNthX5MoERERERERqdgeeughNm/ezEUXXUR8fDyjR48uGlEDMHr0aKpVq8bYsWPZtGkTCQkJtG/fnocffviw5/P7/SxfvpxHH320rH4E2ccwDzdvpxLIyckhPj6eyfPXEx0TG+xyjsowINoeCG0So8NJjLIREW49/M516hxYnnv79rItVEREREREJATs/77ncDiIiyvdxr4iJa3Sj6gpD0wT8pxe8pxedmYXAhBlt1Il2k5StI2kaNvxj7gRERERERERkZCloKacKnD5KHAVsC2zAKvFICnaRrVYOzWGDcealwtKjUVERERERETKHQU1FYDPb7In18WeXBerL7+JxGgbteIjqO43NdJGREREREREpBxRUFPBmCZk5rnJzHOzxpJLtVg7teIjqBJjD3ZpIiIiIiIiInIMCmoqMJ/fJN3hJN3hJMpmJTkxktoJkYRbLcEuTUREREREREQOQ0FNJVHg9rF+Vx4b9+RRIy6CuklRxEaEB7ssERERERERETmIgpoKpvOV52Pbuwd31Wos+GraIa/7/ZCW7SQt20lSjI36VaJJirYFoVIRERERERER+TcFNRWMbe8eInalHde++3vZxEWGU79KFNXjIkq5OhERERERERE5GgU1FYy7arVi98cjp9DDsu0OYiLyOa1aDNVi1XhYREREREREJBjUVbaCWfDVNGb/tuSw056OJc/p5Z9t2SzckklGnqsUqhMREREREZGjGTJkCIZhcNtttx3y2ogRIzAMgyFDhpR9YVJmFNTIIRwFHpakZvN3ahZ5Lm+wyxEREREREalUUlJS+OKLLygsLCza5nQ6+eyzz6hbt24QK5OyoKBGjigzz838TRmsTsvB7fUHuxwREREREZFKoX379qSkpPDtt98Wbfv222+pW7cu7dq1K9rm9/sZO3YsDRo0IDIykjZt2vDNN98Uve7z+bjpppuKXm/SpAmvvPJKsWsNGTKE/v378+KLL1KrVi2qVKnCiBEj8Hg8pf+DymGFRFAzduxYOnXqRGxsLNWrV6d///6sXbv2qMdMnDgRwzCK3SIi1Ay3pJkm7MgqZO7GvWzNyMc0zWCXJCIiIiIiUuHdeOONTJgwoej5Bx98wA033FBsn7Fjx/LRRx/x9ttvs3LlSu6++26uvfZaZs2aBQSCnDp16vD111+zatUqHnvsMR5++GG++uqrYuf5/fff2bhxI7///jsffvghEydOZOLEiaX+M8rhhUQz4VmzZjFixAg6deqE1+vl4Ycf5vzzz2fVqlVER0cf8bi4uLhigY5hGGVRbkir/96rWPNy8cXEsmXonSV2Xq/PZP2uPNIcTprVjCM+KrzEzi0iIiIiIlIWOnaE9PSyvWbNmrBo0Ykfd+211/LQQw+xdetWAObMmcMXX3zBzJkzAXC5XIwZM4YZM2bQrVs3ABo2bMjs2bN555136NmzJ+Hh4Tz55JNF52zQoAHz5s3jq6++4sorryzanpiYyOuvv47VaqVp06b069ePX3/9laFDh578Dy4nLSSCml9++aXY84kTJ1K9enUWL17MWWeddcTjDMOgZs2ax3UNl8uFy3WgQW5OTs7JFRvi6nw+gYhdaThr1CrRoGa/PKeXhVsySU6MpFH1GMKtITEoS0RERERE5JjS02HHjmBXcXyqVatGv379mDhxIqZp0q9fP6pWrVr0+oYNGygoKOC8884rdpzb7S42PeqNN97ggw8+IDU1lcLCQtxuN23bti12TIsWLbBarUXPa9WqxfLly0vnB5NjComg5t8cDgcASUlJR90vLy+PevXq4ff7ad++PWPGjKFFixaH3Xfs2LHFkkQ5NTuyCtmT66JpzViqx2nKmYiIiIiIhL7j/D1/yFzzxhtv5PbbbwcCgcvB8vLyAJgyZQrJycnFXrPb7QB88cUX3HvvvYwbN45u3boRGxvLCy+8wPz584vtHx5efMaEYRj4/epTGiwhF9T4/X5GjhxJjx49aNmy5RH3a9KkCR988AGtW7fG4XDw4osv0r17d1auXEmdOnUO2f+hhx5i1KhRRc9zcnJISUkplZ8hmJaPfw+L24XfZi/1a7m9fpZtd1AjzkWTmrHYwjS6RkREREREQtfJTEEKpr59++J2uzEMgz59+hR7rXnz5tjtdlJTU+nZs+dhj58zZw7du3dn+PDhRds2btxYqjXLqQu5oGbEiBGsWLGC2bNnH3W/bt26Fc3DA+jevTvNmjXjnXfeYfTo0Yfsb7fbi1LFiszRtmOZX3NXjpOsAjdNa8VSPVaja0REREREREqC1Wpl9erVRY8PFhsby7333svdd9+N3+/njDPOwOFwMGfOHOLi4rj++utp3LgxH330EVOnTqVBgwZ8/PHHLFy4kAYNGgTjx5HjFFJBze23385PP/3EH3/8cdhRMUcTHh5Ou3bt2LBhQylVJ0fj9vpZts1BzfjAdKgw9a4RERERERE5ZXFxcUd8bfTo0VSrVo2xY8eyadMmEhISaN++PQ8//DAAt956K0uWLOGqq67CMAwGDhzI8OHD+fnnn8uqfDkJhhkC6y2bpskdd9zBd999x8yZM2ncuPEJn8Pn89GiRQsuvPBCxo8ff8z9c3JyiI+PZ/L89UTHxJ5M2XIEUTYrLZLjiY/UylAiIiIiIlL29n/fczgcRw06REJRSIyoGTFiBJ999hnff/89sbGxpO9bLy0+Pp7IyEgABg8eTHJyMmPHjgXgqaeeomvXrjRq1Ijs7GxeeOEFtm7dys033xy0nyMURG3egOHzYlrDKGjQKCg1FLh9LN6aScOqMdSveuTl1UVERERERESkuJAIat566y0AevXqVWz7hAkTGDJkCACpqalYLAem02RlZTF06FDS09NJTEykQ4cOzJ07l+bNm5dV2SGp/U0Dipbnnv3bkqDV4ffDht15ZBa4aVE7DnuY9dgHiYiIiIiIiFRyITH1KRgq6tSnM85pFxJBzcHs4RZaJycQH6WpUCIiIiIiUvo09UnKs5AYUSMlJ/3C/oTnOPDExQe7lCILine truncated
|
||||
"text/plain": [
|
||||
"<Figure size 1200x800 with 2 Axes>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"plot(x0, y0, model)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "38515339-77f1-4981-8a7e-7c2bc11191ab",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Notice that the model fit is quite close to the actual fit function. We didn't have to guess the actual function shape itself. The uncertainty band is also calculated and the maximum of the acquisition function, calculated using the fit result gives us the next point to probe.\n",
|
||||
"\n",
|
||||
"Let's probe the next point x in the `next_parameter` and add it to our table of values known. After that we can just repeat the process above."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 13,
|
||||
"id": "ae25ddb3-113f-4d7b-b640-9913260b542f",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"next_y = true_f(next_parameter)\n",
|
||||
"\n",
|
||||
"x1 = torch.cat([x0, next_parameter], axis=0)\n",
|
||||
"y1 = torch.cat([y0, next_y], axis=0)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 14,
|
||||
"id": "3c6ac772-1f48-4685-8d06-033aa8d00858",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"/home/daniloefl/workspace/ml-lecture-oct-2023/env/lib/python3.9/site-packages/botorch/models/utils/assorted.py:173: InputDataWarning: Input data is not contained to the unit cube. Please consider min-max scaling the input data.\n",
|
||||
" warnings.warn(msg, InputDataWarning)\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"next_parameter, model = update_params(x1, y1)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 14,
|
||||
"id": "634e111f-71b4-482a-923a-62a4d3e8f0d9",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"image/png": "iVBORw0KGgoAAAANSUhEUgAABGoAAAMfCAYAAACO/zH6AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAAEAAElEQVR4nOzdd3xTZfvH8U+S7j0YhVL2RjaCgD44UEAcuPWHIqC4ByIOfERQFJwIKooLwYkDwYECggKPDJmVIXsVSlsKHelOmpzfH4FApYW2tE3H9/16xZOcc5+TKy0Uz9Xrvm6TYRgGIiIiIiIiIiLicWZPByAiIiIiIiIiIi5K1IiIiIiIiIiIVBJK1IiIiIiIiIiIVBJK1IiIiIiIiIiIVBJK1IiIiIiIiIiIVBJK1IiIiIiIiIiIVBJK1IiIiIiIiIiIVBJK1IiIiIiIiIiIVBJK1IiIiIiIiIiIVBJK1IiIiIiIiIiIVBJeng6gNJYvX85rr73G+vXrSUhIYO7cuQwaNKjI8UOHDmXWrFmn7W/bti1bt24FYPz48Tz//PMFjrdq1Yrt27cXOy6n08nhw4cJDg7GZDIV+zwREREREREpO4ZhkJGRQf369TGbVZ8gVUuVTNRkZWXRsWNHhg8fzvXXX3/W8VOnTuXll192v87Pz6djx47cdNNNBca1a9eOxYsXu197eZXsy3P48GFiYmJKdI6IiIiIiIiUj4MHD9KgQQNPhyFSIlUyUTNgwAAGDBhQ7PGhoaGEhoa6X8+bN4/U1FSGDRtWYJyXlxdRUVGljis4OBhw/TAICQkp9XVERERERESk9KxWKzExMe57NJGqpEomas7Vxx9/TN++fWnUqFGB/bt27aJ+/fr4+fnRs2dPJk2aRMOGDYu8Tl5eHnl5ee7XGRkZAISEhChRIyIiIiIi4mFqSSFVUY2brHf48GF+/fVX7r777gL7e/TowcyZM1mwYAHvvfce+/bt46KLLnInXwozadIkd7VOaGiopj2JiIiIiIiIyDmpcYmaWbNmERYWdlrz4QEDBnDTTTfRoUMH+vXrxy+//EJaWhrffPNNkdcaM2YM6enp7sfBgwfLOXqpMpo2BV9f11ZERERERESkmGrU1CfDMJgxYwZ33HEHPj4+ZxwbFhZGy5Yt2b17d5FjfH198fX1LeswpTqw2U4+RERERERERIqpRlXULFu2jN27d3PXXXeddWxmZiZ79uyhXr16FRCZVDvnnQedO7u2IiIiIiIiIsVUJRM1mZmZxMbGEhsbC8C+ffuIjY0lLi4OcE1JGjJkyGnnffzxx/To0YPzCrl5Hj16NMuWLWP//v2sXLmS6667DovFwm233Vaun0WqqQULYMMG11ZERERERKQKe/vtt2nUqBFeXl6MHj3a0+FUe1Vy6tO6deu45JJL3K9HjRoFwJ133snMmTNJSEhwJ21OSE9PZ86cOUydOrXQax46dIjbbruNY8eOUbt2bS688EJWr15N7dq1y++DiIiIiIiIiBx3tlWqxo0bx/jx4ysmmOP+/vtvRo0axQ8//EDnzp0JDQ2t0PeviUyGYRieDqK6sFqthIaGkp6eruW5RUREREREPKSq3pslJia6n3/99dc899xz7Nixw70vKCiIoKCgAufYbLaz9mA9Fy+99BK//PILK1asKLf3kIKq5NQnERERERERkeomKirK/QgNDcVkMhXYFxQUxMUXX8xDDz3EyJEjqVWrFv369QOgcePGTJkypcD1OnXq5K7AcTqdTJo0iSZNmuDv70/Hjh357rvvzhhP8+bNefbZZ1m5ciUmk6nQFiNS9qrk1CeRSu+JJyA1FcLD4bXXPB2NiIiIiIhUI7NmzeL+++8vUZXLpEmT+Pzzz5k+fTotWrRg+fLl3H777dSuXZs+ffoUes7KlSvp2bMn999/P7fffvtp1TxSPpSoESkPX30F8fEQHa1EjYiIiIhIJTJ51WQmr5p81nFd6nXhx9t+LLDvmq+uYUPChrOeO6rnKEb1HFXqGM+mRYsWvPrqq8Uen5eXx8SJE1m8eDE9e/YEoGnTpvz555+8//77RSZqgoKC2L9/PxdeeCFRUVFlErucnRI1IiIiIiIiUmNY86zEZ8SfdVxMaMxp+5Kzk4t1rjXPWqrYiqtr164lGr97926ys7O5/PLLC+y32Wx07ty5yPM2bdoEQPv27UsepJSaEjUi5WHJEsjPBy/9FRMRERERqUxCfEOIDo4+67jaAaevAFw7oHaxzg3xLd8GxoGBgaftM5vN/HutILvdDkBmZiYA8+fPJzq6YPy+vr5Fvk9sbCzNmzcv9P2k/OguUuQs0nPT2XlsJ7tSdrHr2C52p+4mLTcNPy8/ArwD8Pfyp1ZALV689MWTJ7Vq5bmARURERESkSOcyLenfU6Eqk9q1a5OQkOB+bbVa2bdvHwBt27bF19eXuLi4Iqc5FSY2NpaOHTuWeaxyZkrUiJzB0HlD+WzTZzgN5xnHXdDgAl7kxQL7Vh9aTYOQBjQIaVCeIYqIiIiIiHDppZcyc+ZMrr76asLCwnjuueewWCwABAcHM3r0aB577DGcTicXXngh6enprFixgpCQEO68885CrxkbG8s111xTkR9DUKJG5IzqBtY9a5IGoEd0jwKvDcNg2A/D2HlsJ/2a9ePuLndzVcur8LH4lFeoIiIiIiJSg40ZM4Z9+/Zx1VVXERoayoQJE9wVNQATJkygdu3aTJo0ib179xIWFkaXLl145plnCr2e0+lk8+bNjB07tqI+ghxnMv49iU1KzWq1EhoaSnp6OiEh5TsnUcqOw+lg9pbZvLXmLRbevpAwvzD3sbXxa7l97u30adSHFhEtaBHZghYRLagTWIfc/Fyy7dnk5OcQFRRFVNDJLuib5n3AI/PuJc8LVh/vQVYnsA5P936aB85/AF+voueBioiIiIjIudG9mVRlStSUIf0wqFoMw2DOtjk898dzbDu6DYCx/xnLC5e8UGAMgMlkKtG1HdH1sRxOICHUQv3HHAWONQ1vyit9X+GGNjeU+LoiIiIiInJ2ujeTqszs6QBEPOFo9lFu+OYGbvr2JneSBuDvpL8LdEo3mUylSqZYTK6/WlFBUSy6fRE3tr0RE67r7E3dy03f3kTvGb1ZG7/2HD+JiIiIiIiIVCeqqClDytpWDb/s+oXhPwwnKSvJva93TG8mXDKBS5pcUjZvMmkSWK0QEgJjxgCwMWEjjy96nD/2/+Ee1q1+N9bcvUaVNSIiIiIiZUj3ZlKVKVFThvTDoHLLtmfz+MLHmb5+untfpH8k71/1Pte3ub5CkiWGYTB/13ye+O0JDqQdYMO9G2hdq3W5v6+IiIiISE2iezOpyrTqk9QIefl59JnZh3WH17n3DWg+gBnXzijQBLi8mUwmrmp5Ff2a9ePvpL9PS9Jk2jIJ8gmqsHhERERERESkclGPGqkRfL18ubrl1QAEeAfw3sD3mP9/8ys0SXMqb4s33ep3K7Avx55Dr4978dwfz6FCNxERERERkZpJFTVSY4z9z1jy8vO4rf1tnFfnPE+Hc5onf3uSzUc2s/nIZg6kH+DDqz/Ex+Lj6bBERERERESkAqmiRqqtwxmHC7w2mUy8dNlLFZOk6dYNGjRwbYupeURz98pQn/79KQO/HEh6bnp5RSgiIiIiIiKVkBI1Ui3N2z6P5m815/d9v3smgMREiI93bYvp0Qse5dubvsXPyw+AxXsXc9EnF3HIeqi8ohQREREREZFKRokaqXaW7F3CLd/dQk5+Dtd9fR0H0g5UfBBRURAd7dqWwA1tb2DJkCVE+kcCsPnIZvrM7EO8Nb48ohQREREREZFKRokaqVZWH1rNtbOvxeawAXBtq2uJCY0p9/c1DIN8h5O8fAe5dgfZK1eTt/8ArFt39pP/pVdML1bdtYpm4c0A2Ju6l0s/vZTEzOJX54iIiIiISNU0dOhQTCYT991332nHHnzwQUwmE0OHDq34wKTCqJmwVBubkzZz5RdXkmXPAlxJmhnXzsBsKtt8pGEYpOfYsebkk5FnJyM3n6y8fApbqMliNuHnbcHfx0KQr4XIQF/CArwxmUxnfI8WkS1YNnQZfWb2YU/qHnYe28lln17G0juXUjuwdpl+HhERERERqVxiYmKYPXs2b775Jv7+/gDk5uby5Zdf0rBhQw9HJ+VNFTVSLexL3ccVn19Bam4qAJc2uZTZN87Gy1x2uci0bBvbEqws33WUdftT2ZmUQUJaLpm5hSdpABxOg6y8fI5m5LH/aDbrD6SybGcymw+lk5iei9NZ9DLc0SHR/H7n7zQKbQRAncA6+Hv7l9nnERERERGRyqlLly7ExMTw/fffu/d9//33NGzYkM6dO7v3OZ1OJk2aRJMmTfD396djx45899137uMOh4O77rrLfbxVq1ZMnTq1wHsNHTqUQYMG8frrr1OvXj0iIyN58MEHsdvt5f9BpVCqqJEqL8uWxbWzr3VPDeoe3Z15t8xzN+U9F/kOJwdTc4hPzSHX7jjn67muaZBkzSXJmouPl5mYiAAahPvjbTk9b9owtCG/3/k7Lyx7gXcHvkuAd0CZxCAiIiIiIpXb8OHD+eSTTxg8eDAAM2bMYNiwYSxdutQ9ZtKkSXz++edMnz6dFi1asHz5cm6//XZq165Nnz59cDqdNGjQgG+//ZbIyEhWrlzJPffcQ7169bj55pvd1/njjz+oV68ef/zxB7t37+aWW26hU6dOjBgxoqI/tgAmwyiqFkBKymq1EhoaSnp6OiEhIZ4Op0YwDINb59zKN1u/AaBlZEtW3bWKCP+Ic7quw2kQl5LNgWNZ5DtK/lek8YdvYcnMwBEUzP4Rj5x1vMVson6YP40iA/DztpQmZBEREREROa6oe7Nu3Uq0MGuZiIoqWevKoUOHkpaWxocffkhMTAw7duwAoHXr1hw8eJC7776bsLAw3n//fSIiIli8eDE9e/Z0n3/33XeTnZ3Nl19+Wej1H3roIRILine truncated
|
||||
"text/plain": [
|
||||
"<Figure size 1200x800 with 2 Axes>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"plot(x1, y1, model)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "12bd6504-355a-4b13-b35f-246b9d555c1e",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"This can be repeated many times until we are very confident we are close enough to the maximum:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 15,
|
||||
"id": "f1d29b5d-c0f6-43fc-8a19-57101ab8a783",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"/home/daniloefl/workspace/ml-lecture-oct-2023/env/lib/python3.9/site-packages/botorch/models/utils/assorted.py:173: InputDataWarning: Input data is not contained to the unit cube. Please consider min-max scaling the input data.\n",
|
||||
" warnings.warn(msg, InputDataWarning)\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"image/png": "iVBORw0KGgoAAAANSUhEUgAABGoAAAMeCAYAAABFo+JfAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAAEAAElEQVR4nOzdd3xT1f/H8VeStuluKRTKKHsIsqeIiigIiCgOhoIMFRcOxAlfURQVFygqiqIsQUFF+eECBASUIcvKENmztFBaukfa5P7+CARC2bRN076fj0ceN/fcc28+SUNpPjnnc0yGYRiIiIiIiIiIiIjHmT0dgIiIiIiIiIiIOClRIyIiIiIiIiJSTChRIyIiIiIiIiJSTChRIyIiIiIiIiJSTChRIyIiIiIiIiJSTChRIyIiIiIiIiJSTChRIyIiIiIiIiJSTChRIyIiIiIiIiJSTChRIyIiIiIiIiJSTPh4OoCSxOFwcOjQIUJCQjCZTJ4OR0REREREpFQyDIO0tDQqVaqE2azxCeJdvDJRs3z5ct555x3Wr19PXFwcP/zwAz169Dhr/4EDBzJt2rR87Q0aNGDLli0AjBo1ildeecXteL169fjvv/8uOK5Dhw4RHR19wf1FRERERESk8Bw4cIAqVap4OgyRi+KViZqMjAyaNGnCfffdxx133HHe/uPHj+fNN9907efl5dGkSRN69uzp1u/KK69k0aJFrn0fn4t7eUJCQgDnL4PQ0NCLOldEREREREQKRmpqKtHR0a7PaCLexCsTNV27dqVr164X3D8sLIywsDDX/ty5czl27BiDBg1y6+fj40NUVNQlx3ViulNoaKgSNSIiIiIiIh6mkhTijUrlZL0vvviCjh07Uq1aNbf2HTt2UKlSJWrWrEnfvn3Zv3//Oa+Tk5NDamqq201ERERERERE5FKVukTNoUOH+PXXX3nggQfc2tu0acPUqVOZP38+n3zyCXv27OHaa68lLS3trNcaM2aMa7ROWFiY6tOUZjVrgtXq3IqIiIiIiIhcolKXqJk2bRrh4eH5ig937dqVnj170rhxYzp37swvv/xCcnIy33zzzVmvNXz4cFJSUly3AwcOFHL0UmzZbCdvIiIiIiIiIpfIK2vUXCrDMJg8eTL33nsvfn5+5+wbHh5O3bp12blz51n7WK1WrFZrQYcp3qhhQyhf3nkTERERERERuUSlakTNsmXL2LlzJ/fff/95+6anp7Nr1y4qVqxYBJGJ15s/HzZscG5FRERERERKkA8//JBq1arh4+PDM8884+lwSjyvTNSkp6cTExNDTEwMAHv27CEmJsZV/Hf48OH0798/33lffPEFbdq0oWHDhvmOPfPMMyxbtoy9e/eycuVKbr/9diwWC3fffXehPhcRERERERERcK5Sda7bqFGjijymf/75h2HDhvHJJ59w4MABXn311SKPobTxyqlP69ato0OHDq79YcOGATBgwACmTp1KXFxcvhWbUlJSmDNnDuPHjz/jNQ8ePMjdd99NYmIikZGRXHPNNaxevZrIyMjCeyIiIiIiIiIix8XFxbnuz549m5deeolt27a52oKDg/OdY7PZzlva43L89NNPtG7dmptvvrnQHkPceWWi5vrrr8cwjLMenzp1ar62sLAwMjMzz3rOrFmzCiI0ERERERERkUsSFRXluh8WFobJZHJrA+fn4YYNG+Lj48OMGTNo1KgRv//+O9WrV2fo0KEMHTrU1bdp06b06NGDUaNG4XA4eOutt/jss8+Ij4+nbt26jBw5krvuuuus8dSuXZtdu3YBztE+9957L9OnTy/YJy35eGWiRqTYefZZOHYMypSBd94Bw4DEREhPh+BgKFsWTCZPRykiIiIiIiXAtGnTeOSRR1ixYsUFnzNmzBhmzJjBxIkTqVOnDsuXL6dfv35ERkbSvn37M56zcuVK2rZtyyOPPEK/fv3OOKJHCp4SNSIF4euvITYWKlWCKlXgww/heOYZgFq14PHHYcAACA/3WJgiIiIiIqXduFXjGLdq3Hn7Na/YnHl3z3Nru/XrW9kQt+G85w5rO4xhbYddcoznU6dOHd5+++0L7p+Tk8Mbb7zBokWLaNu2LQA1a9bkzz//5NNPPz1roiY4OJi9e/dyzTXX5BvZI4VHiRqRghQXB089lb99925n+//+B3PmQOfORR+biIiIiIiQmpNKbFrseftFh0Xna0vITLigc1NzUi8ptgvVokWLi+q/c+dOMjMz6dSpk1u7zWajWbNmZz1v48aNADRq1Ojig5RLpkSNSEEYNQoeesg55elM9ZNOtGVlQbdu8PPPStaIiIiIiHhAqDWUyiGVz9svMjD/wjKRgZEXdG6oNfSSYrtQQUFB+drMZnO+Wq65ubmAc+VkgJ9//pnKld3jt1qtZ32cmJgYateufcbHk8KjRI3I5UpOhhMFu85R5BoAhwPMZrjzTjh4UNOgRERERESK2OVMSzp9KlRxEhkZ6bZqVGpqKnv27AGgQYMGWK1W9u/ff9ZpTmcSExNDkyZNCjxWOTezpwMQKc6SspKYtXkWx7KOubUv2rWE4DeCKfd2JC8PrIEjI8OZhLkQDgdkZoKqpYuIiIiISAG54YYb+PLLL/njjz/YtGkTAwYMwGKxABASEsIzzzzDU089xbRp09i1axcbNmzgww8/ZNq0aWe9ZkxMDE2bNi2iZyAnaESNyGkOph5k7n9z+eG/H1i2dxl2w868XgtpFHk1KVm5HE7K5Yc/c8hYfxsZR+tw7+qJgAk4z2ia033wgbPAsFaDEhERERGRyzR8+HD27NnDLbfcQlhYGKNHj3aNqAEYPXo0kZGRjBkzht27dxMeHk7z5s0ZMWLEGa/ncDjYtGkTI0eOLKqnIMeZjNMnscklS01NJSwsjJSUFEJDC3dOohS81QdX88qyV5i/c36+Y0MafETo3n78sdDK2j+t5GQ7kytlOcpR8s9dvWBHjzqX7hYRERERkQKjz2bizTSiRkq9VQdW8cqyV1iwa0G+Y2WP3In1rxf59I3G5NnyzxQMJv2yHjv9aBLBStSIiIiIiIjIcUrUSKl27w/3MmPjDLe2CoFVaGZ7jn3/15etayPynRMe4aD1dTlUr51H3Ug7PH/pj3/HfUE8MjqVG68OINTf99IvJCIiIiIiIiWCEjVSqt1S5xZXoiYqMJqO/m/w3+y7mL/C361fRKSdazvlcO1N2bRpa1Alwp/wQCuh1hrwWS3YteuiHteBid3U5LeVFVl8E9zaJ4snn8ukZb0ggq36ZykiIiIiIlJa6ROhlFqGYdCqQjfuqDOY6KArObpgEF99FoLDfrK4b8XoPPo9nEHH7tlUCLcSHRFMRJCf+4UefxzjqacwXVS5J4MPrPdCjgmHHebODGTxjw76P5bO0Ccs1CofiElFhkVEREREREodFRMuQCpYVbzN/W8uC3ct5ONuH5OTZ2fLoVSS0m3s3Wnh7eFh7Pj35NSjqMp2+j6cTsfu2URH+lOjXBCBfmfJayYnk1exIqbsbCwXEIcdyPXzoeZTYcSteQT+fAFyg1zHm12Vw+j3M7i2SYimQ4mIiIiIXAJ9NhNvlr86qkgJYxgGY1eO5Y7Zd/DJuk+YumEWa/YkcTTVxrdTAnn0rrKuJI3ZYnDvo+lM/ukod96dx1V1ynBlpbCzJ2kAIyyMYVWrYuBMwpyLHeci3s9Wr8kfz/1F275L4PG60GSaq8/fq6306xbGxJnp7E64vGLFIiIiIiIi4l2UqJESLc+Rx6M/P8ozvz2DgXPw2Pf/zic1zcGY58L47N0QcnOdU4yq1szjg6+SGPR4JvWrBNOmRgThgX7nujwANpuNb1JS6AZkAY7jt1OdaMsCbga+S02lSlAVlg9axujbHsHnzgdgwPUQfAiA5EQLLwwOZ+SLsH5PMnn2068oIiIiIiIiJZGmPhUgDa8rXtJy0uj9XW9+3fmrq61/g2foVv5ZRj0ZztZ/nEkYk8ngzgGZDHoinYgwHxpXOfcImjOxNW4Mhw+TFx5O4q23EjlrFv4HD7qOZ1epQkKfPiTecguOkBDKly9PlSpVXMfXxq6lz5w+7D6QRuVFfxAbU891rGW7HN76JJ229cIJ8LuQyVUiIiIiIqWbPpuJN1OipgDpl0HxkZSVRMfpHfk7/m8AfMy+PN3yPWrn9ubFR8pw+JAz4eEf4GDEOym07WAjKsyf+hVDsZgvoYhvlSoQGwuVK8PBg2AYkJQEaWkQEgIREXCe4sBHM48y+e/J3Fn7Ud56x+CL94NdhY2vaJTLm5+lcF2jUMqcXsxYRERERETc6LOZeDMlagqQfhkUDynZKXT8siPrDq0DINQvnJeunoxjV3tGPRlOZrpzxl+5CnZGT0imToM86pQPoWrZwEt/0JYtIT4eoqJg3brLfg5H0rL56v+yePHRMDLSnPFG18jjrUnHuLF1COVD/M9zBRERERGR0kufzcSbqUaNlCgZtgy6fdXNlaQpG1CB9zr8iGNXe158tIwrSVOnQS4fzkriioZ2mlUtc3lJGnAmZw4eLJAkDUD5EH/uvyuYRz9YDCFxABzY48MTfSP48fd0jqRlF8jjiIiIiIhI8TJw4EBMJhMPP/xwvmNDhgzBZDIxcODAog9MiowSNVKi7E/Zz7bEbQCEWyN489rZJG9vwEuPlcGW45xGdFX7HMZOS6JSJWhRrQwRxXQqUWZeEhMO9YD72kJZ53M6etjCU/dGMO/3DCVrRERERERKqOjoaGbNmkVWVparLTs7m6+++oqqVat6MDIpCkrUSIlSP7I+ywYuo1Z4PcZcO5v0XY3438PhZGcdT9Jcn8NL7ycTHmamRbUyhPj7ejjis6sQXIGPun6Eqcx+uO8aqLQWgLRUM8/fH87CVekcSVWyRkRERESkpGnevDnR0dF8//33rrbvv/+eqlWr0qxZM1ebw+FgzJgx1KhRg4CAAJo0acJ3333nOm6327n//vtdx+vVq8f48ePdHmvLine truncated
|
||||
"text/plain": [
|
||||
"<Figure size 1200x800 with 2 Axes>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"next_y = true_f(next_parameter)\n",
|
||||
"\n",
|
||||
"x2 = torch.cat([x1, next_parameter], axis=0)\n",
|
||||
"y2 = torch.cat([y1, next_y], axis=0)\n",
|
||||
"\n",
|
||||
"next_parameter, model = update_params(x2, y2)\n",
|
||||
"\n",
|
||||
"plot(x2, y2, model)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "f3a7a81b-7191-477a-bfc9-e8f19abd3908",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "2ffb46d2-3374-48b0-b176-9dce934e079d",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.10.12"
|
||||
},
|
||||
"widgets": {
|
||||
"application/vnd.jupyter.widget-state+json": {
|
||||
"state": {},
|
||||
"version_major": 2,
|
||||
"version_minor": 0
|
||||
}
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
@@ -0,0 +1,309 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "1c0d6d14-a97f-41f2-8b3e-673399418107",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Basic statistics\n",
|
||||
"\n",
|
||||
"Here, the goal is to get used to several concepts in basic statistics needed later on for Machine Learning.\n",
|
||||
"We are going to generate a simple dataset ourselves and try to calculate some standard quantities to describe them.\n",
|
||||
"\n",
|
||||
"First, let's import some standard libraries and generate the dataset."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"id": "d5a29abd-6af1-4b18-8bec-8a8402b97dc7",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import matplotlib.pyplot as plt\n",
|
||||
"\n",
|
||||
"import pandas as pd\n",
|
||||
"import numpy as np\n",
|
||||
"import seaborn as sns"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"id": "f4bed5ac-0828-4e5f-a166-2e161557b49b",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"rng = np.random.RandomState(0)\n",
|
||||
"n_samples = 10000\n",
|
||||
"cov = [[1, 0.7*1*1.3, 0.3*1.3*1],\n",
|
||||
" [0.7*1*1.3, 1.3**2, 0],\n",
|
||||
" [0.3*1.3*1, 0, 1.5**2]]\n",
|
||||
"data = rng.multivariate_normal(mean=[1, -1, 0], cov=cov, size=n_samples)\n",
|
||||
"data = pd.DataFrame(data, columns=[\"x1\", \"x2\", \"x3\"])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "87d5cd39-4720-4eba-a268-0c38a561976a",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"First, make a scatter plot of this dataset using matplotlib to visualise what are the possible values of $x_1$, $x_2$ and $x_3$.\n",
|
||||
"One can make a 2D plot of $x_1$ versus $x_2$, $x_2$ versus $x_3$ or $x_1$ versus $x_3$ to understand how they relate to one another.\n",
|
||||
"\n",
|
||||
"Often this is cumbersome if one has too many variables to plot, but we are going to discuss methods to get around this later in the lecture.\n",
|
||||
"\n",
|
||||
"One tip: one may use `plt.scatter` for every pair of variables to do this, or use the seaborn module to plot all pairs in one go. Look for documentation on `sns.pairplot`."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"id": "7de735cf-3466-41e9-bcd5-fcc756958019",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<seaborn.axisgrid.PairGrid at 0x7f26ea177bb0>"
|
||||
]
|
||||
},
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAuUAAALlCAYAAABjOpj+AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAEAAElEQVR4nOyde3xT9f3/X7mn6S1p0gIFQltS6I1LuENSBETlLsr8/ixsaymKykWnUwQFAWGK021O0M0pVbev6L6bEwS8bIjTFkUQqrSlQAOVAi2Upk16yf3y+yM9pyc5lxREofB5Ph57TJqTk5Pk88l5f96f1/v1FgWDwSAIBAKBQCAQCATCVUN8tS+AQCAQCAQCgUC40SFBOYFAIBAIBAKBcJUhQTmBQCAQCAQCgXCVIUE5gUAgEAgEAoFwlSFBOYFAIBAIBAKBcJUhQTmBQCAQCAQCgXCVIUE5gUAgEAgEAoFwlSFBOYFAIBAIBAKBcJW5oYLyYDCI1tZWkH5JBMLVhcxFAuHagcxHAuHa4IYKytva2pCYmIi2trarfSkEwg0NmYsEwrUDmY8EwrXBDRWUEwgEAoFAIBAI1yIkKCcQCAQCgUAgEK4yJCgnEAgEAoFAIBCuMiQoJxAIBAKBQCAQrjIkKCcQCAQCgUAgEK4y0qt9AQQCgUAgXEvYHR40tXvQ6vIiIUYGXawciSr51b4sAoGGjNHrExKUEwgEAuGGhCuwcXj8WPHeEZTWNNHHTczUYdO8oUhVx1zFqyUQQtTbnHj8MscoCeavbUhQTiAQCIQbDr7AZslkAw6dbgk79ouaJqx87wg2FxgvOYAhQRDhSmJ3eFjjFujeGO1OME/G69WFBOUEAoFAuKEQCmz8wSCKzenYstfCeqyp3XNJAcoPyWgSCFw0tXtY45ZCaIx2J5jv8PjJeL3KkEJPAoFAINxQCAU2+yxWGPurOR9rc3mjntvu8OBkYztOXGjD4//8jjcIsjs8l3zdBEJrlDHIN0ajBfMNrS4yXq8BSFBOIBAIhBuKaIGN2xfg/Hu8Uib4vHqbE8veKcfNv/8cZ5odKLVYOY+jMppMqGC+vK4FJy+2kyCIwElClDHIN0ajjXmbw3tJ45Xw40DkKwQCgUC4oYgW2Cik7HzVxEwddHH80hVKHnDodAuWTTEgOV6BVxaMgFImweG6FpSU1cLh8dPHMzOaROZC6C66ODkmZurwBUfWW2iMRhvzsXKJ4OPd2SXig+jUuw8JygkEAoFwXREtCBAKbPIzdWhsc4f9bWKmDs/NGyoYSDS1e3DodAteKjDijX21YZp0k0GLlwqMePCdcjowpzKaP6Rwj3DjkaiSY9O8oVj53hF8c7oFxeZ0Wm7VX6PifV6cUgqzQYsyjmy42aCFsjMoV8kl9DndvgC9qEyIEQ7q+SALzkuDBOUEAoFAuG7gCwKevj0PdqcHccpQkE4FNl9EHPfcvKFQySUYk5aENpcX8UoZdHHRM3utLi+Kzel4Y18t9kUEPtS/qQJSZkYzmtb3fKuLBOWEMFLVMdhcYESLw4s12yvCFoB8AW+H24ciUzqCQNj4NBm0KDKlo83pxZSsZMwfO4C1qDQbtLh7VH/6393NfJMF56VDgnICgUAgXBcIBQFPbq+AUa+hg+Ln5g3F5gIjmto9nMH3pQYLCUoZjP3VLNcWin0WK4pN6aysezSt79kWJ3onKEnwQmCxZkclSwfOF/DanV48+E45is3pKDalw+0LQCEVo/yMDQ++U47f/c8w/GrqIDz38THWorLMYsUT71dcskPL5TrF3MiQoJxwzeH3+3Hy5En63wMHDoREIqx3IxAIhGiuKsWmdAChgODx945gw+15V0znqouT43trh+AxiTEyVrAUTesLgAQv1wjXkjb6UgPeBKUMDo+fd9Eol4jR1O5hBeTMc9ocXqzeUdntzPflOsXcyJCgnHDNcfLkSdz78m7E6VLR3lSP15bOxKBBg672ZREIhGucS3FV+aKmCZaL7Vj01jcAfrjONVElRz+N8HM1KnYQp4uTIz9TxxlgmQxalJ+xQRtLAvKrzbWmjb7UgFeojsLcOc5y+iQInrPD47vkhYAQ0dyMbkSIJSLhmiROl4r4XnrE6VKv9qUQCIRrlEgbwTiFcJ4p0lUlMkj/oX7MvROUmJip43yMzxkjUSXHhtvzYDJow/5uMmix0JSOkrJaErxcZaJpo6+GfeWlBrxUgWjk+JyYqcOmO4fiTmNfpGn5C0UBoIPhHsSF3cm9EOAimpvRjQrJlBMIBAKhx8GVuXz2ziFRs85MIoP0H6pzZTpjcBWQ8p1Xo5Jh1tBUTq3vqAEaErxcZa62NppLNnM51ohUgShfHYXd4RE8pzqKA4vL60e9zUnvHFzufLiRIUE5gUAgEHoUfJnLDbuOoqRoNACEPUZlnR98pzzsb5FBOvDDda7RAh8uElVy3DQomQQv1yhXUxvNJ5t5bt7Qywp4EzkkVMzHorkS8QXtJoMWX56y4s+fnwzTll/OfLiR6bFB+aZNm7Bq1So89NBDePHFF6/25RAIBALhJ4Ivc+nw+FH85kH88/7xKJrggscfQD9NDCrO2sM8wrmCdIorIRURCnz4IMHLtcvV0kYLyWYe7yysvNJjJto43DRvKOuamPPJ4fGzdg4uZz7cqPTIoPzgwYN49dVXMXTo0Kt9KYQrBNNxpba2FsHgpT0HIC4tBMKNglDm0uHx43urA0vePgygqxnK5gIj3L4A0nWxQDCIf1dfYD33cnWuV8qVgwQv1yaXKhW5UuMhmmymwe6C0+tHQowM6brYKzZ2hMZhqjoGa2bl4EyzgyW1oha9xFXl8ulxQXl7ezsWLFiA1157DRs3brzal0O4QjAdVxprvkVCv6xLeg5xaSEQbhyYmUuuDoS6ODlUcgkcHj/LBu6VBSOw5O3DMEd02bxcqci15spBuPJcijb6So6HaLKZU00d9OLzpxxzEpEIi976JmzuvXDXsB/c/ZPQA4PypUuXYubMmZg6dSoJyq8zKMeV9qb6S34OgUC4caAyl9/wtLXP52hrT0EVd5ZZrBCJRNix1ASxSHRZ2/6kY+GNQ3fkRVd6PESTzTALlaO9xuVm7/mKTG/JTsH/G6OP2v2TcGn0qKD83XffxeHDh3Hw4MFuHe92u+F2u+l/t7a2/liXRiAQBCBzkXAloTKXn5+4yNnWvtRiRQBdbe0pIos7S2uaIBaJMDAl7rKu42q7clwuZD5eHtHkRVd6PAjJZrgKlblew+7woMXhxZrtFWHdP7uTWRcqMl03Jxcr3jsi2P3zWhz71zo9xqf8zJkzeOihh/D2229DqVR26znPPvssEhMT6f/1709WbwTC1YDMRcKVgvImv9DqgrG/mrcD4T6LFbfm9EJJ0Wgsm2LAlKxk2vebCZf+NdL/nM+Huqd2LCTz8cfhSo8HPm9xpoc9E5VcgkAwSI/dE+fbUH2+DU9GBORAdI/1yKy/Si7BsikGFE5Iw9GGVrS5fILdP5va+c/bnbl1pZ/bU+gxmfJDhw6hsbERI0aMoP/m9/vxxRdfYMuWLXC73awiv1WrVuGRRx6h/93a2kp+fAiEqwCZi4QrATNzp5JL8OefjxQ8/myLE0vePox8gw6/vm0wlm87zNKfayKyeZeiCe6uK4fd4YHN4UWHx4cOjx/qGBlS4hVXLZNI5uOPw4/h0hIpm1HKJNhV0cCSZqnkErxUYMTTO6tQarHSeu/pub2xYOwALDJn4HBdC0rKaunn8WXv7Q4PGuwuFIzRY6EpHUfO2jCsnxqvl52id55eWTACQrQ4PCjv1JdTUhm+ufXMHUPg8Qdgd/JLa26U2o0eE5TffPPNqKioCPvbwoULkZWVhccff5zTdUOhUEChUPxUl0j4EQgGAqitDc8GEJeVngeZi4QfCjNzRwUgErFI8DmU5rbU0gSZVITN80fg+U+OhUlamDf2S9UEd8eVo8HmxGmrA5s/qwnLLOZn6vDsHUPQL0m4i+KPAZmPPw6X09CnOzBlM3aHB0fO2Fi1EsXmdFrKRc2PSL23iVFrQT3H7fOHBc8dHj9rDuQbtBibnoTyOhv9t8jGW5HYnV4seusb+r0/c8cQrNtZxT23/nUEw/Ua+lojg+0bqXajxwTl8fHxyMvLC/tbbGwstFot6++E64eO5vNY934dkvraAIC4rBAINxDMIrMYuYS+KVMBiFGvgcmg5dxGj9Tc5qQm4nefHGMdy7yxX6omOFElx3PzhuK/Jy4iJV5BZ98vtLoweVAyAOC/xy9iV0U9W/feGYw8c8cQtDg8iFNevnUe4aeFr2jycjtYXkoRJt9rTMjQ0kEtM0BnQv37vpsykJuayAran71zCD480oBSS/gc4KrRKD9j6/bc+6KmCaver8Cw/mrsqW5kHV9msWKhKT3seGaw3VNrNy6HHhOUE25cVEm9icMKgXCDEbldzdwuN/ZXY8teC8rrbHipwAgArCz06pnZOG93Y9kUA0rKaunncPFFTRMa29ywOYU1v1xb8kGAFchMzNQh36BDg92FlAQFr/a2zGLFmRYnvjplxZa9lutyO/56I5qM4lKbQF2OLIPrNezOLn31KL0GAFBsSqcXipR0ZZ/FisenZeG5j9kL1JR4BSsgp9hnsaLYlE7LYkbpNZg9tA827qoOew5fY67SmiYUTUjjPDcAuH0BAOEWpyca25EUKw97b1xcq7Ubl0OPDsr/+9//Xu1LIFwjEJkLgXB9QOmvV0cUpzG3y6kbuMPjx4PvlKPYnI5F5gyo5BL4A0F8dcqKO175Eg6Pn96y9wWEO5KdaXEgOV5Y0sG7JW/h3pJfPHEgfIEglk0xhOnYmdpeu9MLY3911/Ous+3464nuyii62wQqWsfOF+4ahl4JSvrYyGw60zXoZGM7gFBQm6pRItUeboiRmqjEy/NHYOm2kK8510KRmld8+ALBMFkMFUA/MGkgFDIxYmQSfFh5ntOKNNr5FVIxr+xLine truncated
|
||||
"text/plain": [
|
||||
"<Figure size 750x750 with 12 Axes>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"sns.pairplot(data)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "89a1c59f-9ff7-4c16-a85e-c390b9f77b31",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"What are the means and covariances of each variable?\n",
|
||||
"Can you explain how they can be geometrically interpreted from the plots above?"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"id": "ee885793-46a8-424c-be66-7b9c642c296a",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"image/png": "iVBORw0KGgoAAAANSUhEUgAAB4QAAAKVCAYAAAAnalYzAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAACmAElEQVR4nOzde3QU9f3/8dfsZjcJARIjgSQaTFpRkCIgKEWtgkS5SUXxgsSv3CStgBUVEBQo4AUviFy0ovaLYA1Sa5VaqSgiFkWKXIw3+AooF1sJaCOkue5md39/8MuUXdiQK7OzeT7OyTmzO7O770k++cxr5rMzYwQCgYAAAAAAAAAAAAAAAFHHYXUBAAAAAAAAAAAAAIDGwYAwAAAAAAAAAAAAAEQpBoQBAAAAAAAAAAAAIEoxIAwAAAAAAAAAAAAAUYoBYQAAAAAAAAAAAACIUgwIAwAAAAAAAAAAAECUYkAYAAAAAAAAAAAAAKIUA8IAAAAAAAAAAAAAEKUYEAYAAAAAAAAAAACAKBVjdQGNxe/367vvvlOLFi1kGIbV5QAAgCYmEAjoP//5j9LT0+Vw8B08OyA/AgAAK5Ef7YkMCQAArFTTDBm1A8LfffedMjIyrC4DAAA0cd9++63OPPNMq8uw1Pr16/X4449r69atOnDggF5//XUNHjxYkuT1ejVt2jT97W9/0zfffKPExERlZ2frkUceUXp6uvkehYWFuuOOO/TXv/5VDodDQ4YM0YIFC9S8eXNzmc8++0zjxo3T5s2blZKSojvuuEOTJ0+ucZ3kRwAAEAnIj/ZChgQAAJHgZBkyageEW7RoIenoL6Bly5YWVwPYU6m3VF2f7SpJ+uRXn6iZq5nFFVmspESqGpz47jspIcHaeixW4ilR+hNHfx/f3fOdEtxN+/cBhCoqKlJGRoaZSZqykpISde7cWaNGjdJ1110XNK+0tFTbtm3T9OnT1blzZ/3444+688479ctf/lJbtmwxl8vJydGBAwe0Zs0aeb1ejRw5Urm5uVq+fLmko7/vq666StnZ2Vq8eLE+//xzjRo1SklJScrNza1RneRH4NQgYx6DfCmJXAlUIT/aExkSTR3ZTk0+05HlAGvVNENG7YBw1SVaWrZsSRgD6sjpcarAWyDp6A5Ok9+YO53/nW7ZssmFu1BOj1OKOzrdsmVL2gcQBpeNk/r376/+/fufcF5iYqLWrFkT9NxTTz2liy66SPv371fbtm21Y8cOrV69Wps3b1b37t0lSYsWLdKAAQM0d+5cpaenKy8vTx6PR0uWLJHb7VbHjh2Vn5+vefPm1XhAmPwInBpkzGOQLyWRK4FQ5Ed7IUOiqSPbqclnOrIcEBlOliGjdkAYQP3FxcTpk199Yk4DAHAqHDlyRIZhKCkpSZK0ceNGJSUlmYPBkpSdnS2Hw6FNmzbp2muv1caNG3XZZZfJ7Xaby/Tt21ePPvqofvzxR5122mnHfU5FRYUqKirMx0VFRY23UgBMZEwAAIDoQbYDAHtgQBhAWE6HU11Su1hdBgCgCSkvL9e9996rm2++2TzDoqCgQK1btw5aLiYmRsnJySooKDCXycrKClqmTZs25rwTDQjPmTNHs2bNaozVAFANMiYAAED0INsBgD04avuC9evXa9CgQUpPT5dhGFq5cqU5z+v16t5771WnTp2UkJCg9PR03Xrrrfruu++C3qOwsFA5OTlq2bKlkpKSNHr0aBUXFwct89lnn+kXv/iF4uLilJGRoccee6xuawgAAABb8Hq9uvHGGxUIBPTMM880+udNnTpVR44cMX++/fbbRv9MAAAAAAAA4FSr9RnCJSUl6ty5s0aNGqXrrrsuaF5paam2bdum6dOnq3Pnzvrxxx9155136pe//KW2bNliLpeTk6MDBw5ozZo18nq9GjlypHJzc7V8+XJJRy/Xd9VVVyk7O1uLFy/W559/rlGjRikpKanG94ADUH9en1d5n+dJknI65cjldFlcEU4Fv98vj8dz0uUqPBU6K+Gso9PlFXL6nSd5BRBdXC6XnE7afUOpGgzet2+f3nvvvaD7r6WmpurQoUNBy1dWVqqwsFCpqanmMgcPHgxapupx1TKhYmNjFRsbW+tafT6fvF5vrV8HNDXh+kkyJgAAQPQg26GuanoMEmjqGuoYZK0HhPv376/+/fufcF5iYqLWrFkT9NxTTz2liy66SPv371fbtm21Y8cOrV69Wps3bzbvA7do0SINGDBAc+fOVXp6uvLy8uTxeLRkyRK53W517NhR+fn5mjdvHgPCwCnk8Xk08i8jJUk3nHcDga4J8Hg82rNnj/x+/0mX9Qf8WnzJYknSgX8ekMOo9UUnANtLSkpSamqqDMOwuhRbqxoM3rVrl9atW6fTTz89aH7Pnj11+PBhbd26Vd26dZMkvffee/L7/erRo4e5zP333y+v1yuX6+j2as2aNTr33HNPeLnouggEAiooKNDhw4cb5P2ApuBE/SQZEwAAIHqQ7VAXtTkGCaBhjkE2+j2Ejxw5IsMwlJSUJEnauHGjkpKSzMFgScrOzpbD4dCmTZt07bXXauPGjbrsssvkdrvNZfr27atHH31UP/744wkP6lVUVKiiosJ8XFRU1HgrBTQRTodTA9oNMKcR3QKBgA4cOCCn06mMjAw5HNUP8PoDfjkPH20XbZPaMiCMJiUQCKi0tNQ8azUtLc3iiiJbcXGxdu/ebT7es2eP8vPzlZycrLS0NF1//fXatm2b3nzzTfl8PvO+wMnJyXK73erQoYP69eunMWPGaPHixfJ6vRo/fryGDh2q9PR0SdKwYcM0a9YsjR49Wvfee6+++OILLViwQE8++WSDrUfVYHDr1q3VrFkzvggAVKO6fpKMiVC0CQAA7IvtOGrbBmp7DBJoyhryGGSjDgiXl5fr3nvv1c0332xe9q+goECtW7cOLiImRsnJyebBv4KCAmVlZQUt06ZNG3PeiQaE58yZo1mzZjXGagBNVlxMnFYNW2V1GThFKisrVVpaqvT0dDVr1qxGr2kf376RqwIiV3x8vCTp0KFDat26NZePrsaWLVvUu3dv8/Hdd98tSRo+fLhmzpypN954Q5LUpUuXoNetW7dOvXr1kiTl5eVp/Pjx6tOnjxwOh4YMGaKFCxeayyYmJuqdd97RuHHj1K1bN7Vq1UozZsxosKvL+Hw+czA49AxmACcWrp8kYyIUbQIAAPtiO47atoG6HIMEmrKGOgbZaAPCVZf+CwQCeuaZZxrrY0xTp041Dy5KR88QzsjIaPTPBYBo4fP5JCno6gwAqle14+L1ehkQrkavXr0UCATCzq9uXpXk5GQtX7682mXOP/98ffDBB7Wuryaq7hnMzipQO/STAAAAAI7FMUig9hpi37pRBoSrBoP37dun9957zzw7WJJSU1PNU5urVFZWqrCwUKmpqeYyBw8eDFqm6nHVMqFiY2MVGxvbkKsBAE0Sl0AFao7/l6aHvzlQO/zPAAAAADgR9hWAmmuI/5cGvzh71WDwrl279O677x53Sb2ePXvq8OHD2rp1q/nce++9J7/frx49epjLrF+/3jwTQ5LWrFmjc88994SXiwbQOEq9pWq3qJ3aLWqnUm+p1eUgwvj8Pm07sE3bDmyTz++zuhwAAGATZEyEKvGUKOHhBCU8nKAST4nV5QAAgFog24EsB9hDrQeEi4uLlZ+fr/z8fEnSnj17lJ+fr/3798vr9er666/Xli1blJeXJ5/Pp4KCAhUUFMjj8UiSOnTooH79+mnMmDH6+OOPtWHDBo0fP15Dhw5Venq6JGnYsGFyu90aPXq0vvzyS/3xj3/UggULgi4JDaDxBQIB7S7crd2Fu2t0OU80Pf6AX/6A3+oyAACAjZAxcSKl3lIOIgMIsn79eg0aNEjp6ekyDEMrV64053m9Xt17773q1KmTEhISlJ6erltvvVXfffdd0HsUFhYqJydHLVu2VFJSkkaPHq3i4uKgZT777DP94he/UFxcnDIyMvTYY4+ditUDogbZDhJZDrCDWg8Ib9myRV27dlXXrl0lSXfffbe6du2qGTNm6F//+pfeeOMN/fOf/1SXLl2UlpZm/nz00Ufme+Tl5al9+/bq06ePBgwYoEsvvVTPPfecOT8xMVHvvPOO9uzZo27duumee+7RjBkzlJub2wCrDKCm4mLi9OHID/XhyA8VFxNndTnACY0YMUKGYejXv/71cfPGjRsnwzA0YsSIU18YAEQI+klEGjImAKAmSkpK1LlzZz399NPHzSstLdW2bds0ffp0bdu2Ta+99pq++uor/fKXvwxaLicnR19++aXWrFmjN998U+vXrw86vlhUVKSrrrpKZ511lrZu3arHH39cM2fODDpOCaB6ZDs0Fexbw+5qfQ/hXr16VftNn5p8Cyg5OVnLly+vdpnzzz9fH3zwQW3LA9CAnA6nLml7idVlACeVkZGhFStW6Mknn1R8fLwkqby8XMuXL1fbtm0trg4ArEc/iUhCxgQA1ET//v3Vv3//E85LTEzUmjVrgp576qmndNFFF2n//v1q27atduzYodWrV2vz5s3q3r27JGnRokUaMGCA5s6dq/T0dOXl5cnj8WjJkiVyu93q2LGj8vPzNW/ePE5MAWqIbIemhH1r2FmD30MYAIBT7YILLlBGRoZee+0187nXXntNbdu2Na9oIUl+v19z5sxRVlaW4uPj1blzZ7366qvmfJ/Pp9GjR5vzzz33XC1YsCDos0aMGKHBgwdr7ty5SktL0+mnn65x48YF3fceACIN/SQAAIh2R44ckWEYSkpKkiRt3LhRSUlJ5mCwJGVnZ8vhcGjTpk3mMpdddpncbre5TN++ffXVV1/pxx9/POHnVFRUqKioKOgHANA0sG8NO6v1GcIAmo5Kf6Ve3/G6JOnaDtcqxkGX0aQEAlJpNff+8PvkKC07Ol1SIjmcDffZzZpJhlGrl4waNUovvPCCcnJyJElLlizRyJEj9f7775vLine truncated
|
||||
"text/plain": [
|
||||
"<Figure size 2400x800 with 3 Axes>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"fig, ax = plt.subplots(figsize=(24, 8), ncols=3)\n",
|
||||
"for i, x in enumerate(data.columns):\n",
|
||||
" mean = data.loc[:, x].mean()\n",
|
||||
" sigma = data.loc[:, x].std()\n",
|
||||
" m_plus = mean + sigma\n",
|
||||
" m_minus = mean - sigma\n",
|
||||
" ax[i].hist(data.loc[:, x], bins=25)\n",
|
||||
" ax[i].axvline(mean, c='r', ls='-', label=\"Mean\")\n",
|
||||
" ax[i].axvline(m_plus, c='g', ls='--', label=r\"Mean + sqrt(variance)\")\n",
|
||||
" ax[i].axvline(m_minus, c='g', ls=':', label=r\"Mean - sqrt(variance)\")\n",
|
||||
" ax[i].legend()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"id": "58a73d63-32a3-469b-bb26-23bb14f49a23",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Correlation[x1, x2] = 0.70 (same as 0.70)\n",
|
||||
"Correlation[x1, x3] = 0.26 (same as 0.26)\n",
|
||||
"Correlation[x2, x3] = -0.00 (same as -0.00)\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"full_cov = data.cov() # covariance matrix\n",
|
||||
"full_corr = data.corr() # correlation coefficients\n",
|
||||
"for i, x in enumerate(data.columns):\n",
|
||||
" for j, y in enumerate(data.columns):\n",
|
||||
" if j <= i:\n",
|
||||
" continue\n",
|
||||
" # we can recalculate the correlation coefficient from the covariance matrix\n",
|
||||
" rho = full_cov.loc[x, y]/np.sqrt(full_cov.loc[x, x]*full_cov.loc[y, y])\n",
|
||||
" print(f\"Correlation[{x}, {y}] = {rho:.2f} (same as {full_corr.loc[x, y]:.2f})\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"id": "df9f429b-e218-49fd-bc72-fca2e77ba770",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Slope between x1 and x2 = 1.45\n",
|
||||
"Slope between x1 and x3 = 3.30\n",
|
||||
"Slope between x2 and x3 = -50.36\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"image/png": "iVBORw0KGgoAAAANSUhEUgAAB4kAAAKnCAYAAACbJnslAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAEAAElEQVR4nOzdd3wUZf4H8M9uei+0BEQIRSBGqnIg2BBOFAVBPUX0rKgYbOediCenHDYsP707EAQVPCk2QEAkHghKkSYhQAxSQihiAqaQkJ7sPr8/4ixbZndnZmdb8nm/Xr5eMjM7+2zLfOf5Ps/3MQghBIiIiIiIiIiIiIiIiIiIqEUw+rsBRERERERERERERERERETkO0wSExERERERERERERERERG1IEwSExERERERERERERERERG1IEwSExERERERERERERERERG1IEwSExERERERERERERERERG1IEwSExERERERERERERERERG1IEwSExERERERERERERERERG1IEwSExERERERERERERERERG1IKH+boAvmc1m/Prrr4iLi4PBYPB3c4iIiKiFEULg3LlzaN++PYxGjtULBowfiYiIyJ8YPwYnxpBERETkT0pjyBaVJP7111/RsWNHfzeDiIiIWriTJ0/iggsu8HczSAHGj0RERBQIGD8GF8aQREREFAjcxZAtKkkcFxcHoOlNiY+P93NriIiIyKe+/hoYP15+X48ewPr1gJfjg4qKCnTs2NESk1DgY/xIRERe98UXwAMPOG6/8UZg8WLft4cCCuPH4MQYkoh09f77wNNPO27fvRvo1s337SGigKc0hmxRSWKpvEt8fDwDNCIiopbkp5+AiRPl9yUlAWvWAD6cmcGSc8GD8SMREXndggXy25980usD2Ch4MH4MLowhiUhXzv6OREQwViAil9zFkEG7mMlrr70Gg8GAJ5980t9NISIiokBWUgKMHg1UVjruCwkBPv8c6NrV9+0iIiIi2rcP2LLFcXvPnsCwYb5vDxEREQWesDD57Q0Nvm0HETU7QZkk3rVrF9577z307t3b300hIiKiQNbYCNx+O3D0qPz+//s/4NprfdsmIiIiIsm778pvf/RRgDNHiYiICADCw+W3M0lMRB4KuiRxZWUlJkyYgPnz5yMpKcnfzSEiIqJA9vTTwLffyu974AHgscd82x4iIiIiSXk5sGiR4/aYGODPf/Z9e4iIiCgwOZtJXF/v23YQUbMTdEnizMxMjBo1CsOHD3d7bF1dHSoqKmz+IyIiohbigw+Af/9bft/llwOzZ3OGDhEREfnPRx8BVVWO2++6C0hI8H17iIiIKDCx3DQReUmovxugxieffILs7Gzs2rVL0fGvvvoqpk+f7uVWERERUcDZuhWYNEl+3wUXAMuXAxERvm0TERERkUQI56WmMzN92xYiIiIKbEwSE5GXBM1M4pMnT+KJJ57A4sWLERkZqegxU6dORXl5ueW/kydPermVRERE5HcnTgDjxsnfLEVFAStXAu3a+b5dRERERJJvvwUOHnTcfsUVwCWX+L49REREFLi4JjEReUnQzCTevXs3zpw5g/79+1u2mUwmbNq0CbNmzUJdXR1CQkJsHhMREYEIzhIiIiJqOaqrgZtvBs6ckd+/YAFgFUsQERER+QVnERMREZFSXJOYiLwkaJLE1157Lfbv32+z7b777kPPnj0xZcoUhwQxERERtTBCAPffD+zZI7//ueeA22/3bZuIiIiI7J082VTZxF67dsDYsb5vDxEREQU2lpsmIi8JmiRxXFwcMjIybLbFxMSgVatWDtuJiIioBXr1VeDTT+X33XQTMGOGb9tDREREJOe99wCz2XH7Qw85LydJRERELReTxETkJUGzJjFpZzAY8OWXX/q7GYp8+eWX6NatG0JCQvDkk0/KHuPu9Xz33XcwGAw4e/as4ud98cUX0bdvX1Vt9Zerr77a6XsT6Dp37ox33nnH380ImHY0R9XV1bjlllsQHx+v+ndI5JFVq4Dnn5ffl54OLFoEGBn2EKnBGNI9xpC+ESixW6C0ozlqUTFkXR0wf77j9pAQ4OGHfd8eIiKiINJi71Fqa+UPkEkS8x7FNwLl3iBQ2tEctZR7lKDuLf3uu+9a/A/gt99+w6RJk3DhhRciIiICKSkpuO6667B161Z/N02Thx9+GLfeeitOnjyJGRpnfF1++eUoLCxEQkKCzq3T7tixYzAYDMjJyfH4XMuXL9f83lCTXbt24aGHHvJ3M3QXCEHgRx99hM2bN+OHH34IuN8hNWM//QRMmNBUbtpeUlJTOcf4eN+3iyiAMYZ0xBiS3GEM6T0tKoZctgw4c8Zx+803Ax06+Lw5REREgYL3KI4s9yjJyfIH+GFNYt6jBBbeo3hPS7lHCZpy0yTvlltuQX19PT766CN06dIFp0+fxrfffouSkhJ/N021yspKnDlzBtdddx3at2+v+Tzh4eFISUnRsWWBJdlZUECKtWnTxt9NCGj19fUI11jmLj8/H7169eIyAOQ7paXAmDFAZaXjvpAQ4LPPgG7dfN8uogDHGNIRY0hyhzGka4whFXr3XfntmZm+bQcREVGA4T2KI8s9yrlz8gcEeblp3qN4jvcorvEexb2gnknsNRMnAoMH+/e/iRPdNvPs2bPYvHkzZs6ciWuuuQadOnXCwIEDMXXqVIwePdrp4/bv349hw4YhKioKrVq1wkMPPYRKq871e++9FzfffDOmT5+ONm3aID4+Ho888gjqrUYmmc1mvPrqq0hLS0NUVBT69OmDL774wmV7y8rK8Oc//xlJSUmIjo7G9ddfj8OHDwNomhUeFxcHABg2bBgMBgO+++47p+cqLi7G2LFjER0dje7du2PVqlWWfXKlAufPn4+OHTsiOjoaY8eOxf/93/8hMTHR4bwff/wxOnfujISEBNxxxx04Z3UBdveay8rKMGHCBLRp0wZRUVHo3r07FixYAABIS0sDAPTr1w8GgwFXX32109eWm5uL66+/HrGxsWjXrh3uvvtuFBcXW/bbl+EoLCzEqFGjEBUVhbS0NCxZssShzMTZs2fx4IMPWj7PYcOGYe/evZb90sgcZ69/3rx5aN++Pcx262aNGTMG999/P4CmP5pjxoxBu3btEBsbi8suuwzr1693+jrlRp2dPXvW4bN3937I2bJlC6644gpERUWhY8eOePzxx1FVVWXZb//+/Pzzzxg6dCgiIyORnp6O9evXO5R7OXnyJP70pz8hMTERycnJGDNmDI4dO2bZL/1u3nzzTaSmpqJVq1bIzMxEw+/B2nPPPYc//OEPDm3t06cP/vnPf1r+/f7776NXr16IjIxEz5498a5dJ9Ivv/yC8ePHIzk5GTExMbj00kuxY8cOLFy4ENOnT8fevXthMBhgMBiwcOFCAMCJEycwZswYxMbGIj4+Hn/6059w+vRpyzmlz//9999HWloaIiMjnb63y5Ytw8UXX4yIiAh07twZb731lmXf1VdfjbfeegubNm1y+z1fvXo1LrvsMkRGRqJ169YYO3asZZ+rvxUVFRWIiorC2rVrbc63YsUKxMXFobq62ulzUjPU2Aj86U9Afr78/v/7P2D4cN+2iVo2xpCMIRlDMoZkDCmLMeTv9u4F5GZD9eoFuHjdREREHvH3fQrvUTy/R6mpsWybD6AjgGgAY+fO5T0K71F4j8J7FM+IFqS8vFwAEOXl5a4PHDRIiKailf77b9Agt6+noaFBxMbGiieffFLU1tY6PQ6AWLFihRBCiMrKSpGamirGjRsn9u/fL7799luRlpYm7rnnHsvx99xzj4iNjRW33367yM3NFV999ZVo06aNeO655yzHvPTSS6Jnz54iKytL5OfniwULFoiIiAjx3XffOW3H6NGjRa9evcSmTZtETk6OuO6660S3bt1EfX29qKurEwcPHhQAxLJly0RhYaGoq6tz+nouuOACsWTJEnH48GHx+OOPi9jYWFFSUiKEEGLjxo0CgCgrKxNCCLFlyxZhNBrFG2+8IQ4ePChmz54tkpOTRUJCguWcL7zwgoiNjbW8L5s2bRIpKSmqXnNmZqbo27ev2LVrlygoKBDr1q0Tq1atEkIIsXPnTgFArF+/XhQWFlraaq+srEy0adNGTJ06VRw4cEBkZ2eLESNGiGuuucZyzFVXXSWeeOIJy7+HDx8u+vbtK7Zv3y52794trrrqKhEVFSXefvttm2NuuukmsWvXLnHo0CHx9NNPi1atWlna4e71l5aWivDwcLF+/XrLOUtKSmy25eTkiLlz54r9+/eLQ4cOieeff15ERkaK48ePWx7TqVMnS7sKCgoEALFnzx6b1w9AbNy4UfH7Ye/IkSMiJiZGvP322+LQoUNi69atol+/fuLee++VbUdjY6Po0aOHGDFihMjJyRGbN28WAwcOtPnd1NfXi169eon7779f7Nu3T+Tl5Yk777xT9OjRw/I9veeee0R8fLx45JFHxIEDB8Tq1atFdHS0mDdvnhBCiNzcXAFAHDlyxNIOadvhw4eFEEIsWrRIpKamimXLlomjR4+KZcuWieTkZLFw4UIhhBDnzp0TXbp0EVdccYXYvHmzOHz4sPj000/FDz/8IKqrq8XTTz8tLr74YlFYWCgKCwtFdXW1MJlMom/fvmLo0KHixx9/FNu3bxcDBgwQV111laUdL7zwgoiJiREjR44U2dnZYu/evbLv7Y8//iiMRqP45z//KQ4ePCgWLFggoqKixIIFCyzfiYkTJ4rBgwe7/J5/9dVXIiQkRPzjH/8QeXl5IicnR7zyyiuW/a7+VgghxK233iruuusum3PecsstDtuoBXj8cefX0fvvF8Js9ncLHSiORShgqPrMGEMyhmQMyRiSMaQDxpBWJk6U/9v9n//4rg0UdBg/Bid+bhRQ/H2fwnsUz+9R8vKEAMQWQBgB8QYgDgJi9tixvEfhPQrvUXiPIktpLMIksRx/XzgVXjyFEOKLL74Line truncated
|
||||
"text/plain": [
|
||||
"<Figure size 2400x800 with 3 Axes>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"fig, ax = plt.subplots(figsize=(24, 8), ncols=3)\n",
|
||||
"count = 0\n",
|
||||
"for i, x in enumerate(data.columns):\n",
|
||||
" for j, y in enumerate(data.columns):\n",
|
||||
" if j <= i:\n",
|
||||
" continue\n",
|
||||
" # get the data for the x and y axis\n",
|
||||
" dx = data.loc[:, x]\n",
|
||||
" dy = data.loc[:, y]\n",
|
||||
" \n",
|
||||
" # find their means, so we can draw a line passing through that point\n",
|
||||
" mi = dx.mean()\n",
|
||||
" mj = dy.mean()\n",
|
||||
"\n",
|
||||
" # calculate the covariance matrix\n",
|
||||
" cov = np.cov(dx, dy)\n",
|
||||
"\n",
|
||||
" # find the eigenvectors of cov:\n",
|
||||
" eigvals, eigvec = np.linalg.eigh(cov)\n",
|
||||
" # find the index of the highest eigenvalue\n",
|
||||
" i_max = np.argmax(np.fabs(eigvals))\n",
|
||||
" # find the slope of the eigenvector corresponding to the highest eigenvalue:\n",
|
||||
" slope = eigvec[1, i_max]/eigvec[0, i_max]\n",
|
||||
" # note: this procedure\n",
|
||||
" # sheds light into an interesting and useful interpretation of the covariance matrix\n",
|
||||
" print(f\"Slope between {x} and {y} = {slope:.2f}\")\n",
|
||||
" # plot the data\n",
|
||||
" ax[count].scatter(dx, dy)\n",
|
||||
" # plot a line with the slope above passing through the mean\n",
|
||||
" ax[count].axline((mi, mj), slope=slope,\n",
|
||||
" label=f\"Slope of highest eigenvalue eigenvector of cov.\",\n",
|
||||
" c=\"r\", ls=\"-\", lw=4)\n",
|
||||
" ax[count].set(xlabel=x, ylabel=y)\n",
|
||||
" ax[count].legend(frameon=False)\n",
|
||||
" count += 1"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "3552a373-ac09-4820-ab33-9c5e0e911dab",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Now let's try \"standardizing\" the data and looking at the correlation coefficients:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"id": "54373271-dd1a-4fdd-8f49-b7db7a595695",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"image/png": "iVBORw0KGgoAAAANSUhEUgAAB4kAAAKnCAYAAACbJnslAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAEAAElEQVR4nOzdeXxU1fk/8M9M9n0BIZE1LCIxgIAiCKIiCEqFVrtI1da9KLi2VbBSQVRA+/2hLRYVFLUItFYUFIwFAZEYFtljUFkSQEjA7JCQde7vj3jDzNx7Z86duXPnTvJ5v16+Wu5sZ7bMc89zzvPYJEmSQEREREREREREREREREREbYI92AMgIiIiIiIiIiIiIiIiIiLzMElMRERERERERERERERERNSGMElMRERERERERERERERERNSGMElMRERERERERERERERERNSGMElMRERERERERERERERERNSGMElMRERERERERERERERERNSGMElMRERERERERERERERERNSGMElMRERERERERERERERERNSGhAd7AGZyOBw4efIkEhISYLPZgj0cIiIiamMkScKZM2dw4YUXwm7nWr1QwPiRiIiIgonxY2hiDElERETBJBpDtqkk8cmTJ9GlS5dgD4OIiIjauOPHj6Nz587BHgYJYPxIREREVsD4MbQwhiQiIiIr8BZDtqkkcUJCAoDmFyUxMTHIoyEiIgoBf/gDsGKF8vgFFwDbtwOpqeaPKYRVVVWhS5cuLTEJWR/jRyIiizhyBBg4UHn87ruB+fPNHw+RSRg/hibGkEQUcNu2Addfrzy+ZAlw883mj4eILEU0hmxTSWK5vEtiYiIDNCIiIm/+9z/1BDEAvPoq0L27qcNpTVhyLnQwfiQisohdu9SPjxkD8O8ztQGMH0MLY0giCrj4ePXjMTGMjYiohbcYks1MiIiISKm6unkXsZoJE4Bf/tLc8RAREVHb9sUX6sevvtrccRARERFZgVbiR5LMHQcRhTQmiYmIiEjpr38FCguVxxMSmncRcycDERERmUktSXzRRUB6uvljISIiIgo2JomJyABMEhMREZGrHTuAl19Wv2zePKBzZ1OHQ0RERG1cYSFw9KjyOHcRExERUVvFJDERGYBJYiIiIjqvoQG4917A4VBeNny4dglqIiIiokBhqWkiIiIiV0wSE5EBmCQmIiKi8/72N2DfPuXxyEhg0SLAztCBiIiITMYkMREREZErrfkZJomJSAfO9BIREVGz774DZs1Sv+zpp4G+fc0dDxERERGgniTu0YMtMIiIiKjt0tpJrFYZjohIA5PERERE1HwScf/9QF2d8rKsLODJJ80fExEREdHx48CRI8rj11xj+lCIiIiILIPlponIAEwSExEREbB4MbB5s/K4zdZ8WWSk+WMiIiIiYqlpIiIiIiUmiYnIAEwSExERtXUnTwJ//rP6ZQ8/DFxxhbnjISIiIpIxSUxERESkxCQxERmASeI2wGaz4aOPPgr2MAjANddcg0cffdQy90NEBACYOhWoqlIe79oVeO4588dDRJbAGNI6GENSm6aWJO7Wrfk/IiIioraKSeJWheff1tHWzr+ZJA5xP/74Ix544AF07doVUVFRSEtLw9ixY5GTkxPsoRlu5cqVuP7669GuXTvYbDbs2bNH87oZGRlYv369eYMLkE2bNsFms6GiosLl+MqVKzF79uzgDIqIWpeVK4EPP1S/7LXXgPh4c8dDRKZgDKmOMSSRxRQVAQcPKo9zFzERERG1dUwShwyef6vj+bc1hAd7AOSfW265BfX19XjnnXfQo0cPnDp1Cp9//jlKS0uDPTTDVVdXY8SIEfj1r3+N++67T/N6+/btQ3l5Oa4OwMSBJEloampCeLjrV6e+vh6RJvbrTE1NNe2xiKgVq6gApkxRv+y224AbbjB1OERkHsaQSowhiSxIq9T0NdeYOgwiIiIiy2GSOGTw/FuJ59/WwZ3Eau67Dxg2LLj/efgCySoqKvDll19i3rx5uPbaa9GtWzcMGTIE06dPx4QJEzRvt3//fowaNQoxMTFo164d7r//fpw9e7bl8jvvvBM///nPMWvWLFxwwQVITEzE5MmTUV9f33Idh8OBOXPmICMjAzExMRgwYAD++9//+ve6e3HHHXfgr3/9K0aPHu3xeqtWrcK4ceMQEREBAMjJycE111yD2NhYpKSkYOzYsSgvLwcA1NXV4eGHH0aHDh0QHR2NESNGYMeOHS33Ja8C+fTTTzF48GBERUVhy5YtuOaaazB16lQ8+uijaN++PcaOHQsAyMvLww033ID4+Hh07NgRd9xxB0pKSjTH+q9//QuXXXYZEhISkJaWht/+9rc4ffo0AKCwsBDXXnstACAlJQU2mw133nknAGWpgvLycvzud79DSkoKYmNjccMNN+Cg04r7t99+G8nJyfjss8/Qt29fxMfHY9y4cSgqKhJ89YmoVXriCaC4WHm8XTtg/nzzx0MU6hhDMob8CWNIIoNs2qR+nDuJiYiIqK2za6R22kqSmOffPP/+Cc+//cOdxGry8oCtW4M9Cq/i4+MRHx+Pjz76CEOHDkVUVJTX21RXV2Ps2LEYNmwYduzYgdOnT+Pee+/F1KlT8fbbb7dc7/PPP0d0dDQ2bdqEwsJC3HXXXWjXrh2ef/55AMCcOXOwdOlSvPbaa+jduzc2b96M22+/HRdccIHm6o/Jkydj6dKlHsfn/IfOV6tXr8bjjz8OANizZw+uu+463H333XjllVcQHh6OjRs3oqmpCQDwxBNP4IMPPsA777yDbt264cUXX8TYsWNx6NAhl5Ue06ZNw9/+9jf06NEDKSkpAIB33nkHDzzwQEtZiIqKCowaNQr33nsv5s+fj3PnzuHJJ5/Er3/9a2zYsEF1rA0NDZg9ezb69OmD06dP4/HHH8edd96JtWvXokuXLvjggw9wyy234LvvvkNiYiJiYmJU7+fOO+/EwYMHsXr1aiQmJuLJJ5/EjTfeiPz8/JY/tDU1Nfjb3/6Gf/3rX7Db7bj99tvxpz/9Ce+9957frzkRhaBNm4BFi9Qve/ll4IILzBwNUevAGJIxJGNIImOp7STu3BnIyDB/LERERERWorWT2OEwdxzBwvNvnn/z/NsYUhtSWVkpAZAqKys9X3HoUElqXnMTvP+GDhV6Tv/973+llJQUKTo6Wrryyiul6dOnS3v37nW5DgDpww8/lCRJkt544w0pJSVFOnv2bMvla9askex2u1RcXCxJkiT9/ve/l1JTU6Xq6uqW6yxcuFCKj4+XmpqapNraWik2Nlb66quvXB7nnnvukSZNmqQ51lOnTkkHDx70+J+IgoICCYC0e/duxWU//PCDFBkZKZWXl0uSJEmTJk2Shg8frno/Z8+elSIiIqT33nuv5Vh9fb104YUXSi+++KIkSZK0ceNGCYD00Ucfudz26quvlgYOHOhybPbs2dL111/vcuz48eMSAOm7775rud0jjzyi+dx27NghAZDOnDnj8vjy83F+fPl+vv/+ewmAlJOT03J5SUmJFBMTI/3nP/+RJEmSlixZIgGQDh061HKdV199VerYsaPmWIioFaupkaRevdR/f8aOlSSHI9gjbLWEYxGyDF3vGWNIxpCMIYmMU1ys/nfuttuCPTIiUzF+DE1834go4A4cUI+VFi4M9sjMwfNvnn/z/Nsj0ViEO4lD3C233ILx48fjyy+/xNatW/Hpp5/ixRdfxOLFi1u2tTs7cOAABgwYgLi4uJZjw4cPh8PhwHfffYeOHTsCAAYMGIDY2NiW6wwbNgxnz57F8ePHcfbsWdTU1GDMmDEu911fX4+BAwdqjrVDhw7o0KGDn8/Ys9WrV2PEiBFITk4G0LwK5Ve/+pXqdQ8fPoyGhgYMHz685VhERASGDBmCAwcOuFz3sssuU9x+8ODBLv/eu3cvNm7ciPj4eNXHuuiiixTHd+7ciZkzZ2Lv3r0oLy+H46eVXseOHUNmZqbnJ/uTAwcOIDw8HFdccUXLsXbt2qFPnz4uzyM2NhY9e/Zs+Xd6enpLWQQiamOefRY4dEh5PC4OeP117dWoRNRqMIZ0xRiyGWNIspTNm9WPsx8xEREREXsShxCef7vi+Xczq5x/M0ncCkRHR2PMmDEYM2YMZsyYgXvvvRfPPPOM6h8YI8jlBNasWYNOnTq5XOapXIIZpQpWr17tUstfa2u/Xs5/kLWOnT17FjfddBPmzZunuG56errimFw2YuzYsXjvvfdwwQUX4NixYxg7dqxL7wCjyCULZDabDRKDBqK2Z88e4KWX1C97/nmgWzdTh0NEwcMY8jzGkNoYQ1LQsB8xERERkTYmiUMKz7/P4/m3tmCcfzNJrCYrK9gj8GsMmZmZ+Oijj1Qv69u3L95++21UV1e3fEFycnJgt9vRp0+fluvt3bsX586da/mCbt26FfHx8ejSpQtSU1MRFRWFY8eOadauV/Pss8/iT3/6k8/Py5uzZ89i48aNWLhwYcux/v374/PPP8esWbMU1+/ZsyciIyORk5ODbj8lRRoaGrBjxw6XhuKiBg0ahA8++ADdu3dHeLj3r9a3336L0tJSzJ07F126dAEAfP311y7XiYyMBICW+vtq+vbti8bGRmzbtg1XXnklAKC0tBTfffed8EoWImojGhuBe+8F1P6mDBkCTJ1q/piIWhPGkIwhGUMSGUetH3F6OtCrl/ljISIiIrKatp4k5vk3z795/m0IJonVLFoU7BEIKS0txa9+9Svcfffd6N+/PxISEvD111/jxRdfxMSJE1Vvc9ttt+GZZ57B73//e8ycORM//vgjHnroIdxxxx0tZQqA5rID99xzD55++mkUFhbimWeewdSpU2G325GQkIA//elPeOyxx+BwODBixAhUVlYiJycHiYmJ+P3vf6/62P6WKigrK8OxY8dw8uRJAMB3330HAEhLS0NaWhqys7Nx0UUXoXv37i23mT59Ovr164cHH3wLine truncated
|
||||
"text/plain": [
|
||||
"<Figure size 2400x800 with 3 Axes>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"fig, ax = plt.subplots(figsize=(24, 8), ncols=3)\n",
|
||||
"count = 0\n",
|
||||
"for i, x in enumerate(data.columns):\n",
|
||||
" for j, y in enumerate(data.columns):\n",
|
||||
" if j <= i:\n",
|
||||
" continue\n",
|
||||
" # get the data for the x and y axis\n",
|
||||
" dx = data.loc[:, x]\n",
|
||||
" dy = data.loc[:, y]\n",
|
||||
" # find their means\n",
|
||||
" mi = dx.mean()\n",
|
||||
" mj = dy.mean()\n",
|
||||
" # find the square root of their variances\n",
|
||||
" si = dx.std()\n",
|
||||
" sj = dy.std()\n",
|
||||
" # standardize them:\n",
|
||||
" di = (dx-mi)/si\n",
|
||||
" dj = (dy-mj)/sj\n",
|
||||
" # plot them again\n",
|
||||
" ax[count].scatter(di, dj)\n",
|
||||
" # what is the correlation coefficient?\n",
|
||||
" rho = np.corrcoef(di, dj)[0, 1]\n",
|
||||
" # plot a line with the slope above passing through the mean\n",
|
||||
" ax[count].axline((0, 0), slope=1/rho,\n",
|
||||
" label=f\"Slope = 1/correlation\",\n",
|
||||
" c=\"r\", ls=\"-\", lw=4)\n",
|
||||
" ax[count].set(xlabel=x, ylabel=y)\n",
|
||||
" ax[count].legend(frameon=False)\n",
|
||||
" count += 1"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "1e7966d8-8c82-4717-8c8d-49e4a31f338a",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.10.6"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
@@ -0,0 +1,126 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "1c0d6d14-a97f-41f2-8b3e-673399418107",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Basic statistics\n",
|
||||
"\n",
|
||||
"Here, the goal is to get used to several concepts in basic statistics needed later on for Machine Learning.\n",
|
||||
"We are going to generate a simple dataset ourselves and try to calculate some standard quantities to describe them.\n",
|
||||
"\n",
|
||||
"First, let's import some standard libraries and generate the dataset."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"id": "d5a29abd-6af1-4b18-8bec-8a8402b97dc7",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import matplotlib.pyplot as plt\n",
|
||||
"\n",
|
||||
"import pandas as pd\n",
|
||||
"import numpy as np\n",
|
||||
"import seaborn as sns"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"id": "f4bed5ac-0828-4e5f-a166-2e161557b49b",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"rng = np.random.RandomState(0)\n",
|
||||
"n_samples = 10000\n",
|
||||
"cov = [[1, 0.7*1*1.3, 0.3*1.3*1],\n",
|
||||
" [0.7*1*1.3, 1.3**2, 0],\n",
|
||||
" [0.3*1.3*1, 0, 1.5**2]]\n",
|
||||
"data = rng.multivariate_normal(mean=[1, -1, 0], cov=cov, size=n_samples)\n",
|
||||
"data = pd.DataFrame(data, columns=[\"x1\", \"x2\", \"x3\"])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "87d5cd39-4720-4eba-a268-0c38a561976a",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"First, make a scatter plot of this dataset using matplotlib to visualise what are the possible values of $x_1$, $x_2$ and $x_3$.\n",
|
||||
"One can make a 2D plot of $x_1$ versus $x_2$, $x_2$ versus $x_3$ or $x_1$ versus $x_3$ to understand how they relate to one another.\n",
|
||||
"\n",
|
||||
"Often this is cumbersome if one has too many variables to plot, but we are going to discuss methods to get around this later in the lecture.\n",
|
||||
"\n",
|
||||
"One tip: one may use `plt.scatter` for every pair of variables to do this, or use the seaborn module to plot all pairs in one go. Look for documentation on `sns.pairplot`."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "7de735cf-3466-41e9-bcd5-fcc756958019",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "89a1c59f-9ff7-4c16-a85e-c390b9f77b31",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"What are the means and covariances of each variable?\n",
|
||||
"Can you explain how they can be geometrically interpreted from the plots above?\n",
|
||||
"\n",
|
||||
"Tip 1: Try subtracting the mean from the variables, dividing by the square root of the variance, and plotting the variables again. If you fit a line to the data, what is the slope? What is the correlation coefficient?\n",
|
||||
"\n",
|
||||
"Tip 2: Try using `np.linalg.eig(covariance_matrix)` to decompose the covariance matrix into eigenvectors and try interpreting it.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "ee885793-46a8-424c-be66-7b9c642c296a",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "df9f429b-e218-49fd-bc72-fca2e77ba770",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "54373271-dd1a-4fdd-8f49-b7db7a595695",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.10.6"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
@@ -0,0 +1,284 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "1bba0128",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Bayesian fits\n",
|
||||
"\n",
|
||||
"This is a notebook showing a visualization of a Byesian interpretation for the line fit.\n",
|
||||
"The idea is that we generate all possible line fits and reject them if they are not compatible with the data.\n",
|
||||
"Naturally, this is not how most fits are done, but it illustrates the logic behind the Bayes Theorem when applied for fitting lines.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"id": "23feddde",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from typing import Tuple\n",
|
||||
"\n",
|
||||
"import pandas as pd\n",
|
||||
"import numpy as np\n",
|
||||
"import matplotlib.pyplot as plt\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "bb1286f0",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We start by generating some fake dataset, which is simple enough that we can visualize the results easily. For this reason, the dataset will contain only two variables.\n",
|
||||
"\n",
|
||||
"The simulated example data will be $f(x) = 2 x + 1 + \\epsilon$, where $\\epsilon \\sim \\mathcal{N}(\\mu=0, \\sigma=0.5)$.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"id": "aef54435-78af-4b13-aee2-2bc576c773cd",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"true_alpha = 0.3\n",
|
||||
"true_beta = 0.5\n",
|
||||
"true_eps = 0.5\n",
|
||||
"def generate_data(N: int) -> np.ndarray:\n",
|
||||
" x = 2*np.random.randn(N, 1)\n",
|
||||
" epsilon = true_eps*np.random.randn(N, 1)\n",
|
||||
" z = true_beta*x + true_alpha + epsilon\n",
|
||||
" return np.concatenate((x, z), axis=1).astype(np.float32)\n",
|
||||
"\n",
|
||||
"data = generate_data(N=100)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "04432613-39cd-4209-8833-7af585e7eed9",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Given some parameters $\\alpha$ and $\\beta$, we don't know the true $f(x)$, but we assume it is something like $f(x) = \\beta x + \\alpha$."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 9,
|
||||
"id": "a795239a-03eb-483a-8b07-635e3d9dbc55",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def f(x: np.ndarray, alpha: float, beta: float) -> np.ndarray:\n",
|
||||
" return beta*x + alpha"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "a2fba887-dc1a-4400-ab40-59fec452dc83",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Let's plot it for reference:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 48,
|
||||
"id": "fead461d-4d3a-48f2-a525-c518be4f1cf8",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAp4AAAH5CAYAAAAobz7sAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAB1XElEQVR4nO3dd3hUZdoG8HsmvQ4EEyZAgFAEQugCUtSAIAFBkKKyIgvroiC6gusnoGICFtauWLCLK7YVRKpBEBBBECSGXgRCCwkEEiYhpM6c74/jhEwy5ZzJnDPt/l1Xrs9M3pl5M7ifN295Ho0gCAKIiIiIiBSmdfcEiIiIiMg/MHgSERERkSoYPImIiIhIFQyeRERERKQKBk8iIiIiUgWDJxERERGpgsGTiIiIiFQR6O4J2GMymXDu3DlERUVBo9G4ezpEREREVIsgCCguLkaTJk2g1dpf0/To4Hnu3DkkJCS4expERERE5MCZM2fQrFkzu2M8OnhGRUUBEH+R6OhoN8+GiIiIiGorKipCQkJCdW6zx6ODp3l7PTo6msGTiIiIyINJORbJy0VEREREpAoGTyIiIiJSBYMnEREREamCwZOIiIiIVMHgSURERESqYPAkIiIiIlUweBIRERGRKhg8iYiIiEgVDJ5EREREpAoGTyIiIiJSBYMnEREREamCwZOIiIiIVMHgCcBoErD9+CWsyMrB9uOXYDQJir/npEmToNFooNFoEBQUhMaNG2Pw4MH45JNPYDKZZL1WSkoKZsyYocxEiYiIiFwk0N0TcLeM/bmYt+ogcg1l1Y/F60KRNiIJqcnxir53amoqPv30UxiNRpw/fx4ZGRl49NFHsXTpUqxcuRKBger+8ezatQsLFy7Eb7/9hnPnzkGv16Nz5864fPkyGjRogIkTJ2LUqFGqzomIiHyP0SRgZ3YBLhSXIS4qFL0SYxCg1bh7WqQCv17xzNifi2lLMi1CJwDkGcowbUkmMvbnKvr+ISEh0Ov1aNq0Kbp3744nn3wSK1aswA8//IDFixeLc8zIQP/+/dGgQQM0atQIw4cPx/Hjx6tfY9KkSfj555/x5ptvVq+gnjx50uHzatu5cyeOHDmC999/HzNmzEBJSQnWrl2L7777Dhs3bsSoUaNw55134u2331b0MyEiIt+WsT8X/V/ciPEf7sCjX2dh/Ic70P/FjYr/N5c8g0YQBOX3lZ1UVFQEnU4Hg8GA6Ohol7620SSg/4sb64ROMw0AvS4UW2cNVORvYZMmTcLly5fx/fff1/lZ165d0aRJE6xduxbLli2DRqNB586dceXKFTzzzDM4efIksrKyoNVqYTAYMHToUCQnJ2P+/PkAgNjYWHz//fd2n2fPvffeiz179mD//v0Wj/ft2xelpaX4448/XPY5EBGR/zAv+NQOHub/yi6a0F3x3UZyPTl5zW+32ndmF9gMnQAgAMg1lGFndgH6tG6k3sQAtG/fHnv37gUAjBkzxuJnn3zyCWJjY3Hw4EEkJydDp9MhODgY4eHh0Ov11eMcPc+WyspKrFmzBtOnT6/zs+bNm2PXrl31+dWIiMhPGU0C5q06WCd0AuJ/czUA5q06iMFJem67+zC/3Wq/UGw7dDozzpUEQYBGI/6P7s8//8T48ePRqlUrREdHo2XLlgCA06dP230NZ5+3adMmGAwGq2c5T58+jVatWsn+fYiIiOQs+JDv8tsVz7ioUJeOc6VDhw4hMTERADBixAi0aNECH374IZo0aQKTyYTk5GRUVFTYfQ1nn7dixQo0a9YMN9xwg8XjJ0+exG+//YaFCxfW75cjIiK/5MkLPqQevw2evRJjEK8LRZ6hzOqyv/mMZ6/EGFXntXHjRuzbtw8zZ87EpUuXcOTIEXz44Ye46aabAABbt26t85zg4GAYjcbq76U+rzZBELBy5Urccccd1SuuAFBRUYEpU6agQ4cO+Oc//1nfX5GIiPyQJy/4kHr8NngGaDVIG5GEaUsyoQEswqc5cqWNSFL0nEl5eTny8vIsyiktWLAAw4cPx8SJE6HRaNCoUSN88MEHiI+Px+nTpzF79uw6r9OyZUv89ttvOHnyJCIjIxETEyPpebVt374dZ8+exYgRIwAAJSUlWL16NebOnYuQkBCsWbMGISEhLv8ciIjI93nqgg+py2/PeAJAanI8Fk3oDr3O8m9Xel2oKjfrMjIyEB8fj5YtWyI1NRWbNm3CwoULsWLFCgQEBECr1eLrr7/G7t27kZycjJkzZ+Lll1+u8zqPP/44AgICkJSUhNjYWJw+fVrS82o6ePAgxo8fDwB44IEHkJycjKSkJHz22WeYM2cOMjMz0aJFC0U+ByIi8n3mBR/g2gKPmVoLPuR+fltOqSYWsiUiIlKHOxu3+IXSUuDAAUCjAa6/HoiKUvwt5eQ1Bk8iIiJSFRd8FPK//8F04AByCktRUlGFsAbRaDZjGgL0jRV9W9bxJCIiIo8VoNWoXiPbp129Crz0Eo5dKMbmI/m4Ul711w/ykJvzPgbMmuIxq8mKnvFctGgROnfujOjoaERHR6NPnz744YcflHxLIiIiIo9hNAnYfvwSVmTlYPvxSzCaXLzRvH9/dehcvTe3RugUVVwuUqUNuFSKrng2a9YM//nPf9C2bVsIgoDPPvsMI0eOxB9//IGOHTsq+dZEREREbqXoeVZBAF5+Gbh6FSZBwOYj+VaHnWwgvo+ndIVSdMVzxIgRGDZsGNq2bYvrr78ezz//PCIjI7Fjxw4l35aIiIhIcfZWM8196Wt3a8ozlNV/BfLkSWDePHGLHUBOYWmdlU6zP5q086iuUKqd8TQajfj2229RUlKCPn36WB1TXl6O8vLy6u+LiorUmh4RERGRZPZWMwcn6ZXrS//aa0BREUyCUH2JqOBK3a6EJxs2wfcdB1g85gldoRQPnvv27UOfPn1QVlaGyMhILF++HElJSVbHLliwAPPmzVN6SkREREROM69m1g6W5tXMGYOul9yXXvIlq6oq4LnnAMDKJSJL3yel4GRM0zqPe0JXKMXLKVVUVOD06dMwGAxYunQpPvroI/z8889Ww6e1Fc+EhASWUyIiIiKPYDQJ6P/iRpvBUgNAFx6Ey1crHb7Wm/d0xciudQNiHVlZwPffA0D1JSJb3r3xLlQEBtWZk14Xiq2zBipyxtOjyikFBwejTZs2AIAePXpg165dePPNN/H+++/XGRsSEsKWjEREROSxdmYXOFzNlBI6AYkrkOnp1f9o7xLR6QZ6fJd8a53HPa0rlOp1PE0mk8WqJhEREZG3kHpOskFYEAyllc73pS8pEW+t12DrEtHG1j2xN/56qy+j97CuUIoGzzlz5mDo0KFo3rw5iouL8eWXX2Lz5s1Yt26dkm9LHm7x4sV46aWXcPLkSbRo0QKvvPIKbr/9dndPi4iIyCGp5yQn92uJNzb8CQ1gET4lrUBu2ABs3Vrn4ZKKuqHz7T53oyrAMs49PKAN2jaO9MiuUIqWU7pw4QImTpyIdu3a4dZbb8WuXbuwbt06DB48WMm39XgajcbuV3qNZXVfs2zZMjz88MOYO3cu9u/fjyFDhmDq1KnunhYREZEkvRJjEK8Lha0op4F4u/3hgW2xaEJ36HWWQVWvC8WiCd1tr0Cmp1sNnQAQEXwtYJYHBuON/vfWCZ0A0K/NdRjZtSn6tG7kUaETYK92t8jLy6v+52+++QbPPPMMjhw5Uv1YZGQkIiMjLZ5TUVGB4OBg1eaolH79+mHQoEHV1QvWr1+PcePG4fLly/V63ZSUFEyaNAmTJk2q/ySJiIjsMN9qB6yvZtYMlpL70ufnA++8Y/d9TYKAT7ZmY0mbm3CyYZM6P1f6EpEtHnW5SDUrVwIXLrh3DnFxwB13OBym1+ur/1mn00Gj0Vg8BohBKjk5GYGBgViyZAk6deqETZs2oWXLlpgxYwZmzJhRPbZr164YNWoU0tPTYTKZ8OKLL+KDDz5AXl4err/+esydOxdjx461OZ9mzZrhySefxEMPPVT92K+//opBgwbh0KFDaNGihYwPwbbi4mLs2LEDr732WvVj69atQ7du3aq//+qrr/CPf/wDJ06cQHy8+D/ayZMnY/fu3fjll1+g0+lcMpedO3fiiSeewG+//YYWLVpgyZIlyMzMxOrVq7Fy5UqXvAcREfmm1OR4LJrQvU4dT2vnKSX1pf/yS+DoUYfvq9Vo0OyN/+DUF384t4XvAXwneF64AJw96+5ZuNRnn32GadOmYdu2bZKfs2DBAixZsgTvvfce2rZtiy1btmDChAmIjY3FLbfcYvU5vXv3xq5du6q/FwQBM2bMwMyZM+uEzhdeeAEvvPCC3TkcPHgQzZs3r/P4nj17oNVq0aVLF1y9ehVffvklFi5ciOXLl1ePueeee/Cf//wHL7zwAt566y2kpaVhw4YN2LFjh8tC544dOzBgwADMnz8fH374IZ544gnMnz8fBw4cwNKlS13yHkRE5NtSk+MxOEkvbTXTFkEQOxBJ0aoVMHEiUgEsmqCRFHo9ke8ETx/Utm1bvPTSS5LHl5eX44UXXsCGDRuqu0O1atUKW7duxfvvv28zeN5444347LPPqr///PPPcebMGcyZM6fO2KlTp+Kuu+6yO48mTeou/wNAVlYW2rdvj927d6N///4AgNGjR2Po0KHVYzQaDZ5//nmMHTsWer0eb731Fn755Rc0bSqhzplEjz32GMaNG4f/+7//AwCMHz8e48ePx8iRIy1WX4mIiOyRtJppy4kTwH//K23sQw+Ju6p/cUnodRMGTw/Wo0cPWeOPHTuGq1ev1rm8VVFRYTdQ3XjjjZg9ezauXLkCjUaDJ598Es8991ydc6YAEBMTg5gYO+Uf7MjKykL37t3RqVMn/Pbbb9i2bRuefvppzJ8/3+JC1fDhw5GUlIT58+fjxx9/RMeOHeu8Vu2V19LSUuzYsQMPP/xw9WPWVl7Pnj2L7du345VXXql+LDAwEIIgsGsWERGLine truncated
|
||||
"text/plain": [
|
||||
"<Figure size 800x600 with 1 Axes>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"fig = plt.figure(figsize=(8, 6))\n",
|
||||
"plt.scatter(data[:,0], data[:,1], label=r\"Data $\\mathcal{D}$\")\n",
|
||||
"plt.plot(data[:,0], f(data[:,0], true_alpha, true_beta), c='r', ls='-', alpha=0.5, lw=4, label=r\"True $y=\\beta x + \\alpha$\")\n",
|
||||
"xs = np.linspace(-2, 2, 10)\n",
|
||||
"plt.plot(xs, f(xs, true_alpha, true_beta) - 2*true_eps, c='r', ls='--', alpha=0.5, lw=4)\n",
|
||||
"plt.plot(xs, f(xs, true_alpha, true_beta) + 2*true_eps, c='r', ls='--', alpha=0.5, lw=4)\n",
|
||||
"xa = 2\n",
|
||||
"ya = f(xa, true_alpha, true_beta)\n",
|
||||
"plt.annotate(xy=(xa, ya + 2*true_eps), xytext=(xa, ya - 2*true_eps), arrowprops=dict(arrowstyle='|-|', lw=3, alpha=0.5, color=\"green\"), text=r\"\")\n",
|
||||
"plt.text(xa+0.5, ya, s=r\"$\\epsilon$ noise\", size=16)\n",
|
||||
"plt.legend(frameon=False)\n",
|
||||
"plt.savefig(\"data.png\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "00e098e8-0082-4451-927a-073cf242787e",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Let's show several random lines and reject the ones that do not agree with the likelihood:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 49,
|
||||
"id": "08d9daaf-25c5-4d06-b4a6-6cecaaa35651",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import matplotlib.pylab as pylab\n",
|
||||
"params = {'legend.fontsize': 'x-large',\n",
|
||||
" 'figure.figsize': (15, 5),\n",
|
||||
" 'axes.labelsize': 'x-large',\n",
|
||||
" 'axes.titlesize':'x-large',\n",
|
||||
" 'xtick.labelsize':'x-large',\n",
|
||||
" 'ytick.labelsize':'x-large'}\n",
|
||||
"pylab.rcParams.update(params)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 59,
|
||||
"id": "4c7002b2-5cb5-4929-8e01-964144ee60b6",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def get_log_prior(A, B):\n",
|
||||
" #return 0.25*A/A/1/5\n",
|
||||
" return -0.5*A**2/1**2 - 0.5*B**2/1**2 #- (2*np.pi*p_width**2)\n",
|
||||
"def sample_prior() -> Tuple[float, float]:\n",
|
||||
" #return (2*np.random.rand() - 1)*1, (2*np.random.rand() - 1)*5\n",
|
||||
" return 1*np.random.randn(), 1*np.random.randn()\n",
|
||||
"def get_log_likelihood(x, y, A, B, sigma):\n",
|
||||
" error = f(x, A, B) - y\n",
|
||||
" return np.sum(-0.5*error**2/sigma**2, axis=0) # - (2*np.pi*sigma**2)\n",
|
||||
"def plot(I: int):\n",
|
||||
" # how many lines to show in last column\n",
|
||||
" K = 20\n",
|
||||
" # assumed noise std. dev.\n",
|
||||
" sigma=0.5\n",
|
||||
" # region of alpha and beta to plot\n",
|
||||
" R = 4\n",
|
||||
" # take a subset of data points\n",
|
||||
" D = data[:I,:]\n",
|
||||
" # make plot\n",
|
||||
" fig, ax = plt.subplots(figsize=(4*5, 4), ncols=4)\n",
|
||||
" # create alpha and beta grid\n",
|
||||
" A = np.linspace(-R, R, 50)\n",
|
||||
" B = np.linspace(-R, R, 50)\n",
|
||||
" A, B = np.meshgrid(A, B)\n",
|
||||
" \n",
|
||||
" # calculate the log prior\n",
|
||||
" log_prior = get_log_prior(A, B)\n",
|
||||
" # calculate the log likelihood\n",
|
||||
" log_likelihood = get_log_likelihood(D[:, 0, np.newaxis, np.newaxis],\n",
|
||||
" D[:, 1, np.newaxis, np.newaxis],\n",
|
||||
" A[np.newaxis, ...],\n",
|
||||
" B[np.newaxis, ...],\n",
|
||||
" sigma=sigma)\n",
|
||||
" # the log posterior is, except for a constant factor, just the sum of those\n",
|
||||
" log_posterior = log_prior + log_likelihood\n",
|
||||
"\n",
|
||||
" m=ax[0].imshow(np.exp(log_prior), extent=(-R, R, -R, R), aspect=\"equal\")\n",
|
||||
" #plt.colorbar(m,ax=ax[0], orientation=\"horizontal\")\n",
|
||||
" ax[0].set(xlabel=r\"$\\alpha$\", ylabel=r\"$\\beta$\", title=r\"$p(\\alpha, \\beta)$\")\n",
|
||||
" \n",
|
||||
" m=ax[1].imshow(np.exp(log_likelihood), extent=(-R, R, -R, R), aspect=\"equal\")\n",
|
||||
" #plt.colorbar(m,ax=ax[1], orientation=\"horizontal\")\n",
|
||||
" \n",
|
||||
" ax[1].set(xlabel=r\"$\\alpha$\", ylabel=r\"$\\beta$\", title=r\"$p(\\mathcal{D}|\\alpha,\\beta)$\")\n",
|
||||
" m=ax[2].imshow(np.exp(log_posterior), extent=(-R, R, -R, R), aspect=\"equal\")\n",
|
||||
" #plt.colorbar(m,ax=ax[2], orientation=\"horizontal\")\n",
|
||||
" ax[2].set(xlabel=r\"$\\alpha$\", ylabel=r\"$\\beta$\", title=r\"$p(\\alpha, \\beta|\\mathcal{D})$\")\n",
|
||||
"\n",
|
||||
" if I > 0:\n",
|
||||
" ax[3].scatter(D[:, 0], D[:,1], label=\"Data\")\n",
|
||||
" ax[3].set(xlabel=\"x\", ylabel=\"y\", title=\"Data\", xlim=(-5, 5), ylim=(-5, 5))\n",
|
||||
"\n",
|
||||
" # sample alpha and beta\n",
|
||||
" def get_line():\n",
|
||||
" a, b = sample_prior()\n",
|
||||
" if len(D) == 0:\n",
|
||||
" return a, b\n",
|
||||
" for _ in range(1000):\n",
|
||||
" ap, bp = a + 0.1*np.random.randn(), b + 0.1*np.random.randn()\n",
|
||||
" A = min(1, np.exp(get_log_likelihood(D[:, 0], D[:, 1], ap, bp, sigma=sigma) - get_log_likelihood(D[:, 0], D[:, 1], a, b, sigma=sigma) ))\n",
|
||||
" if np.random.rand() <= A:\n",
|
||||
" a = ap\n",
|
||||
" b = bp\n",
|
||||
" return a, b\n",
|
||||
" for _ in range(K):\n",
|
||||
" a, b = get_line()\n",
|
||||
" ax[3].axline((0, a), slope=b, lw=2, ls='-', c='r', alpha=0.5)\n",
|
||||
" #plt.legend(frameon=False)\n",
|
||||
" plt.tight_layout()\n",
|
||||
" plt.savefig(f\"bayes_reg_{I}.png\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 67,
|
||||
"id": "693827d3-c5ac-4b6d-b019-233799cd5d78",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"image/png": "iVBORw0KGgoAAAANSUhEUgAAB5EAAAGGCAYAAABSR3Y3AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAADkBElEQVR4nOzdd3hc5Z02/vtM16h3yZbkXmTcK9hgDNgOSZaQhc1uNmWBd8kCSVgCbPKDZJeyyQtJNtkFNnnT2DRIJYEAyQI22KYYcK/YuMpdvZeZ0ZTz++Pm6GisM0YybpLuz3XNJfuZU54zI/vYuuf7fQzTNE2IiIiIiIiIiIiIiIiIiIgAcJ3vCYiIiIiIiIiIiIiIiIiIyIVDIbKIiIiIiIiIiIiIiIiIiPRQiCwiIiIiIiIiIiIiIiIiIj0UIouIiIiIiIiIiIiIiIiISA+FyCIiIiIiIiIiIiIiIiIi0kMhsoiIiIiIiIiIiIiIiIiI9FCILCIiIiIiIiIiIiIiIiIiPRQii4iIiIiIiIiIiIiIiIhID4XIIiIiIiIiIiIiIiIiIiLSQyGyiIiIiIiIiIiIiIiIiIj0UIgsIiIiIiIiIiIiIiIiIiI9FCKLfAA33HADioqK0NnZec7PHYlE8PDDD2PatGlIS0tDWloaZs6cif/5n/9J2m7Tpk0wDAOPP/74OZ+jiIiIiIiIiIiIiIiIDD4KkUVO04YNG/DEE0/gnnvuQXp6+jk9d3d3N5YvX46vfvWr8Hg8uPXWW3HDDTegqqoKN998M1566aWebefMmYOPf/zj+Ld/+zd0dHSc03mKiIicS2frw10PPPAADMPAoUOHzuhx+0sfHBMRkaFsMHw4Gzh7/x7Q/VtERIYSwzCSHn6/H4WFhZg9ezZuvvlmvPDCC4jH42fkXD//+c9hGAZ+/vOfn5HjiUhfCpFFTtPXvvY1ZGVl4bbbbjvn53700Ufx2muv4ZZbbsHmzZvxX//1X/jhD3+IH/zgBwCAN954I2n7e++9FzU1NXjsscfO+VxFRETOhfP54a6zSR8cExGRoWywfDj7bNL9W0REhqL7778f999/P77yla/gk5/8JHJycvDEE0/gIx/5CC6++GLs3bv3fE9RRPrBc74nIDIY7d27Fy+//DJuvvlmpKWlnfPz//CHP0QwGMR3v/tdGIbRM+7x8I90fn5+0vbz58/H5MmT8aMf/Qj33HMPXC59fkRERIaWU3246/jx4/jJT36CV199Ffv370djYyMikQjGjh2LzMxMhEIhjBw5EgsWLMAnP/lJTJs27TxcgbPeHxz7wQ9+0HPfX7x4MT796U/jjTfewIc+9KGe7e+9914sWLAAjz32GL761a+er2mLiIj0y4Xy4ez+3GNP5YP+W0P3bxERGWoeeOCBPmO1tbW4/fbb8dRTT2Hp0qXYuHEjioqKzv3kRKTflCSJAHjrrbdgGAZuv/12/OY3v8Fll12G7Oxs+P1+zJ8/H6tXr07a/qc//SlM08Tf/d3fpTxmNBrFI488gpkzZyItLQ1lZWW488470d3dja6uLhQXF+PTn/70gOd6+PBhHDx4EFdddVWfT2r//ve/BwBceeWVffb75Cc/iSNHjmDlypUDPqeIiMiFzPpw19/+7d8mfbjLNE185zvfwbe//W0sWbIEjz76KNLT0xEKhXD99ddj37592Lx5M3bs2IF/+Zd/wTPPPIPp06fj7rvvPo9Xk+yDfHAskUic07mKiIgMRKr797ky0HuskzP1bw3dv0VEZDgoLi7Gb3/7WyxZsgRHjx7FQw89lPT8pk2bcMcdd2DGjBnIy8tDIBDAhAkTcPfdd6O5uTlp2yVLluCmm24CANx0001JLbStpSdOnDiBf//3f8eiRYtQUlICn8+HESNG4FOf+hR27dp1Tq5ZZLBTiCwC3qAA4OWXX8ZnP/tZ5OXl4dZbb8XixYuxYcMGfOQjH8GRI0d6tn/55Zfhdrtx8cUXOx6vqakJixYtwp133ony8nL88z//M7Kzs/HII4/gscceww9/+EM0NTXhwQcfHPBcN27cCABYsGBBz5hpmnj00Ufxxz/+EUuXLsX06dP77Ldo0SIAUIgsIiIXvDP14a6amhp86EMfwqOPPoolS5Zg+vTpPT8Y/sxnPtOzncfjwdVXX43Vq1cjOzsb//mf/zmg+6U+OCYiIjI8Ppx9sjP5bw3dv0VEZDhwuVz413/9VwDAb37zG5im2fPcT37yE/z2t7/FpEmTcNNNN+G2225DaWkp/vM//xOLFi1Ce3t7z7Y33ngjrr32WgDAtdde29M++/7770dOTg4A4LXXXsM3v/lN5OTk4Prrr8edd96Jiy++GH/4wx8wf/58bNu27dxduMhgZYqIedNNN5kAzKysLPP1119Peu722283AZh33323aZqm2dHRYbrdbnPq1Kkpj7d06VITgPnoo4/2jNXX15t+v9+88sorzZKSEvPmm28+rbnee++9JgDzhRdeMFetWmXefPPN5qRJk0wA5owZM8yamhrH/VpaWkwA5rx5807rvCIiIufKf//3f5sAzMmTJ5tut9v82Mc+Zn7lK1/pub8GAgHz8OHDPdvPmTPHdLvdZkdHxymPe+DAgZ79Ozs7Hbf5x3/8RxOA+bnPfa5n7P777zcBmFVVVX22b2xsNOfNm2cCMP/qr/7K/MpXvmJOmTLFBGD+x3/8h/nd737X9Hg85r59+wb8OvzhD38wAZjf+MY3esYSiYT5yCOPmADMpUuXOu63cuXKpH+7iIiInAtn+v59od1jT/XvAcvp/lvDNHX/FhGRoQGA+X6xUzgcNj0ejwnAPHjwYM/4oUOHzFgs1mf7xx9/3ARgfvOb30wa/9nPfmYCMH/2s585nqe2ttZsa2vrM75161YzPT3dvPrqq/txRSLDmyqRRQBs3rwZAPDf//3fuPTSS5Oeu/nmmwGgp8XF8ePHEY/HUVpa6nisl19+GS+//DIuu+wy3H777T3jBQUFGD16NFatWoXm5mbcd999pzVXq2p67ty5+NGPfoTHH38ce/bsAQBMnjwZ8Xjccb/s7GwEAoGkimoREZELkXVfPnHiBNasWYNnn30W3/rWt7By5UrcfvvtCIfDeOyxxwAAnZ2d2Lp1KyorK/tUEp3sueeeAwBcddVVCAaDjtuMHTsWAHD06NF+zfXv/u7vsGHDBjz66KN4/vnn8a1vfQuvvvoq/H4/XnjhBfzHf/wHbrzxRowfP75fx+vNuufPmTMHq1evxuc+9zlUVlbiS1/6EmbMmIEnn3zScb958+YB4KeuRUREzpUzff++EO+x7+eD/FtD928RERku/H5/z7IR9fX1PeOjRo2C2+3us/3/+T//B1lZWXjppZcGdJ6ioiJkZmb2GZ8xYwauvPJKrF69GtFodICzFxleFCLLsBeJRLBr1y6Ul5cntZuyWDc064bS2NgIAMjNzXU83hNPPAEA+NKXvpS0rhIABAIBAMAtt9yC8vLy05rvpk2bMGrUKBQUFOBXv/oVGhoasGbNGlx//fX43e9+h6uvvjrlvnl5eWhoaDit84qIiJwrZ/LDXb09//zzAIC/+qu/SrlNR0cHAMDn873v8fTBMREREdtw+HD2+/kg/9bQ/VtERIYT87021r1/fh6NRvG9730Pl156KfLy8uB2u2EYBlwuF9ra2nD8+PEBn+cvf/kLrrnmGpSWlsLr9fasm/z8888jEonoZ+Ui78Nzvicgcr7t2LED0WgUH/rQh+By9f1cxaFDhwAAFRUVAIC0tDQAQDgcdjzeq6++Cq/XmzLMDQaD+OpXv3pacz18+DAaGxuxZMkSAIDb7UZ+fj4uv/xyXH755Zg5cya2bduGgwcP9ny6ubdQKNQzfxERkQvRmf5wl6WtrQ2vv/46gFP/YPfw4cMAgIsuuuh953quPzj2/e9/Hzt37sR///d/43e/+x127dqF7du3O+6bl5eH2tra0zqviIjIQA32D2cP5B6bypn4t4bu3yIiMhyEw2E0NTUBAAoLC3vG/+7v/g7PPPMMxo4di2uvvRYlJSXw+/0AgEceeQSRSGRA53n00UfxpS99Cbm5uVi2bBkqKioQDAZhGAb+9Kc/Ydu2bQM+pshwoxBZhj3r09KjR492fP4vf/kLAGD58uUA2AYDsP/T21soFMKRI0cwbty4Pq2rDh48iHfffRcLFy5EcXHxac2196elnVj/AXdq05FIJNDS0oIxY8ac1rlFRETOhTP94S7Liy++iGg0ilmzZqGsrMxxm3g83tMe66Mf/ej7zlUfHBMREaHh9OHsVM7EvzV0/xYRkeHgjTfeQCwWQ3Fxcc/P5Ddu3IhnnnkGS5cuxQsvvACPx46uEokEvv3tbw/oHLFYDA888ABKSkqwefPmPt1P3nrrrQ98HSLDgdpZy7BnhcgtLS19nmtqasKPf/xjlJeX49prrwUAlJaWorCwsKfVVW+hUAimaTr+p/nOO+9EJBJJugEOVO91m5zmunbtWkybNi3pE1yWPXv2wDRNzJw587TPLyIicradyQ939fbnP/8ZAHDNNdek3OZnP/sZGhsbsWDBAixatOiUx7M+ODZq1KiUHxxbsGDBef3gmPXaiIiInG1n48PZF+I99lQ+6L81dP8WEZHhIJFI4P/+3/8LAPjUpz7VM75//34AwMc+9rE+Pz9fv349QqFQn2NZ6yc7LUPR0NCAlpYWLFy4sE+A3NHR0fNvFxE5NYXIMuxZ/4H8wx/+gM7Ozp7xjo4OfOpTn0JjYyMeeeSRnpZZhmFg8eLFaGho6Lm5WXJzc5GRkYH9+/cntb76wQ9+gOeeew6Ac1gNADfeeCMMw8DPf/7z953r7373u551IwCgu7sbt9xyC6LRKO6++27Hfd9++20AwBVXXJHy+CIiIufbmfxwlyWRSOB///d/AaT+we7+/ftx7733wu/34wc/+MH7zlMfHBMREbENlw9np3Im/q2h+7eIiAx1dXVLine truncated
|
||||
"text/plain": [
|
||||
"<Figure size 2000x400 with 4 Axes>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"plot(I=7)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "a1fbd00b-e7f2-4eff-90db-e1d44d104fba",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.10.6"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
@@ -0,0 +1,873 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "1bba0128",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Central Limit Theorem example\n",
|
||||
"\n",
|
||||
"Here, we try adding uniform distributions a few times and we quickly end up with a Gaussian.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"id": "23feddde",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from typing import Optional\n",
|
||||
"\n",
|
||||
"import pandas as pd\n",
|
||||
"import numpy as np\n",
|
||||
"import matplotlib.pyplot as plt\n",
|
||||
"import seaborn as sns\n",
|
||||
"\n",
|
||||
"import matplotlib.pylab as pylab\n",
|
||||
"params = {'legend.fontsize': 'x-large',\n",
|
||||
" 'figure.figsize': (15, 5),\n",
|
||||
" 'axes.labelsize': 'x-large',\n",
|
||||
" 'axes.titlesize':'x-large',\n",
|
||||
" 'xtick.labelsize':'x-large',\n",
|
||||
" 'ytick.labelsize':'x-large'}\n",
|
||||
"pylab.rcParams.update(params)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "bb1286f0",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"The idea is simple: generate random values between $-1$ and $1$ and make a histogram of them. The histogram will look like a box.\n",
|
||||
"\n",
|
||||
"The Central Limit Theorem tells us that the mean of $N$ variables with variance $\\sigma^2$ is a Gaussian with variance $\\sigma^2/N$ in the limit that $N \\rightarrow \\infty$.\n",
|
||||
"\n",
|
||||
"Notice that the variance of this uniform distribution is not 1: calculate it, and you will see that its variance is 1/3 (so $\\sqrt{\\text{var}}=\\sqrt{3}/3\\approxeq 0.57$). We will see that below.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"id": "aef54435-78af-4b13-aee2-2bc576c773cd",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# this generates random uniform samples\n",
|
||||
"def random_samples(n_samples: int) -> np.ndarray:\n",
|
||||
" return (2*np.random.rand(n_samples) - 1)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "04432613-39cd-4209-8833-7af585e7eed9",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Let's make a function to generate samples."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"id": "7ea0322e-a35c-4603-ad46-8b9fd4c216c0",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"n_samples = 10000\n",
|
||||
"x = random_samples(n_samples)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "ae05df16-29b2-4039-a6c2-db7a98e5c22f",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"x has now arbitrary numbers between $-1$ and $1$:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"id": "3f57aeb7-4148-4ba5-b6eb-b600779ef0db",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/html": [
|
||||
"<div>\n",
|
||||
"<style scoped>\n",
|
||||
" .dataframe tbody tr th:only-of-type {\n",
|
||||
" vertical-align: middle;\n",
|
||||
" }\n",
|
||||
"\n",
|
||||
" .dataframe tbody tr th {\n",
|
||||
" vertical-align: top;\n",
|
||||
" }\n",
|
||||
"\n",
|
||||
" .dataframe thead th {\n",
|
||||
" text-align: right;\n",
|
||||
" }\n",
|
||||
"</style>\n",
|
||||
"<table border=\"1\" class=\"dataframe\">\n",
|
||||
" <thead>\n",
|
||||
" <tr style=\"text-align: right;\">\n",
|
||||
" <th></th>\n",
|
||||
" <th>x</th>\n",
|
||||
" </tr>\n",
|
||||
" </thead>\n",
|
||||
" <tbody>\n",
|
||||
" <tr>\n",
|
||||
" <th>0</th>\n",
|
||||
" <td>0.235730</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>1</th>\n",
|
||||
" <td>-0.587471</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>2</th>\n",
|
||||
" <td>0.325833</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>3</th>\n",
|
||||
" <td>0.786101</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>4</th>\n",
|
||||
" <td>0.480421</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>...</th>\n",
|
||||
" <td>...</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>9995</th>\n",
|
||||
" <td>-0.995029</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>9996</th>\n",
|
||||
" <td>-0.299443</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>9997</th>\n",
|
||||
" <td>-0.464221</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>9998</th>\n",
|
||||
" <td>0.751605</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>9999</th>\n",
|
||||
" <td>0.958662</td>\n",
|
||||
" </tr>\n",
|
||||
" </tbody>\n",
|
||||
"</table>\n",
|
||||
"<p>10000 rows × 1 columns</p>\n",
|
||||
"</div>"
|
||||
],
|
||||
"text/plain": [
|
||||
" x\n",
|
||||
"0 0.235730\n",
|
||||
"1 -0.587471\n",
|
||||
"2 0.325833\n",
|
||||
"3 0.786101\n",
|
||||
"4 0.480421\n",
|
||||
"... ...\n",
|
||||
"9995 -0.995029\n",
|
||||
"9996 -0.299443\n",
|
||||
"9997 -0.464221\n",
|
||||
"9998 0.751605\n",
|
||||
"9999 0.958662\n",
|
||||
"\n",
|
||||
"[10000 rows x 1 columns]"
|
||||
]
|
||||
},
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"pd.DataFrame(dict(x=x))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "e6639893-7572-4e93-a13d-2851d8da624f",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Let's check its mean and variance:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"id": "db02f31d-3cde-4dd1-be2e-fdecaff39c2d",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/html": [
|
||||
"<div>\n",
|
||||
"<style scoped>\n",
|
||||
" .dataframe tbody tr th:only-of-type {\n",
|
||||
" vertical-align: middle;\n",
|
||||
" }\n",
|
||||
"\n",
|
||||
" .dataframe tbody tr th {\n",
|
||||
" vertical-align: top;\n",
|
||||
" }\n",
|
||||
"\n",
|
||||
" .dataframe thead th {\n",
|
||||
" text-align: right;\n",
|
||||
" }\n",
|
||||
"</style>\n",
|
||||
"<table border=\"1\" class=\"dataframe\">\n",
|
||||
" <thead>\n",
|
||||
" <tr style=\"text-align: right;\">\n",
|
||||
" <th></th>\n",
|
||||
" <th>x</th>\n",
|
||||
" </tr>\n",
|
||||
" </thead>\n",
|
||||
" <tbody>\n",
|
||||
" <tr>\n",
|
||||
" <th>count</th>\n",
|
||||
" <td>10000.000000</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>mean</th>\n",
|
||||
" <td>0.013094</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>std</th>\n",
|
||||
" <td>0.582749</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>min</th>\n",
|
||||
" <td>-0.999871</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>25%</th>\n",
|
||||
" <td>-0.496063</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>50%</th>\n",
|
||||
" <td>0.025202</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>75%</th>\n",
|
||||
" <td>0.523966</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>max</th>\n",
|
||||
" <td>0.999870</td>\n",
|
||||
" </tr>\n",
|
||||
" </tbody>\n",
|
||||
"</table>\n",
|
||||
"</div>"
|
||||
],
|
||||
"text/plain": [
|
||||
" x\n",
|
||||
"count 10000.000000\n",
|
||||
"mean 0.013094\n",
|
||||
"std 0.582749\n",
|
||||
"min -0.999871\n",
|
||||
"25% -0.496063\n",
|
||||
"50% 0.025202\n",
|
||||
"75% 0.523966\n",
|
||||
"max 0.999870"
|
||||
]
|
||||
},
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"pd.DataFrame(dict(x=x)).describe()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "0f63c4cd-2479-4498-8de4-bf1c378c5545",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We can make a histogram of it to try to double check:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"id": "fead461d-4d3a-48f2-a525-c518be4f1cf8",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"[Text(0, 0.5, 'Fraction of occurrences / bin size'), Text(0.5, 0, 'x')]"
|
||||
]
|
||||
},
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"image/png": "iVBORw0KGgoAAAANSUhEUgAABOAAAAHKCAYAAACqtDSHAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABap0lEQVR4nO3deVyU9f7//+egCIIICqmgIMJxodWlOoimYkdKW6TStDJFW+yTqWie0/H4SbRc6ntajqadVrU8ZeWS5nGjEDUxF1JbBY1UQFNcAdFQ4fr90Q8+jgIOFzMOMz7ut9vcbs71fl/v6zXMxTXjk+u63hbDMAwBAAAAAAAAcAgPZxcAAAAAAAAAuDMCOAAAAAAAAMCBCOAAAAAAAAAAByKAAwAAAAAAAByIAA4AAAAAAABwIAI4AAAAAAAAwIEI4AAAAAAAAAAHIoADAAAAAAAAHKiuswtwJaWlpTp48KD8/PxksVicXQ4AAAAAAACcyDAMFRYWKiQkRB4elZ/nRgBXDQcPHlRoaKizywAAAAAAAEAtkpOToxYtWlTaTgBXDX5+fpL++KE2bNjQydUAAAAAAADAmQoKChQaGlqeGVWGAK4ayi47bdiwIQEcAAAAAAAAJOmytypjEgYAAAAAAADAgQjgAAAAAAAAAAcigAMAAAAAAAAciAAOAAAAAAAAcCACOAAAAAAAAMCBCOAAAAAAAAAAB6rr7AIAAABQuXPnzqmkpMTZZbiUOnXqyNPT09llAAAAlCOAAwAAqIUKCgp09OhRFRcXO7sUl+Tl5aWgoCA1bNjQ2aUAAAAQwAEAANQ2BQUFOnDggBo0aKCgoCB5enrKYrE4uyyXYBiGzp07p/z8fB04cECSCOEAAIDTEcABAADUMkePHlWDBg3UokULgjcT6tevLz8/P+Xm5uro0aMEcAAAwOmYhAEAAKAWOXfunIqLi+Xv70/4VgMWi0X+/v4qLi7WuXPnnF0OAAC4yhHAAQAA1CJlEy4wiUDNlf0MmcQCAAA4GwEcAABALcTZbzXHzxAAANQWBHAAAAAAAACAAzEJAwAAAAAAgIvIzs7W0aNHnV2GaUFBQQoLC3N2GVccARwAAICLqe1fvK/WL9YAADhadna22kVF6czp084uxbT6Pj7K2LXrqvuuQAAHAADgQlzhi/fV+sUaAABHO3r0qM6cPq1HnvunmoZFOrucajucnaWPXv6rjh49etV9TyCAAwAAcCG1/Yv31fzFGgCAK6VpWKRatL7O2WWgGgjgAAAAXBBfvAEAAFwHs6ACAACg1omPj5fFYtHMmTMvaXv++edlsVj02GOPOaEyAACA6uMMOAAAANQ6c+bMUYcOHfS3v/1Nt912mzp06CBJSklJ0bRp03TttdfqjTfecHKVAGBObZ9M53KYbAeoPgI4AAAA1DqNGzfWggUL1L17dw0YMEDbt29XUVGRBg0aJC8vL3322Wfy8fFxdpkAUG2uMJnO5TDZDlB9BHAAAAColWJiYvTiiy9q/PjxGj58uI4cOaJDhw7p3Xff1XXXcf87AK6ptk+mczlMtgOYQwAHAACAWuu5555TamqqPv74Y0nSQw89pMcff9zJVQFAzTGZDnB1IYADABfG/UMAuDuLxaL7779fycnJkqTExETnFgQAAGACARwAuCjuHwLgarBnzx6NGzdOjRo1Un5+vh5//HFt3bpV3t7ezi4NAADAZjUK4M6dO6eUlBTt2rVLp06d0vPPPy9J+v3331VQUKCgoCB5eHjYpVAAgDXuHwLA3RUXF2vAgAEqKirSmjVrtH79ek2dOlWJiYl66623nF0eAACAzUwHcKtXr9Zjjz2mQ4cOyTAMWSyW8gBu586d6tKli/7zn//ooYcesluxAIBLcf8QAO5q3Lhx2rFjh5577jn16tVLPXv21Lp16/T222/r9ttvV//+/Z1dIgAAgE1MnZ6Wnp6u+Ph4WSwWvf7663r44Yet2qOjo9WqVSt9/vnndikSAAAAV5fPP/9cs2bN0p///GdNmTJFklSnTh0tWLBAjRs31uOPP65ff/3VyVUCAADYxtQZcC+++KJ8fHyUnp6uZs2aafLkyZf0ueWWW7R9+/YaFwgAAIBLHc7OcnYJFbJHXdnZ2Xrsscfk7++vTz75RHXr/t9X1tDQUM2ZM0fx8fEaOHCgNm7cqHr16tV4mwAAAI5kKoBLS0tTfHy8mjVrVmmf0NBQrVixwnRhAAAAuFRQUJDq+/joo5f/6uxSKlXfx0dBQUGm1w8LC9Px48crbe/bt68MwzA9PgDX58ozwe/atcvZJQBwAlMB3KlTpy77per06dMqLS01VRQAAAAqFhYWpoxdu2r1fzyDgoKYXAWAw7jDTPDSH/+vBnD1MBXANW/eXD/99FOVfXbu3KmIiAhTRQEAAKByYWFhBFwArlquPhP8rq3rteqDGfr999+dXUqNuPKZfPyhCM5gKoDr3bu33nrrLW3cuFFdu3a9pH3VqlXatGmT/v73v9e4QAAAAAAALuaqM8HX1nt42qrg+BFJ0qBBg5xciXle3t5avGiRgoODnV1Ktbly8Hm1MxXAjR8/Xp988oni4uI0cuRI7du3T5K0YsUKbdiwQbNnz1ZwcLDGjh1rz1oBAAAAAIATnTlVIEm6a/gEtb2xk5Orqb5ff0zX0n9P09133+3sUmqES5hdj+lLUJOTk/Xggw/qn//8Z/nye++9V4ZhKDIyUkuWLKnRzXcBAAAAAEDtFBjS0qXPQHTVANFdLmG+GpkK4CSpY8eOyszM1IoVK/TNN9/o2LFj8vf3V3R0tPr27Ws1XTwAAAAA+3LlWSAl7sEEwLlcPUCE66lRSlanTh3de++9uvfee+1VDwAAAIDLcIdZIOv7+Chj1y5COADAVcFUAPfhhx/Kx8dH/fr1q7TP999/r507d2rw4MGmiwMAALhaGYbh7BJcnjv/DF19FsjD2Vn66OW/6ujRowRwAICrgqkALiEhQRaLRQ888IDmz58vLy+vS/p8/vnneuGFFwjgAAAAqqFOnTqSpHPnzql+/fpOrsa1nTt3TtL//Uwv5sqXcJbNgueqs0ACAHC1MX0JauPGjbVo0SLl5ubqiy++sPuEC7m5uZo4caJWr16tY8eOKTg4WPHx8UpKSlKjRo1sGqNHjx5av359pe1nzpyRt7e3vUp2Wa785VPi/iEAAPfi6ekpLy8v5efny8/PTxaLxdkluSTDMJSfny8vLy95enpe0u4Ol3BKzIIHAICrMB3AjRw5UiUlJXrxxRfVuXNnrVixQm3atLFLUVlZWYqJiVFeXp769u2rdu3aaevWrZoxY4ZWr16ttLQ0BQYG2jxeUlJShcuZKMI9vny6+v1DCECdy5V//mVnPwBwP0FBQTpw4IByc3Pl7+8vT09PgjgbGYahc+fOKT8/X6dOnVLz5s0r7Ofql3AyCx4AAK6lRgnU5MmTFRkZqSeeeEIxMTH6/PPPddttt9W4qKefflp5eXmaOXOmRo4cWb587Nixev311zVhwgS99dZbNo83adKkGtfkrlz9y6er3z+EANS53OHnL3H2A+COGjZsKOmPz+kDBw44uRrX5OXlpebNm5f/LCvjqpdwMgseAACupcangA0ePFihoaG6//77FRcXp/fee0+PPPKI6fGysrKUnJys8PBwjRgxwqpt8uTJeueddzR//ny9+uqr8vX1rWn5+P+56pdPV0cA6lyu/vPn7AfAvTVs2FANGzbUuXPnVFJS4uxyXEqdOnUqvOwUAADAWexyDWZsbKw2bdqkPn36aPDgwfr1119Nj5WamipJiouLk4eHh1Wbn5+funTpouTkZG3evFm33367TWN++umn2rt3r+rVq6eoqCj17NmzwokjAGchAHUuV/35c/YDcHXw9PQkTILbcuXbKbj6bTgAAFeW3W6CFhUVpa1bt+ruu+/WpEmTqnWPtgtlZmZKUqX3k2vdurWSk5O1e/dumwO4gQMHWj1v0qSJZs+erX79+lW5XnFxsYqLi8ufFxQU2LQ9AAAAAJUrOH5EkjRo0CAnV2Kel7e3Fi9apODgYGeXYgoBIgBcWXadheCaa67R+vXr9cgjj+jzzz83dbPg/Px8SZK/v3+F7WXLT548edmx+vbtq3HjxqlDhw4KDAzU/v379cEHH+jVV1/VgAEDtGLFCt15552Vrj99+nRNnjy52q8BAAAAQOXOnPrjD9t3DZ+gtjd2cnI11ffrj+la+u9puvvuu51dimmufB9fAHBFpgK4vXv3KiAgoMI2b29vLVq0SIsWLdJpJ9/YfMyYMVbP27Ztq2nTpikkJEQjR47U+PHjqwzgxo8fr7Fjx5Y/LygoUGhoqMPqhXmuevmCq9YNAABgD4EhLV36NhCuGiC6+n18AcAVmQrgWrZsWWW7xWJR//79TRVUdoZb2ZlwFytbXlkAaIvHH39cY8aM0c6dO1VYWCg/P78K+3l5eXGvuFrOHS5fkJjFEnBV2dnZOnr0qLPLMI3LjwCgZlw1QAQAXHl2vQTVHtq2bStJ2r17d4Xte/bskVT5PeJs4e3tLT8/P504cUJFRUWVBnCo/Vz98gVmsQRcV3Z2ttpFRemMk8/2rgkuPwIAAACuDJsCuIiICFksFn311Vdq1aqVIiIibBrcYrEoK6t6s/TFxsZKkpKTk1VaWmo1E2phYaHS0tLk4+Oj6Ojoao17oczMTJ04cUJ+fn4KCgoyPQ5qD1f96yOzWAKu6+jRozpz+rQeee6fahoW6exyqo3LjwAAAIArx6YArrS01GpChYufV8YwjGoXFBkZqbi4OCUnJ2v27NkaOXJkeVtSUpKKioo0fPhw+fr6li/PyMiQJLVr16582d69e+Xv76/GjRtbjX/kyBENHTpU0h+zo9atW+tOAgQAuJCmYZEu+QcAAAAAAFeOTenTvn37qnxub2+++aZiYmI0atQopaSkKCoqSlu2bFFqaqratGmjqVOnWvWPioqSZB34rV+Line truncated
|
||||
"text/plain": [
|
||||
"<Figure size 1500x500 with 1 Axes>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"sns.histplot(pd.DataFrame(dict(x=x)), stat=\"density\")\n",
|
||||
"plt.gca().set(ylabel=\"Fraction of occurrences / bin size\", xlabel=\"x\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "00e098e8-0082-4451-927a-073cf242787e",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"It looks like the probability of getting a specific x value is indeed the same for all x in the range [-1, 1].\n",
|
||||
"\n",
|
||||
"Let's get values of x independently and sum them $N$ times:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"id": "08d9daaf-25c5-4d06-b4a6-6cecaaa35651",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def get_many_x(N: int) -> pd.DataFrame:\n",
|
||||
" return pd.DataFrame({f\"x{k}\": random_samples(n_samples) for k in range(N)})"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 9,
|
||||
"id": "998baa6c-dc2e-42bc-85a5-295a63db95ea",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"N = 10\n",
|
||||
"data = get_many_x(N=10)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"id": "2195bfb9-acec-47e7-8bae-64a2671eb09d",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/html": [
|
||||
"<div>\n",
|
||||
"<style scoped>\n",
|
||||
" .dataframe tbody tr th:only-of-type {\n",
|
||||
" vertical-align: middle;\n",
|
||||
" }\n",
|
||||
"\n",
|
||||
" .dataframe tbody tr th {\n",
|
||||
" vertical-align: top;\n",
|
||||
" }\n",
|
||||
"\n",
|
||||
" .dataframe thead th {\n",
|
||||
" text-align: right;\n",
|
||||
" }\n",
|
||||
"</style>\n",
|
||||
"<table border=\"1\" class=\"dataframe\">\n",
|
||||
" <thead>\n",
|
||||
" <tr style=\"text-align: right;\">\n",
|
||||
" <th></th>\n",
|
||||
" <th>x0</th>\n",
|
||||
" <th>x1</th>\n",
|
||||
" <th>x2</th>\n",
|
||||
" <th>x3</th>\n",
|
||||
" <th>x4</th>\n",
|
||||
" <th>x5</th>\n",
|
||||
" <th>x6</th>\n",
|
||||
" <th>x7</th>\n",
|
||||
" <th>x8</th>\n",
|
||||
" <th>x9</th>\n",
|
||||
" </tr>\n",
|
||||
" </thead>\n",
|
||||
" <tbody>\n",
|
||||
" <tr>\n",
|
||||
" <th>0</th>\n",
|
||||
" <td>0.557678</td>\n",
|
||||
" <td>-0.973417</td>\n",
|
||||
" <td>0.722963</td>\n",
|
||||
" <td>-0.011507</td>\n",
|
||||
" <td>-0.297092</td>\n",
|
||||
" <td>-0.096422</td>\n",
|
||||
" <td>0.649971</td>\n",
|
||||
" <td>-0.822024</td>\n",
|
||||
" <td>-0.953068</td>\n",
|
||||
" <td>-0.465009</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>1</th>\n",
|
||||
" <td>0.077690</td>\n",
|
||||
" <td>-0.597280</td>\n",
|
||||
" <td>0.659335</td>\n",
|
||||
" <td>0.433345</td>\n",
|
||||
" <td>0.407872</td>\n",
|
||||
" <td>0.026935</td>\n",
|
||||
" <td>0.760887</td>\n",
|
||||
" <td>0.375025</td>\n",
|
||||
" <td>0.436400</td>\n",
|
||||
" <td>0.940421</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>2</th>\n",
|
||||
" <td>0.389252</td>\n",
|
||||
" <td>0.312857</td>\n",
|
||||
" <td>0.744985</td>\n",
|
||||
" <td>-0.547178</td>\n",
|
||||
" <td>0.918958</td>\n",
|
||||
" <td>0.842294</td>\n",
|
||||
" <td>-0.839457</td>\n",
|
||||
" <td>-0.885617</td>\n",
|
||||
" <td>-0.660291</td>\n",
|
||||
" <td>0.799358</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>3</th>\n",
|
||||
" <td>0.002392</td>\n",
|
||||
" <td>-0.452201</td>\n",
|
||||
" <td>0.626875</td>\n",
|
||||
" <td>0.467181</td>\n",
|
||||
" <td>-0.242254</td>\n",
|
||||
" <td>0.342050</td>\n",
|
||||
" <td>-0.804150</td>\n",
|
||||
" <td>-0.757346</td>\n",
|
||||
" <td>0.534880</td>\n",
|
||||
" <td>-0.822235</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>4</th>\n",
|
||||
" <td>0.013865</td>\n",
|
||||
" <td>0.772512</td>\n",
|
||||
" <td>-0.336169</td>\n",
|
||||
" <td>-0.955743</td>\n",
|
||||
" <td>-0.352194</td>\n",
|
||||
" <td>0.788721</td>\n",
|
||||
" <td>0.636567</td>\n",
|
||||
" <td>0.583178</td>\n",
|
||||
" <td>0.733763</td>\n",
|
||||
" <td>-0.785763</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>...</th>\n",
|
||||
" <td>...</td>\n",
|
||||
" <td>...</td>\n",
|
||||
" <td>...</td>\n",
|
||||
" <td>...</td>\n",
|
||||
" <td>...</td>\n",
|
||||
" <td>...</td>\n",
|
||||
" <td>...</td>\n",
|
||||
" <td>...</td>\n",
|
||||
" <td>...</td>\n",
|
||||
" <td>...</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>9995</th>\n",
|
||||
" <td>0.696557</td>\n",
|
||||
" <td>0.653128</td>\n",
|
||||
" <td>-0.452445</td>\n",
|
||||
" <td>-0.474580</td>\n",
|
||||
" <td>0.143816</td>\n",
|
||||
" <td>-0.049960</td>\n",
|
||||
" <td>-0.768933</td>\n",
|
||||
" <td>-0.174698</td>\n",
|
||||
" <td>-0.104592</td>\n",
|
||||
" <td>0.463803</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>9996</th>\n",
|
||||
" <td>-0.897026</td>\n",
|
||||
" <td>-0.488836</td>\n",
|
||||
" <td>-0.014449</td>\n",
|
||||
" <td>-0.757131</td>\n",
|
||||
" <td>0.316350</td>\n",
|
||||
" <td>0.042017</td>\n",
|
||||
" <td>-0.403770</td>\n",
|
||||
" <td>0.548561</td>\n",
|
||||
" <td>0.591823</td>\n",
|
||||
" <td>-0.550146</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>9997</th>\n",
|
||||
" <td>0.834296</td>\n",
|
||||
" <td>-0.850360</td>\n",
|
||||
" <td>0.727586</td>\n",
|
||||
" <td>-0.098943</td>\n",
|
||||
" <td>-0.560702</td>\n",
|
||||
" <td>0.054720</td>\n",
|
||||
" <td>0.673407</td>\n",
|
||||
" <td>-0.041506</td>\n",
|
||||
" <td>0.252130</td>\n",
|
||||
" <td>-0.175144</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>9998</th>\n",
|
||||
" <td>-0.715145</td>\n",
|
||||
" <td>-0.483422</td>\n",
|
||||
" <td>0.874546</td>\n",
|
||||
" <td>0.463453</td>\n",
|
||||
" <td>-0.359133</td>\n",
|
||||
" <td>-0.131677</td>\n",
|
||||
" <td>0.906279</td>\n",
|
||||
" <td>0.422127</td>\n",
|
||||
" <td>-0.231510</td>\n",
|
||||
" <td>-0.981712</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>9999</th>\n",
|
||||
" <td>-0.230620</td>\n",
|
||||
" <td>-0.342262</td>\n",
|
||||
" <td>-0.582301</td>\n",
|
||||
" <td>0.036937</td>\n",
|
||||
" <td>-0.440668</td>\n",
|
||||
" <td>-0.611692</td>\n",
|
||||
" <td>0.633051</td>\n",
|
||||
" <td>0.306141</td>\n",
|
||||
" <td>-0.412429</td>\n",
|
||||
" <td>0.053962</td>\n",
|
||||
" </tr>\n",
|
||||
" </tbody>\n",
|
||||
"</table>\n",
|
||||
"<p>10000 rows × 10 columns</p>\n",
|
||||
"</div>"
|
||||
],
|
||||
"text/plain": [
|
||||
" x0 x1 x2 x3 x4 x5 x6 \n",
|
||||
"0 0.557678 -0.973417 0.722963 -0.011507 -0.297092 -0.096422 0.649971 \\\n",
|
||||
"1 0.077690 -0.597280 0.659335 0.433345 0.407872 0.026935 0.760887 \n",
|
||||
"2 0.389252 0.312857 0.744985 -0.547178 0.918958 0.842294 -0.839457 \n",
|
||||
"3 0.002392 -0.452201 0.626875 0.467181 -0.242254 0.342050 -0.804150 \n",
|
||||
"4 0.013865 0.772512 -0.336169 -0.955743 -0.352194 0.788721 0.636567 \n",
|
||||
"... ... ... ... ... ... ... ... \n",
|
||||
"9995 0.696557 0.653128 -0.452445 -0.474580 0.143816 -0.049960 -0.768933 \n",
|
||||
"9996 -0.897026 -0.488836 -0.014449 -0.757131 0.316350 0.042017 -0.403770 \n",
|
||||
"9997 0.834296 -0.850360 0.727586 -0.098943 -0.560702 0.054720 0.673407 \n",
|
||||
"9998 -0.715145 -0.483422 0.874546 0.463453 -0.359133 -0.131677 0.906279 \n",
|
||||
"9999 -0.230620 -0.342262 -0.582301 0.036937 -0.440668 -0.611692 0.633051 \n",
|
||||
"\n",
|
||||
" x7 x8 x9 \n",
|
||||
"0 -0.822024 -0.953068 -0.465009 \n",
|
||||
"1 0.375025 0.436400 0.940421 \n",
|
||||
"2 -0.885617 -0.660291 0.799358 \n",
|
||||
"3 -0.757346 0.534880 -0.822235 \n",
|
||||
"4 0.583178 0.733763 -0.785763 \n",
|
||||
"... ... ... ... \n",
|
||||
"9995 -0.174698 -0.104592 0.463803 \n",
|
||||
"9996 0.548561 0.591823 -0.550146 \n",
|
||||
"9997 -0.041506 0.252130 -0.175144 \n",
|
||||
"9998 0.422127 -0.231510 -0.981712 \n",
|
||||
"9999 0.306141 -0.412429 0.053962 \n",
|
||||
"\n",
|
||||
"[10000 rows x 10 columns]"
|
||||
]
|
||||
},
|
||||
"execution_count": 10,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"data"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "90c7b417-4b22-4788-9cb4-b9a63f42a1b2",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Each x column has independent values of a uniform distribution. We can double check that:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 11,
|
||||
"id": "aa75ce65-2e94-4d64-93ca-80bb953e81d1",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"[Text(0, 0.5, 'Fraction of occurrences / bin size'), Text(0.5, 0, 'x')]"
|
||||
]
|
||||
},
|
||||
"execution_count": 11,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"image/png": "iVBORw0KGgoAAAANSUhEUgAABO0AAAHKCAYAAABfSr83AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAADSTElEQVR4nOzdeXxU5dk//s+ZJbNkTyAsISGALAFEcAEEEYWCWltBqoItaFWsfagLUiqiP2QR8OnXqg+IFrFFLVYQF0BEIYphNSyBsCckBEKAELIvs2/n98fMhMQkEIb7JBn4vPuymszMNXdmOct17vu6JFmWZRAREREREREREVGroWrpARAREREREREREVFdTNoRERERERERERG1MkzaERERERERERERtTJM2hEREREREREREbUyTNoRERERERERERG1MkzaERERERERERERtTJM2hEREREREREREbUyTNoRERERERERERG1MpqWHsC1zuPxoKCgAOHh4ZAkqaWHQ0RERERERERELUiWZVRXV6Njx45QqRqfT8ekncIKCgqQkJDQ0sMgIiIiIiIiIqJW5MyZM+jUqVOjtzNpp7Dw8HAA3jciIiKihUdDREREREREREQtqaqqCgkJCTU5o8Ywaacw/5LYiIgIJu2IiIiIiIiIiAgALltGjY0oiIiIiIiIiIiIWhkm7YiIiIiIiIiIiFoZJu2IiIiIiIiIiIhaGSbtiIiIiIiIiIiIWhkm7YiIiIiIiIiIiFoZJu2IiIiIiIiIiIhaGU1LD4CIiIiIiIiIiMRyOp1wu90tPYzrglqthlarFR6XSTsiIiIiIiIiomtEVVUVSkpKYLfbW3oo1xWdToc2bdogIiJCWEwm7YiIiIiIiIiIrgFVVVU4d+4cwsLC0KZNG2i1WkiS1NLDuqbJsgyn04nKykqcO3cOAIQl7pi0IyIiIiIiIiK6BpSUlCAsLAydOnVisq4ZGQwGhIeH4+zZsygpKRGWtGMjCiIiIiIiIiKiIOd0OmG32xEZGcmEXQuQJAmRkZGw2+1wOp1CYrbapN3Zs2fx5JNPomPHjtDpdEhKSsLUqVNRXl5+RXHKysowdepUJCUlQafToWPHjnjyySdx9uzZSz5u8+bNePDBB9G+ffuax91zzz347rvvrubPIiIiIiIiIiISzt90QomGCNQ0/tdeVAOQVrk8Njc3F0OGDEFRURHGjBmDXr16Yc+ePVi0aBE2btyInTt3IjY29rJxSktLMWTIEGRnZ2PEiBGYMGECsrKy8NFHH2HDhg1IS0tD165d6z3upZdewptvvolOnTrhgQceQJs2bVBcXIx9+/Zhy5Yt+PWvf63En01EREREREREdFU4y67liH7tW2XSbsqUKSgqKsLixYvx3HPP1fx+2rRpeOedd/Dqq69i6dKll43zyiuvIDs7G9OmTcNbb71V8/vFixfjhRdewJQpU7Bx48Y6j/nwww/x5ptv4vHHH8eyZcsQEhJS53ZRUxyJiIiIiIiIiIgaI8myLLf0IGrLzc3FDTfcgKSkJOTm5kKluriCt7q6Gh06dIAsyygqKkJoaGijcUwmE+Li4qBSqXD+/HmEh4fX3ObxeNC1a1ecPn0aubm5NbPt7HY7EhISYDAYkJOTUy9hF4iqqipERkaisrJSaNtfIiJSXn5+PkpKSoTHbdOmDRITE4XHJSIiEon7QaLgYrPZcOrUKXTp0gV6vb6lh3Ndaup70NRcUaubaZeamgoAGD16dJ2EHQCEh4dj6NChSElJwa5duzBy5MhG4+zatQtWqxWjR4+uk7ADAJVKhXvuuQfLli1DampqTdLuhx9+QHFxMaZOnQqVSoUNGzbgyJEj0Ov1GDhwIG6//XbBfy0REbVW+fn56JXcC1aLVXhsg9GArMwsnrAQEVGr5d0PJsNqsQiPbTAakZWZyf0gUQtQKhkvCpP6dbW6pN3x48cBAD169Gjw9u7duyMlJQXZ2dmXTNo1JQ4AZGdn1/xu7969AAC9Xo8BAwbgyJEjdR5z55134ssvv0Tbtm0bfV673Q673V7zc1VVVaP3JSKi1qukpARWixWPL3gc7bu0Fxa38FQhPnn1E5SUlPCAhIiIWi3vftCCCW8vQdwNNwiLW3TiBFZNe5b7QaIWoGQyXpSWSuofO3YMc+bMwZYtW1BVVYXOnTtjwoQJePnll2EwGJp1LLW1uqRdZWUlACAyMrLB2/2/r6ioEB6nqKgIAPDmm2+id+/e2L59O/r3749Tp05h+vTpSElJwcMPP4wtW7Y0+rxvvPEG5s6de8mxERFR8GjfpT0Sk3lSQURE16e4G25Ap779WnoYRCSAPxn/hxlvol1it5YeTj0X8nPx37//rdmT+rt378aIESPgdDrx0EMPISEhAT/99BPmzZuHzZs3Y/PmzdDpdM02ntpaXdKuJXk8HgCARqPBN998g6SkJADAjTfeiDVr1qBnz57YunUr0tLSGl0qO3PmTEybNq3m56qqKiQkJCg+diIiIiIiIiKiy2mX2A2duvdp6WG0Cm63G0888QQsFgvWrVuHBx54AIA3P/TII4/gq6++wjvvvIOXX365Rcanuvxdmpd/Bpx/ptwv+X8fFRUlPI7/vwcMGFCTsPMzGo245557AAB79uxp9Hl1Oh0iIiLq/ENERERERERERMobO3YsJEnC4sWL6902a9YsSJKEp556CgCwdetWZGZm4s4776xJ2AHeXgj/7//9PwDA0qVL0VI9XFtd0q5nz54A6taaqy0nJwdA47XqriaO/zGNJQSjo6MBAFar+KLkRERERERERER0dZYvX47ExES89NJLyMjIqPn95s2bsXDhQvTu3RvvvvsuAOCnn34CANx777314nTt2hU9evTA6dOncfLkyeYZ/C+0uuWxd999NwAgJSUFHo+nTgfZ6upq7Ny5E0ajEYMHD75knMGDB8NgMGDnzp2orq6u00HW4/EgJSWlzvMBwMiRIyFJEo4dO1bvuQHUNKbo0qXL1f2RRETXIaU6VbHDFBERERER+cXExGDlypUYPnw4xo8fj/3798NsNmPixInQ6XRYvXo1jEYjgKY1Mc3OzkZ2dja6dWv+OoCtLmnXrVs3jB49GikpKXjvvffw3HPP1dw2e/ZsmM1mPPPMMwgNDa35fVZWFgCgV69eNb8LCwvDpEmTsGzZMsyZMwdvvfVWzW1LlixBXl4e7rnnHnTt2rXm9507d8Zvf/tbfPPNN1i0aBFefPHFmttSUlKwadMmREVFNZiBJSKixuXn5yM5uRcsFvEzlY1GAzIzs5i4IyIiIiIiAMCQIUPw+uuvY+bMmXjmmWdQXFyMwsJCfPjhh+jT52I9P1HNUJXS6pJ2APD+++9jyJAheP7557F582YkJydj9+7dSE1NRY8ePbBgwYI6909OTgaAemuMFy5ciC1btuDtt9/GgQMHMHDgQGRmZmLdunWIi4vDe++9V++533vvPWRkZGDatGnYsGEDBgwYgFOnTmHt2rVQq9X417/+1eibSUREDSspKYHFYsWnrzyC5MS2wuJm5hdj4sLVzd5hioiIiIiIWrcZM2YgNTUVn332GQDg0UcfxeTJk1t4VFemVSbtunXrhvT0dLz22mvYuHEjvvvuO3To0AEvvPACZs+eXVNb7nJiY2ORlpaGuXPnYu3atdi+fTtiY2PxxBNPYN68eejUqVO9x3Tq1An79u3DvHnz8M0332Dbtm2IiIjAb3/7W8ycORMDBw4U/ecSEV03khPb4uYe8S09DCIiCnIsuUBERJcjSRLGjRtXUx5t6tSp9e4jqhmqUlpl0g4AEhIS8NFHHzXpvpfq4hETE4NFixZh0aJFTX7utm3b4t13360pTEhERERERK1Dfn4+knv1gkWB5nBGgwGZWSy5QER0LcjJycH06dMRHR2NyspKTJ48GXv27IFer6+5j6hmqEq5qqSd0+nE5s2bkZmZCZPJhFmzZgEAbDYbqqqq0KZNm3rNHIiIiIiIiAJVUlICi9WK9yc9hh7t2wuLm11YiCkr/sOSC0RE1wC73Y7x48fDbDZj06ZN2Lp1KxYsWICpU6di6dKlNfcbMWIEFixYgI0bN2LmzJl1Ypw8eRLZ2dno3LlznX4IzSngpN3GjRvx1FNPobCwELIsQ5KkmqTdgQMHMHToUHz66ad49NFHhQ2WiIiIiIgIAHq0b49+CQktPQwiImqFpk+fjoyMDMyYMQOjRo3CiBEjsGXLFnzwwQcYOXIkHn74YQDA8OHDkZycjG3btuGbb77BAw88AADweDyYMWMGAODPf/4zJElqkb8joKRdeno6xo4dizZt2uCdd97Bnj17sHLlyprbBw8ejC5dumDNmjVM2hERETUgMzNTkbisx0RERERE17M1a9ZgyZIlGDRoEObPnw8AUKvVWLlyJfr374/JkyfjlltuQdeuXaFWq/HRRx9hxIgReOihh/DQQw8hMTERmzdvRnp6OoYOHYoXX3yxxf6WgJJ2r7/+OoxGI9LT09G+fXvMnTu33n1uu+027N+//6oHSEREdC0xmUwAgIkTJyoS32A0Iiszk4k7IiIiImrQhfzclh5Cg0SMKz8/H0899RQiIyOxatUqaDQX014JCQlYvnw5xo4diwkTJmDHjh0ICQnBoEGDsHfvXsyePRspKSmorq5G586d8dprr+Hll1+GTqe76nEFKqCk3c6dOzF27Fi0v0QNiYSEBGzYsCHggREREV2LbDYbAGD0tBnoddfdQmMXnTiBVdOeZT0mIiIiIqqnTZs2MBiN+O/f/9bSQ2mUwWhEmzZtAn58YmIiysrKGr19zJgxDTYz7d27N7744ouAn1cpASXtTCbTZV9Ei8UCj8cT0KCIKHjl5+ejpKREeFwu+aNrTUxCJ3Tq26+lh0FEpBiljgmUKi9ARNSaBLINlWUZGo0GFosFbre73u2xsbHYv28fKioqEBISImqoQvG8r66Aknbx8fE4evToJe9z4MCLine truncated
|
||||
"text/plain": [
|
||||
"<Figure size 1500x500 with 1 Axes>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"sns.histplot(data.loc[:, [f\"x{k}\" for k in range(N)]], stat=\"density\")\n",
|
||||
"plt.gca().set(ylabel=\"Fraction of occurrences / bin size\", xlabel=\"x\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "f7b8daf9-dd92-49d2-96e7-88f8c0f2758e",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"They all look mostly flat. Now let's create a new variable, called \"mean_x_0_{k}\", which will be the mean of the x variables from 0 until the k-th element:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 12,
|
||||
"id": "a4b370e3-bd2b-4ca2-b13f-d71fe875f89f",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"for k in range(N):\n",
|
||||
" data[f\"mean_x_0_{k}\"] = sum(data.loc[:, f\"x{i}\"] for i in range(k+1))/float(k+1)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "fa184b55-e5ae-4b73-947e-747856f0ab05",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Let's histogram these sums:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 13,
|
||||
"id": "437d8afb-414a-4ec6-8f5a-cdaf97136183",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"image/png": "iVBORw0KGgoAAAANSUhEUgAAA10AAALZCAYAAABMLUslAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAC3UUlEQVR4nOzdd1QUZ9sG8GvpgrCIqICCKDaMBbsCiliwK/aSxPYlMc2evIlvVNRY04wmmsTEnqiJFbsoUhQrYlcsKGJDsNCRtvv9sS9EnAVh2N3Zhet3DueE+xlmL4yFe2fmfmRKpVIJIiIiIiIi0gojqQMQERERERGVZ2y6iIiIiIiItIhNFxERERERkRax6SIiIiIiItIiNl1ERERERERaxKaLiIiIiIhIi9h0ERERERERaRGbLiIiIiIiIi0ykTqAIVEoFHj06BGsra0hk8mkjkNERERERBJRKpVITU2Fk5MTjIyKv5bFpqsUHj16BGdnZ6ljEBERERGRnrh//z5q1apV7DFsukrB2toagOoX1sbGRuI0REREREQklZSUFDg7Oxf0CMVh01UK+bcU2tjYsOkiIiIiIqISPXbEQRpERERERERaxKaLiIiIiIhIi9h0ERERERERaRGbLiIiIiIiIi1i00VERERERKRFbLqIiIiIiIi0iE0XERERERGRFrHpIiIiIiIi0iI2XURERERERFrEpouIiIiIiEiL2HQRERERERFpEZsuIiIiIiIiLWLTRUREREREpEVsuoiIiIiIiLSITRcREREREZEWsekiogolPDwc/fv3R82aNSGTybBu3TqpIxEREVE5x6aLqAJLT08v09cvWrQIbdq0gY2NDapVq4Z+/frhypUrGkqnHWlpaWjSpAmWLVuGSpUqaey8Zf21JCIiovKLTRdRBXXs2DHs3LmzTOcIDQ3Fxx9/jBMnTuDo0aMwMTFBt27d8Pz5cw2l1LzevXtj4cKFGDJkCIyMNPdX4LNnz7Bw4UKNnY+IiIjKDxOpAxCR7j1+/BjfffddmZuuQ4cOFfp848aNkMvliIiIQL9+/cp0bkPj4uICY2NjbN26FUOHDpU6DhEREekRXuki0rDY2FjIZDKMHTtW6ihF+vDDDzFnzhyNXukBgNTUVCgUClSpUkWj5zUU06dPxw8//IDExESpoxAREZEeYdNFOnPz5k1MmzYNLVu2hJ2dHUxNTWFnZ4d27drhs88+w7lz56SOWCEcOXIET548QYsWLTR+7smTJ8PDwwMdOnTQ+LkNgYmJCYYMGYKZM2dKHYWIiIj0CG8vJK1TKpWYN28e5s2bB4VCgZYtW2L48OGws7NDamoqLl26hJ9++gnff/89fv75Z3zyySdSRy6TmjVr4vr165DL5VJHUWvhwoUYM2aMxs87bdo0HD9+HMePH4exsbHGz28o3nnnHdStWxezZs1CrVq1pI5DREREeoBNF2ndvHnzMGfOHDg7O2Pz5s3w8vISHJOQkIAff/wRycnJEiTULFNTUzRq1EjqGGrdunULISEh+PPPPzV63qlTp2LLli0ICQlB3bp1NXpuQ1OjRg20bt0aa9aswezZs6WOQ0RERHqAtxeSVt25cwfz58+HmZkZDhw4oLbhAoDq1atj4cKF+M9//lOovm7dOgwePBh169ZFpUqVYGNjAy8vL7VNQ2hoKGQyGebMmaP2NVxdXeHq6iqo7969G127doWjoyPMzc3h5OQEHx8frFy5UtRxRT3TVZrv5fXzxMbGYsSIEbC3t4eFhQVat26NvXv3qv264vz999+oUaMGnJycijwmLS0N8+bNQ4sWLWBtbQ2ZTKb248mTJwBUtxRu3rwZR48e1dtm81VpaWm4cOECLly4AIVCgbi4OFy4cAFxcXEae42WLVti8+bNGjsfERERGTY2XaRVa9euRW5uLoYMGYK33nrrjcebmBS++PrRRx/h3r176NSpE6ZMmYIRI0bg3r17ePfddzFr1qwy51u1ahUGDBiAa9euoV+/fpg+fTp69+6NzMxMrF27ttTHFUfs93Lv3j20bdsWsbGxePfddzF8+HBcuXIFAwYMQEhISKm+36NHjxb7LFdCQgLatGmDgIAAKBQKfPjhh5g4cSIcHBwAqK7iubm5oV27dqhRowY++eQTrF27Fps2bUKVKlUQHx+P+Ph4pKWllSqXLkVGRqJFixZo0aIFMjMzERAQgBYtWmj0qlSLFi0QHR2N+Ph4jZ2TiIiIDBdvL5SKpgcNjBgBTJ5c/DHLlgFbtpTsfCdPlj0TgIiICABAly5dRH39lStX4ObmVqiWnZ2NXr16YfHixfjwww9Rs2ZN0fl+++03mJmZ4eLFi6hevXqhtadPn5b6uOKI/V5CQ0MxZ84cBAQEFNRGjRqFnj174ttvv4Wvr2+JXh9QNRzvvfdekeujRo1CdHQ0/vOf/2Dx4sWQyWQAgM8//xz169dHXl4eTp06BXt7ewAouMrXtWvXQucJCAgo8opjvh9//BFJSUklzu7h4QF/f/8SH1+Uzp07Q6lUlvk8xWncuDEA4PTp0xgwYIBWX4uIiIj0H5suqZw6pdnzlaSJu3dP86/7Bvnv9KtrJmJjY7Fu3bpCNVtbW0yZMqXg89ebFAAwMzPDJ598gqNHjyI4OBijR48uU0YTExOYmpoK6vmNRWmPK4rY76V27dqCaXg9evSAi4sLzpw5U6LXBoAnT54gNTUVtra2atcPHz6M4OBgeHt7Y9GiRQUNFwA4OzujY8eOOHLkCC5cuIBu3boBQJmalx9//BH37t0r8fFjxozRSNOlC/m/xjExMdIGISIiIr3AposkExsbi7lz5xaq1a5du1DTFRcXhyVLliA4OBhxcXHIzMwsdPzDhw/LlOHtt9/G9OnT0bhxY4wYMQI+Pj7w8vJCtWrVRB1XHLHfi4eHh9ppgM7OzjhZiiuSjx8/BgDY2NioXc9/tmzKlClq9+/Kn8aoUChK/JrFiY2N1ch59FH+r3H+rzkRERFVbGy6SKscHBxw/fp1PHr0SLD26m1eubm5gqtId+7cQdu2bfHixQt07NgRfn5+kMvlMDY2RmxsLNavX4+srKwy5Zs2bRrs7e2xcuVKLF++HD/++CNkMhl8fHzw7bffonXr1qU6rihl+V6KujJlYmJSqgYoPT0dQNFN17Fjx2BkZISePXuqXX/w4AEAoF69eiV+zYoqv0HN/zUnIiKiio1NF2mVl5cXQkJCEBwcjPHjx5fqa3/44Qc8e/YMa9euFUwC3Lx5M9avX1+oln91Jjc3V+35kpKS1DYwo0ePxujRo5GUlIQTJ05g586dWLNmDXr06IHo6OiCq1klPU4T34s2qbslMC8vD/fu3UP16tVhZWUlWH/y5AnOnj2LOnXqaGwkvKae6Xr1NkipvP5rmv+5PmQjIiIi6bHpkkr79po9X+3aJTtG06/7BmPHjsXixYuxbds2zJw5E+7u7iX+2tu3bwMABg8eLFgLCwsT1KpUqQIAuH//vtpzJScnF3nVCFBdUerduzd69+4NhUKBNWvWIDw8XPD6JT2uLN+LNuQ3UykpKYK1/IY1NTUVCoVCcHvhN998A4VCgQkTJmgsj6ae6dL2UAwx8n+N1TWwREREVPGw6ZKKhqYDlsrkyW+ecKhhbm5umDlzJubMmYNevXph06ZN8PT0FByn7opH/p5aoaGh6NevX0H90KFD+OOPPwTHN2rUCDY2NggMDERCQkLBlMHMzExMmjRJbb6QkBB07txZcEUiISEBAGBpaVmq44pS2u9FGxwdHQGob7pkMhmaN2+O8+fPY/PmzXj77bcL1rZt24Yff/wRjRo1wmQN/v4pz8905f8aF7cfGhEREVUcbLpI62bPng2lUomvv/4aXl5eaNWqFdq2bQs7OzskJSUhNjYWR44cAQB06tSp4Os+/vhjrF27FkOHDsWQIUPg5OSEK1eu4ODBgxg2bBj+/vvvQq9jamqKyZMn4+uvv0aLFi0wcOBA5Obm4vDhw3ByclL7A/DAgQNRuXJltG/fHq6urlAqlTh27BjOnj2LVq1aFUzpK+lxRSnt96INNWrUgLW1dZG39M2ePRuDBg3CuHHjcPDgQTg7O+Ps2bM4cuQI6tevj/3798PCwkLrOcVYsWIFfvvtt4JG7q233sLMmTPRp0+fYr+uc+fOCA0NLXI9PDwc3333Hc6dO4dHjx6pvT1UnRcvXgDg829ERESkwqaLtE4mk2HOnDkYOXIkfv31V4SEhGDTpk1IT0+HtbU13Nzc8NFHH+Hdd99Fy5YtC76uWbNmCAkJwcyZM7Fv3z7k5uaiefPm2LFjB2xtbdU2KnPnzoWlpSV+//13rFq1Cg4ODhgxYgTmzJlTsHfSqxYvXoxDhw4hKiqqoKmoXbs2lixZgo8++qhguEdJjyuKmO9FG1q3bo1r166pXfP398fOnTuxZMkS7NixA0qlEg0aNMCCBQswadIkVK5cWScZxahVqxaWLFmC+vXrQ6FQYP369fD398e5c+fQrFmzQsceO3YMOTk5hfaOO3r0KExNTdGxY8dCx6alpaFJkyYFz/OV1NWrVwEA7dq1K8N3RUREROWFTKmPD0ToqZSUFMjlciQnJxc5AY5In82fPx/Lly8vuC2yPLOzs8OiRYsEz6HFxcVh2rRpqF69Ok6fPo127dohISEBP/zwA1xcXIo8X+XKlfHzzz+X6ErX5MmTERISgkuXLpX12yAiIiI9VZreQLgZDxGVWyNGjEBiYmLB+PfyKC8vD1u2bEFaWpra5wddXFywbds2yOVyREVFQS6XY9u2bcU2XKUVFRWFESNGaOx8REREZNjYdBFVIPXq1UOfPn2wc+dOqaNo3OXLl1G5cmWYm5vjww8/xM6dO9G0aVPBcQ8ePMDw4cORnJyMli1bIjk5GcOHD9dYI/r48WOcP38e48aN08j5iIiIyPCx6SKqYBYsWICNGzdKHUPjGjZsiAsXLuD06dP46KOPMGbMGFyLine truncated
|
||||
"text/plain": [
|
||||
"<Figure size 1000x800 with 1 Axes>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"image/png": "iVBORw0KGgoAAAANSUhEUgAAA10AAALZCAYAAABMLUslAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAC1YUlEQVR4nOzdeViVZf7H8c8BFAUFxBUVRE3FHfd9Tc0ll8yyMXP71VRaactMOS1qado0rVPWNJNatliZW7mRivuKilugpiLuiMqikizn/P5gZMJzUDicc54DvF/XxTWd7/Nwnw9ek/E99/3ct8lisVgEAAAAAHAKD6MDAAAAAEBxRtMFAAAAAE5E0wUAAAAATkTTBQAAAABORNMFAAAAAE5E0wUAAAAATkTTBQAAAABORNMFAAAAAE7kZXSAosRsNuvs2bMqX768TCaT0XEAAAAAGMRisSg1NVXVq1eXh8ft57Jougrg7NmzCg4ONjoGAAAAADdx6tQp1axZ87b30HQVQPny5SVl/8H6+fkZnAYAAACAUVJSUhQcHJzTI9wOTVcB3FxS6OfnR9MFAAAAIF+PHbGRBgAAAAA4EU0XAAAAADgRTRcAAAAAOBFNFwAAAAA4EU0XAAAAADgRTRcAAAAAOBFNFwAAAAA4EU0XAAAAADgRTRcAAAAAOBFNFwAAAAA4EU0XAAAAADgRTRcAAAAAOBFNFwAAAAA4EU0XAAAAADgRTRcAAAAAOBFNF4ASZePGjRo0aJBq1Kghk8mkefPmGR0JAAAUczRdQAl27dq1Qn3/zJkz1aZNG/n5+aly5coaOHCgDh486KB0znH16lU1adJEH3zwgcqWLeuwcQv7ZwkAAIovmi6ghNq0aZMWL15cqDHWr1+v8ePHa+vWrVq3bp28vLzUq1cvXb582UEpHa9///568803NWzYMHl4OO6vwEuXLunNN9902HgAAKD48DI6AADXO3funP7xj38UuulavXp1rtfz58+Xv7+/tmzZooEDBxZq7KImJCREnp6e+uGHH/TAAw8YHQcAALgRZroAB4uLi5PJZNKYMWOMjpKnJ554QlOnTnXoTI8kpaamymw2q0KFCg4dt6h4/vnn9e677+rixYtGRwEAAG6Epgsuc+TIET333HNq2bKlAgMDVapUKQUGBqpdu3Z64YUXtHv3bqMjlghr1qzRhQsX1KJFC4ePPXHiRIWHh6tDhw4OH7so8PLy0rBhw/TKK68YHQUAALgRlhfC6SwWi15//XW9/vrrMpvNatmypYYPH67AwEClpqZq//79+uc//6l33nlHH330kSZMmGB05EKpUaOGYmJi5O/vb3QUm958802NHj3a4eM+99xz2rx5szZv3ixPT0+Hj19UjBw5UnXq1NGrr76qmjVrGh0HAAC4AZouON3rr7+uqVOnKjg4WN9++606depkdU9CQoLef/99JScnG5DQsUqVKqWwsDCjY9h09OhRRUZG6quvvnLouM8++6wWLFigyMhI1alTx6FjFzVVq1ZV69atNWfOHL322mtGxwEAAG6A5YVwquPHj2v69OkqXbq0Vq5cabPhkqQqVarozTff1F//+tdc9Xnz5un+++9XnTp1VLZsWfn5+alTp042m4b169fLZDJp6tSpNt8jNDRUoaGhVvVly5bp7rvvVlBQkLy9vVW9enV169ZNs2fPtuu+vJ7pKsjPcus4cXFxeuihh1SpUiWVKVNGrVu31s8//2zz+27nu+++U9WqVVW9evU877l69apef/11tWjRQuXLl5fJZLL5deHCBUnZSwq//fZbrVu3zm2bzT+6evWqoqOjFR0dLbPZrPj4eEVHRys+Pt5h79GyZUt9++23DhsPAAAUbTRdcKq5c+cqMzNTw4YNU+PGje94v5dX7snXJ598UidPnlTXrl01adIkPfTQQzp58qQeeeQRvfrqq4XO99lnn2nw4MH69ddfNXDgQD3//PPq37+/0tLSNHfu3ALfdzv2/iwnT55U27ZtFRcXp0ceeUTDhw/XwYMHNXjwYEVGRhbo5123bt1tn+VKSEhQmzZtNGXKFJnNZj3xxBN6+umnVa1aNUnZs3h169ZVu3btVLVqVU2YMEFz587VN998owoVKuj8+fM6f/68rl69WqBcrhQVFaUWLVqoRYsWSktL05QpU9SiRQuHzkq1aNFCsbGxOn/+vMPGBAAARRfLC43i6I0GHnpImjjx9vd88IG0YEH+xtu2rfCZJG3ZskWS1LNnT7u+/+DBg6pbt26uWnp6uvr166dZs2bpiSeeUI0aNezO969//UulS5fWvn37VKVKlVzXEhMTC3zf7dj7s6xfv15Tp07VlClTcmojRoxQ37599fbbb6tHjx75en8pu+F49NFH87w+YsQIxcbG6q9//atmzZolk8kkSfrLX/6ievXqKSsrS9u3b1elSpUkKWeW7+677841zpQpU/Kccbzp/fffV1JSUr6zh4eHa8iQIfm+Py/du3eXxWIp9Di306hRI0nSjh07NHjwYKe+FwAAcH80XUbZvt2x4+WniTt50vHvewc3P+m31UzExcVp3rx5uWoBAQGaNGlSzutbmxRJKl26tCZMmKB169Zp7dq1GjVqVKEyenl5qVSpUlb1m41FQe/Li70/S61atax2w7vnnnsUEhKinTt35uu9JenChQtKTU1VQECAzeu//PKL1q5dq86dO2vmzJk5DZckBQcHq0uXLlqzZo2io6PVq1cvSSpU8/L+++/r5MmT+b5/9OjRDmm6XOHmn/GxY8eMDQIAANwCTRcMExcXp2nTpuWq1apVK1fTFR8fr7feektr165VfHy80tLSct1/5syZQmV4+OGH9fzzz6tRo0Z66KGH1K1bN3Xq1EmVK1e2677bsfdnCQ8Pt7kbYHBwsLYVYEby3LlzkiQ/Pz+b128+WzZp0iSb53fd3I3RbDbn+z1vJy4uziHjuKObf8Y3/8wBAEDJRtMFp6pWrZpiYmJ09uxZq2t/XOaVmZlpNYt0/PhxtW3bVleuXFGXLl3Up08f+fv7y9PTU3Fxcfriiy9048aNQuV77rnnVKlSJc2ePVsffvih3n//fZlMJnXr1k1vv/22WrduXaD78lKYnyWvmSkvL68CNUDXrl2TlHfTtWnTJnl4eKhv3742r58+fVqSdNddd+X7PUuqmw3qzT9zAABQstF0wak6deqkyMhIrV27VuPGjSvQ97777ru6dOmS5s6da7UT4LfffqsvvvgiV+3m7ExmZqbN8ZKSkmw2MKNGjdKoUaOUlJSkrVu3avHixZozZ47uuecexcbG5sxm5fc+R/wszmRrSWBWVpZOnjypKlWqyNfX1+r6hQsXtGvXLtWuXdthW8I76pmuPy6DNMqtf6Y3X7tDNgAAYDyaLqO0b+/Y8WrVyt89jn7fOxgzZoxmzZqlhQsX6pVXXlHDhg3z/b2//fabJOn++++3urZhwwarWoUKFSRJp06dsjlWcnJynrNGUvaMUv/+/dW/f3+ZzWbNmTNHGzdutHr//N5XmJ/FGW42UykpKVbXbjasqampMpvNVssL//73v8tsNuvxxx93WB5HPdPl7E0x7HHzz9hWAwsAAEoemi6jOGh3wAKZOPHOOxw6WN26dfXKK69o6tSp6tevn7755ht17NjR6j5bMx43z9Rav369Bg4cmFNfvXq1/vOf/1jdHxYWJj8/Py1dulQJCQk5uwympaXpmWeesZkvMjJS3bt3t5qRSEhIkCT5+PgU6L68FPRncYagoCBJtpsuk8mk5s2ba+/evfr222/18MMP51xbuHCh3n//fYWFhWmiA///U5yf6br5Z3y789AAAEDJQdMFp3vttddksVj0xhtvqFOnTmrVqpXatm2rwMBAJSUlKS4uTmvWrJEkde3aNef7xo8fr7lz5+qBBx7QsGHDVL16dR08eFCrVq3Sgw8+qO+++y7X+5QqVUoTJ07UG2+8oRYtWui+++5TZmamfvnlF1WvXt3mL8D33XefypUrp/bt2ys0NFQWi0WbNm3Srl271KpVq5xd+vJ7X14K+rM4Q9WqVVW+fPk8l/S99tprGjp0qMaOHatVq1YpODhYu3bt0po1a1SvXj2tWLFCZcqUcXpOe3z88cf617/+ldPINW7cWK+88ooGDBhw2+/r3r271q9fn+f1mTNnatGiRTp8+LC8vb3Vvn17zZw5U02aNLntuFeuXJHE828AACCbWx6OvHDhQj399NPq0qWL/Pz8ZDKZNHLkSLvGOn36tMaNG6fq1avL29tboaGhmjRpUs4vRXA+k8mkqVOn6tdff9WkSZOUmZmpb775Rm+99Za++eYbXbhwQU8++aR2796tL7/8Muf7mjVrpsjISHXs2FHLly/XJ598opSUFC1atEhPPPGEzfeaNm2aZs6cqTJlyuizzz7TihUrdP/992v16tU2t3ufNWuW2rRpoz179mj27NmaO3euMjIy9NZbbykyMjLne/J7X17s+VmcoXXr1vr1119tXhsyZIgWL16s1q1ba9GiRXr//fd18eJFzZgxQ3v27FHt2rVdlrOgatasqbfeekt79uxRVFSUevbsqSFDhmj//v1W927atEnr1q3LVVu3bp02bdpkde/69es1fvx4bd26VevWrZOXl5d69eqly5cv3zbPoUOHJEnt2rUrxE8FAACKC5PFDR+ICA8P1759+1SuXDnVrFlTsbGxevjhh3O2tM6vY8eOqWPHjkpISNDgwYMVFhamnTt3KjIyUg0aNNCWLVtUsWLFfI+XkpIif39/JScn57kDHODOpk+frg8//DBnWWRxFhgYqJkzZ1o9hxYfH6/nnntOVapU0Y4dO9SuXTslJCTo3XffVUhIyG3HvHr1qvz9/bVkyZJcy0RvNXHiREVGRtps+gAAQPFQkN7ALWe63nvvPR05ckQpKSn65JNP7B5n/PjxSkhI0IcffqglS5Zo1qxZWrdunZ599lkdPnxYL7/8sgNLine truncated
|
||||
"text/plain": [
|
||||
"<Figure size 1000x800 with 1 Axes>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"image/png": "iVBORw0KGgoAAAANSUhEUgAAA10AAALZCAYAAABMLUslAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAACzQklEQVR4nOzdeVxVdf7H8fcFFQEFRFxAQRTBNRX3fddSK221zEydmprKtGWmnMqlNGva+2XTNFNaTmVlppZrKe77vgJuiBsiKqCICNz7+4ORwntRuFzuuRdez8eDR8PnnPu9bxwf6ud+l2OyWCwWAQAAAABKhYfRAQAAAACgLKPpAgAAAIBSRNMFAAAAAKWIpgsAAAAAShFNFwAAAACUIpouAAAAAChFNF0AAAAAUIpougAAAACgFFUwOoA7MZvNOnXqlKpWrSqTyWR0HAAAAAAGsVgsunjxokJCQuThceO5LJquYjh16pRCQ0ONjgEAAADARRw/flx169a94T00XcVQtWpVSXm/sH5+fganAQAAAGCU9PR0hYaG5vcIN0LTVQzXlhT6+fnRdAEAAAAo0rYjDtIAAAAAgFJE0wUAAAAApYimCwAAAABKEU0XAAAAAJQimi4AAAAAKEU0XQAAAABQimi6AAAAAKAU0XQBAAAAQCmi6QIAAACAUkTTBQAAAACliKYLAAAAAEoRTRcAAAAAlCKaLgAAAAAoRTRdAAAAAFCKaLoAAAAAoBTRdAEoV1avXq0777xTderUkclk0syZM42OBAAAyjiaLqAcy8jIKNHrp02bpnbt2snPz081atTQHXfcob179zooXem4dOmSmjdvrg8//FDe3t4OG7ekv5YAAKDsoukCyqk1a9bop59+KtEYK1eu1JNPPqn169drxYoVqlChgvr27avz5887KKXjDRw4UG+88YbuvfdeeXg47o/Ac+fO6Y033nDYeAAAoOyoYHQAAM53+vRpvfPOOyVuupYuXVrg+1mzZsnf31/r1q3THXfcUaKx3U1YWJg8PT31ww8/6L777jM6DgAAcCHMdAEOlpCQIJPJpJEjRxodpVBPPPGEJk2a5NCZHkm6ePGizGazqlWr5tBx3cXzzz+v9957T2fPnjU6CgAAcCE0XXCa+Ph4Pffcc2rdurUCAwNVsWJFBQYGqkOHDnrhhRe0bds2oyOWC7/99pvOnDmj6Ohoh489duxYtWrVSp06dXL42O6gQoUKuvfee/XKK68YHQUAALgQlhei1FksFr322mt67bXXZDab1bp1aw0dOlSBgYG6ePGidu/erf/7v//Tu+++q48//lhPPfWU0ZFLpE6dOjpw4ID8/f2NjmLTG2+8oUceecTh4z733HNau3at1q5dK09PT4eP7y6GDx+uBg0a6NVXX1XdunWNjgMAAFwATRdK3WuvvaZJkyYpNDRU3377rbp06WJ1T3Jysj744AOlpaUZkNCxKlasqMaNGxsdw6aDBw8qJiZG//3vfx067rPPPqvZs2crJiZGDRo0cOjY7qZWrVpq27atvvjiC02YMMHoOAAAwAWwvBCl6siRI5oyZYoqVaqkxYsX22y4JKlmzZp644039Le//a1AfebMmbrnnnvUoEEDeXt7y8/PT126dLHZNKxcuVImk0mTJk2y+R7h4eEKDw+3qi9YsEB9+vRRcHCwvLy8FBISoh49euiTTz6x677C9nQV52e5fpyEhAQ98MADCgoKUuXKldW2bVv98ssvNl93I999951q1aqlkJCQQu+5dOmSXnvtNUVHR6tq1aoymUw2v86cOSMpb0nht99+qxUrVrhss/lHly5d0s6dO7Vz506ZzWYlJiZq586dSkxMdNh7tG7dWt9++63DxgMAAO6NpgulasaMGcrJydG9996rZs2a3fT+ChUKTr7+5S9/0bFjx9S9e3eNGzdODzzwgI4dO6aHH35Yr776aonzffbZZxo8eLD279+vO+64Q88//7wGDhyozMxMzZgxo9j33Yi9P8uxY8fUvn17JSQk6OGHH9bQoUO1d+9eDR48WDExMcX6eVesWHHDvVzJyclq166dJk6cKLPZrCeeeEJjxoxR7dq1JeXN4kVERKhDhw6qVauWnnrqKc2YMUPffPONqlWrpqSkJCUlJenSpUvFyuVMW7duVXR0tKKjo5WZmamJEycqOjraobNS0dHRio2NVVJSksPGBAAA7ovlhUZx9EEDDzwgjR1743s+/FCaPbto423YUPJMktatWydJ6t27t12v37t3ryIiIgrUrl69qgEDBujNN9/UE088oTp16tid71//+pcqVaqkXbt2qWbNmgWupaSkFPu+G7H3Z1m5cqUmTZqkiRMn5teGDRum2267TW+//bZ69epVpPeX8hqORx99tNDrw4YNU2xsrP72t7/pzTfflMlkkiT99a9/VWRkpHJzc7Vx40YFBQVJUv4sX58+fQqMM3HixEJnHK/54IMPlJqaWuTsrVq10pAhQ4p8f2F69uwpi8VS4nFupGnTppKkTZs2afDgwaX6XgAAwPXRdBll40bHjleUJu7YMce/701c+6TfVjORkJCgmTNnFqgFBARo3Lhx+d9f36RIUqVKlfTUU09pxYoVWr58uUaMGFGijBUqVFDFihWt6tcai+LeVxh7f5Z69epZnYZ36623KiwsTJs3by7Se0vSmTNndPHiRQUEBNi8/uuvv2r58uXq2rWrpk2blt9wSVJoaKi6deum3377TTt37lTfvn0lqUTNywcffKBjx44V+f5HHnnEIU2XM1z7NT58+LCxQQAAgEug6YJhEhISNHny5AK1evXqFWi6EhMT9dZbb2n58uVKTExUZmZmgftPnjxZogwPPfSQnn/+eTVt2lQPPPCAevTooS5duqhGjRp23Xcj9v4srVq1snkaYGhoqDYUY0by9OnTkiQ/Pz+b16/tLRs3bpzN53ddO43RbDYX+T1vJCEhwSHjuKJrv8bXfs0BAED5RtOFUlW7dm0dOHBAp06dsrr2x2VeOTk5VrNIR44cUfv27XXhwgV169ZN/fv3l7+/vzw9PZWQkKAvv/xSWVlZJcr33HPPKSgoSJ988ok++ugjffDBBzKZTOrRo4fefvtttW3btlj3FaYkP0thM1MVKlQoVgOUkZEhqfCma82aNfLw8NBtt91m8/qJEyckSQ0bNizye5ZX1xrUa7/mAACgfKPpQqnq0qWLYmJitHz5co0ePbpYr33vvfd07tw5zZgxw+okwG+//VZffvllgdq12ZmcnByb46WmptpsYEaMGKERI0YoNTVV69ev108//aQvvvhCt956q2JjY/Nns4p6nyN+ltJka0lgbm6ujh07ppo1a8rX19fq+pkzZ7RlyxbVr1/fYUfCO2pP1x+XQRrl+l/Ta9+7QjYAAGA8mi6jdOzo2PHq1SvaPY5+35sYOXKk3nzzTc2ZM0evvPKKmjRpUuTXHjp0SJJ0zz33WF1btWqVVa1atWqSpOPHj9scKy0trdBZIylvRmngwIEaOHCgzGazvvjiC61evdrq/Yt6X0l+ltJwrZlKT0+3unatYb148aLMZrPV8sJ//OMfMpvNevzxxx2Wx1F7ukr7UAx7XPs1ttXAAgCA8oemyygOOh2wWMaOvfkJhw4WERGhV155RZMmTdKAAQP0zTffqHPnzlb32ZrxuPZMrZUrV+qOO+7Iry9dulT/+c9/rO5v3Lix/Pz8NH/+fCUnJ+efMpiZmalnnnnGZr6YmBj17NnTakYiOTlZkuTj41Os+wpT3J+lNAQHB0uy3XSZTCa1bNlSO3bs0LfffquHHnoo/9qcOXP0wQcfqHHjxhrrwN8/ZXlP17Vf4xs9Dw0AAJQfNF0odRMmTJDFYtHrr7+uLl26qE2bNmrfvr0CAwOVmpqqhIQE/fbbb5Kk7t2757/uySef1IwZM3Tffffp3nvvVUhIiPbu3aslS5bo/vvv13fffVfgfSpWrKixY8fq9ddfV3R0tO666y7l5OTo119/VUhIiM1/AN91112qUqWKOnbsqPDwcFksFq1Zs0ZbtmxRmzZt8k/pK+p9hSnuz1IaatWqpapVqxa6pG/ChAm6++67NWrUKC1ZskShoaHasmWLfvvtN0VGRmrRokWqXLlyqee0x/Tp0/Wvf/0rv5Fr1qyZXnnlFQ0aNOiGr+vZs6dWrlzp8HEvXLggif1vAAAgD00XSp3JZNKkSZP04IMP6tNPP1VMTIy++eYbZWRkqGrVqoqIiNBf/vIXPfzww2rdunX+61q0aKGYmBi98sorWrhwoXJyctSyZUvNnTtXAQEBNhuVyZMny8fHR//+97/12WefqXbt2nrggQc0adKk/Gcn/dGbb76ppUuXavv27flNRb169fTWW2/pL3/5S/7hHkW9rzD2/CyloW3bttq/f7/Na0OGDNFPP/2kt956S3PnzpXFYlFUVJSmTp2qZ555RlWqVHFKRnvUrVtXb731liIjI2U2m/Xll19qyJAh2rZtm1q0aFHg3jVr1ig7O7vAs+NWrFihihUrqlu3bnaP+0f79u2TJHXo0MGBPyUAAHBXJosrbohwUenp6fL391daWlqhJ8ABrmzKlCn66KOP8pdFlmWBgYGaNm2a1T60xMREPffcc6pZs6Y2bdqkDh06KDk5We+9957CwsLsHvePxo4dq5iYGO3evbvEPwcAAHBNxekNrB/GA6DMeuCBB3T27Nn849/LotzcXM2ePVuXLl2yuX8wLCxMc+bMkb+/v7Zv3y5/f3/NmTPnpg3Xzcb9o+3bt+uBBx4o0c8BAADKDpouoBxp2LChBg0apJ9++snoKA63Z88eValSRV5eXnriiSf0008/6ZZbbrG678SJExo6dKjS0tLUunVrpaWlaejQoYU2okUd95rTp09rx44dGjVqlMN+NgAA4N5YXlgMLC9EWbBr1y499thj2rx5s9FRHOrq1atKTExUWlqa5syZo3//+99auXKlmjdvXuC+1atLine truncated
|
||||
"text/plain": [
|
||||
"<Figure size 1000x800 with 1 Axes>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"image/png": "iVBORw0KGgoAAAANSUhEUgAAA10AAALZCAYAAABMLUslAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAACyzElEQVR4nOzdeViU9f7/8dewiIICoqKoIIIiuOS+75aWWmml1TFz6dt22rTlnPJkLuVSv06lnrJOp1KzPXOpzKUUl9wS9wVcUAR3UVlUZJ3fHySJMyAMw9wDPB/XxXU17/vmnhdeXeJ7PpvJbDabBQAAAAAoFS5GBwAAAACA8oymCwAAAABKEU0XAAAAAJQimi4AAAAAKEU0XQAAAABQimi6AAAAAKAU0XQBAAAAQCmi6QIAAACAUuRmdICyJCcnRydPnlS1atVkMpmMjgMAAADAIGazWampqapbt65cXAofy6LpKoaTJ08qMDDQ6BgAAAAAnERCQoLq169f6D00XcVQrVo1Sbl/sN7e3ganAQAAAGCUlJQUBQYG5vUIhaHpKoZrUwq9vb1pugAAAAAUadkRG2kAAAAAQCmi6QIAAACAUkTTBQAAAACliKYLAAAAAEoRTRcAAAAAlCKaLgAAAAAoRTRdAAAAAFCKaLoAAAAAoBTRdAEAAABAKaLpAgAAAIBSRNMFAAAAAKWIpgsAAAAAShFNFwAAAACUIpouAAAAAChFNF0AAAAAUIpougBUKOvWrdPdd9+tevXqyWQyae7cuUZHAgAA5RxNF1CBXb58uUTfP336dLVv317e3t6qVauW7rrrLu3du9dO6UrHpUuX1Lx5c82cOVNVqlSx23NL+mcJAADKL5ouoIJav369Fi1aVKJnrFmzRk899ZQ2btyo1atXy83NTbfddpsuXLhgp5T2N2DAAE2bNk1DhgyRi4v9/go8f/68pk2bZrfnAQCA8sPN6AAAHO/UqVP697//XeKma8WKFflez58/Xz4+PtqwYYPuuuuuEj27rAkKCpKrq6u+//57DR061Og4AADAiTDSBdhZXFycTCaTRo0aZXSUAj355JOaNGmSXUd6JCk1NVU5OTmqXr26XZ9bVrz44ot69913de7cOaOjAAAAJ0LTBYc5ePCgXnjhBbVp00Z+fn5yd3eXn5+fOnbsqJdeeknbtm0zOmKF8Ntvv+nMmTNq3bq13Z89ZswYtWrVSp07d7b7s8sCNzc3DRkyROPHjzc6CgAAcCJML0SpM5vNev311/X6668rJydHbdq00QMPPCA/Pz+lpqZq9+7d+s9//qN33nlH77//vp5++mmjI5dIvXr1FB0dLR8fH6OjWDVt2jSNHDnS7s994YUX9Pvvv+v333+Xq6ur3Z9fVgwfPlwhISF67bXXVL9+faPjAAAAJ0DThVL3+uuva9KkSQoMDNTXX3+trl27Wtxz9uxZzZgxQ8nJyQYktC93d3eFh4cbHcOqQ4cOKTIyUl988YVdn/v888/rm2++UWRkpEJCQuz67LKmdu3aateunT777DNNmDDB6DgAAMAJML0QperIkSOaMmWKKlWqpGXLllltuCTJ399f06ZN0z//+c989blz5+q+++5TSEiIqlSpIm9vb3Xt2tVq07BmzRqZTCZNmjTJ6nsEBwcrODjYov7jjz/q1ltvVUBAgDw8PFS3bl317NlTs2fPtum+gtZ0FednufE5cXFxevDBB1WzZk1VrlxZ7dq1088//2z1+wrz7bffqnbt2qpbt26B91y6dEmvv/66WrdurWrVqslkMln9OnPmjKTcKYVff/21Vq9e7bTN5vUuXbqknTt3aufOncrJyVF8fLx27typ+Ph4u71HmzZt9PXXX9vteQAAoGyj6UKpmjNnjrKysjRkyBA1a9bspve7ueUffP373/+uY8eOqUePHho7dqwefPBBHTt2TA8//LBee+21Euf7+OOPNWjQIO3fv1933XWXXnzxRQ0YMEBpaWmaM2dOse8rjK0/y7Fjx9ShQwfFxcXp4Ycf1gMPPKC9e/dq0KBBioyMLNbPu3r16kLXcp09e1bt27fXxIkTlZOToyeffFLPPvus6tSpIyl3FC80NFQdO3ZU7dq19fTTT2vOnDn66quvVL16dZ0+fVqnT5/WpUuXipXLkaKiotS6dWu1bt1aaWlpmjhxolq3bm3XUanWrVsrJiZGp0+fttszAQBA2cX0QqPYe6OBBx+Uxowp/J6ZM6Vvvina8zZtKnkmSRs2bJAk9enTx6bv37t3r0JDQ/PVMjIy1L9/f7355pt68sknVa9ePZvz/fe//1WlSpW0a9cu+fv757uWmJhY7PsKY+vPsmbNGk2aNEkTJ07Mqw0bNkx33HGH3n77bfXu3btI7y/lNhyPPvpogdeHDRummJgY/fOf/9Sbb74pk8kkSfrHP/6hxo0bKzs7W5s3b1bNmjUlKW+U79Zbb833nIkTJxY44njNjBkzlJSUVOTsrVq10uDBg4t8f0F69eols9lc4ucUpmnTppKkLVu2aNCgQaX6XgAAwPnRdBll82b7Pq8oTdyxY/Z/35u49km/tWYiLi5Oc+fOzVfz9fXV2LFj817f2KRIUqVKlfT0009r9erVWrVqlUaMGFGijG5ubnJ3d7eoX2ssintfQWz9WRo0aGCxG97tt9+uoKAg/fHHH0V6b0k6c+aMUlNT5evra/X6r7/+qlWrVqlbt26aPn16XsMlSYGBgerevbt+++037dy5U7fddpsklah5mTFjho4dO1bk+0eOHGmXpssRrv0Zx8bGGhsEAAA4BZouGCYuLk6TJ0/OV2vQoEG+pis+Pl5vvfWWVq1apfj4eKWlpeW7/8SJEyXK8NBDD+nFF19U06ZN9eCDD6pnz57q2rWratWqZdN9hbH1Z2nVqpXV3QADAwO1qRgjkqdOnZIkeXt7W71+bW3Z2LFjrZ7fdW03xpycnCK/Z2Hi4uLs8hxndO3P+NqfOQAAqNhoulCq6tSpo+joaJ08edLi2vXTvLKysixGkY4cOaIOHTro4sWL6t69u/r16ycfHx+5uroqLi5O8+bNU3p6eonyvfDCC6pZs6Zmz56tWbNmacaMGTKZTOrZs6fefvtttWvXrlj3FaQkP0tBI1Nubm7FaoAuX74sqeCma/369XJxcdEdd9xh9frx48clSY0aNSrye1ZU1xrUa3/mAACgYqPpQqnq2rWrIiMjtWrVKj3yyCPF+t53331X58+f15w5cyx2Avz66681b968fLVrozNZWVlWn5eUlGS1gRkxYoRGjBihpKQkbdy4UYsWLdJnn32m22+/XTExMXmjWUW9zx4/S2myNiUwOztbx44dk7+/v7y8vCyunzlzRlu3blXDhg3ttiW8vdZ0XT8N0ig3/plee+0M2QAAgPFouozSqZN9n9egQdHusff73sSoUaP05ptvasGCBRo/frwiIiKK/L2HDx+WJN13330W19auXWtRq169uiQpISHB6rOSk5MLHDWSckeUBgwYoAEDBignJ0efffaZ1q1bZ/H+Rb2vJD9LabjWTKWkpFhcu9awpqamKicnx2J64f/7f/9POTk5euKJJ+yWx15rukp7UwxbXPszttbAAgCAioemyyh22h2wWMaMufkOh3YWGhqq8ePHa9KkSerfv7+++uordenSxeI+ayMe187UWrNmje666668+ooVK/TJJ59Y3B8eHi5vb28tWbJEZ8+ezdtlMC0tTc8995zVfJGRkerVq5fFiMTZs2clSZ6ensW6ryDF/VlKQ0BAgCTrTZfJZFLLli21Y8cOff3113rooYfyri1YsEAzZsxQeHi4xtjx/5/yvKbr2p9xYeehAQCAioOmC6VuwoQJMpvNeuONN9S1a1e1bdtWHTp0kJ+fn5KSkhQXF6fffvtNktSjR4+873vqqac0Z84cDR06VEOGDFHdunW1d+9eLV++XPfff7++/fbbfO/j7u6uMWPG6I033lDr1q11zz33KCsrS7/++qvq1q1r9R/A99xzj6pWrapOnTopODhYZrNZ69ev19atW9W2bdu8XfqKel9BivuzlIbatWurWrVqBU7pmzBhgu69916NHj1ay5cvV2BgoLZu3arffvtNjRs31i+//KLKlSuXek5bfPDBB/rvf/+b18g1a9ZM48eP18CBAwv9vl69emnNmjVFeo/p06frX//6l55++mm9//77hd578eJFSax/AwAAuWi6UOpMJpMmTZqkv/3tb/roo48UGRmpr776SpcvX1a1atUUGhqqv//973r44YfVpk2bvO+75ZZbFBkZqfHjx2vp0qXKyspSy5YttXDhQvn6+lptVCZPnixPT0/973//08cff6w6derowQcf1KRJk/LOTrrem2++qRUrVmj79u15TUWDBg301ltv6e9//3ve5h5Fva8gtvwspaFdu3bav3+/1WuDBw/WokWL9NZbb2nhwoUym80KCwvT1KlT9dxzz6lq1aoOyWiL+vXr66233lLjxo2Vk5OjefPmafDgwdq2bZtuueWWfPeuX79emZmZ+c6OW716tdzd3dW9e3erz9+8ebM+/vhji2cVZN++fZKkjh072vgTAQCA8sRkdsYFEU4qJSVFPj4+Sk5OLnAHOMCZTZkyRbNmzcqbFlme+fn5afr06Rbr0OLj4/XCCy/I399fW7ZsUceOHXX27Fm9++67CgoKsnhOcnKy2rRpo08++USTJ09W8+bNbzrSNWbMGEVGRmr37t12/ZkAAIDzKE5vYHkYD4By68EHH9S5c+fytn8vj7Kzs/XNN9/o0qVLVtcPBgUFacGCBfLx8dH27dvl4+OjBQsWWG24JOnxxx/XkCFD1Lt37yJn2L59ux588EGbfwYAAFC+0HQBFUijRo00cOBALVq0yOgodrdnzx5VrVpVHh4eevLJJ7Vo0SK1aNHC4r7jx4/rgQceyBvBSk5O1gMPPGC1Ef3f//6nw4cPa8qUKUXOcerUKe3YsUOLine truncated
|
||||
"text/plain": [
|
||||
"<Figure size 1000x800 with 1 Axes>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"image/png": "iVBORw0KGgoAAAANSUhEUgAAA10AAALZCAYAAABMLUslAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAC0HUlEQVR4nOzdeViVdf7/8ddhEQEFVFzYEQSXzF3T3C0ta0zby8ylb02ractUTouaa9O06LRNU2nZtFoujWsprqm54Y4LyCKKiAu4oGzn+wcjI54DwuFw7gM8H9fF9Rve983Ni37fS31xf+7PbTKbzWYBAAAAACqFi9EBAAAAAKA6o3QBAAAAQCWidAEAAABAJaJ0AQAAAEAlonQBAAAAQCWidAEAAABAJaJ0AQAAAEAlonQBAAAAQCVyMzpAVVJQUKCjR4+qbt26MplMRscBAAAAYBCz2ayzZ88qMDBQLi6l38uidJXD0aNHFRISYnQMAAAAAE4iJSVFwcHBpZ5D6SqHunXrSir8D+vj42NwGgAAAABGycrKUkhISFFHKA2lqxwuLyn08fGhdAEAAAAo02NHbKQBAAAAAJWI0gUAAAAAlYjSBQAAAACViNIFAAAAAJWI0gUAAAAAlYjSBQAAAACViNIFAAAAAJWI0gUAAAAAlYjSBQAAAACViNIFAAAAAJWI0gUAAAAAlYjSBQAAAACViNIFAAAAAJWI0gUAAAAAlYjSBQAAAACViNIFoEZZs2aN7rjjDgUFBclkMmn27NlGRwIAANUcpQuowc6fP1+hr582bZo6d+4sHx8fNWzYUIMGDdLu3bvtlK5ynDt3Tq1bt9aMGTPk6elpt+tW9L8lAACovihdQA21du1azZs3r0LXWLVqlZ566in9/vvvWrlypdzc3HTzzTfr1KlTdkppf7fddpumTp2qe+65Ry4u9vsj8OTJk5o6dardrgcAAKoPN6MDAHC8Y8eO6e9//3uFS9eyZcuKfT5nzhz5+vpq/fr1GjRoUIWuXdWEhobK1dVVP/74o+69916j4wAAACfCnS7AzhITE2UymTRy5Eijo5ToiSee0IQJE+x6p0eSzp49q4KCAtWrV8+u160qXnjhBb377rs6ceKE0VEAAIAToXTBYQ4cOKDnn39eHTp0UP369eXu7q769evrhhtu0IsvvqitW7caHbFG+O2333T8+HG1b9/e7tceM2aM2rVrp27dutn92lWBm5ub7rnnHr322mtGRwEAAE6E5YWodGazWW+++abefPNNFRQUqEOHDrr//vtVv359nT17Vjt37tQ//vEPvfPOO/rggw/09NNPGx25QoKCgrRv3z75+voaHcWqqVOnasSIEXa/7vPPP69169Zp3bp1cnV1tfv1q4phw4YpIiJCr7/+uoKDg42OAwAAnAClC5XuzTff1IQJExQSEqJvv/1W3bt3tzgnPT1d77//vjIzMw1IaF/u7u5q0aKF0TGsOnjwoGJiYvT111/b9brPPfecvvvuO8XExCgiIsKu165qGjdurE6dOumLL77QG2+8YXQcAADgBFheiEqVkJCgyZMnq1atWlqyZInVwiVJjRo10tSpU/XSSy8Vm8+ePVt33323IiIi5OnpKR8fH3Xv3t1qaVi1apVMJpMmTJhg9XuEh4crPDzcYr5w4ULddNNNCggIkIeHhwIDA9W7d2999NFHNp1X0jNd5flZrr5OYmKiHnjgAfn7+6t27drq1KmT/vOf/1j9utJ8//33aty4sQIDA0s859y5c3rzzTfVvn171a1bVyaTyerH8ePHJRUuKfz222+1cuVKpy2bVzp37pxiY2MVGxurgoICJScnKzY2VsnJyXb7Hh06dNC3335rt+sBAICqjdKFSjVr1izl5eXpnnvu0XXXXXfN893cit98ffLJJ5WUlKRevXpp7NixeuCBB5SUlKSHH35Yr7/+eoXzffrppxo8eLD27t2rQYMG6YUXXtBtt92m7OxszZo1q9znlcbWnyUpKUldunRRYmKiHn74Yd1///3avXu3Bg8erJiYmHL9vCtXriz1Wa709HR17txZ48ePV0FBgZ544gmNHj1aTZo0kVR4Fy8yMlI33HCDGjdurKefflqzZs3SN998o3r16iktLU1paWk6d+5cuXI50pYtW9S+fXu1b99e2dnZGj9+vNq3b2/Xu1Lt27dXXFyc0tLS7HZNAABQdbG80Cj23mjggQekMWNKP2fGDOm778p2vQ0bKp5J0vr16yVJ/fr1s+nrd+/ercjIyGKznJwcDRw4UNOnT9cTTzyhoKAgm/P985//VK1atbRjxw41atSo2LGMjIxyn1caW3+WVatWacKECRo/fnzRbOjQobr11lv19ttvq2/fvmX6/lJh4Xj00UdLPD506FDFxcXppZde0vTp02UymSRJf/nLXxQVFaX8/Hxt3LhR/v7+klR0l++mm24qdp3x48eXeMfxsvfff19nzpwpc/Z27dppyJAhZT6/JH369JHZbK7wdUrTqlUrSdKmTZs0ePDgSv1eAADA+Tll6Zo7d65Wr16t2NhY7dixQ2fPntVDDz1k83MoK1as0AcffKANGzbo9OnTatCgga6//nqNGTNGt912m53Tl9HGjfa9XllKXFKS/b/vNVz+Tb+1MpGYmKjZs2cXm/n5+Wns2LFFn19dUiSpVq1aevrpp7Vy5UqtWLFCw4cPr1BGNzc3ubu7W8wvF4vynlcSW3+WsLAwi93wbrnlFoWGhuqPP/4o0/eWpOPHj+vs2bPy8/OzevzXX3/VihUr1KNHD02bNq2ocElSSEiIevbsqd9++02xsbG6+eabJalC5eX9999XUlJSmc8fMWKEXUqXI1z+bxwfH29sEAAA4BScsnRNnjxZO3bsUJ06dRQcHKy4uDibr/XSSy/p7bffVnBwsO644w75+/vrxIkT2rp1q1atWmVc6YISExM1ceLEYrOwsLBipSs5OVlvvfWWVqxYoeTkZGVnZxc7PzU1tUIZHnroIb3wwgtq1aqVHnjgAfXu3Vvdu3dXw4YNbTqvNLb+LO3atbO6G2BISIg2lOOO5LFjxyRJPj4+Vo9f/qXG2LFjrb6/6/JujAUFBWX+nqVJTEy0y3Wc0eX/xpf/mwMAgJrNKUvXe++9p+DgYDVr1kyrV68u1/KpK/3rX//S22+/rREjRujTTz9VrVq1ih3Pzc21R1yUokmTJtq3b5+OHj1qcezKZV55eXkWd5ESEhLUpUsXnT59Wj179tSAAQPk6+srV1dXJSYm6ssvv9SlS5cqlO/555+Xv7+/PvroI82cOVPvv/++TCaTevfurbfffludOnUq13klqcjPUtKdKTc3t3IVoPPnz0squXStXbtWLi4uuvXWW60eP3LkiCSpWbNmZf6eNdXlgnr5vzkAAKjZnLJ02VqyrnTp0iW9+uqrCg0NtVq4JFldKgb76t69u2JiYrRixQo98sgj5frad999VydPntSsWbMsdgL89ttv9eWXXxabXb47k5eXZ/V6Z86csVpghg8fruHDh+vMmTP6/fffNW/ePH3xxRe65ZZbFBcXV3Q3q6zn2eNnqUzWlgTm5+crKSlJjRo1kre3t8Xx48ePa/PmzWratKndtoS31zNdVy6DNMrV/00vf+4M2QAAgPGcsnTZw6+//qoTJ04ULZVatGiRdu/erdq1a6tLly7qZu+NLMqra1f7Xi8srGzn2Pv7XsPIkSM1ffp0zZ07V6+99ppatmxZ5q89dOiQJOnuu++2OLZ69WqLWb169SRJKSkpVq+VmZlZ4l0jqfCO0m233abbbrtNBQUF+uKLL7RmzRqL71/W8yrys1SGy2UqKyvL4tjlwnr27FkVFBRYLC/829/+poKCAj3++ON2y2OvZ7oqe1MMW1z+b2ytwAIAgJqn2pauzZs3S5Jq166t9u3ba/fu3cWO9+rVS3Pnzi317sSlS5eKLfmy9o9Vm9lpd8ByGTPm2jsc2llkZKRee+01TZgwQQMHDtQ333yjG2+80eI8a3c8Lr9Ta9WqVRo0aFDRfNmyZfrss88szm/RooV8fHy0YMECpaenF+0ymJ2drWeffdZqvpiYGPXp08fijkR6erokycvLq1znlaS8P0tlCAgIkGT9/45NJpPatm2r7du369tvv9VDDz1UdGzu3Ll6//331aJFC42x4//9VOdnui7/Ny7tfWgAAKDmqLal6/I/ht9++221atVKa9euVbt27XT48GG9+OKLWr58ue69916tWrWqxGtMmzbNYqMHlN8bb7whs9msSZMmqXv37urYsaO6dOmi+vXr68yZM0pMTNRvv/0mqbAMX/bUU09p1qxZuvfee3XPPfcoMDBQu3fv1tKlS3Xffffp+++/L/Z93N3dNWbMGE2aNEnt27fXnXfeqby8PP36668KDAy0+g/gO++8U3Xq1FHXrl0VHh4us9mstWvXavPmzerYsWPRLn1lPa8k5f1ZKkPjxo1Vt27dEpf0vfHGG7rrrrs0atQoLV26VCEhIdq8ebN+++03RUVFafHixapdu3al57TFhx9+qH/+859FRe66667Ta6+9pttvv73Ur+vTp0+pfwZMmDDB4s+Axo0bX/P9W6dPn5bE828AAKBQtS1dlzcYcHNz08KFC4vuNFx//fWaN2+emjdvrtWrV2vDhg0lLjUcN26cnn/++aLPs7KyFBISUunZqxuTyaQJEybowQcf1CeffKKYmBh98803On/+vOrWravIyEg9+eSTevjhh9WhQ4eir2vTpo1iYmL02muvadGiRcrLy1Pbtm31888/y8/Pz2pRmThxory8vPSvf/1Ln376qZo0aaIHHnhAEyZMKHp30pWmT5+uZcuWadu2bUWlIiwsTG+99ZaefPLJouf+ynpeSWz5WSpDp06dtHfvXqvHhgwZonnz5umtt97Szz//LLPZrOjoaE2ZMkXPPvus6tSp45CMtggODtZbb72lqKgoFRQU6Msvv9SQIUOLine truncated
|
||||
"text/plain": [
|
||||
"<Figure size 1000x800 with 1 Axes>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"image/png": "iVBORw0KGgoAAAANSUhEUgAAA2oAAALZCAYAAADGAtJkAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAC150lEQVR4nOzdd3xUVcLG8WdSKIGQEEIJ6QktgBA60kFFwQIqCioguLa1oeiuuipFEXRXRX3tq6Li2kABXQVBukiHUBNKCgkhAYKkACGEZN4/IlmHuQnJZDIzmfy+n08+65x7584zvL7Cwzn3XJPZbDYLAAAAAOAyPJwdAAAAAABgiaIGAAAAAC6GogYAAAAALoaiBgAAAAAuhqIGAAAAAC6GogYAAAAALoaiBgAAAAAuhqIGAAAAAC7Gy9kB3F1xcbGOHDkiX19fmUwmZ8cBAAAA4CRms1l5eXlq2bKlPDzKnzOjqFWzI0eOKDQ01NkxAAAAALiItLQ0hYSElHsORa2a+fr6Sir5P0ajRo2cnAYAAACAs+Tm5io0NLS0I5SHolbNLix3bNSoEUUNAAAAQIVuiWIzEQAAAABwMRQ1AAAAAHAxFDUAAAAAcDEUNQAAAABwMRQ1AAAAAHAxFDUAAAAAcDEUNQAAAABwMRQ1AAAAAHAxFDUAAAAAcDEUNQAAAABwMRQ1AAAAAHAxFDUAAAAAcDEUNQAAAABwMRQ1AAAAAHAxFDUAAAAAcDEUNQC4hDVr1uiGG25QcHCwTCaTPvnkE2dHAgAAbo6iBqBSTp8+XaX3z5o1Sz169FCjRo3UtGlTXX/99dq9e7ed0lWPU6dOqWPHjnrjjTdUv359u123qr+WAADAfVHUAFTY2rVrtWDBgipdY9WqVXrggQf022+/acWKFfLy8tKVV16p33//3U4p7W/48OGaOXOmRo0aJQ8P+/1n88SJE5o5c6bdrgcAANyHl7MDAKgZMjIy9Morr1S5qP38888Wr+fOnSs/Pz+tW7dO119/fZWuXdOEhYXJ09NT8+bN0y233OLsOAAAwIUwowa4gJSUFJlMJk2YMMHZUcp0//33a9q0aXadUZKkvLw8FRcXq3Hjxna9bk3x+OOP67XXXtPx48edHQUAALgQihpc2v79+zV58mR17dpVAQEB8vb2VkBAgHr16qUnnnhCW7dudXbEWuGXX37R0aNH1aVLF7tfe9KkSYqNjdXll19u92vXBF5eXho1apSeffZZZ0cBAAAuhKWPcElms1nPP/+8nn/+eRUXF6tr164aPXq0AgIClJeXp507d+r//u//9Oqrr+qtt97Sgw8+6OzIVRIcHKz4+Hj5+fk5O4qhmTNn6s4777T7dSdPnqxff/1Vv/76qzw9Pe1+/Zpi7NixioqK0nPPPaeQkBBnxwEAAC6AogaX9Pzzz2vatGkKDQ3Vl19+qb59+1qdc+zYMb3++uvKyclxQkL78vb2Vrt27Zwdw9CBAwe0cuVKff7553a97mOPPaavvvpKK1euVFRUlF2vXdM0b95c3bt318cff6wpU6Y4Ow4AAHABLH2Ey0lKStKMGTNUp04dLV682LCkSVKzZs00c+ZM/f3vf7cY/+STT3TzzTcrKipK9evXV6NGjdS3b1/DorFq1SqZTCZNmzbN8DMiIiIUERFhNf7999/riiuuUFBQkOrWrauWLVtq4MCBeuedd2w6r6x71CrzXS6+TkpKisaMGaPAwEDVq1dP3bt313//+1/D95Xn66+/VvPmzdWyZcsyzzl16pSef/55denSRb6+vjKZTIY/R48elVSy3PHLL7/UihUrXLag/tmpU6cUFxenuLg4FRcXKzU1VXFxcUpNTbXbZ3Tt2lVffvml3a4HAABqNooaXM6cOXN0/vx5jRo1Sh06dLjk+V5elhPDf/3rX3Xo0CENGDBAjz76qMaMGaNDhw5p3Lhxeu6556qc74MPPtCIESO0d+9eXX/99Xr88cc1fPhw5efna86cOZU+rzy2fpdDhw6pZ8+eSklJ0bhx4zR69Gjt3r1bI0aM0MqVKyv1fVesWFHuvWnHjh1Tjx49NHXqVBUXF+v+++/Xww8/rBYtWkgqmS2Mjo5Wr1691Lx5cz344IOaM2eOvvjiCzVu3FiZmZnKzMzUqVOnKpXLkbZs2aIuXbqoS5cuys/P19SpU9WlSxe7zn516dJFCQkJyszMtNs1AQBAzcXSx5rE3pstjBkjTZpU/jlvvCF99VXFrrd+fdUzSVq3bp0kaciQITa9f/fu3YqOjrYYO3funIYNG6aXXnpJ999/v4KDg23O9/7776tOnTrasWOHmjVrZnEsKyur0ueVx9bvsmrVKk2bNk1Tp04tHbv99tt1zTXX6F//+pcGDx5coc+XSkrK3XffXebx22+/XQkJCfr73/+ul156SSaTSZL0t7/9Ta1bt1ZRUZE2bNigwMBASSqdTbziiissrjN16tQyZzYveP3115WdnV3h7LGxsRo5cmSFzy/LoEGDZDabq3yd8rRv316StHHjRo0YMaJaPwsAALg+lyxq8+fP1+rVqxUXF6cdO3YoLy9Pd9xxR6Xukfnkk080ceLEcs/x8PBQUVFR6euUlBRFRkaWef7o0aP1VUVLS3XYsMG+16tI8Tt0yP6fewkXZhSMCkhKSoo++eQTizF/f389+uijpa8vLjaSVKdOHT344INasWKFli9frvHjx1cpo5eXl7y9va3GL5SRyp5XFlu/S3h4uNUugldffbXCwsK0adOmCn22JB09elR5eXny9/c3PL5s2TItX75c/fr106xZs0pLmiSFhoaqf//++uWXXxQXF6crr7xSkqpUeF5//XUdOnSowuffeeeddilqjnDh1zgxMdG5QQAAgEtwyaI2Y8YM7dixQw0bNlRISIgSEhIqfY3Y2FiL2YQ/W7t2rVasWKFhw4YZHu/cubPhH+46duxY6Rywr5SUFE2fPt1iLDw83KKopaam6uWXX9by5cuVmpqq/Px8i/PT09OrlOGOO+7Q448/rvbt22vMmDEaOHCg+vbtq6ZNm9p0Xnls/S6xsbGGuyiGhoZqfSVmPjMyMiRJjRo1Mjx+4S9PHn30UcPnq13YxbK4uLjCn1melJQUu1zHFV34Nb7waw4AAGo3lyxqs2fPVkhIiFq1aqXVq1dXapnWBbGxsYqNjTU8duF5Tffee2+Z773UEixUnxYtWig+Pl5HjhyxOvbnJWjnz5+3mq1KSkpSz549dfLkSfXv319Dhw6Vn5+fPD09lZKSok8//VQFBQVVyjd58mQFBgbqnXfe0ZtvvqnXX39dJpNJAwcO1L/+9S917969UueVpSrfpawZMC8vr0qVptOnT0squ6itXbtWHh4euuaaawyPHz58WJLUqlWrCn9mbXWh1F74NQcAALWbSxY1W4pZRe3atUsbNmxQcHCwrr322mr7HNiub9++WrlypZYvX6677rqrUu997bXXdOLECc2ZM8dqB8Uvv/xSn376qcXYhVmg8+fPG14vOzvbsPSMHz9e48ePV3Z2tn777TctWLBAH3/8sa6++molJCSUzppV9Dx7fJfqZLRcsaioSIcOHVKzZs3UoEEDq+NHjx7V5s2bFRkZabft9+11j9qfl2g6y8W/phdeu0I2AADgfC5Z1KrTBx98IEn6y1/+UuYDdo8cOaL3339fJ06cUJMmTXT55ZerU6dOjoxprHdv+14vPLxi59j7cy9hwoQJeumllzR//nw9++yziomJqfB7Dx48KEm6+eabrY6tXr3aaqxx48aSpLS0NMNr5eTklDk7JZXMXA0fPlzDhw9XcXGxPv74Y61Zs8bq8yt6XlW+S3W4UMByc3Otjl0ouXl5eSouLrZa+vjPf/5TxcXFuu++++yWx173qFX3xiC2uPBrbFR6AQBA7VOrilp+fr4+//xzeXp6lruL3bJly7Rs2TKLsUGDBunTTz9VWFhYuZ9RUFBgsRzN6A+4NrPTroqVMmnSpXeGtLPo6Gg9++yzmjZtmoYNG6YvvvhCffr0sTrPaGblwjPPVq1apeuvv750/Oeff9aHH35odX67du3UqFEjLVq0SMeOHSvdnTE/P1+PPPKIYb6VK1dq0KBBVjMfx44dkyT5+PhU6ryyVPa7VIegoCBJxv8em0wmde7cWdu3b9eXX36pO+64o/TY/Pnz9frrr6tdu3aaZMd/f9z5HrULv8blPa8OAADUHrWqqH3zzTfKzs7Wtddeq9DQUKvjPj4+eu655zRy5MjSpVo7d+7UtGnTtHLlSl1xxRWKi4sr92+8Z82aZbXZBSpvypQpMpvNeuGFF9S3b19169ZNPXv2VEBAgLKzs5WSkqJffvlFkjRgwIDS9z3wwAOaM2eObrnlFo0aNUotW7bU7t27tWTJEt166636+uuvLT7H29tbkyZN0gsvvKAuXbroxhtv1Pnz57Vs2TK1bNnS8A/NN954oxo2bKjevXsrIiJCZrNZa9eu1ebNm9WtW7fS3Q0rel5ZKvtdqkPz5s3l6+tb5nLDKVOm6KabbtLEiRO1ZMkShYaGavPmzfrll1/UunVr/fTTT6pXr16157TF22+/rffff7+0/HXo0EHPPvvsJZdEDxo0SKtWrSr3nIyMDD311FP66aeflJeXp6ioKL377rsaOHBgme85efKkJO7nAwAAfzC7uJUrV5olme+4444qX6tPnz5mSebvv/++Uu8rLCw09+rVyyzJ/Prrr5d77tmzZ805OTmlP2lpaWZJ5pycnKpEr7USEhLMjz76qLlz585mPz8/s5eXl7lx48bm7t27mx999FHz1q1brd6zbt068+DBg83+/v7mhg0bmvv27WtesGBB6b9LU6dOtTi/uLjYPGvWLHNUVJTZ29vbHBoaav7b3/5mPn36tDk8PNwcHh5ucf67775rHjlypDkyMtJcv359c+PGjc2xsbHml19+2Zybm1vp88xmszk5OdksyXznnXdW6buUdZ0LBg4Line truncated
|
||||
"text/plain": [
|
||||
"<Figure size 1000x800 with 1 Axes>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"image/png": "iVBORw0KGgoAAAANSUhEUgAAA2oAAALfCAYAAAAQWzF5AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAC4EElEQVR4nOzdeViVdf7/8ddhEUQFRER2ENwzd8vctbKyMa1MrdS0abHVcppJf5VomdbMlNW3mqaZMmumrMytMc1yzy3NNBcQNwQRRFQWFVnP7w9GxuM5B+FwOAs8H9d1ruJ93+c+74Necl58lttgNBqNAgAAAAC4DA9nNwAAAAAAMEVQAwAAAAAXQ1ADAAAAABdDUAMAAAAAF0NQAwAAAAAXQ1ADAAAAABdDUAMAAAAAF0NQAwAAAAAXQ1ADAAAAABdDUAMAAAAAF+NyQe306dP65z//qTvvvFOtWrVSw4YNFRAQoL59++qjjz5SWVlZta53/PhxPfjggwoPD5ePj49iY2P1zDPP6OzZs1afs3//fo0aNUohISHy9fVV27ZtlZCQoIKCgpq+PQAAAAC4KoPRaDQ6u4nLffDBB3rssccUFhamQYMGKTo6WidPntSiRYuUm5uru+++W19//bUMBsNVr3X48GH17t1bWVlZGj58uNq1a6eff/5Za9euVdu2bbVp0yY1a9bM5Dnbtm3T4MGDVVxcrJEjRyoqKkpr1qzRjh071KdPH61evVo+Pj619fYBAAAAwPWC2po1a3T+/Hndfvvt8vD434BfZmamrrvuOqWlpWnhwoW6++67r3qtW265RatWrdI777yjp556qqI+ZcoUzZ07V48++qg++OCDinppaamuvfZaJSYmaunSpbrjjjskSWVlZRo1apS++eYbzZkzR1OnTrXjOwYAAAAAUy4X1Coze/ZsvfDCC3ryySf1f//3f5Wee/jwYbVq1UqxsbE6fPiwSejLz89XWFiYjEajsrKy1KhRI0nlIfHGG29U//79tX79epPrHTlyRPHx8YqJidHRo0erNKIHAAAAALbwcnYD1eHt7S1J8vK6ettr166VJA0ZMsQkpElSkyZN1KdPH61atUpbt27VjTfeKKk8qEnSrbfeana9uLg4tWnTRsnJyRWhrSrKysp04sQJNWnShHAHAAAA1GNGo1H5+fkKDw83yyhXcpugVlJSok8//VSS5SB1pQMHDkiS2rRpY/F469attWrVKiUnJ1cEtao8Jzk5WcnJyVaDWmFhoQoLCyu+Tk9PV4cOHa7aLwAAAID6IS0tTZGRkZWe4zZBberUqdq7d6+GDh2qW2655arn5+bmSpICAgIsHr9Uz8nJqdFzrjRnzhzNnDnTrJ6WliZ/f/+r9g0AAACgbsrLy1NUVJSaNGly1XPdIqi98847euONN9SuXTt99tlnzm6nUtOmTdOUKVMqvr70h+Hv709QAwAAAFClJVEuH9TeffddTZ48WR06dNDq1asVFBRUpeddGv26NEp2pUv1wMDAGj3nSj4+PmzfDwAAAKBGXO6G15d766239NRTT6ljx45au3atQkNDq/zctm3bSpKSk5MtHj948KAk0/VotjwHAAAAAOzNZYPa66+/rmeffVZdunTR2rVrFRISUq3nDxo0SJK0atUqlZWVmRzLz8/Xpk2b5Ofnp169elXUBw8eLElauXKl2fWOHDmi5ORkxcTEKC4urrpvBwAAAACqzCWD2iuvvKKpU6eqe/fuWr16tYKDg62eW1xcrKSkJB0+fNikHh8fryFDhiglJUXvvfeeybGEhASdP39e48aNq7iHmiQNGDBA7du314YNG7Rs2bKKellZmZ5//nlJ0qRJk9hmHwAAAECtcrkbXs+fP18TJkyQp6ennnrqKYs7MMbGxmrChAmSpJSUFLVs2VIxMTFKSUkxOe/w4cPq3bu3srKyNHz4cLVv317btm3T2rVr1aZNG23evFnNmjUzec62bds0ePBgFRcXa+TIkYqOjtbq1au1Y8cO9enTR6tXr67WGrS8vDwFBAQoNzeXzUQAAACAeqw62cDlNhM5evSoJKm0tFRvvfWWxXMGDBhQEdQqEx8frx07dmj69OlauXKlvvvuO4WFhWny5MlKSEhQ06ZNzZ5z/fXXa/v27UpISNCqVauUn5+vmJgYTZ8+XVOnTmWjEAAAAAC1zuVG1OoaRtQAAAAASNXLBi65Rg0AAAAA6jOCGgAAAAC4GIIaAAAAALgYghoAAAAAuBiCGgAAAAC4GIIaAAAAALgYghoAAAAAuBiCGgAAAAC4GIIaAFzFhg0bdMcddygiIkIGg0GffPKJs1sCAAB1HEENQLWcP3++Rs+fM2eOevbsKX9/fzVv3lzDhg3T3r177dRd7Th37pw6duyot99+Ww0bNrTbdWv6vQQAAHUXQQ1AlW3cuFGLFy+u0TXWrVunxx9/XJs3b9aaNWvk5eWlm266SWfOnLFTl/Y3dOhQzZ49WyNHjpSHh/3+2Tx9+rRmz55tt+sBAIC6w8vZDQBwDxkZGfrrX/9a46D2/fffm3z92WefKSAgQJs2bdKwYcNqdG13Ex0dLU9PT3399de65557nN0OAABwIYyoAS4gJSVFBoNBEyZMcHYrVk2aNEkzZsyw64iSJOXn56usrExNmza163XdxR/+8Ae9+eabOnXqlLNbAQAALoSgBpeWnJysKVOmqFu3bgoKCpK3t7eCgoJ0/fXX67nnntMvv/zi7BbrhR9//FEnT55U165d7X7tyZMnq0uXLrrhhhvsfm134OXlpZEjR+rFF190disAAMCFMPURLsloNOrll1/Wyy+/rLKyMnXr1k2jR49WUFCQ8vPz9dtvv+n//u//9MYbb+jdd9/VE0884eyWayQiIkKJiYkKCAhwdisWzZ49Ww888IDdrztlyhT99NNP+umnn+Tp6Wn367uLsWPHKi4uTi+99JIiIyOd3Q4AAHABBDW4pJdfflkzZsxQVFSUvvjiC/Xp08fsnKysLL311lvKzc11Qof25e3trXbt2jm7DYsOHjyotWvX6l//+pddr/vss89qwYIFWrt2reLi4ux6bXfTokUL9ejRQx9//LGmT5/u7HYAAIALYOojXM6RI0c0a9YsNWjQQCtWrLAY0iQpJCREs2fP1p/+9CeT+ieffKK7775bcXFxatiwofz9/dWnTx+LQWPdunUyGAyaMWOGxdeIjY1VbGysWX3ZsmW68cYbFRYWJh8fH4WHh2vAgAF6//33bTrP2hq16ryXK6+TkpKiMWPGKDg4WL6+vurRo4f+85//WHxeZb788ku1aNFC4eHhVs85d+6cXn75ZXXt2lVNmjSRwWCw+Dh58qSk8umOX3zxhdasWeOyAfVy586d065du7Rr1y6VlZUpNTVVu3btUmpqqt1eo1u3bvriiy/sdj0AAODeCGpwOfPmzVNJSYlGjhypa6655qrne3mZDgw/9thjOnbsmPr3769nnnlGY8aM0bFjxzRu3Di99NJLNe7vww8/1PDhw7V//34NGzZMf/jDHzR06FAVFBRo3rx51T6vMra+l2PHjum6665TSkqKxo0bp9GjR2vv3r0aPny41q5dW633u2bNmkrXpmVlZalnz55KSEhQWVmZJk2apKeeekqhoaGSykcL4+Pjdf3116tFixZ64oknNG/ePH3++edq2rSpMjMzlZmZqXPnzlWrL0fasWOHunbtqq5du6qgoEAJCQnq2rWrXUe/unbtqqSkJGVmZtrtmgAAwH0x9dGd2HuzhTFjpMmTKz/n7belBQuqdr0tW2rek6RNmzZJkgYPHmzT8/fu3av4+HiTWlFRkW677Ta99tprmjRpkiIiImzu7+9//7saNGig3bt3KyQkxORYdnZ2tc+rjK3vZd26dZoxY4YSEhIqavfdd59uvfVW/eUvf9GgQYOq9PpSeUh56KGHrB6/7777lJSUpD/96U967bXXZDAYJEl//OMf1bp1a5WWlmrr1q0KDg6WpIrRxBtvvNHkOgkJCVZHNi956623lJOTU+Xeu3TpohEjRlT5fGsGDhwoo9FY4+tUpkOHDpKkbdu2afjw4bX6WgAAwPUR1NzJ1q32vV5Vgt+xY/Z/3au4NKJgKYCkpKTok08+MakFBgbqmWeeqfj6ymAjSQ0aNNATTzyhNWvWaPXq1Ro/fnyNevTy8pK3t7dZ/VIYqe551tj6XmJiYsx2EbzlllsUHR2tn3/+uUqvLUknT55Ufn6+AgMDLR7/4YcftHr1avXt21dz5sypCGmSFBUVpX79+unHH3/Url27dNNNN0lSjQLPW2+9pWPHjlX5/AceeMAuQc0RLn2PDx8+7NxGAACASyCowa2kpKRo5syZJrWYmBiToJaamqrXX39dq1evVmpqqgoKCkzOT09Pr1EP999/v/7whz+oQ4cOGjNmjAYMGKA+ffqoefPmNp1XGVvfS5cuXSzuohgVFaUt1Rj5zMjIkCT5+/tbPH5prdwzzzxj8f5ql3axLCsrq/JrViYlJcUu13FFl77Hl77nAACgfiOoweWEhoYqMTFRJ06cMDt2+RS0kpISs9GqI0eO6LrrrtPZs2fVr18/DRkyRAEBAfL09FRKSormz5+vwsLCGvU3ZcoUBQcH6/3339c777yjt956SwaDQQMGDNBf/vIX9ejRo1rnWVOT92JtBMzLy6taoen8+fOSrAe1jRs3ysPDQ7feeqvF48ePH5cktWrVqsqvWV9dCrWXvucAAKB+I6jB5fTp00dr167V6tWr9eCDD1bruW+++aZOnz6tefPmme2g+MUXX2j+/PkmtUujQCUlJRavl5OTYzH0jB8/XuPHj1dOTo42b96sxYsX6+OPP9Ytt9yipKSkilGzqp5nj/dSmyxNVywtLdWxY8cUEhKiRo0amR0/efKktm/frpYtW9pt+317rVG7fIqms1z5Pb30tSv0BgAAnI+g5k569bLv9WJiqnaOvV/3KiZMmKDXXntLine truncated
|
||||
"text/plain": [
|
||||
"<Figure size 1000x800 with 1 Axes>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"image/png": "iVBORw0KGgoAAAANSUhEUgAAA2oAAALZCAYAAADGAtJkAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAC3P0lEQVR4nOzdeVyVZcLG8euwiKICIiLIKu5m7mm5a2VpmVa2Oa6tTpvt6UyJlmVNmdW0TTNl5bRbLk1qFm655l4auKAIIgi4ACqynvcPkjc8z0E4HM4Cv+/nw2eG+3nOc67jO694cd/P/ZjMZrNZAAAAAACX4eHsAAAAAACA8ihqAAAAAOBiKGoAAAAA4GIoagAAAADgYihqAAAAAOBiKGoAAAAA4GIoagAAAADgYihqAAAAAOBivJwdoLYrKSnR0aNH1bhxY5lMJmfHAQAAAOAkZrNZubm5atGihTw8Kp4zo6jVsKNHjyoiIsLZMQAAAAC4iJSUFIWHh1d4DkWthjVu3FhS6f8x/Pz8nJwGAAAAgLPk5OQoIiKirCNUhKJWw84vd/Tz86OoAQAAAKjULVFsJgIAAAAALoaiBgAAAAAuhqIGAAAAAC6GogYAAAAALoaiBgAAAAAuhqIGAAAAAC6GogYAAAAALoaiBgAAAAAuhqIGAAAAAC6GogYAAAAALoaiBgAAAAAuhqIGAAAAAC6GogYAAAAALoaiBgAAAAAuhqIGAAAAAC6GogYAF7F27VrdcMMNCgsLk8lk0kcffeTsSAAAoJajqAGokjNnzlTr9bNnz9Zll10mPz8/NWvWTCNGjNDu3bvtlK5mnD59Wp06ddIbb7yhBg0a2O261f2zBAAAtRdFDUCl/fzzz1q4cGG1rrF69Wrdf//92rBhg1auXCkvLy9dddVVOnHihJ1S2t/w4cP14osvavTo0fLwsN9fm8ePH9eLL75ot+sBAIDaw8vZAQC4h7S0NL366qvVLmo//PBDue/nz58vf39/rV+/XiNGjKjWtd1NZGSkPD099fXXX+uWW25xdhwAAOBCmFEDXEBSUpJMJpMmTpzo7ChWTZ48WTNmzLDrjJIk5ebmqqSkRE2aNLHrdd3F448/rtdee02ZmZnOjgIAAFwIRQ0ubd++fXrsscfUvXt3BQYGytvbW4GBgerdu7eeeOIJbdu2zdkR64SffvpJx44dU7du3ex+7SlTpqhr16664oor7H5td+Dl5aXRo0frmWeecXYUAADgQlj6CJdkNpv13HPP6bnnnlNJSYm6d++u2267TYGBgcrNzdWvv/6qf/7zn5ozZ47eeustPfDAA86OXC1hYWGKj4+Xv7+/s6MYevHFFzVhwgS7X/exxx7TunXrtG7dOnl6etr9+u5i7NixiomJ0bPPPqvw8HBnxwEAAC6AogaX9Nxzz2nGjBmKiIjQ559/rr59+1qck5GRoddff13Z2dlOSGhf3t7eat++vbNjGNq/f79WrVql//73v3a97qOPPqovvvhCq1atUkxMjF2v7W6aN2+unj176sMPP9T06dOdHQcAALgAlj7C5Rw8eFCzZs1SvXr1tGzZMsOSJknBwcF68cUX9dRTT5Ub/+ijj3TzzTcrJiZGDRo0kJ+fn/r27WtYNFavXi2TyaQZM2YYvkd0dLSio6MtxpcsWaIrr7xSoaGh8vHxUYsWLTRw4EC98847Np1n7R61qnyWC6+TlJSk22+/XUFBQapfv7569uyp//3vf4avq8iXX36p5s2bq0WLFlbPOX36tJ577jl169ZNjRs3lslkMvw6duyYpNLljp9//rlWrlzpsgX1z06fPq2dO3dq586dKikpUXJysnbu3Knk5GS7vUf37t31+eef2+16AADAvVHU4HLmzZunoqIijR49WpdccslFz/fyKj8x/Ne//lWHDx/WgAED9Mgjj+j222/X4cOHNW7cOD377LPVzvf+++9r5MiR+v333zVixAg9/vjjGj58uPLy8jRv3rwqn1cRWz/L4cOH1atXLyUlJWncuHG67bbbtHv3bo0cOVKrVq2q0udduXJlhfemZWRk6LLLLlNsbKxKSko0efJkPfTQQwoJCZFUOlvYqlUr9e7dW82bN9cDDzygefPm6bPPPlOTJk2Unp6u9PR0nT59ukq5HGnr1q3q1q2bunXrpry8PMXGxqpbt252nf3q1q2bEhISlJ6ebrdrAgAA98XSR3di780Wbr9dmjKl4nPeeEP64ovKXW/jxupnkrR+/XpJ0pAhQ2x6/e7du9WqVatyYwUFBRo2bJheeuklTZ48WWFhYTbn+9e//qV69epp165dCg4OLncsKyuryudVxNbPsnr1as2YMUOxsbFlY2PGjNG1116rV155RYMHD67U+0ulJeXuu++2enzMmDFKSEjQU089pZdeekkmk0mS9OSTT6pNmzYqLi7Wpk2bFBQUJElls4lXXnlluevExsZandk87/XXX9epU6cqnb1r164aNWpUpc+3ZtCgQTKbzdW+TkU6duwoSdq8ebNGjhxZo+8FAABcH0XNnWzaZN/rVab4HT5s//e9iPMzCkYFJCkpSR999FG5sYCAAD3yyCNl319YbCSpXr16euCBB7Ry5UrFxcVp/Pjx1cro5eUlb29vi/HzZaSq51lj62eJioqy2EXwmmuuUWRkpH755ZdKvbckHTt2TLm5uQoICDA8/uOPPyouLk79+vXT7Nmzy0qaJEVERKh///766aeftHPnTl111VWSVK3C8/rrr+vw4cOVPn/ChAl2KWqOcP7PODEx0blBAACAS6Cowa0kJSVp5syZ5caioqLKFbXk5GS9/PLLiouLU3JysvLy8sqdn5qaWq0Mf/nLX/T444+rY8eOuv322zVw4ED17dtXzZo1s+m8itj6Wbp27Wq4i2JERIQ2VmHmMy0tTZLk5+dnePz8vXKPPPKI4fPVzu9iWVJSUun3rEhSUpJdruOKzv8Zn/8zBwAAdRtFDS4nJCRE8fHxOnr0qMWxPy9BKyoqspitOnjwoHr16qWTJ0+qf//+Gjp0qPz9/eXp6amkpCR9/PHHys/Pr1a+xx57TEFBQXrnnXf05ptv6vXXX5fJZNLAgQP1yiuvqGfPnlU6z5rqfBZrM2BeXl5VKk1nzpyRZL2o/fzzz/Lw8NC1115rePzIkSOSpNatW1f6Peuq86X2/J85AACo2yhqcDl9+/bVqlWrFBcXpzvvvLNKr33ttdd0/PhxzZs3z2IHxc8//1wff/xxubHzs0BFRUWG1zt16pRh6Rk/frzGjx+vU6dOacOGDVq4cKE+/PBDXXPNNUpISCibNavsefb4LDXJaLlicXGxDh8+rODgYDVs2NDi+LFjx7Rlyxa1bNnSbtvv2+setT8v0XSWC/9Mz3/vCtkAAIDzUdTcyeWX2/d6UVGVO8fe73sREydO1EsvvaQFCxbomWeeUYcOHSr92gMHDkiSbr75Zotja9assRhr0qSJJCklJcXwWtnZ2VZnp6TSmavhw4dr+PDhKikp0Ycffqi1a9davH9lz6vOZ6kJ5wtYTk6OxbHzJTc3N1clJSUWSx//8Y9/qKSkRPfdd5/d8tjrHrWa3hjEFuf/jI1KLwAAqHsoau7ETrsqVsmUKRffGdLOWrVqpWeeeUYzZszQsGHD9Nlnn6lPnz4W5xnNrJx/5tnq1as1YsSIsvEffvhB//nPfyzOb9++vfz8/LR48WJlZGSU7c6Yl5enhx9+2DDfqlWrNGjQIIuZj4yMDEmSr69vlc6zpqqfpSaEhoZKMi5qJpNJXbp00Y4dO/T555/rL3/5S9mxBQsW6PXXX1f79u01xY7/+6nN96id/zOu6Hl1AACg7nC556gdP35c//nPf3TjjTeqdevWatCggfz9/dWvXz998MEHVd6U4MiRI7rzzjvVokUL+fj4KDo6Wo888ohOnjxp9TW///67br31VgUHB6t+/fpq166dYmNjLTZyQM2ZPn26YmNjlZKSor59+6pnz566//779cwzz+jBBx/U9ddfr/DwcEnSgAEDyl53//33q169errllls0duxYPfXUUxo+fLiGDRum0aNHW7yPt7e3pkyZouzsbHXr1k0PPvigJk+erE6dOik3N9fwH8033nijIiIiNHr0aD3xxBN6/PHH1atXL/3vf/9Tjx49ynY3rOx51lT1s9SE5s2bq3HjxlaXG06fPl0mk0mTJk3SuHHj9Le//U1XX321brnlFrVq1UpLly5V/fr1HZK1qt5++2117txZfn5+8vPz0xVXXKHvv//+oq8bNGhQhceLi4v17LPPqmXLlqpfv75atmypZ555xury2vPO/53E/XwAAECSZHYx7777rlmSOTQ01DxmzBjz1KlTzZMmTTL7+/ubJZlvvvlmc0lJSaWudeDAAXNwcLBZknnkyJHmp59+2jx48GCzJHO7du3MWVlZFq/ZtGmT2dfX1+zt7W2+4447zE899ZS5Z8+eZknmvn37ms+dO1elz5OdnW2WZM7Ozq7S61AqISHB/Mgjj5i7dOli9vf3N3t5eZmbNGli7tmzp/mRRx4xb9u2zeI169evNw8ePNgcEBBgbtSokblv377mhQsXmletWmWWZI6NjS13fklJiXn27NnmmJgYs7e3tzkiIsL85JNPms+cOWOOiooyR0VFlTv/3XffNY8aNcrcsmVLc4MGDcxNmjQxd+3a1fzyyy+bc3Jyqnye2Ww2Hzp0yCzJPGHChGp9FmvXOW/gwIHmqv6//eDBg81Dhw61enzRokXmK664wuzr62tu0KCBuUuXLuYXXnjBnJubW6X3cbRFixaZly5dat6/f79579695r/97W9mLy8v865duyzOXbt2rTkuLs5sNpf+GZrNZnNcXJx57dq1Fue+8MIL5iZNmpiXLFliPnTokHnx4sXmgIAA83PPPVdhnnnz5pklmTMyMqr/4QAAgEuqSjdwuaIWFxdnXrJkibm4uLjceFpamjkiIsIsybxLine truncated
|
||||
"text/plain": [
|
||||
"<Figure size 1000x800 with 1 Axes>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"image/png": "iVBORw0KGgoAAAANSUhEUgAAA10AAALZCAYAAABMLUslAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAACi6UlEQVR4nOzdd3hUZcL+8XtSSUIKIb0RerHQRFGQpqBigbWvooCvv7Ui1l19VYqi6LoWXNvqKii7VhTQFymKIEUUERCBBKSEQEgFUoCQOr8/smQNcyYmk5k5M5Pv57pyrfOcM2fusLuGO89znmOxWq1WAQAAAABcws/sAAAAAADgyyhdAAAAAOBClC4AAAAAcCFKFwAAAAC4EKULAAAAAFyI0gUAAAAALkTpAgAAAAAXonQBAAAAgAsFmB3Am9TW1urgwYMKDw+XxWIxOw4AAAAAk1itVpWVlSkpKUl+fo3PZVG6muHgwYNKTU01OwYAAAAAD7F//36lpKQ0eg6lqxnCw8Ml1f3BRkREmJwGAAAAgFlKS0uVmppa3xEaQ+lqhpNLCiMiIihdAAAAAJp02xEbaQAAAACAC1G6AAAAAMCFKF0AAAAA4EKULgAAAABwIUoXAAAAALgQpQsAAAAAXIjSBQAAAAAuROkCAAAAABeidAEAAACAC1G6AAAAAMCFKF0AAAAA4EKULgAAAABwIUoXAAAAALgQpQsAAAAAXIjSBQAAAAAuROkC0KqsWrVKV1xxhZKTk2WxWDRnzhyzIwEAAB9H6QJasWPHjrXo/TNnztSAAQMUERGh2NhYXX755dq6dauT0rnG0aNHdfrpp2vWrFkKCQlx2nVb+mcJAAB8F6ULaKVWr16t+fPnt+gaK1eu1J133qnvvvtO33zzjQICAnThhRfq8OHDTkrpfKNHj9bTTz+tq6++Wn5+zvtX4KFDh/T000877XoAAMB3BJgdAID75ebm6m9/+1uLS9fSpUsbvJ47d64iIyO1du1aXX755S26trdJS0uTv7+/PvnkE11zzTVmxwEAAB6EmS7AybKysmSxWDRhwgSzo9h1++23a9q0aU6d6ZGksrIy1dbWql27dk69rrd44IEH9MILL6iwsNDsKAAAwINQuuA2O3fu1P33369+/fopOjpagYGBio6O1jnnnKMHH3xQP/30k9kRW4Wvv/5a+fn56tu3r9OvPXnyZPXp00fnnnuu06/tDQICAnT11VfrscceMzsKAADwICwvhMtZrVY98cQTeuKJJ1RbW6t+/frpuuuuU3R0tMrKyrRlyxb9/e9/1/PPP69XXnlFd911l9mRWyQ5OVkZGRmKjIw0O4qhp59+WuPHj3f6de+//36tWbNGa9askb+/v9Ov7y3GjRunTp066fHHH1dKSorZcQAAgAegdMHlnnjiCU2bNk2pqan64IMPNGjQIJtzCgoK9NJLL6mkpMSEhM4VGBioHj16mB3D0K+//qoVK1boX//6l1Ove9999+nDDz/UihUr1KlTJ6de29vEx8frrLPO0jvvvKMpU6aYHQcAAHgAlhfCpfbs2aMZM2YoKChIixcvNixckhQXF6enn35af/7znxuMz5kzR1dddZU6deqkkJAQRUREaNCgQYalYeXKlbJYLJo2bZrhZ6Snpys9Pd1m/PPPP9cFF1ygxMREBQcHKykpSUOHDtVrr73m0Hn27ulqzvdy6nWysrJ0/fXXKyYmRm3atNFZZ52l//u//zN8X2M++ugjxcfHKykpye45R48e1RNPPKG+ffsqPDxcFovF8Cs/P19S3ZLCDz74QN98843Hls3fOnr0qDZv3qzNmzertrZW2dnZ2rx5s7Kzs532Gf369dMHH3zgtOsBAADvRumCS82ePVvV1dW6+uqrddppp/3u+QEBDSdf77jjDu3bt09DhgzRvffeq+uvv1779u3TTTfdpMcff7zF+d58802NGTNG27dv1+WXX64HHnhAo0ePVnl5uWbPnt3s8xrj6Peyb98+nX322crKytJNN92k6667Tlu3btWYMWO0YsWKZn2/33zzTaP3chUUFGjAgAGaOnWqamtrdfvtt2vSpElKSEiQVDeL17lzZ51zzjmKj4/XXXfdpdmzZ+v9999Xu3btlJeXp7y8PB09erRZudxpw4YN6tu3r/r27avy8nJNnTpVffv2deqsVN++fZWZmam8vDynXRMAAHgvlheaxdkbDVx/vTR5cuPnzJolffhh0663bl3LM0lau3atJGnEiBEOvX/r1q3q3Llzg7HKykpdcskleuaZZ3T77bcrOTnZ4Xz/+Mc/FBQUpJ9//llxcXENjhUVFTX7vMY4+r2sXLlS06ZN09SpU+vHbrjhBl188cV67rnnNHz48CZ9vlRXOG699Va7x2+44QZlZmbqz3/+s5555hlZLBZJ0kMPPaSuXbuqpqZG33//vWJiYiSpfpbvggsuaHCdqVOn2p1xPOmll15ScXFxk7P36dNHY8eObfL59gwbNkxWq7XF12lMr169JEk//PCDxowZ49LPAgAAno/SZZbvv3fu9ZpS4vbtc/7n/o6Tv+k3KhNZWVmaM2dOg7GoqCjde++99a9PLSmSFBQUpLvuukvffPONli9frptvvrlFGQMCAhQYGGgzfrJYNPc8exz9Xjp06GCzG95FF12ktLQ0rV+/vkmfLUn5+fkqKytTVFSU4fGvvvpKy5cv1+DBgzVz5sz6wiVJqampOv/88/X1119r8+bNuvDCCyWpReXlpZde0r59+5p8/vjx451Sutzh5J/x7t27zQ0CAAA8AqULpsnKytL06dMbjHXo0KFB6crOztazzz6r5cuXKzs7W+Xl5Q3Oz8nJaVGGG2+8UQ888IB69eql66+/XkOHDtWgQYMUGxvr0HmNcfR76dOnj+FugKmpqVrXjBnJ3NxcSVJERITh8ZP3lt17772Gz+86uRtjbW1tkz+zMVlZWU65jic6+Wd88s8cAAC0bpQuuFRCQoIyMjJ08OBBm2O/XeZVXV1tM4u0Z88enX322Tpy5IjOP/98jRo1SpGRkfL391dWVpbeffddVVRUtCjf/fffr5iYGL322mt6+eWX9dJLL8lisWjo0KF67rnndNZZZzXrPHta8r3Ym5kKCAhoVgE6duyYJPula/Xq1fLz89PFF19sePzAgQOSpC5dujT5M1urkwX15J85AABo3ShdcKlBgwZpxYoVWr58uW655ZZmvfeFF17QoUOHNHv2bJudAD/44AO9++67DcZOzs5UV1cbXq+4uNiwwNx88826+eabVVxcrO+++07z58/XO++8o4suukiZmZn1s1lNPc8Z34srGS0JrKmp0b59+xQXF6ewsDCb4/n5+frxxx/VsWNHp20J76x7un67DNIsp/6ZnnztCdkAAID5KF1mGTjQudfr0KFp5zj7c3/HhAkT9Mwzz2jevHl67LHH1LNnzya/d9euXZKkq666yubYt99+azPWrl07SdL+/fsNr1VSUmJ31kiqm1EaPXq0Ro8erdraWr3zzjtatWqVzec39byWfC+ucLJMlZaW2hw7WVjLyspUW1trs7zwr3/9q2pra3Xbbbc5LY+z7uly9aYYjjj5Z2xUYAEAQOtD6TKLk3YHbJbJk39/h0Mn69y5sx577DFNmzZNl1xyid5//32dd955NucZzXicfKbWypUrdfnll9ePL126VP/85z9tzu/Ro4ciIiK0cOFCFRQU1O8yWF5ernvuuccw34oVKzRs2DCbGYmCggJJUmhoaLPOs6e534srJCYmSjIuXRaLRb1799amTZv0wQcf6MYbb6w/Nm/ePL300kvq0aOHJjvxfz++fE/XyT/jxp6HBgAAWg9KF1xuypQpslqtevLJJzVo0CD1799fZ599tqKjo1VcXKysrCx9/fXXkqQhQ4bUv+/OO+/U7Nmzdc011+jqq69WUlKStm7dqiVLlujaa6/VRx991OBzAgMDNXnyZD355JPq27ev/vCHP6i6ulpfffWVkpKSDP8C/Ic//EFt27bVwIEDlZ6eLqvVqtWrV+vHH39U//7963fpa+p59jT3e3GF+Ph4hYeH213SN2XKFF155ZWaOHGilixZotTUVP3444/6+uuv1bVrV3355Zdq06aNy3M64tVXX9U//vGP+iJ32mmn6bHHHtOll17a6PuGDRumlStX2j1eVlamxx9/XPPnz1dBQYH69u2rWbNmacCAAY1e98iRI5K4/w0AANShdMHlLBaLpk2bpj/+8Y964403tGLFCr3//vs6duyYwsPD1blzZ91xxx266aab1K9fv/r3nXnmmVqxYoUee+wxLVq0SNXV1erdu7c+++wzRUVFGRaV6dOnKzQ0VG+99ZbefPNNJSQk6Prrr9e0adPqn530W88884yWLl2qjRs31peKDh066Nlnn9Udd9xRv7lHU8+zx5HvxRXOOussbd++3fDY2LFjNX/+fD377LP67LPPZLVa1a1bNz311FO655571LZtW7dkdERKSoqeffZZde3aVbW1tXr33Xc1duxY/fTTTzrzzDMbnLt69WpVVVU1eHbcN998o8DAQJ1//vkNzr311lu1ZcsWvfvuu0pJSdG//vUvXXjhhdq+fXujz4fbtm2bJOmcc85x4ncJAAC8lcXqiTdEeKjS0lJFRkaqpKTE7g5wgCebMWOGXn755fplkb4sOjpaM2fOtLkPLTs7W/fff7/i4uL0ww8/6JxzzlFBQYFeeOEFpaWl1Z9XXl6u8PBwffrppw0ecNy/f39dcsklmjFjht3Pnjx5slasWKEtW7Y4/xsDAAAeoTndwPZhPAB81vXXX6/CwsL67d99UU1NjT788EMdPXrU8P7BtLQ0zZs3T5GRkdq4caMiIyM1b968BoVLqtsFs6amxmZJZUhIiNasWdNoho0bN+r6669v+TcDAAB8AqULaEW6dOmiSy+9VPPnzzc7itP98ssvatu2rYKDg3X77bdr/vz5OuOMM2zLine truncated
|
||||
"text/plain": [
|
||||
"<Figure size 1000x800 with 1 Axes>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"image/png": "iVBORw0KGgoAAAANSUhEUgAAA10AAALZCAYAAABMLUslAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAACiqElEQVR4nOzdd3hUVeLG8XcSICFAEkJ6I/QiShNEQJqKggVUrAsIrrtiBVlddVcpiqBrA1dZ110FRcWCirpURYoiIF2BhJqQAAlJgBQgpM7vj/yIhrkTwmT6fD/Pk2fJOXfuvMF9Et6ce881mc1mswAAAAAADuHn6gAAAAAA4M0oXQAAAADgQJQuAAAAAHAgShcAAAAAOBClCwAAAAAciNIFAAAAAA5E6QIAAAAAB6J0AQAAAIAD1XN1AE9SUVGhI0eOqEmTJjKZTK6OAwAAAMBFzGazCgsLFRsbKz+/mteyKF0X4MiRI0pISHB1DAAAAABuIiMjQ/Hx8TUeQ+m6AE2aNJFU+RcbHBzs4jQAAAAAXKWgoEAJCQlVHaEmlK4LcPaSwuDgYEoXAAAAgFrddsRGGgAAAADgQJQuAAAAAHAgShcAAAAAOBClCwAAAAAciNIFAAAAAA5E6QIAAAAAB6J0AQAAAIADUboAAAAAwIEoXQAAAADgQJQuAAAAAHAgShcAAAAAOBClCwAAAAAciNIFAAAAAA5E6QIAAAAAB6J0AQAAAIADUboA+LQ1a9boxhtvVFxcnEwmk+bOnevqSAAAwMtQugAfdurUqTq9fsaMGerRo4eCg4MVERGhG264QTt27LBTOuc4efKkOnXqpFmzZqlhw4YOfa+6/n0DAADPROkCfNQPP/ygL7/8sk7nWLVqlR544AH99NNP+v7771WvXj1dddVVOn78uJ1SOt7QoUM1ffp0jRgxQn5+jv2WeOzYMU2fPt2h7wEAANxPPVcHAOB8mZmZevnll+tcupYtW1bt83nz5ikkJERr167VDTfcUKdze6PExET5+/vrs88+06233urqOAAAwElY6QLsLC0tTSaTSWPGjHF1FKvGjRunKVOm2H1lp7CwUBUVFWratKldz+tN/vKXv+jVV19VTk6Oq6MAAAAnoXTBafbs2aOJEyeqW7duCgsLU/369RUWFqbLLrtMjz32mDZv3uzqiD7hu+++09GjR9W1a1e7n3v8+PHq0qWLLr/8cruf21vUq1dPI0aM0NNPP+3qKAAAwEm4vBAOZzab9eyzz+rZZ59VRUWFunXrpttvv11hYWEqLCzUL7/8on/+85965ZVX9MYbb+jBBx90deQ6iYuLU3JyskJCQlwdxdD06dN199132/28EydO1I8//qgff/xR/v7+dj+/Nxk5cqRatmypZ555RvHx8a6OAwAAHIzSBYd79tlnNWXKFCUkJGj+/Pnq06ePxTHZ2dmaOXOm8vPzXZDQvurXr6/27du7OoahvXv3auXKlfrggw/set5HH31UH3/8sVauXKmWLVva9dzeKCoqSpdeeqneffddTZo0ydVxAACAg3F5IRzqwIEDmjZtmho0aKAlS5YYFi5JioyM1PTp0/XXv/612vjcuXN1yy23qGXLlmrYsKGCg4PVp08fw9KwatUqmUwmTZkyxfA9kpKSlJSUZDH+9ddf68orr1RMTIwCAgIUGxur/v37a/bs2TYdZ+2ergv5Ws49T1pamu644w6Fh4crMDBQl156qf73v/8Zvq4mn3zyiaKiohQbG2v1mJMnT+rZZ59V165d1aRJE5lMJsOPo0ePSqq8pHD+/Pn6/vvv3bZs1uTkyZPatm2btm3bpoqKCqWnp2vbtm1KT0936Pt269ZN8+fPd+h7AAAA90DpgkPNmTNHZWVlGjFihC666KLzHl+vXvXF1/vvv18HDx5Uv379NGHCBN1xxx06ePCgRo0apWeeeabO+d5++20NGzZMu3bt0g033KC//OUvGjp0qIqKijRnzpwLPq4mtn4tBw8eVM+ePZWWlqZRo0bp9ttv144dOzRs2DCtXLnygr7e77//vsZ7ubKzs9WjRw9NnjxZFRUVGjdunB5++GFFR0dLqlzFa9WqlS677DJFRUXpwQcf1Jw5c/TRRx+padOmysrKUlZWlk6ePHlBuVxp06ZN6tq1q7p27aqioiJNnjxZXbt2dfgKVNeuXZWSkqKsrCyHvg8AAHA9Li90FXtvNHDHHdL48TUfM2uW9PHHtTvfunV1zyRp7dq1kqRBgwbZ9PodO3aoVatW1cZKSko0ZMgQvfDCCxo3bpzi4uJszvfvf/9bDRo00Pbt2xUZGVltLjc394KPq4mtX8uqVas0ZcoUTZ48uWrsrrvu0rXXXquXXnpJAwcOrNX7S5UF495777U6f9dddyklJUV//etf9cILL8hkMkmSHn/8cbVp00bl5eVav369wsPDJalqle/KK6+sdp7JkydbXXE8a+bMmcrLy6t19i5dumj48OG1Pr62BgwYILPZbPfznk/Hjh0lSRs2bNCwYcOc/v4AAMB5KF2usn69fc9XmxJ38KD93/c8zv4W36hMpKWlae7cudXGQkNDNWHChKrPzy0pktSgQQM9+OCD+v7777VixQqNHj26Thnr1aun+vXrW4yfLRYXepw1tn4tzZs3t9jp7pprrlFiYqJ+/vnnWr23JB09elSFhYUKDQ01nP/222+1YsUK9e3bVzNmzKgqXJKUkJCgK664Qt999522bdumq666SpLqVFZmzpypgwcP1vr4u+++2yGly1XO/nfYv3+/a4MAAACHo3TBZdLS0jR16tRqY82bN69WutLT0/Xiiy9qxYoVSk9PV1FRUbXjDx8+XKcMf/jDH/SXv/xFHTt21B133KH+/furT58+ioiIsOm4mtj6tXTp0sVwN8CEhAStu4AVyczMTElScHCw4fzZe8smTJhg+Pyus7sxVlRU1Po9a5KWlmaX83iqs/8dzv53AQAA3ovSBYeKjo5WcnKyjhw5YjH3+8u6ysrKLFaRDhw4oJ49e+rEiRO64oorNHjwYIWEhMjf319paWl67733VFxcXKd8EydOVHh4uGbPnq3XX39dM2fOlMlkUv/+/fXSSy/p0ksvvaDjrKnL12JtZapevXoXVIBOnTolyXrp+uGHH+Tn56drr73WcP7QoUOSpNatW9f6PWHd2RJ79r8LAADwXpQuOFSfPn20cuVKrVixQvfcc88FvfbVV1/VsWPHNGfOHIudAOfPn6/33nuv2tjZ1ZmysjLD8+Xl5RkWmNGjR2v06NHKy8vTTz/9pC+//FLvvvuurrnmGqWkpFStZtX2OHt8LY5kdElgeXm5Dh48qMjISDVq1Mhi/ujRo9q4caNatGhhty3h7XVP1+8vg3Rn5/69n/3cU/IDAADbUbpcpVcv+56vefPaHWPv9z2PMWPG6IUXXtCCBQv09NNPq0OHDrV+7b59+yRJt9xyi8Xc6tWrLcaaNm0qScrIyDA8V35+vtVVI6lyRWno0KEaOnSoKioq9O6772rNmjUW71/b4+rytTjC2TJVUFBgMXe2sBYWFqqiosLi8sJ//OMfqqio0H333We3PPa6p8sVm2DYw9n/DkYlFwAAeBdKl6vYaXfACzJ+/Pl3OLSzVq1a6emnn9aUKVM0ZMgQffTRR+rdu7fFcUYrHmefqbVq1SrdcMMNVePLli3Tf//7X4vj27dvr+DgYH311VfKzs6u2mWwqKhIjzzyiGG+lStXasCAARarDdnZ2ZKkoKCgCzrOmgv9WhwhJiZGknHpMplM6ty5s7Zu3ar58+frD3/4Q9XcggULNHPmTLVv317j7fj/H1+/p+vsf4eanpkGAAC8A6ULDjdp0iSZzWY999xz6tOnj7p3766ePXsqLCxMeXl5SktL03fffSdJ6tevX9XrHnjgAc2ZM0e33nqrRowYodjYWO3YsUNLly7Vbbfdpk8++aTa+9SvX1/jx4/Xc889p65du+qmm25SWVmZvv32W8XGxhr+4/amm25S48aN1atXLyUlJclsNuuHH37Qxo0b1b1796pd+mp7nDUX+rU4QlRUlJo0aWL1kr5Jkybp5ptv1tixY7V06VIlJCRo48aN+u6779SmTRstXrxYgYGBDs9pizfffFP//ve/q4rcRRddpKefflrXXXddja8bMGCAVq1aZXV+zZo1evnll7V582YdOXLE8PJQqXLr/JdeekmZmZm66KKLNHPmTF1xxRU1vveJEyckcY8cAAC+gIcjw+FMJpOmTJmiXbt2acKECSorK9NHH32kF198UR999JGOHj2q+++/X5s3b9b7779f9bpLLrlEK1euVO/evbVo0SL961//UkFBgb744guNGzfO8L2mTp2qGTNmKDAwUG+//bYWL16sW265RcuWLTPc7v2FF15Qjx49tGXLFs2ePVtz5sxRaWmpXnzxRa1cubLqNbU9zhpbvhZHuPTSS7Vr1y7DueHDh+vLL7/UpZdeqi+++EIzZ85UTk6Onn/+eW3ZskUtWrRwWs4LFR8frxdffFFbtmzRpk2bNGjQIA0fPly//PKLxbE//PCDvv/++2pj33//vX744QeLY0+ePKlOnTpp1qxZatiwoeF7f/LJJxo/frz+9re/aevWrerdu7eGDBmi9PT0GjPv3LlTknTZZZfV9ssEAAAeymT21BsiXKCgoEAhISHKz8+3ugMc4M6mTZum119/veqySG8WFhamGTNmWNyHlp6erokTJyoyMlIbNmzQZZddpuzsbL366qtKTEy0er7GjRvrjTfesFjpuuyyy3TJJZfoP//5T9VYmzZtNGLECM2YMcPq+caPH6+VK1caFkMAAOD+LqQbsNIF+JA77rhDOTk5Vdu/e6Py8nJ9/PHHOnnypOH9g4mJiVqwYIFCQkK0ZcsWhYSEaMGCBTUWLmtKSkq0efNmDR48uNr44MGD9dNPP9X42i1btuiOO+644PcEAACeh9IF+JDWrVvruuuu05dLine truncated
|
||||
"text/plain": [
|
||||
"<Figure size 1000x800 with 1 Axes>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"original_variance = 1/3\n",
|
||||
"for k in range(N):\n",
|
||||
" nbins = 20\n",
|
||||
" fig = plt.figure(figsize=(10, 8))\n",
|
||||
" sns.histplot(data=data, x=f\"mean_x_0_{k}\", bins=nbins, stat=\"density\", label=\"Data\")\n",
|
||||
" xg = np.linspace(-1.5, 1.5, 100)\n",
|
||||
" yg = 1/np.sqrt(2*np.pi*original_variance/(k+1))*np.exp(-xg**2/2/(original_variance/(k+1)))\n",
|
||||
" plt.plot(xg, yg, label=r\"Gaussian $\\left(\\sigma^2 = \\frac{1}{3 * \"+f\"{k+1}\"+r\"}\\right)$\", ls=\"--\", color=\"r\", lw=4)\n",
|
||||
" plt.gca().set(xlabel=r\"$\\frac{1}{\"+f\"{k+1}\"+r\"}\\sum_{k=0}^{\"+f\"{k+1}\"+r\"} x_{k}$\", ylabel=\"Fraction of occurrences / bin size\", ylim=(0, np.amax(yg)*1.1))\n",
|
||||
" plt.legend(frameon=False, loc=\"upper left\")\n",
|
||||
" plt.savefig(f\"clt_{k}.png\")\n",
|
||||
" plt.show()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 15,
|
||||
"id": "a1fbd00b-e7f2-4eff-90db-e1d44d104fba",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# mean for 1000 variables:\n",
|
||||
"mean_many = sum(random_samples(n_samples) for k in range(1000))/1000"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 17,
|
||||
"id": "20617a26-0fd6-4b00-a096-53b917760a35",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"image/png": "iVBORw0KGgoAAAANSUhEUgAAA1YAAALZCAYAAAC0zbDSAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAACqUElEQVR4nOzdd3hUZd7G8XuSAEmAJIQAoaRCQpXeO6goKIqKgixSfC249rbqrgoqiLqrouuqa0NhFVSUtoCAVGnSQSB0ktAChJJQQtrM+0eWaJgzIUwmc2Ym38915VrzOyczd9Q13DnneY7FZrPZBAAAAABwmp/ZAQAAAADA21GsAAAAAKCUKFYAAAAAUEoUKwAAAAAoJYoVAAAAAJQSxQoAAAAASoliBQAAAAClRLECAAAAgFIKMDuAJ7JarTpy5IiqVq0qi8VidhwAAAAAJrHZbDp79qzq1KkjPz/H16UoVgaOHDmiqKgos2MAAAAA8BAHDx5UvXr1HB6nWBmoWrWqpIK/eSEhISanAQAAAGCWzMxMRUVFFXYERyhWBi7d/hcSEkKxAgAAAHDFJUJsXgEAAAAApUSxAgAAAIBSolgBAAAAQClRrAAAAACglChWAAAAAFBKFCsAAAAAKCWKFQAAAACUEsUKAAAAAEqJYgUAAAAApUSxAgAAAIBSolgBAAAAQClRrAAAAACglChWAAAAAFBKFCsAAAAAKCWKFQAAAACUEsUKgE9bvny5brnlFtWtW1cWi0Vffvml2ZEAAIAPolgB5cD58+dL9fXjx49Xu3btFBISoho1aqh///7atm2bi9KVrXPnzqlZs2Z67733FBQUVGbvU9q/xwAAwLtRrAAf98svv2j69Omleo2lS5fqz3/+s1atWqXFixcrICBA1113nU6dOuWilGWnX79+ev311zVw4ED5+ZXdf/JOnjyp119/vcxeHwAAeLYAswMAKDtHjx7VP/7xj1IXq/nz5xf5fPLkyQoNDdXKlSvVv3//Ur22r4iOjpa/v7++//573XnnnWbHAQAAbsYVK8BJycnJslgsGjFihNlRHBo1apTGjBnj8is1Z8+eldVqVbVq1Vz6ut7u6aef1jvvvKMTJ06YHQUAALgZxQout3v3bj311FNq3bq1wsPDVaFCBYWHh6tDhw565plntGHDBrMjlgs///yzjh07platWrn8tR9//HG1bNlSnTp1cvlre7OAgAANHDhQL774otlRAACAm3ErIFzGZrPp1Vdf1auvviqr1arWrVtr0KBBCg8P19mzZ7V161b985//1Ntvv60PPvhADz/8sNmRS6Vu3bpKSkpSaGio2VEMvf766xo+fLjLX/epp57SihUrtGLFCvn7+7v89b3d0KFDFR8fr5deekn16tUzOw4AAHATihVc5tVXX9WYMWMUFRWlKVOmqEuXLnbnHD9+XBMmTFBGRoYJCV2rQoUKatSokdkxDO3Zs0dLlizRf/7zH5e+7pNPPqmpU6dqyZIlio+Pd+lr+4patWqpbdu2+uKLL/Tyyy+bHQcAALgJtwLCJfbv36+xY8eqYsWKmjdvnmGpkqSaNWvq9ddf11/+8pci8y+//FJ33HGH4uPjFRQUpJCQEHXp0sWwGCxdulQWi0VjxowxfI/Y2FjFxsbazWfNmqVrr71WtWvXVqVKlVSnTh316NFDH374oVPnOVpjdTXfy+Wvk5ycrMGDBysiIkKBgYFq27at/vvf/xp+XXG+/fZb1apVS3Xq1HF4zrlz5/Tqq6+qVatWqlq1qiwWi+HHsWPHJBXc/jdlyhQtXrzYYwulkXPnzmnz5s3avHmzrFarUlNTtXnzZqWmppbZe7Zu3VpTpkwps9cHAACeh2IFl5g4caLy8vI0cOBANW3a9IrnBwQUvVj60EMPKSUlRd27d9cTTzyhwYMHKyUlRffcc49eeumlUuf75JNPdOutt2rHjh3q37+/nn76afXr109ZWVmaOHHiVZ9XHGe/l5SUFLVv317Jycm65557NGjQIG3btk233nqrlixZclXf7+LFi4tdW3X8+HG1a9dOo0ePltVq1ahRo/Too48qMjJSUsHVuPr166tDhw6qVauWHn74YU2cOFHffPONqlWrprS0NKWlpencuXNXlcsM69evV6tWrdSqVStlZWVp9OjRatWqVZleTWrVqpV27typtLS0MnsPAADgWbgVsKy5enH/4MHS448Xf85770lTp5bs9VavLn0mSStXrpQk9e7d26mv37Ztm+rXr19klpOTo759++qNN97QqFGjVLduXafz/fvf/1bFihW1ZcsW1axZs8ix9PT0qz6vOM5+L0uXLtWYMWM0evTowtmQIUN044036u9//7t69epVoveXCsrEfffd5/D4kCFDtHPnTv3lL3/RG2+8IYvFIkl69tlnlZCQoPz8fK1Zs0YRERGSVHi17tprry3yOqNHj3Z45fCSCRMm6MyZMyXO3rJlSw0YMKDE519Jz549ZbPZXPZ6JdGkSRNJ0q+//qpbb73Vre8NAADMQbEqa2vWuPb1SlLUUlJc/75XcOk380aFITk5WV9++WWRWVhYmJ544onCzy8vIpJUsWJFPfzww1q8eLEWLVqkYcOGlSpjQECAKlSoYDe/VB6u9jxHnP1eYmJi7HaTu+GGGxQdHa21a9eW6L0l6dixYzp79qzCwsIMjy9cuFCLFi1S165dNX78+MJSJUlRUVHq1q2bfv75Z23evFnXXXedJJWqmEyYMEEpKSklPn/48OEuLVZmuPT3ft++feYGAQAAbkOxQplLTk7WK6+8UmQWExNTpFilpqbqzTff1KJFi5SamqqsrKwi5x8+fLhUGf70pz/p6aefVpMmTTR48GD16NFDXbp0UY0aNZw6rzjOfi8tW7Y03GUvKipKq6/iyuLRo0clSSEhIYbHL631euKJJwyfb3Vpl0Or1Vri9yxOcnKyS17Hm1z6e3/pnwUAAPB9FCu4RGRkpJKSknTkyBG7Y3+8FSsvL8/uatD+/fvVvn17nT59Wt26dVOfPn0UGhoqf39/JScn66uvvlJ2dnap8j311FOKiIjQhx9+qPfff18TJkyQxWJRjx499Pe//11t27a9qvMcKc334ugKU0BAwFWVnPPnz0tyXKx++eUX+fn56cYbbzQ8fujQIUlSgwYNSvyeKOpSOb30zwIAAPg+ihVcokuXLlqyZIkWLVqke++996q+9p133tHJkyc1ceJEux32pkyZoq+++qrI7NJVlry8PMPXO3PmjGFJGTZsmIYNG6YzZ85o1apVmj59ur744gvdcMMN2rlzZ+FVqZKe54rvpSwZ3b6Xn5+vlJQU1axZU5UrV7Y7fuzYMa1bt05xcXEu207dVWus/njLoicx+vt8aeapmQEAgOtRrMpax46ufb2YmJKd4+r3vYIRI0bojTfe0LRp0/Tiiy+qcePGJf7avXv3SpLuuOMOu2PLli2zm1WrVk2SdPDgQcPXysjIcHj1Ryq4MtSvXz/169dPVqtVX3zxhZYvX273/iU9rzTfS1m4VJgyMzPtjl0qpWfPnpXVarW7FfCtt96S1WrVgw8+6LI8rlpj5e4NKErj0t97o/IKAAB8E8WqrLlo172r8vjjV9450MXq16+vF198UWPGjFHfvn31zTffqHPnznbnGV25uPTMqaVLl6p///6F8/nz5+uzzz6zO79Ro0YKCQnRzJkzdfz48cLd+7KysvTYY48Z5luyZIl69uxpdwXh+PHjkqTg4OCrOs+Rq/1eykLt2rUlGRcri8WiFi1aaNOmTZoyZYr+9Kc/FR6bNm2aJkyYoEaNGulxF/77Ux7XWF36e1/cc8QAAIBv4TlWcJmXX35Zo0eP1sGDB9WlSxe1bdtWf/7zn/Xiiy/qkUce0c0336x69epJkrp37174dX/+859VsWJF3XnnnRo6dKj+8pe/qF+/furbt68GDhxo9z4VKlTQ448/royMDLVq1UqPPPKIRo0apWbNmuns2bOGf5i97bbbFBUVpYEDB+qZZ57R008/rfbt2+u///2v2rRpU7j7XUnPc+Rqv5eyUKtWLVWtWtXh7Xcvv/yyLBaLRo4cqXvuuUd//etfdf311+vOO+9U/fr1NXfuXAUGBrol69X617/+pebNmyskJEQhISHq1KmT5syZY3fehx9+qLi4OAUGBqpNmzb65ZdfyuwcI6dPn5bEOjUAAMoVG+xkZGTYJNkyMjLMjuKVdu7caXviiSdsLVq0sIWGhtoCAgJs1apVs7Vt29b2xBNP2DZs2GD3NStXrrT16tXLFhYWZqtSpYqtS5cutunTp9uWLFlik2QbPXp0kfOtVqtt/Pjxtvj4eFuFChVsUVFRtmeffdZ2/vx5W0xMjC0mJqbI+R999JFtwIABtri4OFtQUJCtWrVqtpYtW9refPNNW2Zm5lWfZ7PZbAcOHLBJsg0fPrxU34uj17mkR48etqv9v2qvXr1sffr0cXh8xowZtk6dOtmCg4NtQUFBthYtWtjGjRtnO3v27FW9j7vNmDHDNnfuXNuePXtsu3btsv31r3+1BQQE2LZs2VJ4ztSpU20BAQG2Tz75xLZjxw7bI488YqtcubItJSXF5ec4MnHiRJsk2/Hjx137NwAAALhdSbuBxWbzooULbpKZmanQ0FBlZGQ43FkN8GRjx47V+++/X3gLoy8LDw/X+PHjC9eFdejQQc2bN9enn35aeE5CQoIGDhyo8ePHu/QcRx5//HEtWbJEW7duddn3CQAAzFHSbsCtgIAPGjx4sE6cOFG4dbovys/P19SpU3Xu3LnC9Xw5OTnasGGD+vTpU+TcPn36aNWqVS49pzgbN27U4MGDnf7eAACA96FYAT6oQYMGuummmzR9+nSzo7jcb7/9pipVqqhSpUoaNWqUpk+frmuuuUaSlJ6ervz8fNWqVavI19SqVUtpaWkuPceRo0ePatOmTRo5cmSpvk8AAOBdKFaAjxo3bpwmT55Line truncated
|
||||
"text/plain": [
|
||||
"<Figure size 1000x800 with 1 Axes>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"nbins = 20\n",
|
||||
"fig = plt.figure(figsize=(10, 8))\n",
|
||||
"sns.histplot(data=mean_many, bins=nbins, stat=\"density\", label=\"Data\")\n",
|
||||
"xg = np.linspace(-0.1, 0.1, 100)\n",
|
||||
"yg = 1/np.sqrt(2*np.pi/3/1000)*np.exp(-xg**2/2/(1/3/1000))\n",
|
||||
"plt.plot(xg, yg, label=r\"Gaussian $\\left(\\sigma^2 = \\frac{1}{3000}\\right)$\", ls=\"--\", color=\"r\", lw=4)\n",
|
||||
"plt.gca().set(xlabel=r\"$\\frac{1}{1000}\\sum_{k=0}^{1000} x_{k}$\", ylabel=\"Fraction of occurrences / bin size\", ylim=(0, np.amax(yg)*1.1))\n",
|
||||
"plt.legend(frameon=False, loc=\"upper left\")\n",
|
||||
"plt.savefig(f\"clt_many.png\")\n",
|
||||
"plt.show()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "4f2b94b4-465d-4cfc-bed9-d70813939592",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.10.6"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
@@ -0,0 +1,667 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "6a764d92",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Gaussian Processes\n",
|
||||
"\n",
|
||||
"If one is ready to assume the data available is such that the variable being fit is Gaussian for any pair of samples taken, one can use fit the data and obtain a reliable uncertainty with a very small set of assumptions.\n",
|
||||
"\n",
|
||||
"For this example, we will use the dataset collected in Mauna Loa showing the amount of CO2 present in the atmosphere. This data may be downloaded from https://gml.noaa.gov/ccgg/trends/data.html and it should already be available in this repository.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "c8131588",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We can take a quick look at the file to see what is inside:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"id": "07ecc0d3",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"# --------------------------------------------------------------------\n",
|
||||
"# USE OF NOAA GML DATA\n",
|
||||
"# \n",
|
||||
"# These data are made freely available to the public and the\n",
|
||||
"# scientific community in the belief that their wide dissemination\n",
|
||||
"# will lead to greater understanding and new scientific insights.\n",
|
||||
"# The availability of these data does not constitute publication\n",
|
||||
"# of the data. NOAA relies on the ethics and integrity of the user to\n",
|
||||
"# ensure that GML receives fair credit for their work. If the data \n",
|
||||
"# are obtained for potential use in a publication or presentation, \n",
|
||||
"# GML should be informed at the outset of the nature of this work. \n",
|
||||
"# If the GML data are essential to the work, or if an important \n",
|
||||
"# result or conclusion depends on the GML data, co-authorship\n",
|
||||
"# may be appropriate. This should be discussed at an early stage in\n",
|
||||
"# the work. Manuscripts using the GML data should be sent to GML\n",
|
||||
"# for review before they are submitted for publication so we can\n",
|
||||
"# ensure that the quality and limitations of the data are accurately\n",
|
||||
"# represented.\n",
|
||||
"# \n",
|
||||
"# Contact: Pieter Tans (303 497 6678; pieter.tans@noaa.gov)\n",
|
||||
"# \n",
|
||||
"# File Creation: Fri Jan 21 05:00:12 2022\n",
|
||||
"# \n",
|
||||
"# RECIPROCITY\n",
|
||||
"# \n",
|
||||
"# Use of these data implies an agreement to reciprocate.\n",
|
||||
"# Laboratories making similar measurements agree to make their\n",
|
||||
"# own data available to the general public and to the scientific\n",
|
||||
"# community in an equally complete and easily accessible form.\n",
|
||||
"# Modelers are encouraged to make available to the community,\n",
|
||||
"# upon request, their own tools used in the interpretation\n",
|
||||
"# of the GML data, namely well documented model code, transport\n",
|
||||
"# fields, and additional information necessary for other\n",
|
||||
"# scientists to repeat the work and to run modified versions.\n",
|
||||
"# Model availability includes collaborative support for new\n",
|
||||
"# users of the models.\n",
|
||||
"# --------------------------------------------------------------------\n",
|
||||
"# \n",
|
||||
"# \n",
|
||||
"# See www.esrl.noaa.gov/gmd/ccgg/trends/ for additional details.\n",
|
||||
"# \n",
|
||||
"# NOTE: DATA FOR THE LAST SEVERAL MONTHS ARE PRELIMINARY, ARE STILL SUBJECT\n",
|
||||
"# TO QUALITY CONTROL PROCEDURES.\n",
|
||||
"# NOTE: The week \"1 yr ago\" is exactly 365 days ago, and thus does not run from\n",
|
||||
"# Sunday through Saturday. 365 also ignores the possibility of a leap year.\n",
|
||||
"# The week \"10 yr ago\" is exactly 10*365 days +3 days (for leap years) ago.\n",
|
||||
"# \n",
|
||||
"# Start of week CO2 molfrac (-999.99 = no data) increase\n",
|
||||
"# (yr, mon, day, decimal) (ppm) #days 1 yr ago 10 yr ago since 1800\n",
|
||||
" 1974 5 19 1974.3795 333.37 5 -999.99 -999.99 50.40\n",
|
||||
" 1974 5 26 1974.3986 332.95 6 -999.99 -999.99 50.06\n",
|
||||
" 1974 6 2 1974.4178 332.35 5 -999.99 -999.99 49.60\n",
|
||||
" 1974 6 9 1974.4370 332.20 7 -999.99 -999.99 49.65\n",
|
||||
" 1974 6 16 1974.4562 332.37 7 -999.99 -999.99 50.06\n",
|
||||
" 1974 6 23 1974.4753 331.73 5 -999.99 -999.99 49.72\n",
|
||||
" 1974 6 30 1974.4945 331.68 6 -999.99 -999.99 50.02\n",
|
||||
" 1974 7 7 1974.5137 331.46 6 -999.99 -999.99 50.20\n",
|
||||
" 1974 7 14 1974.5329 330.83 5 -999.99 -999.99 50.01\n",
|
||||
" 1974 7 21 1974.5521 330.76 7 -999.99 -999.99 50.41\n",
|
||||
" 1974 7 28 1974.5712 329.80 4 -999.99 -999.99 49.97\n",
|
||||
" 1974 8 4 1974.5904 329.85 5 -999.99 -999.99 50.54\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"with open(\"co2_weekly_mlo.txt\", \"r\") as f:\n",
|
||||
" for i, line in enumerate(f.readlines()):\n",
|
||||
" if i > 60: # let's print only the first 60 lines only\n",
|
||||
" break\n",
|
||||
" print(line[:-1])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "fce4d8e8",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We start by loading the necessary Python modules."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"id": "300cf8d3",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import matplotlib.pyplot as plt\n",
|
||||
"\n",
|
||||
"import pandas as pd\n",
|
||||
"import numpy as np\n",
|
||||
"from sklearn.gaussian_process import GaussianProcessRegressor\n",
|
||||
"from sklearn.gaussian_process.kernels import RBF, ExpSineSquared, WhiteKernel, RationalQuadratic"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "0ecd6a69",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Let us now load the data downloaded using pandas. If you are not familiar with Pandas, take a look at the documentation for a quick introduction at https://pandas.pydata.org/docs/user_guide/10min.html\n",
|
||||
"\n",
|
||||
"We can easily parse the data file using Pandas' read_csv function. It would require some coding to do it manually. As Pandas does not detect the column names, we have to specify it manually in the names parameter below."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"id": "4959a292",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"data = pd.read_csv(\"co2_weekly_mlo.txt\",\n",
|
||||
" delim_whitespace=True,\n",
|
||||
" comment='#',\n",
|
||||
" header=None,\n",
|
||||
" names=[\"year\", \"month\", \"day\", \"decimal\",\n",
|
||||
" \"co2\",\n",
|
||||
" \"ndays\",\n",
|
||||
" \"last_year\", \"last_decade\",\n",
|
||||
" \"increase\"])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "d8295e8a",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Let's print out the dataset read first."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"id": "024fb65a",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/html": [
|
||||
"<div>\n",
|
||||
"<style scoped>\n",
|
||||
" .dataframe tbody tr th:only-of-type {\n",
|
||||
" vertical-align: middle;\n",
|
||||
" }\n",
|
||||
"\n",
|
||||
" .dataframe tbody tr th {\n",
|
||||
" vertical-align: top;\n",
|
||||
" }\n",
|
||||
"\n",
|
||||
" .dataframe thead th {\n",
|
||||
" text-align: right;\n",
|
||||
" }\n",
|
||||
"</style>\n",
|
||||
"<table border=\"1\" class=\"dataframe\">\n",
|
||||
" <thead>\n",
|
||||
" <tr style=\"text-align: right;\">\n",
|
||||
" <th></th>\n",
|
||||
" <th>year</th>\n",
|
||||
" <th>month</th>\n",
|
||||
" <th>day</th>\n",
|
||||
" <th>decimal</th>\n",
|
||||
" <th>co2</th>\n",
|
||||
" <th>ndays</th>\n",
|
||||
" <th>last_year</th>\n",
|
||||
" <th>last_decade</th>\n",
|
||||
" <th>increase</th>\n",
|
||||
" </tr>\n",
|
||||
" </thead>\n",
|
||||
" <tbody>\n",
|
||||
" <tr>\n",
|
||||
" <th>0</th>\n",
|
||||
" <td>1974</td>\n",
|
||||
" <td>5</td>\n",
|
||||
" <td>19</td>\n",
|
||||
" <td>1974.3795</td>\n",
|
||||
" <td>333.37</td>\n",
|
||||
" <td>5</td>\n",
|
||||
" <td>-999.99</td>\n",
|
||||
" <td>-999.99</td>\n",
|
||||
" <td>50.40</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>1</th>\n",
|
||||
" <td>1974</td>\n",
|
||||
" <td>5</td>\n",
|
||||
" <td>26</td>\n",
|
||||
" <td>1974.3986</td>\n",
|
||||
" <td>332.95</td>\n",
|
||||
" <td>6</td>\n",
|
||||
" <td>-999.99</td>\n",
|
||||
" <td>-999.99</td>\n",
|
||||
" <td>50.06</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>2</th>\n",
|
||||
" <td>1974</td>\n",
|
||||
" <td>6</td>\n",
|
||||
" <td>2</td>\n",
|
||||
" <td>1974.4178</td>\n",
|
||||
" <td>332.35</td>\n",
|
||||
" <td>5</td>\n",
|
||||
" <td>-999.99</td>\n",
|
||||
" <td>-999.99</td>\n",
|
||||
" <td>49.60</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>3</th>\n",
|
||||
" <td>1974</td>\n",
|
||||
" <td>6</td>\n",
|
||||
" <td>9</td>\n",
|
||||
" <td>1974.4370</td>\n",
|
||||
" <td>332.20</td>\n",
|
||||
" <td>7</td>\n",
|
||||
" <td>-999.99</td>\n",
|
||||
" <td>-999.99</td>\n",
|
||||
" <td>49.65</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>4</th>\n",
|
||||
" <td>1974</td>\n",
|
||||
" <td>6</td>\n",
|
||||
" <td>16</td>\n",
|
||||
" <td>1974.4562</td>\n",
|
||||
" <td>332.37</td>\n",
|
||||
" <td>7</td>\n",
|
||||
" <td>-999.99</td>\n",
|
||||
" <td>-999.99</td>\n",
|
||||
" <td>50.06</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>...</th>\n",
|
||||
" <td>...</td>\n",
|
||||
" <td>...</td>\n",
|
||||
" <td>...</td>\n",
|
||||
" <td>...</td>\n",
|
||||
" <td>...</td>\n",
|
||||
" <td>...</td>\n",
|
||||
" <td>...</td>\n",
|
||||
" <td>...</td>\n",
|
||||
" <td>...</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>2482</th>\n",
|
||||
" <td>2021</td>\n",
|
||||
" <td>12</td>\n",
|
||||
" <td>12</td>\n",
|
||||
" <td>2021.9466</td>\n",
|
||||
" <td>416.45</td>\n",
|
||||
" <td>7</td>\n",
|
||||
" <td>414.00</td>\n",
|
||||
" <td>392.19</td>\n",
|
||||
" <td>137.16</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>2483</th>\n",
|
||||
" <td>2021</td>\n",
|
||||
" <td>12</td>\n",
|
||||
" <td>19</td>\n",
|
||||
" <td>2021.9658</td>\n",
|
||||
" <td>417.49</td>\n",
|
||||
" <td>6</td>\n",
|
||||
" <td>414.86</td>\n",
|
||||
" <td>392.54</td>\n",
|
||||
" <td>137.93</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>2484</th>\n",
|
||||
" <td>2021</td>\n",
|
||||
" <td>12</td>\n",
|
||||
" <td>26</td>\n",
|
||||
" <td>2021.9849</td>\n",
|
||||
" <td>417.46</td>\n",
|
||||
" <td>7</td>\n",
|
||||
" <td>415.32</td>\n",
|
||||
" <td>392.43</td>\n",
|
||||
" <td>137.66</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>2485</th>\n",
|
||||
" <td>2022</td>\n",
|
||||
" <td>1</td>\n",
|
||||
" <td>2</td>\n",
|
||||
" <td>2022.0041</td>\n",
|
||||
" <td>417.43</td>\n",
|
||||
" <td>6</td>\n",
|
||||
" <td>415.44</td>\n",
|
||||
" <td>393.18</td>\n",
|
||||
" <td>137.40</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>2486</th>\n",
|
||||
" <td>2022</td>\n",
|
||||
" <td>1</td>\n",
|
||||
" <td>9</td>\n",
|
||||
" <td>2022.0233</td>\n",
|
||||
" <td>418.33</td>\n",
|
||||
" <td>7</td>\n",
|
||||
" <td>415.06</td>\n",
|
||||
" <td>393.99</td>\n",
|
||||
" <td>138.10</td>\n",
|
||||
" </tr>\n",
|
||||
" </tbody>\n",
|
||||
"</table>\n",
|
||||
"<p>2487 rows × 9 columns</p>\n",
|
||||
"</div>"
|
||||
],
|
||||
"text/plain": [
|
||||
" year month day decimal co2 ndays last_year last_decade \n",
|
||||
"0 1974 5 19 1974.3795 333.37 5 -999.99 -999.99 \\\n",
|
||||
"1 1974 5 26 1974.3986 332.95 6 -999.99 -999.99 \n",
|
||||
"2 1974 6 2 1974.4178 332.35 5 -999.99 -999.99 \n",
|
||||
"3 1974 6 9 1974.4370 332.20 7 -999.99 -999.99 \n",
|
||||
"4 1974 6 16 1974.4562 332.37 7 -999.99 -999.99 \n",
|
||||
"... ... ... ... ... ... ... ... ... \n",
|
||||
"2482 2021 12 12 2021.9466 416.45 7 414.00 392.19 \n",
|
||||
"2483 2021 12 19 2021.9658 417.49 6 414.86 392.54 \n",
|
||||
"2484 2021 12 26 2021.9849 417.46 7 415.32 392.43 \n",
|
||||
"2485 2022 1 2 2022.0041 417.43 6 415.44 393.18 \n",
|
||||
"2486 2022 1 9 2022.0233 418.33 7 415.06 393.99 \n",
|
||||
"\n",
|
||||
" increase \n",
|
||||
"0 50.40 \n",
|
||||
"1 50.06 \n",
|
||||
"2 49.60 \n",
|
||||
"3 49.65 \n",
|
||||
"4 50.06 \n",
|
||||
"... ... \n",
|
||||
"2482 137.16 \n",
|
||||
"2483 137.93 \n",
|
||||
"2484 137.66 \n",
|
||||
"2485 137.40 \n",
|
||||
"2486 138.10 \n",
|
||||
"\n",
|
||||
"[2487 rows x 9 columns]"
|
||||
]
|
||||
},
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"data"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "1c178424",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We can plot this fairly easily using Matplotlib."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"id": "e63b38c5",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAtAAAAK/CAYAAABA9foTAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAByMklEQVR4nO3dd3gU1eLG8XfTE1IIpFFC6L2DQkAUFBO9gKKCikoTERBUwIvIVQH1KioqogK2K3AVRFHxWhAITcFEei+hF4GEnkCA1Pn9kV/WrJsNGVI2JN/P8+QhO3N25syeZPNy9sw5FsMwDAEAAAAoEBdnVwAAAAC4nhCgAQAAABMI0AAAAIAJBGgAAADABAI0AAAAYAIBGgAAADCBAA0AAACYQIAGAAAATCBAAwAAACYQoAEAAAATCNAAAACACQRoAEVi4sSJslgsOn36dImcb9asWbJYLDp06FCJnA9lkzN/jvgZBq5fBGigmEyfPl0Wi0Xt2rXLc39sbKwmTpyo8+fPl2zF4ND+/fs1ZMgQ1a5dW15eXvL391fHjh01depUXb582absjh079Mgjj6hatWry9PRU1apV9fDDD2vHjh1Oqv31rbh/H8rL71tOKLdYLFq9erXdfsMwFB4eLovFou7duzuhhkUv55rXr1/v7KqgHCFAA8Vkzpw5qlmzptauXat9+/bZ7Y+NjdVLL71U5v+gF5e+ffvq8uXLioiIKJLj/fzzz2rWrJm+/vpr9ejRQ++//74mTZqkGjVqaMyYMXr66aetZb/77ju1bt1ay5Yt08CBAzV9+nQNGjRIK1asUOvWrbVgwYIiqVN5Uty/D46OX9Q/R2YU57m9vLw0d+5cu+2//vqr/vzzT3l6ehb5OYHyxM3ZFQDKooMHDyo2NlbfffedhgwZojlz5mjChAnOrlaZkJKSogoVKsjV1VWurq5FcsyDBw/qwQcfVEREhJYvX64qVapY9w0fPlz79u3Tzz//LCm7l7pv376qXbu2fvvtNwUHB1vLPv300+rUqZP69u2rrVu3qnbt2kVSP9jL+TkorKL8OSpN5/7HP/6h+fPn67333pOb219/6ufOnas2bdqU2FAroKyiBxooBnPmzFFgYKC6deumXr16ac6cOTb7J06cqDFjxkiSatWqZf3I9dChQ9axxHv27NEjjzyigIAABQcH68UXX5RhGDp69Kjuvvtu+fv7KywsTG+//Xaeddi0aZPuvPNO+fv7y9fXV7fddpv++OMPmzIXLlzQyJEjVbNmTXl6eiokJES33367Nm7caK2nxWLR7t27df/998vf31+VK1fW008/rStXruR53vPnz2vAgAGqWLGiAgICNHDgQF26dMmu3LFjx/Too48qNDRUnp6eatKkiT777DO718lisWjnzp166KGHFBgYqJtuuklS3uNHjx07pkGDBqlq1ary9PRUrVq1NGzYMKWlpeXTWtKbb76pixcv6j//+Y9NeM5Rt25daw/05MmTdenSJX388cc24VmSgoKC9NFHHyklJUVvvvlmvucsqIJcU0HaOue13LdvX4Hb52rnNdOG+Z03v9+H3MfI6+fg8OHDeuKJJ9SgQQN5e3urcuXK6t27t83PRX7HdzQOuThe07/L69yFPWaOPn366MyZM4qJibFuS0tL0zfffKOHHnrIrnxBXkdJGjBggGrWrGn3/Jx657XtatdS0HMXVkHatKTqgusfPdBAMZgzZ47uvfdeeXh4qE+fPpoxY4bWrVunG264QZJ07733as+ePfryyy81ZcoUBQUFSZJNIHvggQfUqFEjvf766/r555/173//W5UqVdJHH32kW2+9VW+88YbmzJmjf/7zn7rhhht08803W5+7Y8cOderUSf7+/nr22Wfl7u6ujz76SJ07d9avv/5qHZc9dOhQffPNNxoxYoQaN26sM2fOaPXq1dq1a5dat25tPd7999+vmjVratKkSfrjjz/03nvv6dy5c/rvf/9rd+3333+/atWqpUmTJmnjxo369NNPFRISojfeeMNaJjExUe3bt5fFYtGIESMUHBysX375RYMGDVJycrJGjhxpc8zevXurXr16eu2112QYRp6v+fHjx3XjjTfq/Pnzevzxx9WwYUMdO3ZM33zzjS5duiQPDw+H7fXjjz+qdu3a6tChg8MyucvWrFlTnTp1ynP/zTffrJo1a1p7rAujINdU0LbOUZD2Kch5zbZhfuctyO+DlPfPwbp16xQbG6sHH3xQ1atX16FDhzRjxgx17txZO3fulI+PT4GPn6M4XlOzCnvMmjVrKjIyUl9++aXuvPNOSdIvv/yipKQkPfjgg3rvvfdsyhfkdSyuaynOc+coaJuWRF1QRhgAitT69esNSUZMTIxhGIaRlZVlVK9e3Xj66adtyk2ePNmQZBw8eNBm+4QJEwxJxuOPP27dlpGRYVSvXt2wWCzG66+/bt1+7tw5w9vb2+jfv7/NMXr27Gl4eHgY+/fvt247fvy44efnZ9x8883WbQEBAcbw4cMdXktOXe666y6b7U888YQhydiyZYtd2UcffdSm7D333GNUrlzZZtugQYOMKlWqGKdPn7bZ/uCDDxoBAQHGpUuXbI7Zp08fu7rNnDnT5vXr16+f4eLiYqxbt86ubFZWlsNrTEpKMiQZd999t8MyOc6fP1+gsnfddZchyUhOTr7qMfNTkGsqaFubaZ+CnNdsG17tvI5+H3IfI6+fg5zz5BYXF2dIMv773/9e9fh//zkyjOJ5TfOS17mL6pjr1q0zPvjgA8PPz8/6GvXu3dvo0qWLYRiGERERYXTr1s36vIK+jv379zciIiLsyubUO69tV7uWgp67INfsSEHbtLB1QfnBEA6giM2ZM0ehoaHq0qWLJMliseiBBx7QvHnzlJmZWeDjPPbYY9bvXV1d1bZtWxmGoUGDBlm3V6xYUQ0aNNCBAwes2zIzM7VkyRL17NnTZgxulSpV9NBDD2n16tVKTk62Pn/NmjU6fvx4vnUZPny4zeMnn3xSkrRw4UK7skOHDrV53KlTJ505c8Z6TsMw9O2336pHjx4yDEOnT5+2fkVHRyspKck6hMTRMf8uKytL33//vXr06KG2bdva7f/7R8u55dTLz88v33NI2UNeClI2Z3/Osa9FQa7JTFvnuFr7FOS8RdGGfz9vQeT1c+Dt7W39Pj09XWfOnFHdunVVsWJFuzoURHG8pteiKI55//336/Lly/rpp5904cIF/fTTT3kO35CK/nXM7WrXUpznlsy1aXHXBWUHARooQpmZmZo3b566dOmigwcPat++fdq3b5/atWunxMRELVu2rMDHqlGjhs3jgIAAeXl5WT9+zr393Llz1senTp3SpUuX1KBBA7tjNmrUSFlZWTp69Kik7LG/27dvV3h4uG688UZNnDjRJoznqFevns3jOnXqyMXFJc9xgX+vd2BgoCRZ63jq1CmdP3/eOoY499fAgQMlSSdPnrQ5Rq1atezOk9upU6eUnJyspk2b5lsuL/7+/pL+Csf5yQnGVyubO2inpqbq0UcfVY0aNeTv76/27dsrLi7uqucqyDWZaescBWmfgpzXbBte7bwFkdfPweXLlzV+/HiFh4fL09NTQUFBCg4O1vnz55WUlFTgY+cojtf0WhTFMYODg9W1a1fNnTtX3333nTIzM9WrV688yxb165jb1a6lOM8tmWvT4q4Lyg7GQANFaPny5Tpx4oTmzZunefPm2e2fM2eOoqKiCnSsvO7Od3THvuFgXPDV3H///erUqZMWLFigJUuWaPLkyXrjjTf03XffWcdN5iW/Ht2r1TErK0uS9Mgjj6h///55lm3evLnN49y9QkXN399fVatW1fbt269aNiAgQFWqVNHWrVvzLbd161ZVq1ZN/v7+SklJUc2aNbV69WpVr17dOk3eoUOH5OvrW1SXUWBF8TN0LW1YFOfN6+fgySef1MyZMzVy5EhFRkYqICBAFotFDz74oLWexa2ofy+L8pgPPfSQBg8erISEBN15552qWLFinuUK+jo6+t3P79O1q11LaWjDHKWpLijdCNBAEZozZ45CQkI0bdo0u33fffedFixYoA8//FDe3t75htDCCA4Olo+Pj+Lj4+327d69Wy4uLgoPD7duq1Klip544gk98cQTOnnypFq3bq1XX33VJkDv3bvXpvdv3759ysrKyvNu/ILUz8/PT5mZmeratavp5zs6pr+/f4FCcF66d++ujz/+WHFxcYqMjLxq2U8++USrV6+2zgSR26pVq3To0CENGTJEklShQgWNHz/euv/BBx/U6NGjFR8frzZt2hTqmsy2dUEU9LxF3YbX+vvwzTffqH///jaz0Vy5csVuvueCHr84XlNnuueeezRkyBD98ccf+uqrrxyWK+jrGBgYmOdc3YcPH77mOhb03NfKTJsWd11QdjCEAygily9f1nfffafu3burV69edl8jRozQhQsX9MMPP0iSdQ7bon5jdnV1VVRUlP73v//ZDLFITEzU3LlzddNNN8nf31+ZmZl2H0mGhISoatWqSk1Ntdn+9/8QvP/++5KUby91fvW777779O233+YZ0k6dOmX6mC4uLurZs6d+/PHHPFcju1qv3bPPPqsKFSroscceU2Jiot3+/fv3a+rUqZKkMWPGyNvbW0OGDNGZM2dsyp09e1ZDhw6Vj4+Pddq0v9u7d6/Onj2runXrFvqaCtrWZhT0vEXdhtf6++Dq6mrXvu+//75dj2hBj18cr6kz+fr6asaMGZo4caJ69OjhsFxBX8c6deooKSnJ5lOYEydOFGrxoIKeuzDHL2ibFnddUHbQAw0UkR9++EEXLlzQXXfdlef+9u3bKzg4WHPmzNEDDzxg7X18/vnn9eCDD8rd3T3fP3Bm/Pvf/1ZMTIxuuukmPfHEE3Jzc9NHH32k1NRU6/zEFy5cUPXq1dWrVy+1aNFCvr6+Wrp0qdatW2c3t/TBgwd111136Y477lBcXJyLine truncated
|
||||
"text/plain": [
|
||||
"<Figure size 800x800 with 1 Axes>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"fig, ax = plt.subplots(figsize=(8, 8))\n",
|
||||
"data.plot(x=\"decimal\", y=\"co2\", legend=False, ax=ax)\n",
|
||||
"ax.set(xlabel=\"Time [years]\", ylabel=r\"CO$_2$ [ppm]\", title=\"Atmospheric CO$_2$ concentration in Mauna Loa\")\n",
|
||||
"plt.show()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "efd249fe",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"The spikes are related to missing data. We can clean those up and plot again."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"id": "0837b3ff",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"data = data[data.co2 > -999]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"id": "333581b5",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"image/png": "iVBORw0KGgoAAAANSUhEUgAAArsAAAK/CAYAAABp8gWUAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAADHGUlEQVR4nOzdd3gc5fU24Gf7SiutenGRe8cFbINx6JhuSggldEgI1RDKF5LwI4WQAiEJCb0kBFIMJBAgQOi9uGBswL13q9eVtpf5/ph5Z2d2Z1YrW7JX0nNfly9mdt+dGVkuh+PznmORJEkCEREREdEAZD3QD0BERERE1FcY7BIRERHRgMVgl4iIiIgGLAa7RERERDRgMdglIiIiogGLwS4RERERDVgMdomIiIhowGKwS0REREQDFoNdIiIiIhqwGOwSERER0YDFYJeIiIiIBiwGu0SD1J133gmLxYLm5ub9cr+nn34aFosF27dv3y/3o4HpQP464q9hov6JwS6R4pFHHoHFYsGcOXMM31+0aBHuvPNOtLe3798HI1NbtmzBNddcgzFjxsDtdsPr9eKII47A/fffj2AwqFu7Zs0aXHLJJRg2bBhcLheGDh2Kiy++GGvWrDlAT9+/9fXvh8Hy+00E0BaLBZ9++mna+5IkoaamBhaLBaeffvoBeMLeJ77mL7744kA/Cg0SDHaJFAsXLsSoUaPw+eefY/PmzWnvL1q0CL/4xS8G/F++feXSSy9FMBjEyJEje+V6//vf/zBt2jT8+9//xhlnnIEHH3wQd999N0aMGIHbbrsNN910k7r2xRdfxMyZM/Hee+/hO9/5Dh555BFceeWV+OCDDzBz5ky89NJLvfJMg0lf/34wu35v/zrqib68t9vtxjPPPJP2+kcffYTdu3fD5XL1+j2JBgv7gX4Aolywbds2LFq0CC+++CKuueYaLFy4ED//+c8P9GMNCH6/Hx6PBzabDTabrVeuuW3bNlxwwQUYOXIk3n//fQwZMkR9b8GCBdi8eTP+97//AZCzv5deeinGjBmDjz/+GBUVFeram266CUcddRQuvfRSrFy5EmPGjOmV56N04tfBvurNX0e5dO/TTjsNzz//PB544AHY7cm/mp955hnMmjVrv5UbEQ1EzOwSQc7qlpSUYP78+Tj33HOxcOFC3ft33nknbrvtNgDA6NGj1X923L59u1r7unHjRlxyySUoKipCRUUFfvrTn0KSJOzatQtnnXUWvF4vqqur8Yc//MHwGb788kuceuqp8Hq9KCgowLx587BkyRLdms7OTtx8880YNWoUXC4XKisrceKJJ2LFihXqc1osFqxfvx7nn38+vF4vysrKcNNNNyEUChnet729HVdccQWKi4tRVFSE73znOwgEAmnr9uzZg+9+97uoqqqCy+XCQQcdhL/+9a9pP08WiwVr167FRRddhJKSEhx55JEAjOsd9+zZgyuvvBJDhw6Fy+XC6NGjcd111yESiWT4bgH33nsvurq68OSTT+oCXWHcuHFqZvd3v/sdAoEAnnjiCV2gCwDl5eV4/PHH4ff7ce+992a8Z7ay+Zqy+V6Ln8vNmzdn/f3p7r49+R5mum+m3w/aaxj9OtixYweuv/56TJw4EXl5eSgrK8N5552n+3WR6fpmdbN98XOayuje+3pN4cILL0RLSwveeecd9bVIJIIXXngBF110Udr6bH4eAeCKK67AqFGj0j4vntvote6+lmzvva+y+Z7ur2eh/o2ZXSLIwe63vvUtOJ1OXHjhhXj00UexbNkyHHrooQCAb33rW9i4cSOeffZZ/PGPf0R5eTkA6IKnb3/725g8eTLuuece/O9//8OvfvUrlJaW4vHHH8fxxx+P3/72t1i4cCF+8IMf4NBDD8XRRx+tfnbNmjU46qij4PV68cMf/hAOhwOPP/44jj32WHz00UdqHfG1116LF154ATfccAOmTJmClpYWfPrpp1i3bh1mzpypXu/888/HqFGjcPfdd2PJkiV44IEH0NbWhr///e9pX/v555+P0aNH4+6778aKFSvwl7/8BZWVlfjtb3+rrmloaMDhhx8Oi8WCG264ARUVFXjjjTdw5ZVXwufz4eabb9Zd87zzzsP48ePxm9/8BpIkGf6c19bW4rDDDkN7ezuuvvpqTJo0CXv27MELL7yAQCAAp9Np+v169dVXMWbMGHzjG98wXaNdO2rUKBx11FGG7x999NEYNWqUmgneF9l8Tdl+r4Vsvj/Z3Len38NM983m9wNg/Otg2bJlWLRoES644AIMHz4c27dvx6OPPopjjz0Wa9euRX5+ftbXF/ri57Sn9vWao0aNwty5c/Hss8/i1FNPBQC88cYb6OjowAUXXIAHHnhAtz6bn8e++lr68t5Ctt/T/fEsNABIRIPcF198IQGQ3nnnHUmSJCmRSEjDhw+XbrrpJt263/3udxIAadu2bbrXf/7zn0sApKuvvlp9LRaLScOHD5csFot0zz33qK+3tbVJeXl50uWXX667xje/+U3J6XRKW7ZsUV+rra2VCgsLpaOPPlp9raioSFqwYIHp1yKe5cwzz9S9fv3110sApK+//jpt7Xe/+13d2rPPPlsqKyvTvXbllVdKQ4YMkZqbm3WvX3DBBVJRUZEUCAR017zwwgvTnu2pp57S/fxddtllktVqlZYtW5a2NpFImH6NHR0dEgDprLPOMl0jtLe3Z7X2zDPPlABIPp+v22tmks3XlO33uiffn2zu29PvYXf3Nfv9oL2G0a8DcR+txYsXSwCkv//9791eP/XXkST1zc+pEaN799Y1ly1bJj300ENSYWGh+nN03nnnSccdd5wkSZI0cuRIaf78+ernsv15vPzyy6WRI0emrRXPbfRad19LtvfO5ms2k+33dF+fhQYHljHQoLdw4UJUVVXhuOOOAwBYLBZ8+9vfxnPPPYd4PJ71db73ve+pxzabDbNnz4YkSbjyyivV14uLizFx4kRs3bpVfS0ej+Ptt9/GN7/5TV3N6JAhQ3DRRRfh008/hc/nUz+/dOlS1NbWZnyWBQsW6M5vvPFGAMDrr7+etvbaa6/VnR911FFoaWlR7ylJEv7zn//gjDPOgCRJaG5uVn+cfPLJ6OjoUMsozK6ZKpFI4OWXX8YZZ5yB2bNnp72f+s+rWuK5CgsLM94DkMs+slkr3hfX3hvZfE09+V4L3X1/srlvb3wPU++bDaNfB3l5eepxNBpFS0sLxo0bh+Li4rRnyEZf/Jzujd645vnnn49gMIjXXnsNnZ2deO211wxLGIDe/3nU6u5r6ct7Az37nvb1s9DAwGCXBrV4PI7nnnsOxx13HLZt24bNmzdj8+bNmDNnDhoaGvDee+9lfa0RI0bozouKiuB2u9V/gtW+3tbWpp43NTUhEAhg4sSJadecPHkyEokEdu3aBUCuVV29ejVqampw2GGH4c4779QFzsL48eN152PHjoXVajWsY0t97pKSEgBQn7GpqQnt7e1qzav2x3e+8x0AQGNjo+4ao0ePTruPVlNTE3w+H6ZOnZpxnRGv1wsgGchmIoLY7tZqg+JwOIzvfve7GDFiBLxeLw4//HAsXry423tl8zX15HstZPP9yea+Pf0ednffbBj9OggGg/jZz36GmpoauFwulJeXo6KiAu3t7ejo6Mj62kJf/Jzujd64ZkVFBU444QQ888wzePHFFxGPx3Huuecaru3tn0et7r6Wvrw30LPvaV8/Cw0MrNmlQe39999HXV0dnnvuOTz33HNp7y9cuBAnnXRSVtcy2qVttnNbMqlj7c7555+Po446Ci+99BLefvtt/O53v8Nvf/tbvPjii2qdn5FMmdLunjGRSAAALrnkElx++eWGa6dPn64712ZbepvX68XQoUOxevXqbtcWFRVhyJAhWLlyZcZ1K1euxLBhw+D1euH3+zFq1Ch8+umnGD58uNrabPv27SgoKOitLyNrvfFraG++h71xX6NfBzfeeCOeeuop3HzzzZg7dy6KiopgsVhwwQUXqM/Z13r792VvXvOiiy7CVVddhfr6epx66qkoLi42XJftz6PZ7/1M/2rV3deSC99DIZeehXIXg10a1BYuXIjKyko8/PDDae+9+OKLeOmll/DYY48hLy8vY8C4LyoqKpCfn48NGzakvbd+/XpYrVbU1NSorw0ZMgTXX389rr/+ejQ2NmLmzJn49a9/rQt2N23apMuqbd68GYlEwnBXdjbPV1hYiHg8jhNOOKHHnze7ptfrzSpgNXL66afjiSeewOLFizF37txu1/75z3/Gp59+qnYE0Prkk0+wfft2XHPNNQAAj8eDn/3sZ+r7F1xwAW699VZs2LABs2bN2qevqaff62xke9/e/h7u7e+HF154AZdffrmuK0koFErrp5vt9fvi5/RAOvvss3HNNddgyZIl+Ne//mW6Ltufx5KSEsNeyDt27NjrZ8z23nurJ9/Tvn4WGhhYxkCDVjAYxIsvvojTTz8d5557btqPG264AZ2dnXjllVcAQO0R2tt/iNpsNpx00kn473//qyszaGhowDPPPIMjjzwSXq8X8Xg87Z/lKisrMXToUITDYd3rqcH7gw8+CAAZs7+Znu+cc87Bf/7zH8OAqqmpqcfXtFqt+OY3v4lXX33VcIpSd9mwH/7wh/B4PPje976HhoaGtPe3bNmC+++/HwBw2223IS8vD9dccw1aWlp061pbW3HttdciPz9fbXWVatOmTWhtbcW4ceP2+WvK9nvdE9net7e/h3v7+8Fms6V9fx988MG0TGO21++Ln9MDqaCgAI8++ijuvPNOnHHGGabrsv15HDt2LDo6OnT/ulFXV7dPg1Syvfe+XD/b72lfPwsNDMzs0qD1yiuvoLOzE2eeeabh+4cffjgqKiqwcOFCfPvb31azenfccQcuuOACOByOjH8Z9cSvfvUrvPPOOzjyyCNx/fXXw2634/HHH0c4HFb7v3Z2dmL48OE499xzMWPGDBQLine truncated
|
||||
"text/plain": [
|
||||
"<Figure size 800x800 with 1 Axes>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"fig, ax = plt.subplots(figsize=(8, 8))\n",
|
||||
"data.plot(x=\"decimal\", y=\"co2\", legend=False, ax=ax)\n",
|
||||
"ax.set(xlabel=\"Time [years]\", ylabel=r\"CO$_2$ [ppm]\", title=\"Atmospheric CO$_2$ concentration in Mauna Loa\")\n",
|
||||
"plt.show()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "c62fd7f3",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"That's better.\n",
|
||||
"\n",
|
||||
"We can now try to model this behaviour with Gaussian Processes. The key parameter we must choose is the covariance matrix analytic form, which establishes how correlated two points in the time axis are.\n",
|
||||
"\n",
|
||||
"One simple assumption would be that two values of the CO2 concentration are highly correlated they happened at close by times. We use a Gaussian-based model (named here \"Radial Basis Function\", as this does not refer to a probability distribution) to establish that if two points are one year apart they are likely to be highly correlated.\n",
|
||||
"\n",
|
||||
"Additionally, we assume that there is a correlation between points within certain period, since we know that data from the same season in an year is expected to be correlated to data if that same season in the year (ie: CO$_2$ emissions have a rough periodicity of one year due to the season changes).\n",
|
||||
"\n",
|
||||
"Another term of the covariance matrix must be related to the random noise appearing in the data.\n",
|
||||
"\n",
|
||||
"All those assumptions can be motivated by an analysis of the covariance matrix of the data. That is, we can take the covariance matrix of all the data points and analyse it to establish whether our assumptions here are valid or not. A very long and detailed explanation on this point can be found in Bishop (2006), which has a very pedagogical chapter on Gaussian Processes. Online resources can be found on: http://www.gaussianprocess.org/\n",
|
||||
"\n",
|
||||
"Note that all of those are very strong assumptions, but they are made directly on the covariance matrix of the fit function. The actual numbers used below are taken from an example, but they will be optimized in the fit below (unless explicitly asked not to).\n",
|
||||
"\n",
|
||||
"Note: this notebook is a simplified example with extra explanations, with lots of material taken from https://scikit-learn.org/stable/auto_examples/gaussian_process/plot_gpr_co2.html -- take a look at the original example for more details.\n",
|
||||
"\n",
|
||||
"We also assume the data within an year from each other is highly correlated and should show a periodicity of 1 year."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"id": "a31d7eca",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Kernel with parameters given in GPML book\n",
|
||||
"\n",
|
||||
"# these hyper parameters are optimised in the GP fit\n",
|
||||
"# we provide here only a starting value\n",
|
||||
"\n",
|
||||
"# Kernel with optimized parameters\n",
|
||||
"k1 = 50.0 ** 2 * RBF(length_scale=50.0) # long term smooth rising trend\n",
|
||||
"k2 = (\n",
|
||||
" 2.0 ** 2\n",
|
||||
" * RBF(length_scale=100.0)\n",
|
||||
" * ExpSineSquared(length_scale=1.0, periodicity=1.0, periodicity_bounds=\"fixed\")\n",
|
||||
") # seasonal component\n",
|
||||
"# medium term irregularities\n",
|
||||
"k3 = 0.5 ** 2 * RationalQuadratic(length_scale=1.0, alpha=1.0)\n",
|
||||
"k4 = 0.1 ** 2 * RBF(length_scale=0.1) + WhiteKernel(\n",
|
||||
" noise_level=0.1 ** 2, noise_level_bounds=(1e-5, np.inf)\n",
|
||||
") # noise terms\n",
|
||||
"kernel = k1 + k2 + k3 + k4"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "0ad8ddf0",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"The kernel hyper-parameters are:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 9,
|
||||
"id": "032ada0d",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"50**2 * RBF(length_scale=50) + 2**2 * RBF(length_scale=100) * ExpSineSquared(length_scale=1, periodicity=1) + 0.5**2 * RationalQuadratic(alpha=1, length_scale=1) + 0.1**2 * RBF(length_scale=0.1) + WhiteKernel(noise_level=0.01)\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"print(kernel)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"id": "4412a9a7",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"gp = GaussianProcessRegressor(kernel=kernel, normalize_y=True)\n",
|
||||
"time = data.decimal.to_numpy()[:, np.newaxis]\n",
|
||||
"co2 = data.co2.to_numpy()[:, np.newaxis]\n",
|
||||
"gp = gp.fit(time, co2)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "eb69bb80",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"These are the hyper-parameters after the fit and the log-likelihood given the data:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 11,
|
||||
"id": "667db862",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"GPML kernel: 3.19**2 * RBF(length_scale=42.7) + 0.114**2 * RBF(length_scale=174) * ExpSineSquared(length_scale=1.31, periodicity=1) + 0.295**2 * RationalQuadratic(alpha=0.000879, length_scale=6.84) + 0.0101**2 * RBF(length_scale=0.0188) + WhiteKernel(noise_level=0.00014)\n",
|
||||
"Log-marginal-likelihood: 6660.284\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"print(\"GPML kernel: %s\" % gp.kernel_)\n",
|
||||
"print(\"Log-marginal-likelihood: %.3f\" % gp.log_marginal_likelihood(gp.kernel_.theta))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "4968439e",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Now we can use the fit model to predict how the CO2 concentration will evolve at other times."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 12,
|
||||
"id": "ef6b7762",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"new_time = np.linspace(time.min(), time.max()+20, 1000)[:, np.newaxis]\n",
|
||||
"co2_pred, co2_std = gp.predict(new_time, return_std=True)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 14,
|
||||
"id": "5b263358",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"image/png": "iVBORw0KGgoAAAANSUhEUgAAArsAAAK/CAYAAABp8gWUAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAD3pUlEQVR4nOzdeXxcddn//9fMZCYzmWSyL23TlZallIKsFnEDZCtFRUCgQNEKqAVBvz9UbjdcQeTWWxbZREEtoiAoVGSTnRYsIFIKQgvd0jb7MllmSTLn98dnzpyZZNKmbZJmeT8fjzySOefMmXMakLefXue6XJZlWYiIiIiIjEPuvX0BIiIiIiLDRWFXRERERMYthV0RERERGbcUdkVERERk3FLYFREREZFxS2FXRERERMYthV0RERERGbcUdkVERERk3FLYFREREZFxS2FXRERERMYthV0RERERGbcUdkUmqKuvvhqXy0VjY+OIfN5dd92Fy+Vi48aNI/J5Mj7tzX+O9M+wyNiksCuS9Ktf/QqXy8VRRx2Vdf/KlSu5+uqraW1tHdkLkwG99957XHLJJcyaNQu/308oFOJDH/oQv/zlL4lEIhnHrl27lvPOO48pU6aQm5vL5MmTWbx4MWvXrt1LVz+2Dfe/DxPl3zc7QLtcLl544YV++y3LYurUqbhcLk499dS9cIVDz77nV155ZW9fikwQCrsiScuXL2fGjBn861//Yv369f32r1y5ku9///vj/j++w+X8888nEokwffr0ITnf3//+dw466CD+/Oc/s2jRIm688UauueYapk2bxpVXXsnll1+eOvaBBx7g0EMP5Z///Cef+9zn+NWvfsXSpUt5+umnOfTQQ3nwwQeH5JomkuH+92Gg8w/1P0e7Yjg/2+/3c8899/Tb/uyzz1JTU0Nubu6Qf6bIRJGzty9AZDTYsGEDK1eu5IEHHuCSSy5h+fLlfO9739vblzUudHZ2EgwG8Xg8eDyeITnnhg0bOPvss5k+fTpPPfUUkyZNSu1btmwZ69ev5+9//ztgVn/PP/98Zs2axXPPPUd5eXnq2Msvv5wPf/jDnH/++bzxxhvMmjVrSK5P+rP/OdhTQ/nP0Wj67FNOOYX77ruPG264gZwc5z/N99xzD4cddtiIlRuJjEda2RXBrOoWFxezcOFCzjjjDJYvX56x/+qrr+bKK68EYObMmam/dty4cWOq9vXdd9/lvPPOo7CwkPLycr7zne9gWRZbtmzhk5/8JKFQiKqqKv73f/836zX8+9//5uSTTyYUCpGfn89xxx3HSy+9lHFMe3s7V1xxBTNmzCA3N5eKigo+8YlP8Nprr6Wu0+Vy8d///pezzjqLUChEaWkpl19+OdFoNOvntra2cuGFF1JUVERhYSGf+9zn6Orq6nfc1q1b+fznP09lZSW5ubkceOCB/OY3v+n35+RyuXjrrbc499xzKS4u5phjjgGy1ztu3bqVpUuXMnnyZHJzc5k5cyZf+tKXiMfjO/htwXXXXUdHRwd33nlnRtC1zZ49O7Wy+7Of/Yyuri5uv/32jKALUFZWxm233UZnZyfXXXfdDj9zsAZzT4P5Xdt/luvXrx/072dnn7srv8Mdfe6O/n1IP0e2fw42bdrEl7/8Zfbbbz8CgQClpaWceeaZGf9c7Oj8A9XNDsefaV/ZPntPz2k755xzaGpq4oknnkhti8fj3H///Zx77rn9jh/MnyPAhRdeyIwZM/q9377ubNt2di+D/ew9NZjf6Uhdi4xtWtkVwYTd008/HZ/PxznnnMMtt9zC6tWrOeKIIwA4/fTTeffdd/njH//IL37xC8rKygAywtNnP/tZDjjgAK699lr+/ve/86Mf/YiSkhJuu+02jj32WH7605+yfPly/r//7//jiCOO4CMf+UjqvWvXruXDH/4woVCIr3/963i9Xm677TY+9rGP8eyzz6bqiL/4xS9y//33c+mllzJ37lyampp44YUXePvttzn00ENT5zvrrLOYMWMG11xzDS+99BI33HADLS0t/O53v+t372eddRYzZ87kmmuu4bXXXuPXv/41FRUV/PSnP00dU1dXxwc/+EFcLheXXnop5eXl/OMf/2Dp0qWEw2GuuOKKjHOeeeaZzJkzh5/85CdYlpX1z3zbtm0ceeSRtLa2cvHFF7P//vuzdetW7r//frq6uvD5fAP+vh5++GFmzZrF0UcfPeAx6cfOmDGDD3/4w1n3f+QjH2HGjBmpleA9MZh7Guzv2jaY389gPndXf4c7+tzB/PsA2f85WL16NStXruTss8+murqajRs3csstt/Cxj32Mt956i7y8vEGf3zYcf6a7ak/POWPGDBYsWMAf//hHTj75ZAD+8Y9/0NbWxtlnn80NN9yQcfxg/hyH616G87Ntg/2djsS1yDhgiUxwr7zyigVYTzzxhGVZlpVIJKzq6mrr8ssvzzjuZz/7mQVYGzZsyNj+ve99zwKsiy++OLWtp6fHqq6utlwul3Xttdemtre0tFiBQMBasmRJxjk+9alPWT6fz3rvvfdS27Zt22YVFBRYH/nIR1LbCgsLrWXLlg14L/a1nHbaaRnbv/zlL1uA9Z///KffsZ///Oczjv30pz9tlZaWZmxbunSpNWnSJKuxsTFj+9lnn20VFhZaXV1dGec855xz+l3bb3/724w/vwsuuMByu93W6tWr+x2bSCQGvMe2tjYLsD75yU8OeIyttbV1UMeedtppFmCFw+GdnnNHBnNPg/1d78rvZzCfu6u/w5197kD/PqSfI9s/B/bnpFu1apUFWL/73e92ev6+/xxZ1vD8mWaT7bOH6pyrV6+2brrpJqugoCD1Z3TmmWdaH//4xy3Lsqzp06dbCxcuTL1vsH+OS5YssaZPn97vWPu6s23b2b0M9rMHc88DGezvdE+vRSYGlTHIhLd8+XIqKyv5+Mc/DoDL5eKzn/0s9957L729vYM+zxe+8IXUzx6Ph8MPPxzLsli6dGlqe1FREfvttx/vv/9+altvby+PP/44n/rUpzJqRidNmsS5557LCy+8QDgcTr3/5ZdfZtu2bTu8lmXLlmW8vuyyywB45JFH+h37xS9+MeP1hz/8YZqamlKfaVkWf/nLX1i0aBGWZdHY2Jj6OvHEE2lra0uVUQx0zr4SiQR//etfWbRoEYcffni//X3/ejWdfV0FBQU7/AwwZR+DOdbeb597dwzmnnbld23b2e9nMJ87FL/Dvp87GNn+OQgEAqmfu7u7aWpqYvbs2RQVFfW7hsEYjj/T3TEU5zzrrLOIRCKsWLGC9vZ2VqxYkbWEAYb+zzHdzu5lOD8bdu13OtzXIuODwq5MaL29vdx77718/OMfZ8OGDaxfv57169dz1FFHUVdXxz//+c9Bn2vatGkZrwsLC/H7/am/gk3f3tLSknrd0NBAV1cX++23X79zHnDAASQSCbZs2QKYWtU333yTqVOncuSRR3L11VdnBGfbnDlzMl7vs88+uN3urHVsfa+7uLgYIHWNDQ0NtLa2pmpe078+97nPAVBfX59xjpkzZ/b7nHQNDQ2Ew2HmzZu3w+OyCYVCgBNkd8QOsTs7Nj0Ux2IxPv/5zzNt2jRCoRAf/OAHWbVq1U4/azD3tCu/a9tgfj+D+dxd/R3u7HMHI9s/B5FIhO9+97tMnTqV3NxcysrKKC8vp7W1lba2tkGf2zYcf6a7YyjOWV5ezvHHH88999zDAw88QG9vL2eccUbWY4f6zzHdzu5lOD8bdu13OtzXIuODanZlQnvqqafYvn079957L/fee2+//cuXL+eEE04Y1LmyPaU90JPb1gB1rDtz1lln8eEPf5gHH3yQxx9/nJ/97Gf89Kc/5YEHHkjV+WWzo5XSnV1jIpEA4LzzzmPJkiVZj50/f37G6/TVlqEWCoWYPHkyb7755k6PLSwsZNKkSbzxxhs7PO6NN95gypQphEIhOjs7mTFjBi+88ALV1dWp1mYbN24kPz9/qG5j0Ibin6Hd+R0Oxedm++fgsssu47e//S1XXHEFCxYsoLCwEJfLxdlnn526zuE21P9eDuU5zz33XC666CJqa2s5+eSTKSoqynrcYP8cB/p3f0d/a7WzexkNv0PbaLoWGb0UdmVCW758ORUVFdx888399j3wwAM8+OCD3HrrrQQCgR0Gxj1RXl5OXl4e77zzTr99//3vf3G73UydOjW1bdKkSXz5y1/my1/+MvX19Rx66KH8+Mc/zgi769aty1hVW79+PYlEIutT2YO5voKCAnp7ezn++ON3+f0DnTMUCg0qsGZz6qmncvvtt7Nq1SoWLFiw02PvuOMOXnjhhVRHgHTPP/88Gzdu5JJLLgEgGAzy3e9+N7X/7LPP5mtf+xrvvPMOhx122B7d067+rgdjsJ871L/D3f334f7772fJkiUZXUmi0Wi/frqDPf9w/JnuTZ/+9Ke55JJLeOmll/jTn/404HGD/XMsLi7O2gt506ZNu32Ng/3s3bUrv9PhvhYZH1TGIBNWJBLhgQce4NRTT+WMM87o93XppZfS3t7OQw89BJDqETrU/yPq8Xg44YQT+Nvf/pZRZlBXV8c999zDMcccQygUore3t99fy1VUVDB58mRisVjG9r7h/cYbbwTY4ervjq7vM5/5DH/5y1+yBqqGhoZdPqfb7eZTn/oUDz/8cNYpSjtbDfv6179OMBjkC1/4AnV1df32v/fee/zyl78E4MorryQQCHDJJZfQ1NSUcVxzczNf/OIXycvLS7W66mvdunU0Nzcze/bsPb6nwf6ud8VgP3eof4e7+++Dx+Pp9/u98cYb+600Dvb8w/Fnujfl5+dzyy23cPXVV7No0aIBjxvsn+M+++xDW1tbxt9ubN++fY8GqQz2s/fk/IP9nQ73tcj4oJVdmbAeeugh2tvbOe2007Lu/+AHP0h5eTnLly/ns5/9bGpV71vf+hZnn302Xq93h/8x2hU/+tGPeOKJJzjmmGP48peLine truncated
|
||||
"text/plain": [
|
||||
"<Figure size 800x800 with 1 Axes>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"fig, ax = plt.subplots(figsize=(8, 8))\n",
|
||||
"ax.scatter(time, co2, color='red', marker='.', label=\"Data\")\n",
|
||||
"ax.fill_between(new_time[:,0],\n",
|
||||
" co2_pred - co2_std,\n",
|
||||
" co2_pred + co2_std,\n",
|
||||
" color=\"b\",\n",
|
||||
" alpha=0.4,\n",
|
||||
" label=\"Predictions\")\n",
|
||||
"ax.set(xlabel=\"Time [years]\", ylabel=r\"CO$_2$ [ppm]\", title=\"Atmospheric CO$_2$ concentration in Mauna Loa\")\n",
|
||||
"ax.legend(frameon=False, loc=\"lower right\")\n",
|
||||
"plt.show()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "b625b695",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Many more details on the kernel selection as well as further details in this example can be seen in the following references:\n",
|
||||
"\n",
|
||||
"https://scikit-learn.org/stable/auto_examples/gaussian_process/plot_gpr_co2.html\n",
|
||||
"\n",
|
||||
"https://scikit-learn.org/stable/auto_examples/gaussian_process/plot_gpr_noisy_targets.html#sphx-glr-auto-examples-gaussian-process-plot-gpr-noisy-targets-py\n",
|
||||
"\n",
|
||||
"To read more on Gaussian Processes:\n",
|
||||
"\n",
|
||||
"http://www.gaussianprocess.org/gpml/chapters/RW.pdf"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "6d44d78a",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.10.6"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
@@ -0,0 +1,199 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "1bba0128",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Simple fits\n",
|
||||
"\n",
|
||||
"Here we are fitting a line from scratch.\n",
|
||||
"In the next notebook, we will do fancier fits with neural networks, but let's start with a basic problem and complicate it as we go along.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"id": "23feddde",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from typing import Tuple\n",
|
||||
"\n",
|
||||
"import pandas as pd\n",
|
||||
"import numpy as np\n",
|
||||
"import matplotlib.pyplot as plt\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "bb1286f0",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We start by generating some fake dataset, which is simple enough that we can visualize the results easily. For this reason, the dataset will contain only two variables.\n",
|
||||
"\n",
|
||||
"The simulated example data will be $f(x) = 3 x + \\epsilon$, where $\\epsilon \\sim \\mathcal{N}(\\mu=0, \\sigma=0.5)$.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"id": "5d457cd8",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def generate_data(N: int) -> np.ndarray:\n",
|
||||
" x = 2*np.random.randn(N, 1)\n",
|
||||
" epsilon = 0.5*np.random.randn(N, 1)\n",
|
||||
" z = 3*x + epsilon\n",
|
||||
" return np.concatenate((x, z), axis=1).astype(np.float32)\n",
|
||||
"\n",
|
||||
"data = generate_data(N=1000)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "48433f6f",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We can fit this line from scratch, assuming $y = f(x) = \\beta x + \\alpha + \\epsilon$, where $\\epsilon$ is a zero-mean Gaussian noise."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "6a170344-34ff-4af6-9a6c-daa44b95408e",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We can do this in two ways. In both cases, we start assuming the knowledge of $f(x)$ and use the Bayes Theorem:\n",
|
||||
"\n",
|
||||
"$ p(\\alpha, \\beta|\\mathcal{D}) \\, \\propto p(\\mathcal{D}|\\alpha, \\beta) \\, p(\\alpha) \\, p(\\beta)$\n",
|
||||
"\n",
|
||||
"We assume $p(\\alpha)$ and $p(\\beta)$ are constants, so we just need to maximize the likelihood $p(\\mathcal{D}|\\alpha, \\beta)$.\n",
|
||||
"That likelihood is a Gaussian distribution, because we assumed $y$ is exactly equals to a line plus some Gaussian noise coming from $\\epsilon$.\n",
|
||||
"\n",
|
||||
"$p(\\alpha,\\beta|\\mathcal{D}) \\propto \\prod \\mathcal{N}(y_i|\\text{mean}=f(x_i), \\text{std. dev.}=\\sigma_\\epsilon)$\n",
|
||||
"\n",
|
||||
"This leads to minimizing $L = \\sum_i (y_i - \\beta x_i - \\alpha)^2$, as seen in the lecture (simply take $- \\log p$ and to maximize $p$, one has to minimize $-\\log p$).\n",
|
||||
"\n",
|
||||
"We can try finding out $\\beta$ and $\\alpha$ by taking the derivative of $L$ and setting it to zero:\n",
|
||||
"\n",
|
||||
"$\\frac{d L}{d\\beta} = 2 \\sum_i (y_i - \\beta x_i - \\alpha) x_i = 0$\n",
|
||||
"\n",
|
||||
"$\\frac{d L}{d\\alpha} = -2 \\sum_i (y_i - \\beta x_i - \\alpha) = 0$\n",
|
||||
"\n",
|
||||
"Re-arranging:\n",
|
||||
"\n",
|
||||
"$2 \\sum_i (y_i x_i - \\beta x_i^2 - \\alpha x_i) = 0$\n",
|
||||
"\n",
|
||||
"$2 \\sum_i (y_i - \\beta x_i - \\alpha) = 0$\n",
|
||||
"\n",
|
||||
"This leads to:\n",
|
||||
"\n",
|
||||
"$ \\beta \\sum_i x_i^2 = \\sum_i (y_i - \\alpha) x_i$\n",
|
||||
"\n",
|
||||
"$ \\alpha = 1/N \\sum_i y_i - \\beta x_i = \\mathbb{E}[y - \\beta x]$\n",
|
||||
"\n",
|
||||
"We can then substitute $\\alpha$ in $\\beta$:\n",
|
||||
"\n",
|
||||
"$ \\beta \\sum_i x_i^2 = \\sum_i y_i x_i - 1/N \\sum_i \\sum_j (y_j - \\beta x_j) x_i$\n",
|
||||
"\n",
|
||||
"And expand it:\n",
|
||||
"\n",
|
||||
"$ \\beta (\\sum_i x_i^2 - 1/N \\sum_i \\sum_j x_i x_j) = \\sum_i y_i x_i - 1/N \\sum_i \\sum_j y_j x_i$\n",
|
||||
"\n",
|
||||
"Taking $x_j$ from a sum that does not depend on $j$, we see that the second term is just the square of the sum over $x_i$:\n",
|
||||
"\n",
|
||||
"$ \\beta (\\sum_i x_i^2 - 1/N(\\sum_{i} x_i)^2 ) = \\sum_i y_i x_i - 1/N \\sum_i \\sum_j y_j x_i$\n",
|
||||
"\n",
|
||||
"Dividing by the number of points $N$, several sums reduce to averages, which make this much easier to interpret if we use some standard identities from statistics.\n",
|
||||
"For example $1/N \\sum_i x_i^2 - 1/N^2 (\\sum_i x_i)^2$ is a known expression for the variance of $X$. \n",
|
||||
"\n",
|
||||
"$ \\beta \\text{var}[X] = \\mathbb{E}[XY] - \\mathbb{E}[X]\\mathbb{E}[Y]$\n",
|
||||
"\n",
|
||||
"And the right-hand-side is a known expression for the covariance between $X$ and $Y$:\n",
|
||||
"\n",
|
||||
"$ \\beta \\text{var}[X] = \\text{cov}[X,Y]$\n",
|
||||
"\n",
|
||||
"This leads to:\n",
|
||||
"\n",
|
||||
"$\\beta = \\frac{\\text{cov}[X,Y]}{\\text{var}[X]}$\n",
|
||||
"\n",
|
||||
"This, together with the expression from above for $\\alpha$, allows us to easily fit a line in a few lines of code:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"id": "30205402",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"cov = np.cov(data[:,0], data[:,1])[0,1]\n",
|
||||
"varX = np.std(data[:,0])**2\n",
|
||||
"beta = cov/varX\n",
|
||||
"alpha = np.mean(data[:,1] - beta*data[:,0])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"id": "cabdfc12-3033-4a2f-96ed-4ab1ea8312ab",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<matplotlib.lines._AxLine at 0x7f5db8662e30>"
|
||||
]
|
||||
},
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAioAAAGdCAYAAAA8F1jjAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABZHElEQVR4nO3de1zT1f8H8Nc2YFyEKXJPVLyHeENTUSvvooaYZZlpaX2tFO+WguWFsvCSt7x21czMrCzFjPJuGUZqqIiaKN5geEM2FBmwfb5/EIPJgG0MNtjr+Xjw+LWzz+ezI7++7tW5vI9IEAQBRERERFZIbOkOEBEREZWFQYWIiIisFoMKERERWS0GFSIiIrJaDCpERERktRhUiIiIyGoxqBAREZHVYlAhIiIiq2Vn6Q5UlkajQXp6OlxdXSESiSzdHSIiIjKAIAjIzs6Gn58fxOKyx01qfFBJT0+Hv7+/pbtBREREJrh27RoaNGhQ5vtVGlRiYmKwfft2nDt3Dk5OTujWrRsWLVqEli1baq/Jzc3FjBkzsHXrVqhUKgwYMABr166Ft7e3QZ/h6uoKoPAP6ubmViV/DiIiIjIvpVIJf39/7fd4WURVedZPaGgoRowYgcceewwFBQWYPXs2kpKSkJycDBcXFwDA+PHj8fPPP2Pjxo2QyWSYOHEixGIxjhw5YtBnKJVKyGQyKBQKBhUiIqIawtDv7yoNKg+7desWvLy8cOjQITzxxBNQKBTw9PTEli1b8OyzzwIAzp07h0cffRTx8fHo2rVrhc9kUCEiIqph1Goob96EzM+vwu/val2jolAoAADu7u4AgOPHjyM/Px99+/bVXtOqVSs0bNiwzKCiUqmgUqm0r5VKZRX3moiIiMzmxg3gxRcBZ2eDLq+27ckajQZTp05F9+7dERQUBADIyMiAg4MD6tatq3Ott7c3MjIy9D4nJiYGMplM+8OFtERERDXImjXAvn1AbKxBl1dbUImIiEBSUhK2bt1aqedERUVBoVBof65du2amHhIREVGVe+cdoHNnwMvLoMurZepn4sSJ2LVrFw4fPqyzBcnHxwd5eXnIysrSGVW5ceMGfHx89D5LKpVCKpVWdZeJiIjIHAoKALsSccPBAfj+e0ClApo3r/D2Kh1REQQBEydOxI8//oj9+/cjICBA5/2OHTvC3t4e+/bt07adP38eV69eRUhISFV2jYiIiKra4cNAq1bAqVO67f7+1jGiEhERgS1btmDHjh1wdXXVrjuRyWRwcnKCTCbDq6++iunTp8Pd3R1ubm6YNGkSQkJCDNrxQ0RERFZIowEWLSqc5tFogOHDgWPHgApqpuhTpUFl3bp1AICePXvqtG/YsAFjxowBACxfvhxisRjPPPOMTsE3IiIiqoFu3wZGjwbi4orbGjQAcnNNCirVWkelKrCOChERkfmpNQISUjNxMzsXXq6O6BzgDom4gjP1jhwBnn8eSEsrfC0SAXPnAnPmABKJzqWGfn/X+LN+iIiIyLzikuSIjk2GXJGrbfOVOWJeWCBCg3xL36DRAB9+CMyeDajVhW1eXsDXXwMlaqWZotq2JxMREZH1i0uSY/zmEzohBQAyFLkYv/kE4pLkujfcuQMMGQLMmlUcUnr2BBITKx1SAAYVIiIi+o9aIyA6Nhn61oQUtUXHJkOtKXHFv/8Wr0cRiQqnefbuBXz1jLyYgEGFiIiIAAAJqZmlRlJKEgDIFblISM0sbgwJAT74APD0LAws775baj1KZTCoEBEREQDgZnbZIaWIW+493My6r9v45ptAUhLQv7/Z+8SgQkRERAAAL1fHct/vkHYOuzdMQofN63TfEIsNLuBmLAYVIiIiAgB0DnCHr8wRpTYhCwJeTfgR27bMQgPlLfh/tBg4cKBa+sSgQkRERAAAiViEeWGBOm1uuffwyY/vY86Bz2GvKdzVI+rWzaBzesyBQYWIiIh0yJztAQDt0s9j94bJ6H/haPGbkZGFoyklDhmuSiz4RkRERACKa6gIgoCxx3ci6sAGOGgKAAB3HV1xceladJowqlr7xKBCRERE2hoqLqocfLh7OUL/jde+d9yvFSaFz4Kg8MUfGqHiUvpmxKkfIiIi0tZQKRCLEZCZpm1f33kYnh+5EOlunqVrqFQDBhUiIiLS1lDJtXfEhPAoXHfzxCvPzMXCXq+gQGJX6rrqwqkfIiIiW6ZQAAoFvFxdtE0XPfzR67VPkC+xL3V5RbVWzI0jKkRERLbqxAkgOBgID0dnX2edGioPhxQRCk9Q7hzgXq1dZFAhIiKyNYIArF1beE7PpUtAYiIk77ytraHy8FLZotfzwgKrdSEtwKBCRERkW5RKYMQIICICyMsrbOvcGZg8GaFBvlg3Khg+Mt3pHR+ZI9aNCkZokHlORDYG16gQERHZisREYPhwICWluG3qVGDRIsDBAQAQGuSLfoE+SEjNxM3sXHi5Fk73VPdIShEGFSIiotpOEIBPPgGmTAFUqsI2mQzYsAF4+ulSl0vEIoQ0rV/NndSPQYWIiKg2EwRg9Gjg66+L2zp1ArZtAwICLNcvA3GNChERUS2l1giIv5SJ804lRkcmTwb++KNGhBSAIypERES1UlySHNGxyZArciGp1wfrmifgwGP98eS4NxAqlVq6ewYTCYIgWLoTlaFUKiGTyaBQKODm5mbp7hAREVnWvXs49nUshqe64eEv+KLlsJbawVOSod/fnPohIiKqLU6fhtCpE9pFvIS26edLvV0UXKJjk6HW1IxxCgYVIiKiGkatERB/8Q52JKYh/uIdqNUa4PPPgc6dITp/HvbqAiyMW1W4kPYhAmCRwwVNxTUqRERENUjJtScA4JSXi2UH12PgP3u115zxaoIJQyMBUdm1T6r7cEFTMagQERHVEHFJcozffEI7hdP81hWs3bEQze9c016TMXIshvmGQWXnUO6zqvtwQVNx6oeIiKgGUGsERMcma0PKs6f3Yuem6dqQcs/BCXOefxvuX34G9/pupc7rKWKpwwVNxaBCRERUAySkZmqne2Yc/gof7l4Bp4LCKrNnPRtjyEvL8VXjEBy/ctcqDxc0FYMKERFRDVByTcmvLUKgkhSu3tjSbgCGjl6KS/UbaK+zxsMFTcU1KkRERDVAyTUlST7NMK/vG3hgL8WO1r30XmdthwuaikGFiIjImuXkAKtWofO06fCVOSJDkQsBwNb2oTqXiVA4YlJy7Yk1HS5oKk79EBERWZmiOin7fjyEnA6dgMhISObNrVVrTwxVpUHl8OHDCAsLg5+fH0QiEX766Sed98eMGQORSKTzExoaqv9hRERENiAuSY4ei/bjm2kL0fX5UDj/exYAULDyI4R6iGrN2hNDVenUz/3799GuXTu88sorGDZsmN5rQkNDsWHDBu1raQ06KImIiMic4pLkmLohHnP3fYKRJ3/Vtv9bvyEihkZixm2h1qw9MVSVBpWBAwdi4MCB5V4jlUrh4+NTld0gIiKyemqNgC82/Ibtm6MReDNV2/59UB/M6TceuQ6OiI5NRr9An1qx9sRQFl9Me/DgQXh5eaFevXro3bs3FixYgPr1y/7lq1QqqFQq7WulUlkd3SQiIqpSF1d/ji9WT0advAcAgAd2UszpPx7ft+mrvabojB5bCSmAhRfThoaGYtOmTdi3bx8WLVqEQ4cOYeDAgVCr1WXeExMTA5lMpv3x9/evxh4TERFVgW3b0GLKOG1ISXFvgPCXluqElCI15YwecxEJgp6jFavig0Qi/Pjjjxg6dGiZ11y6dAlNmzbF3r170adPH73X6BtR8ff3h0KhgJubm7m7TUREVPVUKtx7rCvqnE7ED617YU7/CchxcNJ76TfjutaKERWlUgmZTFbh97fFp35KatKkCTw8PJCSklJmUJFKpVxwS0RENYZaI2gXvnq4SAERcPueSncRrFQKpx9/wIJZ6/B50ych6Dn1WF+dFFtgVUHl+vXruHPnDnx9a9/2KiIisj1xSXJExyZrz+gBAIeCfMw6tBGr2/bHvWYtMS8sEKFBvpA0bYJO86fj880nIAJQcrqjNtdJqUiVrlG5d+8eEhMTkZiYCABITU1FYmIirl69inv37uGtt97C0aNHcfnyZezbtw/h4eFo1qwZBgwYUJXdIiIiqnJxSXKM33xCJ6Q0vCvHD5vfxKvHdmDtjoVQ3MrC+M0nEJckB4BadUaPuVTpGpWDBw+iV69epdpffvllrFu3DkOHDsU///yDrKws+Pn5oX///njvvffg7e1t8GcYOsdFRERUlXSmeOpIMWNbIjKUxWsqQ88fweLdK+GWlwMAUEnsMWZ4NI42agsfmSP+mNVbO1pS8lm1tU6Kod/f1baYtqowqBARkaXpm+Ip4lCQj6iDX2Ds8Vht20X3RzAxfBbOejXRttWWRbKGqpGLaYmIiGqaoikeff/V3yArA6t3LkJ7+QVt285Hn0DUgIm4L3XWudbWth0bikGFiIjICGqNgKOX7iD+4h0IgoAtCVf1hpT+/8Zjye4VkKnuAyic6onu+xq2tAsF9Ozq8XJ1LNVGDCpEREQGi0uSI3L7aWTl5Jd7XYOsDKz9KQZ2ggYAkFrPFxPDI3HGu2mpa21127GhLFqZloiIqKaIS5Ljjc0nKgwpAHC9rg+WPT4KALCrZQ+EvbyyzJAC2Oa2Y0NxRIWIiKgCao2A+TuTjbpnXddnccGjIfY066J3qgcoHEkpqqNC+jGoEBERVeDopTvIUOpf7GqnLsCsQxuR6SzDuq7Dte2CSIw9zbvqvaeukz3WvBiMrk3qcySlAgwqRERE/9FXv2RPcgYifzit93o/5U2s3rEIwennoRaJcfyRR5HgH1Th52Q9yIdYJGJIMQCDChEREfTXQqnrbF/mmpTeKQlY9vMy1M29BwBQi8RomJVhUFABuB3ZUAwqRERk88qqhaIvpNipC/DW4U14PWG7tu2qzBsR4ZE47dvc4M/kdmTDMKgQEZFNU2sERMcm662Line truncated
|
||||
"text/plain": [
|
||||
"<Figure size 640x480 with 1 Axes>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"plt.scatter(data[:,0], data[:,1])\n",
|
||||
"plt.axline((0, alpha), slope=beta, c='r', lw=2, ls='--', label=\"Line fit\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "a1fbd00b-e7f2-4eff-90db-e1d44d104fba",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.10.6"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
+104
@@ -0,0 +1,104 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "1bba0128",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Simple fits\n",
|
||||
"\n",
|
||||
"Here we are fitting a line from scratch.\n",
|
||||
"In the next notebook, we will do fancier fits with neural networks, but let's start with a basic problem and complicate it as we go along.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"id": "23feddde",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from typing import Tuple\n",
|
||||
"\n",
|
||||
"import pandas as pd\n",
|
||||
"import numpy as np\n",
|
||||
"import matplotlib.pyplot as plt\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "bb1286f0",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We start by generating some fake dataset, which is simple enough that we can visualize the results easily. For this reason, the dataset will contain only two variables.\n",
|
||||
"\n",
|
||||
"The simulated example data will be $f(x) = 3 x + \\epsilon$, where $\\epsilon \\sim \\mathcal{N}(\\mu=0, \\sigma=0.5)$.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"id": "5d457cd8",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def generate_data(N: int) -> np.ndarray:\n",
|
||||
" x = 2*np.random.randn(N, 1)\n",
|
||||
" epsilon = 0.5*np.random.randn(N, 1)\n",
|
||||
" z = 3*x + epsilon\n",
|
||||
" return np.concatenate((x, z), axis=1).astype(np.float32)\n",
|
||||
"\n",
|
||||
"data = generate_data(N=1000)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "48433f6f",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We can fit this line from scratch, assuming $y = f(x) = \\beta x + \\alpha + \\epsilon$, where $\\epsilon$ is a zero-mean Gaussian noise.\n",
|
||||
"\n",
|
||||
"How would you do it? Feel free to use standard Python modules. Look at the solution for a simple mathematical expression for this fit with a full derivation.\n",
|
||||
"\n",
|
||||
"Tip: Look for the documentation for `numpy.linalg.lstsq`."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "85040eef-3558-4531-9bce-4f29d520f86b",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "5e9170af-ca77-4526-b3d2-e19e77fef44e",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.10.6"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
@@ -0,0 +1,703 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "6a764d92",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Mixture Models\n",
|
||||
"\n",
|
||||
"One common objective is to find similarities in data and cluster it. There are several heuristic methods to cluster them. We are going to focus on a few strongly theoretically motivated methods. Other methods use heuristics to identify similarities in data and are mentioned in the end.\n",
|
||||
"\n",
|
||||
"For the purposes of this example, we will generate a fake dataset."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "fce4d8e8",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We start by loading the necessary Python modules."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"id": "300cf8d3",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import matplotlib.pyplot as plt\n",
|
||||
"\n",
|
||||
"import pandas as pd\n",
|
||||
"import numpy as np\n",
|
||||
"from sklearn.mixture import GaussianMixture, BayesianGaussianMixture\n",
|
||||
"from sklearn.cluster import KMeans"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "0ecd6a69",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Let's generate the fake data now to have something to cluster."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"id": "4959a292",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def generate_clusters(mu: np.ndarray, sigma: np.ndarray, N: int) ->np.ndarray:\n",
|
||||
" assert len(mu) == len(sigma)\n",
|
||||
" assert N > 1\n",
|
||||
" D = len(mu[0].shape)\n",
|
||||
" data = np.concatenate([np.random.default_rng().multivariate_normal(mean=mu_k, cov=sigma_k, size=N)\n",
|
||||
" for mu_k, sigma_k in zip(mu, sigma)], axis=0)\n",
|
||||
" source = np.concatenate([k*np.ones([N, 1]) for k in range(len(mu))], axis=0)\n",
|
||||
" return np.concatenate([data, source], axis=1)\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"id": "82929490",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"data = generate_clusters(mu=[np.array([5.0, -2.0]),\n",
|
||||
" np.array([1.0, 5.0]),\n",
|
||||
" np.array([-5.0, -1.0])],\n",
|
||||
" sigma=[np.array([[0.4, 0.2],\n",
|
||||
" [0.2, 0.4]]),\n",
|
||||
" np.array([[1.0, 0.5],\n",
|
||||
" [0.5, 1.0]]),\n",
|
||||
" np.array([[2.0, 0.0],\n",
|
||||
" [0.0, 5.0]])],\n",
|
||||
" N=1000)\n",
|
||||
"data = pd.DataFrame(data, columns=[\"x\", \"y\", \"source\"])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "d8295e8a",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Let's print out the dataset read first."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"id": "024fb65a",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/html": [
|
||||
"<div>\n",
|
||||
"<style scoped>\n",
|
||||
" .dataframe tbody tr th:only-of-type {\n",
|
||||
" vertical-align: middle;\n",
|
||||
" }\n",
|
||||
"\n",
|
||||
" .dataframe tbody tr th {\n",
|
||||
" vertical-align: top;\n",
|
||||
" }\n",
|
||||
"\n",
|
||||
" .dataframe thead th {\n",
|
||||
" text-align: right;\n",
|
||||
" }\n",
|
||||
"</style>\n",
|
||||
"<table border=\"1\" class=\"dataframe\">\n",
|
||||
" <thead>\n",
|
||||
" <tr style=\"text-align: right;\">\n",
|
||||
" <th></th>\n",
|
||||
" <th>x</th>\n",
|
||||
" <th>y</th>\n",
|
||||
" <th>source</th>\n",
|
||||
" </tr>\n",
|
||||
" </thead>\n",
|
||||
" <tbody>\n",
|
||||
" <tr>\n",
|
||||
" <th>0</th>\n",
|
||||
" <td>4.960065</td>\n",
|
||||
" <td>-2.902600</td>\n",
|
||||
" <td>0.0</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>1</th>\n",
|
||||
" <td>6.231718</td>\n",
|
||||
" <td>-1.404399</td>\n",
|
||||
" <td>0.0</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>2</th>\n",
|
||||
" <td>3.985313</td>\n",
|
||||
" <td>-2.833486</td>\n",
|
||||
" <td>0.0</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>3</th>\n",
|
||||
" <td>5.314968</td>\n",
|
||||
" <td>-1.930364</td>\n",
|
||||
" <td>0.0</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>4</th>\n",
|
||||
" <td>5.652647</td>\n",
|
||||
" <td>-1.523516</td>\n",
|
||||
" <td>0.0</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>...</th>\n",
|
||||
" <td>...</td>\n",
|
||||
" <td>...</td>\n",
|
||||
" <td>...</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>2995</th>\n",
|
||||
" <td>-6.289552</td>\n",
|
||||
" <td>-1.923758</td>\n",
|
||||
" <td>2.0</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>2996</th>\n",
|
||||
" <td>-7.690072</td>\n",
|
||||
" <td>1.032256</td>\n",
|
||||
" <td>2.0</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>2997</th>\n",
|
||||
" <td>-2.930869</td>\n",
|
||||
" <td>-0.503878</td>\n",
|
||||
" <td>2.0</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>2998</th>\n",
|
||||
" <td>-3.721900</td>\n",
|
||||
" <td>3.096010</td>\n",
|
||||
" <td>2.0</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>2999</th>\n",
|
||||
" <td>-5.862448</td>\n",
|
||||
" <td>-0.028824</td>\n",
|
||||
" <td>2.0</td>\n",
|
||||
" </tr>\n",
|
||||
" </tbody>\n",
|
||||
"</table>\n",
|
||||
"<p>3000 rows × 3 columns</p>\n",
|
||||
"</div>"
|
||||
],
|
||||
"text/plain": [
|
||||
" x y source\n",
|
||||
"0 4.960065 -2.902600 0.0\n",
|
||||
"1 6.231718 -1.404399 0.0\n",
|
||||
"2 3.985313 -2.833486 0.0\n",
|
||||
"3 5.314968 -1.930364 0.0\n",
|
||||
"4 5.652647 -1.523516 0.0\n",
|
||||
"... ... ... ...\n",
|
||||
"2995 -6.289552 -1.923758 2.0\n",
|
||||
"2996 -7.690072 1.032256 2.0\n",
|
||||
"2997 -2.930869 -0.503878 2.0\n",
|
||||
"2998 -3.721900 3.096010 2.0\n",
|
||||
"2999 -5.862448 -0.028824 2.0\n",
|
||||
"\n",
|
||||
"[3000 rows x 3 columns]"
|
||||
]
|
||||
},
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"data"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "1c178424",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We can plot this fairly easily using Matplotlib."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"id": "e63b38c5",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAq8AAAKsCAYAAADC9VQzAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAEAAElEQVR4nOzdd3yb1dXA8d99JFneM3b23nH2JJMEEkIIIcwAZYTZAbRQ2rLeMtrSptABpaVQ9iaMQIAAgewQsvfey0nsLO8lWXru+4dsxYokr9iyHZ/v5yOCn3HvlePYx/e59xyltdYIIYQQQgjRCBj1PQAhhBBCCCGqSoJXIYQQQgjRaEjwKoQQQgghGg0JXoUQQgghRKMhwasQQgghhGg0JHgVQgghhBCNhgSvQgghhBCi0ZDgVQghhBBCNBoSvAohhBBCiEZDglchhBBCCNFoSPAqhBBCCNHIzZgxgyFDhhATE0NKSgpXXnklu3btqvS+Tz75hB49ehAeHk6fPn345ptvfM5rrXniiSdo2bIlERERjB8/nj179tTV26gSCV6FEEIIIRq5JUuWcO+997Jy5UrmzZtHSUkJl1xyCQUFBUHvWb58OTfeeCN33nknGzZs4Morr+TKK69k69at3mueffZZXnjhBV5++WVWrVpFVFQUEydOpLi4OBRvKyCltdb11rsQQgghhKh1J0+eJCUlhSVLljBmzJiA11x//fUUFBQwZ84c77ELLriA/v378/LLL6O1plWrVvzmN7/ht7/9LQA5OTk0b96ct956ixtuuCEk7+Vs1nrptZ6YpsmxY8eIiYlBKVXfwxFCCCFEFWitycvLo1WrVhhG6B8aFxcX43Q6Q94veN772TGL3W7HbrdXeF9OTg4AiYmJQa9ZsWIFDz74oM+xiRMnMnv2bAAOHDhARkYG48eP956Pi4tj2LBhrFixQoLXUDh27Bht27at72EIIYQQogbS0tJo06ZNSPssLi6mY/toMk64Q9pvmejoaPLz832OPfnkkzz11FNB7zFNkwceeICRI0fSu3fvoNdlZGTQvHlzn2PNmzcnIyPDe77sWLBr6kOTCl5jYmIAzxd/bGxsPY9GCCGEEFWRm5tL27ZtvT/HQ8npdJJxws2hdR2IjQntrG9unkn7QQf94pbKZl3vvfdetm7dyrJly+p6iPWiSQWvZdPusbGxErwKIYQQjUx9LvmLjlFEx4S2f5Pqxy333Xcfc+bMYenSpZXOUrdo0YLjx4/7HDt+/DgtWrTwni871rJlS59r+vfvX9W3Uesk24AQQgghRCOntea+++7j888/Z+HChXTs2LHSe4YPH86CBQt8js2bN4/hw4cD0LFjR1q0aOFzTW5uLqtWrfJeUx+a1MyrEEIIIcT56N577+WDDz7giy++ICYmxrsmNS4ujoiICABuvfVWWrduzYwZMwC4//77ufDCC/nHP/7B5MmTmTlzJmvXruWVV14BPDPdDzzwAE8//TRdu3alY8eOPP7447Rq1Yorr7yyXt4nSPAqhBBCCFEptzZxhzi5qFubVb72pZdeAmDs2LE+x998801uu+02AA4fPuyTrWHEiBF88MEH/P73v+exxx6ja9euzJ4922eT10MPPURBQQE//elPyc7OZtSoUcydO5fw8PCav7Fz1KTyvObm5hIXF0dOTo6seRVCCCEaifr8+V3W94ld7etlw1ZK90MSt5xFZl6FEEIIISphojEJ7XxfqPtrLGTDlhBCCCGEaDRk5lUIIYQQohImJlVfgVp7fQp/MvMqhBBCCCEaDQlehRBCCCFEoyHLBoQQQgghKuHWGneIEzSFur/GQmZehRBCCCFEoyEzr0IIIYQQlZBUWQ2HzLwKIYQQQohGQ4JXIYQQQgjRaMiyASGEEEKISpho3LJsoEGQmVchhBBCCNFoyMyrEEIIIUQlZMNWwyEzr0IIIYQQotGQmVchhBBCiEpIkYKGQ2ZehRBCCCFEoyHBqxBCNFFaa3KKiylxu+t7KEIIUWWybEAIIZqgj7Zt4YVVK0jPzyPcauX61D78bsRoIm22+h6aEA2SWfoKdZ/Cn8y8CiFEE/P+lk08uuB70vPzACh2uXh380Z+PucLtKyxE0I0cDLzKoQQTYjLNHlu5Y9+x02tWZZ2iI0Z6Qxo2aoeRiZEw+auhyIFoe6vsZCZVyGEaELS8/LILCoKeE6h2JCRHuIRCSFE9UjwKoQQTUis3Y4Kck6jSYyICOl4hBCiuiR4FUKIJiQuPJwJnbpgUb4hrAKibDYmdOpSPwMTooFz6/p5CX8SvAohRBPzp4vG0zEhEQCrYaAAu9XKfydfQVRYWP0OTgghKiEbtoQQoolJjozim5/cysID+9h28gTJkVFM6daDuPDwOu138/EMPty6mWN5efRs1oyb+vSnbVxcnfYpRG2RVFkNhwSvQgjRBFkNg0s6d+WSzl1D0t/H27bw6ILvMZTCrTXL0w7xzuaNvHfVdQyU7AZCiGqQZQNCCCHqVE5xMU8sXoDmTK12t9Y43W4emf+d5JYVjYKJwh3ilxl0e2XTJsGrEEKIOrX40AGcAUrQmlqzNyuTA9lZ9TAqIURjJcGrEEKIOuUyK165V9l5IYQoT9a8CiGEqFOj2rbHUArzrOUBCmgRHUPn0swHQjRkpva8Qt2n8Cczr0IIIepU8+ho7htyAQBGaX5Zi1IopXjywnFYDPlRJISoOpl5FUIIUefuHzacrolJvLN5I0fzckhNTuHugUMY3Kp1fQ9NiCop20QV6j6FPwlehRBC1DmlFJO7dWdyt+71PRQhRCMnz2qEEEIIIUSjITOvQgghhBCVkGUDDYfMvAohhBBCiEZDZl6FEEI0eScLCvh4+1YOZmfRPi6e63r1pnl0dH0PSzQgplaYOrQzoaHur7GQ4FUIIUSTtj79GLfO/pRilwsFaODFNat4c+rVXNCmbX0PTwhxlkazbMDtdvP444/TsWNHIiIi6Ny5M3/605+kJrYQQogaM7XmgblfU+xyYWqNW2tMrSkx3Tww92up/iW8yta8hvol/DWamddnnnmGl156ibfffpvU1FTWrl3L7bffTlxcHL/61a/qe3hCCCEaoS3HMziSl+t33NSaE4UFrD12VGZfhWhgGk3wunz5cqZOncrkyZMB6NChAx9++CGrV68Oeo/D4cDhcHg/zs31/wYlhBCi6SooKankvDNEIxFCVFWjWTYwYsQIFixYwO7duwHYtGkTy5YtY9KkSUHvmTFjBnFxcd5X27by27MQQogz+qQ0J9waeB7HZhgMatkqxCMSDZUbo15ewl+j+aw88sgj3HDDDfTo0QObzcaAAQN44IEHuOmmm4Le8+ijj5KTk+N9paWlhXDEQgghGroYu50Hho0A8K4uLPvzniHDiA+PqJdxCSGCazTLBj7++GPef/99PvjgA1JTU9m4cSMPPPAArVq1Yvr06QHvsdvt2O32EI9UCCFEY/LTQUNIiYrilXVrOJCdTbu4OO4aOJhre6bW99BEA6LrIVWWllRZATWa4PV3v/udd/YVoE+fPhw6dIgZM2YEDV6FEEKIqriyRy+u7NGrvochhKiCRrNsoLCwEMPwHa7FYsGUNCZCCCGEEE1Go5l5nTJlCn/+859p164dqampbNiwgX/+85/ccccd9T00IYQQQpzn6iPvquR5DazRBK///ve/efzxx7nnnns4ceIErVq14mc/+xlPPPFEfQ9NCCGEEEKESKMJXmNiYnj++ed5/vnn63soQggh6sCOUyf5bu8eTK0Z16Ej/Vu0RCmZeRINg1sbuHVoV1u6pYhoQI0meBVCCNEw5DocfLt3N6cKC+mb0pyR7dpjnEOQqbXmmR+X8sr6tVhK2/nPmpVc1aMXf5tw6Tm1LYQ4/0jwKoQQosp+TDvEz+Z8QWFJCRalcGtNn5TmvH3lNTXOibro4AFeWb8WALc+M9X0+c7tDG3dhutT+9TK2IU4FyYKM8T73E1k6jWQRpNtQAghRP3Kczj4+ZwvKCotqVoWaG4/eYI/LFlY43Zn7djqnXEtTwGfbNta43aFEOcnCV6FEEJUyXf79lBQUuI3F+TWmjm7d1HgdNao3aziYp8Z1zIayCouqlGbQojzlwSvQgghquR0UWHQ9adurcl1OGrU7pBWrQO2a1GKYa3b1KhNIWpbWaqsUL+EPwlehRBCVEnflBaYAWZIAZpFRpISFVWjdm/u0584u91n6YBFKcIsFu4aOLhGbQohzl8SvAohhKiSC9q0ZWDLVgHPWZVBYela2OpKjori02k/YVyHTt4Z2AvatOXja2+gU0JijccrRG0qS5UV6pfwJ9kGhBBCVIlSimt7prI+/ZjfueMF+byxcR33DxtRo7Y7xifwypQrcbrdaK2xW+XHkxAiMPnuIIQQosqWHDqAoZTf8gENzNm9q0rB66HsbN7cuI41x47SLDKSab36cFnXbqjSpQJCCFERCV6FEEJUWYlpooOsey0x3ZXev/3kCaZ9OhOHy4Vbawyl+OHwIdamD+DJCy+q7eEKUWs8eV5Du4Eq1P01FrKYQgghRJWN69ApYNp0i1IMbtma382by9i3X+Oajz/go21b/GZo/7R0EcWlgSvgPf/2pg3sPHWyrocvhDgPyMyrEEKIKrumZy9mbt3MjlMnvYGnRSni7OF8vWcXLtPErTVpOTlsyEhnXfpRnh1/KQAFTierjh4J2K5FKRYc2EePZskhey9CVIeJgVsqbDUIErwKIYSoUK6jmE+2b2PrieM0i4zkT+PGs+zwIb7avROn282ETp1Zc+woW04c9wa0ZT9yP92+jVv6DqBPSnNUkByxZ8gjUiFE5SR4FUKIJiqzqJCVR45gt1gY0bYdETab3zUHs7OY9ulMThcWopRCAW9sWMeMiy/hu5tvA6CwpITeL70QsA+LUiw6sJ8+Kc2JtNkY3qYtq48e8auo5daaCZ061/ZbFEKchyR4FUKIJui/a1bx/KrluEwLine truncated
|
||||
"text/plain": [
|
||||
"<Figure size 800x800 with 2 Axes>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"fig, ax = plt.subplots(figsize=(8, 8))\n",
|
||||
"data.plot.scatter(x=\"x\", y=\"y\", c=\"source\", colormap='viridis', ax=ax)\n",
|
||||
"ax.set(xlabel=\"x\", ylabel=r\"y\", title=\"\")\n",
|
||||
"plt.show()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "a376636d",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"There are several ways of finding similarities in the data. The Gaussian Mixture Model assumes that the data has been produced exactly as in this example. For each sample, the procedure assumed to be used for the generation of each sample is the following:\n",
|
||||
" * a random integer is chosen according to a discrete probability distribution, which identifies to which cluster the sample belongs to (this is the `source` variable in the generation procedure above);\n",
|
||||
" * that random integer is assumed to be used to choose the mean and covariance matrix for a Gaussian random variable, which is then used to produce the observed sample.\n",
|
||||
" \n",
|
||||
"Our objective here is then to find out the probability of producing a sample for each cluster (the probabilities with which a sample belongs to one cluster instead of the others), the means and covariance matrices for the model. Using Bayes' theorem, one may write down the posterior probability for the true means, covariances and cluster probabilities ($\\theta$) given the data ($x$), but it is very hard to calculate the parameters from it. Let us call the cluster that a specific sample belongs to, $z$.\n",
|
||||
"\n",
|
||||
"The true posterior would look like this:\n",
|
||||
"\n",
|
||||
"$p(\\theta|x) = \\frac{p(x|\\theta) p(\\theta)}{p(x)}$\n",
|
||||
"\n",
|
||||
"We assume $p(\\theta)$ is constant for all $\\theta$ and we know $p(x)$ is independent of $\\theta$ (because it is just the probability we obtained the existing data). We also know that $p(x|\\theta) = \\sum_z p(x,z|\\theta)$, since summing over all $z$, we obtain a total probability 1 of getting any $z$.\n",
|
||||
"\n",
|
||||
"$p(\\theta|x) \\propto \\sum_z p(x,z|\\theta) = \\sum_z p(x|\\theta,z) p(z|\\theta)$\n",
|
||||
"\n",
|
||||
"Where we have expanded it on $z$ using $p(a,b) = p(a|b)p(b)$.\n",
|
||||
"\n",
|
||||
"Now if we want to find the most probable $\\theta$ from this posterior (the \"maximum a posteriori\", or \"MAP\"), we just need to maximize the right-hand side of the last equation. This is very hard to do, but we can use the Expectation-Maximization (EM) algorithm, on which we iteratively improve our probability estimates. This method works as follows.\n",
|
||||
"\n",
|
||||
"First we define the log-likelihood $LL(\\theta|z) = \\log L(\\theta|z) = \\log p(x|\\theta,z) = \\sum_{k=\\text{sample}} \\log p(x_k|\\theta,z_k)$. Since we are assuming that the probability for each data sample $x_k$ is a Gaussian distribution, each of the terms in the sum is a Gaussian probability, so if we know $\\theta$ and $z_k$, this full sum can be calculated. The issue is that we do not know $z_k$ (that is the whole point!).\n",
|
||||
"\n",
|
||||
"We start by assuming $\\theta=\\theta_0$ for some random initial value of the parameters. The EM method iterates on two steps:\n",
|
||||
"\n",
|
||||
" 1. Calculate the *average* value of $LL(\\theta|z)$ over $z$ assuming the current $\\theta_i$ to calculate $p(z|\\theta)$, that is: $Q(\\theta) = \\sum_z LL(\\theta|z) p(z|\\theta_i)$. This means we calculate the weighted sum of the log-likelihood a point belongs to each Gaussian, weighted by the probability that that Gaussian was the reason that sample was generated. This avoids needing to know the correct $z_k$, since we just sum over all possibilities. The weights of this weighted sum are simply one of the parameters $\\theta$, namely, the cluster probabilities.\n",
|
||||
" \n",
|
||||
" 2. Find the $\\theta$ which maximizes $Q(\\theta)$ and use this as the next $\\theta_{i+1}$.\n",
|
||||
"\n",
|
||||
"These two steps are iterated to improve the $\\theta$ for several iterations. It can be shown that an improvement on $Q$ following this procedure leads to a new $\\theta$ that improves the posterior probability $p(\\theta|x)$ above (see https://en.wikipedia.org/wiki/Expectation%E2%80%93maximization_algorithm#Proof_of_correctness ).\n",
|
||||
"\n",
|
||||
"We will not write all of this from scratch. Instead, we use the `GaussianMixture` function in `scikit-learn`, which has this ready for us. It is nevertheless important to understand the assumptions made here: the underlying samples are assumed to come from a discrete combination of Gaussians. Other mixture models are possible using other underlying distributions, but note that this method will not work if the samples do not follow this generative pattern."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 9,
|
||||
"id": "0837b3ff",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"gmm = GaussianMixture(n_components=3, covariance_type=\"full\", max_iter=20)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"id": "8798f857",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/html": [
|
||||
"<style>#sk-container-id-1 {color: black;background-color: white;}#sk-container-id-1 pre{padding: 0;}#sk-container-id-1 div.sk-toggleable {background-color: white;}#sk-container-id-1 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-1 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-1 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-1 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-1 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-1 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-1 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-1 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-1 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-1 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-1 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-1 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-1 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-1 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-1 div.sk-item {position: relative;z-index: 1;}#sk-container-id-1 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-1 div.sk-item::before, #sk-container-id-1 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-1 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-1 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-1 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-1 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-1 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-1 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-1 div.sk-label-container {text-align: center;}#sk-container-id-1 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-1 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-1\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>GaussianMixture(max_iter=20, n_components=3)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-1\" type=\"checkbox\" checked><label for=\"sk-estimator-id-1\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">GaussianMixture</label><div class=\"sk-toggleable__content\"><pre>GaussianMixture(max_iter=20, n_components=3)</pre></div></div></div></div></div>"
|
||||
],
|
||||
"text/plain": [
|
||||
"GaussianMixture(max_iter=20, n_components=3)"
|
||||
]
|
||||
},
|
||||
"execution_count": 10,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"gmm.fit(data.loc[:, [\"x\", \"y\"]])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "e928f498",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Now that we have model fit, we can even check the fit parameters $\\theta$, which include the means, the covariance matrices and the probabilities given to each Gaussian in the mixture:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 11,
|
||||
"id": "fb5796e5",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"array([[ 0.99087075, 5.04466165],\n",
|
||||
" [-5.06356078, -0.94701357],\n",
|
||||
" [ 4.97867027, -1.98953164]])"
|
||||
]
|
||||
},
|
||||
"execution_count": 11,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"gmm.means_"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 12,
|
||||
"id": "182904d1",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"array([[[ 9.80327267e-01, 4.34733962e-01],\n",
|
||||
" [ 4.34733962e-01, 9.46532120e-01]],\n",
|
||||
"\n",
|
||||
" [[ 1.94905096e+00, -1.15878394e-03],\n",
|
||||
" [-1.15878394e-03, 4.73462596e+00]],\n",
|
||||
"\n",
|
||||
" [[ 3.86398255e-01, 1.99692091e-01],\n",
|
||||
" [ 1.99692091e-01, 3.77042067e-01]]])"
|
||||
]
|
||||
},
|
||||
"execution_count": 12,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"gmm.covariances_"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 13,
|
||||
"id": "2faa1f72",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"array([0.33400535, 0.33266132, 0.33333333])"
|
||||
]
|
||||
},
|
||||
"execution_count": 13,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"gmm.weights_"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "b54001fc",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We can predict to which cluster each sample belongs now by selecting the cluster for each sample that maximizes the probability $p(z|\\theta,x)$, now that we know $\\theta$."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 14,
|
||||
"id": "cc8fc1f1",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"guess = gmm.predict(data.loc[:, [\"x\", \"y\"]])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "68560fb4",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Let's plot it!"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 15,
|
||||
"id": "88982b21",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"data.loc[:, \"guess\"] = guess"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 16,
|
||||
"id": "333581b5",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"image/png": "iVBORw0KGgoAAAANSUhEUgAAA3oAAANXCAYAAABjYS2QAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAEAAElEQVR4nOzdd3hT5dvA8e85SfektIWWWfYesjfIRhEQFXAB4kZRcf9cOHG8Kk5wgnuwFdl77733LqWF7p2c5/2jNFCatAXaJm3vj1cu7Fm5E2ju3OdZmlJKIYQQQgghhBCi1NCdHYAQQgghhBBCiMIlhZ4QQgghhBBClDJS6AkhhBBCCCFEKSOFnhBCCCGEEEKUMlLoCSGEEEIIIUQpI4WeEEIIIYQQQpQyUugJIYQQQgghRCkjhZ4QQgghhBBClDJS6AkhhBBCCCFEKSOFnhAFcPz4cTRNY8qUKc4ORQghhCgS1atXZ8SIEYV6zREjRlC9evVCvaYQomCk0BP5OnbsGE888QR16tTB29sbb29vGjRowOjRo9m5c6ezwxNCCFEGaJpWoMfy5cudHWqZc/bsWcaNG8f27dudHYoQ4gpmZwcgXNucOXMYMmQIZrOZe+65h6ZNm6LrOvv372fGjBlMnDiRY8eOUa1aNWeHKoQQohT75Zdfcvz8888/s2jRolzb69evX5xhlSoHDhxA16+9DeDs2bO8+eabVK9enWbNmuXY991332EYRiFFKIS4FlLoCYeOHDnC0KFDqVatGkuWLCEsLCzH/g8++ICvv/76upKCKFqGYZCRkYGnp6ezQxFCiEJx77335vh5/fr1LFq0KNf2q6WkpODt7V2UoZUaHh4ehX5NNze3Qr+mEKJg5Bu6cOjDDz8kOTmZyZMn5yryAMxmM2PGjKFKlSq2bV27dqVr1665jrXXR98wDCZMmEDDhg3x9PSkQoUKPPLII8TGxuY4bvPmzfTu3Zvg4GC8vLyIiIjggQceyHHMn3/+SYsWLfDz88Pf35/GjRvz2Wef5TgmLi6Op59+mipVquDh4UGtWrX44IMPct1pjIuLY8SIEQQEBBAYGMjw4cOJi4srwDsGmZmZvPnmm9SuXRtPT0/Kly9Px44dWbRoUY7jli5dSqdOnfDx8SEwMJABAwawb9++fN8zgHHjxqFpWo5tmqbxxBNP8Ntvv9GwYUM8PDyYP38+AGfOnGHUqFGEh4fj4eFBREQEjz32GBkZGdf83gghhCvr2rUrjRo1YsuWLXTu3Blvb2/+97//AVmfk+PGjct1jr1xaTfymTh79mxuueUW22duzZo1efvtt7FarTmOO3ToEIMHD6ZixYp4enpSuXJlhg4dSnx8vO2YRYsW0bFjRwIDA/H19aVu3bq215Pt/PnzjBo1igoVKuDp6UnTpk356aefcsVlGAafffYZjRs3xtPTk5CQEPr06cPmzZsdvhcXL17kueeeo3Hjxvj6+uLv70/fvn3ZsWOH7Zjly5fTqlUrAEaOHGnrQps9pt1eLktOTubZZ5+1vb9169bl//7v/1BK5TguO7fNmjWLRo0a4eHhQcOGDW35TQiRN2nREw7NmTOHWrVq0aZNmyK5/iOPPMKUKVMYOXIkY8aM4dixY3z55Zds27aNNWvW4Obmxvnz5+nVqxchISG89NJLBAYGcvz4cWbMmGG7zqJFixg2bBjdu3fngw8+AGDfvn2sWbOGp556Csi6o9ulSxfOnDnDI488QtWqVVm7di0vv/wykZGRTJgwAQClFAMGDGD16tU8+uij1K9fn5kzZzJ8+PACvaZx48Yxfvx4HnzwQVq3bk1CQgKbN29m69at9OzZE4DFixfTt29fatSowbhx40hNTeWLL76gQ4cObN269boHrS9dupS///6bJ554guDgYKpXr87Zs2dp3bo1cXFxPPzww9SrV48zZ84wbdo0UlJScHd3L/B7I4QQJcGFCxfo27cvQ4cO5d5776VChQrXdP6NfiZOmTIFX19fxo4di6+vL0uXLuX1118nISGBjz76CICMjAx69+5Neno6Tz75JBUrVuTMmTPMmTOHuLg4AgIC2LNnD7feeitNmjThrbfewsPDg8OHD7NmzRrbc6WmptK1a1cOHz7ME088QUREBFOnTmXEiBHExcXZciDAqFGjmDJlCn379uXBBx/EYrGwatUq1q9fT8uWLe2+lqNHjzJr1izuvPNOIiIiiIqK4ptvvqFLly7s3buX8PBw6tevz1tvvcXrr7/Oww8/TKdOnQBo37693WsqpbjttttYtmwZo0aNolmzZixYsIDnn3+eM2fO8Omnn+Y4fvXq1cyYMYPHH38cPz8/Pv/8cwYPHszJkycpX758vn+fQpRpSgg74uPjFaAGDhyYa19sbKyKjo62PVJSUmz7unTporp06ZLrnOHDh6tq1arZfl61apUC1G+//ZbjuPnz5+fYPnPmTAWoTZs2OYz1qaeeUv7+/spisTg85u2331Y+Pj7q4MGDOba/9NJLymQyqZMnTyqllJo1a5YC1Icffmg7xmKxqE6dOilATZ482eFzKKVU06ZN1S233JLnMc2aNVOhoaHqwoULtm07duxQuq6r+++/37bt6vcs2xtvvKGu/tUFlK7ras+ePTm233///UrXdbvvn2EYSqmCvzdCCOFKRo8eneuzsEuXLgpQkyZNynU8oN54441c26tVq6aGDx9u+/lGPxOvzInZHnnkEeXt7a3S0tKUUkpt27ZNAWrq1KkOr/Ppp58qQEVHRzs8ZsKECQpQv/76q21bRkaGateunfL19VUJCQlKKaWWLl2qADVmzJhc18jOBUrlfi/S0tKU1WrNcfyxY8eUh4eHeuutt2zbNm3a5DBHXp3LsvPsO++8k+O4O+64Q2mapg4fPmzbBih3d/cc23bs2KEA9cUXXzh4V4QQ2aTrprArISEBAF9f31z7unbtSkhIiO3x1VdfXfP1p06dSkBAAD179iQmJsb2aNGiBb6+vixbtgyAwMBAIKt1MTMz0+61AgMDSU5OztU98urn69SpE+XKlcvxfD169MBqtbJy5UoA5s6di9ls5rHHHrOdazKZePLJJwv0ugIDA9mzZw+HDh2yuz8yMpLt27czYsQIgoKCbNubNGlCz549mTt3boGex54uXbrQoEED28+GYTBr1iz69+9v925tdvfPgr43QghREnh4eDBy5MjrPv9GPxO9vLxs/5+YmEhMTAydOnUiJSWF/fv3AxAQEADAggULSElJsXud7Pw3e/Zsh11G586dS8WKFRk2bJhtm5ubG2PGjCEpKYkVK1YAMH36dDRN44033sh1jauHAlzJw8PDNg7farVy4cIFWxfSrVu3OjwvL3PnzsVkMjFmzJgc25999lmUUsybNy/H9h49elCzZk3bz02aNMHf35+jR49e1/MLUZZIoSfs8vPzAyApKSnXvm+++YZFixbx66+/Xvf1Dx06RHx8PKGhoTmKxpCQEJKSkjh//jyQVbwMHjyYN998k+DgYAYMGMDkyZNJT0+3Xevxxx+nTp069O3bl8qVK/PAAw/k6r9/6NAh5s+fn+u5evToAWB7vhMnThAWFparwK1bt26BXtdbb71FXFwcderUoXHjxjz//PM5lqA4ceKEw+vVr1+fmJgYkpOTC/RcV4uIiMjxc3R0NAkJCTRq1CjP8wr63gghRElQqVIl3N3dr/v8G/1M3LNnD4MGDSIgIAB/f39CQkJsE8Zkj7+LiIhg7NixfP/99wQHB9O7d2+++uqrHOPzhgwZQocOHXjwwQepUKECQ4cO5e+//85R9J04cYLatWvnmhQte+bR7Jxz5MgRwsPDc9xgLAjDMPj000+pXbs2Hh4eBAcHExISws6dO3PEei1OnDhBeHi47XuGo5izVa1aNdc1ypUrl2s8vxAiNxmjJ+wKCAggLCyM3bt359qXPWbv+PHjufZpmpZrMDWQaxC6YRiEhoby22+/2X3+kJAQ2/WmTZvG+vXr+ffff1mwYAEPPPAAH3/8MevXr8fX15fQ0FC2b9/OggULmDdvHvPmzWPy5Mncf//9tgHphmHQs2dPXnjhBbvPV6dOHcdvxjXo3LkzR44cYfbs2SxcuJDvv/+eTz/9lEmTJvHggw9e07Uc3WW9+r3MduVd5GtRXO+NEEIUh2v9LLSXn673MzEuLo4uXbrg7+/PW2+9Rc2aNfH09GTr1q28+OKLOYq0jz/+mBEjRtjyxZgxYxg/fjzr16+ncuXKeHl5sXLlSpYtW8Z///3H/Pnz+euvv7j55ptZuHAhJpPpml7n9Xjvvfd47bXXeOCBB3j77bcJCgpC13WefvrpYpusy9HrtPddQwiRkxR6wqFbbrmF77//no0bN9K6desCnVOuXDm73SmuvkNXs2ZNFi9eTIcOHQqUlNu2bUvbtm159913+f3337nnnnv4888/bcWTu7s7/fv3p3///hiGweOPP84333zDa6+9Rq1atahZsyZJSUm2O7KOZC8lkZSUlKNV78CBAwV5+QAEBQUxcuRIRo4cSVJSEp07d2bcuHE8+OCDtvUG7V1v//79BAcH4+PjA2S9l/Zm+7z6vXQkJCQEf39/u8X6lQr63gghRElm7zM1IyODyMjIHNtu5DNx+fLlXLhwgRkzZtC5c2fb9mPHjtk9vnHjxjRu3JhXX32VtWvX0qFDByZNmsQ777wDgK7rdO/ene7du/PJJ5/w3nvv8corr7Bs2TJ69OhBtWrV2LlzJ4Zh5GjVy+4imp1zatasyYIFC7h48eI1tepNmzaNbt268cMPP+TYHhcXR3BwsO3nvLp/Xq1atWosXryYxMTEHK16V8cshLhx0nVTOPTCCy/g7e3NAw88QFRUVK799u6m1axZk/379xMdHW3btmPHjhyzhAHcddddWK1W3n777VzXsFgstmQcGxub63myF2PN7r554cKFHPt1XadJkyY5jrnrrrtYt24dCxYsyPV8cXFxWCwLine truncated
|
||||
"text/plain": [
|
||||
"<Figure size 1000x1000 with 4 Axes>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"fig, ax = plt.subplots(figsize=(10, 10), ncols=2)\n",
|
||||
"data.plot.scatter(x=\"x\", y=\"y\", c=\"guess\", colormap='viridis', ax=ax[0])\n",
|
||||
"data.plot.scatter(x=\"x\", y=\"y\", c=\"source\", colormap='viridis', ax=ax[1])\n",
|
||||
"ax[0].set(xlabel=\"x\", ylabel=r\"y\", title=\"Guessed source\")\n",
|
||||
"ax[1].set(xlabel=\"x\", ylabel=r\"y\", title=\"True association\")\n",
|
||||
"plt.show()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "9b1363ec",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Note that if the sample clusters were not \"blobs\" of data, but were in concentric circles, the assumption of this method would be false and the method would simply not work well. This is why it is important to understand the underlying assumptions made in the method."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "fbee128c",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Determining the number of clusters\n",
|
||||
"\n",
|
||||
"So far, we have assumed that the number of clusters is known. This is not often the case. One method to determine the number of clusters is to fit the GMM with different number of clusters and after each fit, calculate `gmm.bic()`. This method returns the Bayesian Information Criteria (see also AIC), which should be small when the fit worked well and the number of components is compatible with your sample.\n",
|
||||
"\n",
|
||||
"See a full example here: https://scikit-learn.org/stable/auto_examples/mixture/plot_gmm_selection.html#sphx-glr-auto-examples-mixture-plot-gmm-selection-py"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "7076f779",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## K-Means\n",
|
||||
"\n",
|
||||
"Another common method used for clustering is the K-Means, on which one simply tries to find the cluster centers which minimize in-cluster distances (in an Euclidean sense) while maximizing distances between the centers. It can be shown that this method is a special case of the Gaussian Mixture Model when the covariance matrices are diagonal, which would mean that within each blob, there is no correlation between the variables (see https://en.wikipedia.org/wiki/K-means_clustering#Gaussian_mixture_model and references).\n",
|
||||
"\n",
|
||||
"While this is an approximation of the GMM model, it is still a very useful approach, since there are faster algorithms to achieve the clustering than for GMMs.\n",
|
||||
"\n",
|
||||
"The scikit-learn module also provides an easy-to-use implementation of this algorithm:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 17,
|
||||
"id": "2f280e1d",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"/home/daniloefl/workspace/ml-lecture-problems/ml/lib/python3.10/site-packages/sklearn/cluster/_kmeans.py:870: FutureWarning: The default value of `n_init` will change from 10 to 'auto' in 1.4. Set the value of `n_init` explicitly to suppress the warning\n",
|
||||
" warnings.warn(\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/html": [
|
||||
"<style>#sk-container-id-2 {color: black;background-color: white;}#sk-container-id-2 pre{padding: 0;}#sk-container-id-2 div.sk-toggleable {background-color: white;}#sk-container-id-2 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-2 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-2 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-2 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-2 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-2 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-2 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-2 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-2 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-2 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-2 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-2 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-2 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-2 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-2 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-2 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-2 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-2 div.sk-item {position: relative;z-index: 1;}#sk-container-id-2 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-2 div.sk-item::before, #sk-container-id-2 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-2 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-2 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-2 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-2 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-2 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-2 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-2 div.sk-label-container {text-align: center;}#sk-container-id-2 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-2 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-2\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>KMeans(n_clusters=3)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-2\" type=\"checkbox\" checked><label for=\"sk-estimator-id-2\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">KMeans</label><div class=\"sk-toggleable__content\"><pre>KMeans(n_clusters=3)</pre></div></div></div></div></div>"
|
||||
],
|
||||
"text/plain": [
|
||||
"KMeans(n_clusters=3)"
|
||||
]
|
||||
},
|
||||
"execution_count": 17,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"kmeans = KMeans(n_clusters=3)\n",
|
||||
"kmeans.fit(data.loc[:, [\"x\", \"y\"]])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 18,
|
||||
"id": "950a7eec",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"data.loc[:, \"guess_kmeans\"] = kmeans.predict(data.loc[:, [\"x\", \"y\"]])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 19,
|
||||
"id": "16a56489",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"image/png": "iVBORw0KGgoAAAANSUhEUgAAA3oAAANXCAYAAABjYS2QAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAEAAElEQVR4nOzdd3xUZdbA8d+dSTLpCSGFhIQQQu+9VwEREUVRQVEBce269tVXXbGva9e1rQVcFUWqgvTepfeWQGghnfQ+M8/7R8hIyEwygSQzSc53P/Nxc1vOXGDOnPs0TSmlEEIIIYQQQghRb+gcHYAQQgghhBBCiOolhZ4QQgghhBBC1DNS6AkhhBBCCCFEPSOFnhBCCCGEEELUM1LoCSGEEEIIIUQ9I4WeEEIIIYQQQtQzUugJIYQQQgghRD0jhZ4QQgghhBBC1DNS6AkhhBBCCCFEPSOFnqiSU6dOoWkaM2fOdHQo9dr06dPRNM3RYQghhGhAmjdvzpQpU6r1mlOmTKF58+bVek0hhH2k0KuCuLg4Hn30UVq3bo2npyeenp60b9+eRx55hP379zs6PCGqxZQpU/D29i63ff/+/QQGBtK8eXNOnTpl81xN0/D19SU/P7/c/piYGDRNQ9M03nvvveoOXQhRj5V+dlT2WrdunaNDbXDOnz/P9OnT2bt3r6NDEUJcwsXRAdQVixcvZsKECbi4uDBp0iS6dOmCTqfj6NGjzJ8/ny+++IK4uDgiIyMdHaqoB1566SWef/55R4dhcfDgQYYPH46Xlxdr166t8Omsi4sLeXl5LFq0iNtvv73Mvp9++gl3d3cKCgpqOGIhRH3zww8/lPn5f//7HytXriy3vV27drUZVr1y7NgxdLqqtwGcP3+eV199lebNm9O1a9cy+77++mvMZnM1RSiEqAop9Oxw4sQJJk6cSGRkJKtXryY0NLTM/nfeeYfPP//8ij4cRc0ym80UFRXh7u7u6FCqxMXFBRcX5/jneejQIa655ho8PDxYu3YtUVFRFR5vMBgYMGAAP//8c7lCb9asWYwZM4Z58+bVZMhCiHrorrvuKvPztm3bWLlyZbntl8vLy8PT07MmQ6s3DAZDtV/T1dW12q8phLCPVCZ2+Pe//01ubi4zZswoV+RByZfyxx9/nIiICMu2oUOHMnTo0HLHWuurbjab+eijj+jQoQPu7u6EhITwwAMPkJ6eXua4nTt3MmrUKAIDA/Hw8CAqKop77723zDG//PILPXr0wMfHB19fXzp16sTHH39c5piMjAyeeOIJIiIiMBgMtGzZknfeeafcE7eMjAymTJmCn58f/v7+TJ48mYyMDDvuGBQXF/Pqq6/SqlUr3N3dady4MQMHDmTlypVljluzZg2DBg3Cy8sLf39/brrpJo4cOVLpPQPr49g0TePRRx/lp59+okOHDhgMBpYtWwZAfHw806ZNIywsDIPBQFRUFA899BBFRUVVvjfWaJrG9OnTy22/fMyDPfemove2cOFCOnbsiMFgoEOHDpb3d6l169bRs2dP3N3diY6O5quvvrqicX9Hjhxh+PDhGAwG1q5dS4sWLew6784772Tp0qVl/r7s2LGDmJgY7rzzTqvn2Hvv33vvPfr370/jxo3x8PCgR48ezJ07t9z17L1f2dnZPPHEEzRv3hyDwUBwcDAjR45k9+7ddr1XIYTzGDp0KB07dmTXrl0MHjwYT09P/u///g+w/zMari4X/Pbbb4wZM8aSa6Kjo3n99dcxmUxljouJiWH8+PE0adIEd3d3wsPDmThxIpmZmZZjVq5cycCBA/H398fb25s2bdpY3k+p5ORkpk2bRkhICO7u7nTp0oXvv/++XFxms5mPP/6YTp064e7uTlBQENdddx07d+60eS8uXLjAM888Q6dOnfD29sbX15fRo0ezb98+yzHr1q2jV69eAEydOtXShbZ0LL+1HJ6bm8vTTz9tub9t2rThvffeQylV5riq5D0hRHnO0WTg5BYvXkzLli3p06dPjVz/gQceYObMmUydOpXHH3+cuLg4/vOf/7Bnzx42b96Mq6srycnJXHvttQQFBfH888/j7+/PqVOnmD9/vuU6K1eu5I477mD48OG88847QMkX9c2bN/P3v/8dKHmyOWTIEOLj43nggQdo1qwZW7Zs4YUXXiAhIYGPPvoIAKUUN910E5s2beLBBx+kXbt2LFiwgMmTJ9v1nqZPn87bb7/NfffdR+/evcnKymLnzp3s3r2bkSNHArBq1SpGjx5NixYtmD59Ovn5+Xz66acMGDCA3bt3X/Hg7TVr1vDrr7/y6KOPWsaUnT9/nt69e5ORkcH9999P27ZtiY+PZ+7cueTl5eHm5mb3vbla9twbWzZt2sT8+fN5+OGH8fHx4ZNPPmH8+PGcOXOGxo0bA7Bnzx6uu+46QkNDefXVVzGZTLz22msEBQVVKc5jx45xzTXX4OLiwtq1a4mOjrb73FtuuYUHH3yQ+fPnWx5GzJo1i7Zt29K9e/dyx1fl3n/88cfceOONTJo0iaKiIn755Rduu+02Fi9ezJgxY6p8vx588EHmzp3Lo48+Svv27UlLS2PTpk0cOXLEaqxCCOeWlpbG6NGjmThxInfddRchISFVOv9qc8HMmTPx9vbmqaeewtvbmzVr1vDPf/6TrKws3n33XQCKiooYNWoUhYWFPPbYYzRp0oT4+HgWL15MRkYGfn5+HDp0iBtuuIHOnTvz2muvYTAYiI2NZfPmzZbflZ+fz9ChQ4mNjeXRRx8lKiqKOXPmMGXKFDIyMiy5H2DatGnMnDmT0aNHc99992E0Gtm4cSPbtm2jZ8+eVt/LyZMnWbhwIbfddhtRUVEkJSXx1VdfMWTIEA4fPkxYWBjt2rXjtdde45///Cf3338/gwYNAqB///5Wr6mU4sYbb2Tt2rVMmzaNrl27snz5cp599lni4+P58MMPyxxvz+e4EMIGJSqUmZmpADVu3Lhy+9LT01VKSorllZeXZ9k3ZMgQNWTIkHLnTJ48WUVGRlp+3rhxowLUTz/9VOa4ZcuWldm+YMECBagdO3bYjPXvf/+78vX1VUaj0eYxr7/+uvLy8lLHjx8vs/35559Xer1enTlzRiml1MKFCxWg/v3vf1uOMRqNatCgQQpQM2bMsPk7lFKqS5cuasyYMRUe07VrVxUcHKzS0tIs2/bt26d0Op265557LNsuv2elXnnlFXX5X2FA6XQ6dejQoTLb77nnHqXT6azeP7PZrJSy/97YAqhXXnml3PbIyEg1efJky8/23Btb783NzU3FxsZatu3bt08B6tNPP7VsGzt2rPL09FTx8fGWbTExMcrFxaXcNa2ZPHmycnV1VaGhoSosLKzc/ajsXC8vL6WUUrfeeqsaPny4Ukopk8mkmjRpol599VUVFxenAPXuu+9azqvKvb/035lSShUVFamOHTuqa665psx2e++Xn5+feuSRR+x+j0II5/DII4+U+0wbMmSIAtSXX35Z7nh7P6OvNhdc/hmllFIPPPCA8vT0VAUFBUoppfbs2aMANWfOHJvX+fDDDxWgUlJSbB7z0UcfKUD9+OOPlm1FRUWqX79+ytvbW2VlZSmllFqzZo0C1OOPP17uGqU5UKny96KgoECZTKYyx8fFxSmDwaBee+01y7YdO3bY/G5weQ4v/X7xxhtvlDnu1ltvVZqmlfnMtvdzXAhhnXTdrERWVhaA1VkIhw4dSlBQkOX12WefVfn6c+bMwc/Pj5EjR5Kammp59ejRA29vb9auXQuAv78/UNK6WFxcbPVa/v7+5ObmluseefnvGzRoEI0aNSrz+0aMGIHJZGLDhg0ALFmyBBcXFx566CHLuXq9nscee8yu9+Xv78+hQ4eIiYmxuj8hIYG9e/cyZcoUAgICLNs7d+7MyJEjWbJkiV2/x5ohQ4bQvn17y89ms5mFCxcyduxYq08tS7sz2ntvrlZl96YiI0aMKNOy1rlzZ3x9fTl58iQAJpOJVatWMW7cOMLCwizHtWzZktGjR9v9e0wmE6mpqQQEBBAYGFjlOKGk++a6detITExkzZo1JCYm2uy2WZV77+HhYfn/6enpZGZmMmjQIKtdLSu7X1Dy5/Hnn39y/vz5K3qfQgjnYjAYmDp16hWff7W54NLPqOzsbFJTUxk0aBB5eXkcPXoUAD8/PwCWL19OXl6e1euU5v3ffvvNZpfRJUuW0KRJE+644w7LNldXVx5//HFycnJYv349APPmzUPTNF555ZVy16ioS7/BYLDMP2AymUhLS7N0Ib3S7u1LlixBr9fz+OOPl9n+9NNPo5Ri6dKlZbbb8zkuhLBOCr1K+Pj4AJCTk1Nu31dffcXKlSv58ccfr/j6MTExZGZmEhwcXKZoDAoKIicnh+TkZKCkeBk/fjyvvvoqgYGB3HTTTcyYMYPCwkLLtR5++GFat27N6NGjCQ8P59577y3Xjz0mJoZly5aV+10jRowAsPy+06dPExoaWq7AbdOmjV3v67XXXiMjI4PWrVvTqVMnnn322TJLUJw+fdrm9dq1a0dqaiq5ubl2/a7LXT5ZSEpKCllZWXTs2LHC8+y9N1ersntTkWbNmpXb1qhRI8t4zuTkZPLz82nZsmW546xts8XDw4P//e9/HD58mDFjxpT5s8jPzycxMbHMy5rrr78eHx8fZs+ezU8//USvXr1sxlCVe7948WL69u2Lu7s7AQEBBAUF8cUXX5QZ11KqsvsFJWNwDx48SEREBL1792b69OnyBUKIOqxp06a4ubld8flXmwsOHTrEzTffjJ+fH76+vgQFBVkmjCn9nIqKiuKpp57im2++ITAwkFGjRvHZZ5+V+RybMGECAwYM4L777iMkJISJEyfy66+/lin6Tp8+TatWrcpNBlc682hprj1x4gRhYWFlHqzaw2w28+GHH9KqVSsMBgOBgYEEBQWxf/9+q5+59jh9+jRhYWGW71e2Yi5lz+e4EMI6GaNXCT8/P0JLine truncated
|
||||
"text/plain": [
|
||||
"<Figure size 1000x1000 with 4 Axes>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"fig, ax = plt.subplots(figsize=(10, 10), ncols=2)\n",
|
||||
"data.plot.scatter(x=\"x\", y=\"y\", c=\"guess_kmeans\", colormap='viridis', ax=ax[0])\n",
|
||||
"data.plot.scatter(x=\"x\", y=\"y\", c=\"source\", colormap='viridis', ax=ax[1])\n",
|
||||
"ax[0].set(xlabel=\"x\", ylabel=r\"y\", title=\"Guessed source using K-Means\")\n",
|
||||
"ax[1].set(xlabel=\"x\", ylabel=r\"y\", title=\"True association\")\n",
|
||||
"plt.show()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "2fe3cad2",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Variational Inference on Gaussian Mixture Models\n",
|
||||
"\n",
|
||||
"While the Gaussian Mixture Model presented beforehand had a very strong theoretical background, it still assumes that there is a single correct value for the Gaussian parameters. This may not be the case if the Gaussian model is only approximate (as is very often the case!).\n",
|
||||
"\n",
|
||||
"One improvement to the Gaussian Mixture Models to allow for the usage of prior knowledge (a prior) on the parameters to be discovered. This helps us, for instance, to determine the ideal number of components automatically, since the number of components itself can be fit with it (assuming a prior distribution on them).\n",
|
||||
"\n",
|
||||
"Optimizing this model becomes even more complicated as the derivation shown before and if we wanted to be fully general, we would need to use very slow algorithms, such as Monte-Carlo sampling to obtain uncertainties with the least amount of extra assumptions. This is often undesirable, since we need fast clustering and often Monte-Carlo sampling is very slow with even more data! An alternative is to assume some underlying prior probability for the means and covariances and find those parameters as well in an optimization algorithm. This is what is done in Variational Inference. Further details can be seen in Bishop (2006), or in a more practical approach, here: https://scikit-learn.org/stable/modules/mixture.html#bgmm\n",
|
||||
"\n",
|
||||
"We can also easily use this method in our toy data, taking the code from scikit-learn:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 20,
|
||||
"id": "0a8642bd",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"bgmm = BayesianGaussianMixture(n_components=3)\n",
|
||||
"bgmm.fit(data.loc[:, [\"x\", \"y\"]])\n",
|
||||
"data.loc[:, \"guess_bgmm\"] = bgmm.predict(data.loc[:, [\"x\", \"y\"]])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 21,
|
||||
"id": "ccbf4019",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"image/png": "iVBORw0KGgoAAAANSUhEUgAAA38AAANXCAYAAACFSObUAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAEAAElEQVR4nOzdd3hT5dvA8e856d5AW9oyS9kb2Uu2bAURZSjLgYqi4njFiYr6c4ITcOHEASi42HsKyN6r7EKhe4/kef8ojYSkbYC2Sdr745ULc85zTu6k7Xlyn2dpSimFEEIIIYQQQogyTXd0AEIIIYQQQgghSp4kf0IIIYQQQghRDkjyJ4QQQgghhBDlgCR/QgghhBBCCFEOSPInhBBCCCGEEOWAJH9CCCGEEEIIUQ5I8ieEEEIIIYQQ5YAkf0IIIYQQQghRDkjyJ4QQQgghhBDlgCR/1+nEiRNomsbXX3/t6FDKtClTpqBpmqPDKHEl8T5Xr16NpmmsXr26WM9bGrZu3UqHDh3w9fVF0zR27tzp6JCEEKLcqlmzJmPGjCnWc44ZM4aaNWsW6zmFEEW7ruQvOjqaRx55hLp16+Lj44OPjw8NGzZkwoQJ7N69u7hjFKJU5eTkEBwcTKdOnQoso5SiWrVq3HTTTaUYmW2ffvqp092E6Nq1K40bN76uY3Nychg6dCjx8fFMmzaN7777jho1ahRzhK7nWq67+TcTdF3n9OnTVudKTk7G29sbTdN45JFHzNvzb2ppmsbUqVNtxjFy5Eg0TcPPz69436AQ1yn/d7aohyveCHN1586dY8qUKXIDTwgn4natB/z555/cdddduLm5MXLkSJo1a4au6xw8eJBff/2VGTNmEB0dLV/WRLF44YUXePbZZ0v1Nd3d3Rk6dCizZs3i5MmTNn+X165dy5kzZ3jiiSeK5TVv5H1++umnBAcHW92Vvfnmm8nIyMDDw6MYIiw9x44d4+TJk3z++efcd999jg7HKVzvddfT05Mff/yRZ555xmL7r7/+WujreXl58eOPP/LCCy9YbE9LS2PhwoV4eXkVzxsTohh89913Fs+//fZbli1bZrW9QYMGpRlWmXLo0CF0/drbC86dO8crr7xCzZo1ad68ucW+zz//HJPJVEwRCiHsdU3J37Fjxxg2bBg1atRgxYoVhIeHW+x/6623+PTTT6/rAiFKlslkIjs72+W+tLm5ueHmds33KG7YyJEjmTlzJj/++KPNpGzOnDnous6wYcNu6HXS0tLw9fUtkfep67rL/bwBYmNjAQgKCiq2c+Z/zq7oRq67/fr1s5n8zZkzh/79+zN//nybr9mvXz9+/fVXdu3aRbNmzczbFy5cSHZ2Nn369GHlypXF8O6EuHF33323xfPNmzezbNkyq+1XS09Px8fHpyRDKzM8PT2L/Zzu7u7Ffk4hRNGuKUt7++23SUtLY/bs2VZfQCDvi/rEiROpVq2aeVvXrl3p2rWrVVlbfb1NJhPTp0+nUaNGeHl5UblyZcaPH09CQoJFuW3bttG7d2+Cg4Px9vYmMjKScePGWZT56aefaNmyJf7+/gQEBNCkSRM++OADizKJiYk8/vjjVKtWDU9PT2rXrs1bb71ldScqMTGRMWPGEBgYSFBQEKNHjyYxMdGOTyyvC9srr7xCnTp18PLyolKlSnTq1Illy5ZZlFu5ciWdO3fG19eXoKAgbrvtNg4cOFDkZwa2x4vld+f64YcfaNSoEZ6enixevBiAs2fPcu+99xIREYGnpyeRkZE89NBDZGdnX/NnY4umaUyZMsVq+9VjBuz5bAp7bwsWLKBx48Z4enrSqFEj8/u70urVq2nVqhVeXl5ERUUxa9Ysu8bXdezYkZo1azJnzhyrfTk5OcybN49u3boRERHB7t27GTNmDLVq1cLLy4uwsDDGjRtHXFycxXH5r7t//35GjBhBhQoVzF1LbcU0e/ZsunfvTmhoKJ6enjRs2JAZM2ZYfab79u1jzZo15q5N+X9vBY35mzt3Li1btsTb25vg4GDuvvtuzp49a1FmzJgx+Pn5cfbsWQYNGoSfnx8hISE89dRTGI3GQj+7gtjzcxszZgxdunQBYOjQoRbvB+DgwYPccccdVKxYES8vL1q1asXvv/9u8Tpff/01mqaxZs0aHn74YUJDQ6latap5/6JFi8x/a/7+/vTv3599+/Zd9/s3mUx88MEHNGnSBC8vL0JCQujTpw/btm2zKPf999+bP/eKFSsybNgwm10yr3Y91918I0aMYOfOnRw8eNC87fz586xcuZIRI0YU+Jrt27cnMjLS6vf/hx9+oE+fPlSsWLHIuIVwJvld0f/9919uvvlmfHx8eO655wD76yy4sbpx4cKF9O/f31z3RkVF8dprr1ldU44cOcKQIUMICwvDy8uLqlWrMmzYMJKSksxlli1bRqdOnQgKCsLPz4969eqZ30++2NhY7r33XipXroyXlxfNmjXjm2++sYrLnmvY1Z9FfHw8Tz31FE2aNMHPz4+AgAD69u3Lrl27zGVWr15N69atARg7dqy5jsofpmDrO01aWhpPPvmk+fOtV68e7777Lkopi3LX8j1ACGHpmpoa/vzzT2rXrk3btm1LJJjx48fz9ddfM3bsWCZOnEh0dDQff/wxO3bsYMOGDbi7uxMbG8stt9xCSEgIzz77LEFBQZw4ccKiG9OyZcsYPnw4PXr04K233gLgwIEDbNiwgcceewzIu+PXpUsXzp49y/jx46levTobN25k8uTJxMTEMH36dCBvbNdtt93G+vXrefDBB2nQoAG//fYbo0ePtus9TZkyhTfffJP77ruPNm3akJyczLZt29i+fTu9evUCYPny5fTt25datWoxZcoUMjIy+Oijj+jYsSPbt2+/7gHRK1eu5JdffuGRRx4hODiYmjVrcu7cOdq0aUNiYiIPPPAA9evX5+zZs8ybN4/09HQ8PDzs/mxulD2fTUHWr1/Pr7/+ysMPP4y/vz8ffvghQ4YM4dSpU1SqVAmAHTt20KdPH8LDw3nllVcwGo28+uqrhISEFBmbpmmMGDGCN954g3379tGoUSPzvsWLFxMfH8/IkSOBvN+348ePM3bsWMLCwti3bx+fffYZ+/btY/PmzVZJ3dChQ6lTpw5vvPGGVYV2pRkzZtCoUSNuvfVW3Nzc+OOPP3j44YcxmUxMmDABgOnTp/Poo4/i5+fH888/D0DlypULPGf+31fr1q158803uXDhAh988AEbNmxgx44dFq1tRqOR3r1707ZtW959912WL1/Oe++9R1RUFA899FCRn6EtRf3cxo8fT5UqVXjjjTeYOHEirVu3Nr+fffv20bFjR6pUqcKzzz6Lr68vv/zyC4MGDWL+/PkMHjzY4rUefvhhQkJCeOmll0hLSwPyuoeNHj2a3r1789Zbb5Gens6MGTPo1KkTO3bssPhbs/f933vvvXz99df07duX++67j9zcXNatW8fmzZtp1aoVAK+//jovvvgid955J/fddx8XL17ko48+4uabb7b63K92I9fdm2++mapVqzJnzhxeffVVAH7++Wf8/Pzo379/occOHz6c77//nv/9739omsalS5dYunQp3333nXzBEi4pLi6Ovn37MmzYMO6+++5Cr5W23Gjd+PXXX+Pn58ekSZPw8/Nj5cqVvPTSSyQnJ/POO+8AkJ2dTe/evcnKyuLRRx8lLCyMs2fP8ueff5KYmEhgYCD79u1jwIABNG3alFdffRVPT0+OHj3Khg0bzK+VkZFB165dOXr0KI888giRkZHMnTuXMWPGkJiYaP4uBPZdw652/PhxFixYwNChQ4mMjOTChQvMmjWLLl26sH//fiIiImjQoAGvvvoqL730Eg888ACdO3cGoEOHDjbPqZTi1ltvZdWqVdx77700b96cJUuW8PTTT3P27FmmTZtmUd6e7wFCCBuUnZKSkhSgBg0aZLUvISFBXbx40fxIT0837+vSpYvq0qWL1TGjR49WNWrUMD9ft26dAtQPP/xgUW7x4sUW23/77TcFqK1btxYY62OPPaYCAgJUbm5ugWVee+015evrqw4fPmyx/dlnn1UGg0GdOnVKKaXUggULFKDefvttc5nc3FzVuXNnBajZs2cX+BpKKdWsWTPVv3//Qss0b95chYaGqri4OPO2Xbt2KV3X1ahRo8zbrv7M8r388svq6h8loHRdV/v27bPYPmrUKKXrus3Pz2QyKaXs/2wKAqiXX37ZanuNGjXU6NGjzc/t+WwKem8eHh7q6NGj5m27du1SgProo4/M2wYOHKh8fHzU2bNnzduOHDmi3NzcrM5py759+xSgJk+ebLF92LBhysvLSyUlJSmllMXve74ff/xRAWrt2rVW72X48OF2vU9b5+3du7eqVauWxbZGjRrZ/BtbtWqVAtSqVauUUkplZ2er0NBQ1bhxY5WRkWEu9+effypAvfTSS+Zto0ePVoB69dVXLc7ZokUL1bJlS6vXulqXLl1Uo0aNLLbZ+3PLj3vu3LkWx/fo0UM1adJEZWZmmreZTCbVoUMHVadOHfO22bNnK0B16tTJ4hqQkpKigoKC1P33329x3vPnz6vAwECL7fa+/5UrVypATZw40eozyP97OnHihDIYDOr111+32L9nzx7l5uZmtf1K13vdzf99unjxonrqqadU7dq1zftat26txo4dq5TK+5lMmDDBvC86OloB6p133lF79+5VgFq3bp1SSqlPPvlE+fn5qbS0NDV69Gjl6+tbYNxCONKECROsrqddunRRgJo5c6ZVeXvrrButG21d08ePH698fHzM17UdO3bYvP5dadq0aea/74JMnz5dAer77783b8vOzlbt27dXfn5+Kjk5WSll3zVMKevPIjMzUxmNRovy0dHRytPT0+K6uXXr1gK/K139nSb/+9bUqVMtyt1xxx1K0zSLusPe+kQIYc3ubp/JyckANmd469q1KyEhIebHJ598Yu9pzebOnUtLine truncated
|
||||
"text/plain": [
|
||||
"<Figure size 1000x1000 with 4 Axes>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"fig, ax = plt.subplots(figsize=(10, 10), ncols=2)\n",
|
||||
"data.plot.scatter(x=\"x\", y=\"y\", c=\"guess_bgmm\", colormap='viridis', ax=ax[0])\n",
|
||||
"data.plot.scatter(x=\"x\", y=\"y\", c=\"source\", colormap='viridis', ax=ax[1])\n",
|
||||
"ax[0].set(xlabel=\"x\", ylabel=r\"y\", title=\"Guessed source using Variation Inference GMM\")\n",
|
||||
"ax[1].set(xlabel=\"x\", ylabel=r\"y\", title=\"True association\")\n",
|
||||
"plt.show()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "72508ead",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Other heuristic approaches\n",
|
||||
"\n",
|
||||
"Many other heuristic approaches for clustering have been developed. Some of them rely on a strong foundation in statistics, others take a more practical approach and make assumptions on how the data samples are expected to be distributed in the $xy$ scatter plot. The following methods deserve a look if you are not familiar with them already:\n",
|
||||
"\n",
|
||||
" * DBSCAN (see https://scikit-learn.org/stable/modules/generated/sklearn.cluster.DBSCAN.html ). Uses the euclidean distances between the samples to define if a sample belongs in a cluster or another. The assumption is related to GMMs. While the GMM requires fixing the number of clusters, DBSCAN relies on the maximum distance parameter do define which samples belong to a cluster.\n",
|
||||
" * Agglomerative clustering (see https://scikit-learn.org/stable/modules/generated/sklearn.cluster.AgglomerativeClustering.html#sklearn.cluster.AgglomerativeClustering). It uses the nearest neighbour to each sample to define which samples belong together in the same cluster. Since the cluster association is topological, it does not assume the samples make a \"blob\" of some sort, as in the case of GMM. One could for example have samples close to one another making a circle in the $xy$-scatter plane and those samples would belong to the same cluster (depending on the definition of closeness and other parameters of the algorithm).\n",
|
||||
" * Spectral clustering (see https://scikit-learn.org/stable/modules/clustering.html#spectral-clustering). Uses K-means on another representation of the data. This representation of the data transforms the data points from the $xy$ space shown before into a space where connectedness of nearest neighbours is the criteria for similarity. For the mathematical foundation, this document may be useful: https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.165.9323&rep=rep1&type=pdf\n",
|
||||
" "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "941c7cb8",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.10.6"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
@@ -1 +1,92 @@
|
||||
# ml-lecture-eiroforum
|
||||
|
||||
|
||||
|
||||
## Getting started
|
||||
|
||||
To make it easy for you to get started with GitLab, here's a list of recommended next steps.
|
||||
|
||||
Already a pro? Just edit this README.md and make it your own. Want to make it easy? [Use the template at the bottom](#editing-this-readme)!
|
||||
|
||||
## Add your files
|
||||
|
||||
- [ ] [Create](https://docs.gitlab.com/ee/user/project/repository/web_editor.html#create-a-file) or [upload](https://docs.gitlab.com/ee/user/project/repository/web_editor.html#upload-a-file) files
|
||||
- [ ] [Add files using the command line](https://docs.gitlab.com/ee/gitlab-basics/add-file.html#add-a-file-using-the-command-line) or push an existing Git repository with the following command:
|
||||
|
||||
```
|
||||
cd existing_repo
|
||||
git remote add origin https://git.xfel.eu/machineLearning/ml-lecture-eiroforum.git
|
||||
git branch -M main
|
||||
git push -uf origin main
|
||||
```
|
||||
|
||||
## Integrate with your tools
|
||||
|
||||
- [ ] [Set up project integrations](https://git.xfel.eu/machineLearning/ml-lecture-eiroforum/-/settings/integrations)
|
||||
|
||||
## Collaborate with your team
|
||||
|
||||
- [ ] [Invite team members and collaborators](https://docs.gitlab.com/ee/user/project/members/)
|
||||
- [ ] [Create a new merge request](https://docs.gitlab.com/ee/user/project/merge_requests/creating_merge_requests.html)
|
||||
- [ ] [Automatically close issues from merge requests](https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically)
|
||||
- [ ] [Enable merge request approvals](https://docs.gitlab.com/ee/user/project/merge_requests/approvals/)
|
||||
- [ ] [Automatically merge when pipeline succeeds](https://docs.gitlab.com/ee/user/project/merge_requests/merge_when_pipeline_succeeds.html)
|
||||
|
||||
## Test and Deploy
|
||||
|
||||
Use the built-in continuous integration in GitLab.
|
||||
|
||||
- [ ] [Get started with GitLab CI/CD](https://docs.gitlab.com/ee/ci/quick_start/index.html)
|
||||
- [ ] [Analyze your code for known vulnerabilities with Static Application Security Testing(SAST)](https://docs.gitlab.com/ee/user/application_security/sast/)
|
||||
- [ ] [Deploy to Kubernetes, Amazon EC2, or Amazon ECS using Auto Deploy](https://docs.gitlab.com/ee/topics/autodevops/requirements.html)
|
||||
- [ ] [Use pull-based deployments for improved Kubernetes management](https://docs.gitlab.com/ee/user/clusters/agent/)
|
||||
- [ ] [Set up protected environments](https://docs.gitlab.com/ee/ci/environments/protected_environments.html)
|
||||
|
||||
***
|
||||
|
||||
# Editing this README
|
||||
|
||||
When you're ready to make this README your own, just edit this file and use the handy template below (or feel free to structure it however you want - this is just a starting point!). Thank you to [makeareadme.com](https://www.makeareadme.com/) for this template.
|
||||
|
||||
## Suggestions for a good README
|
||||
Every project is different, so consider which of these sections apply to yours. The sections used in the template are suggestions for most open source projects. Also keep in mind that while a README can be too long and detailed, too long is better than too short. If you think your README is too long, consider utilizing another form of documentation rather than cutting out information.
|
||||
|
||||
## Name
|
||||
Choose a self-explaining name for your project.
|
||||
|
||||
## Description
|
||||
Let people know what your project can do specifically. Provide context and add a link to any reference visitors might be unfamiliar with. A list of Features or a Background subsection can also be added here. If there are alternatives to your project, this is a good place to list differentiating factors.
|
||||
|
||||
## Badges
|
||||
On some READMEs, you may see small images that convey metadata, such as whether or not all the tests are passing for the project. You can use Shields to add some to your README. Many services also have instructions for adding a badge.
|
||||
|
||||
## Visuals
|
||||
Depending on what you are making, it can be a good idea to include screenshots or even a video (you'll frequently see GIFs rather than actual videos). Tools like ttygif can help, but check out Asciinema for a more sophisticated method.
|
||||
|
||||
## Installation
|
||||
Within a particular ecosystem, there may be a common way of installing things, such as using Yarn, NuGet, or Homebrew. However, consider the possibility that whoever is reading your README is a novice and would like more guidance. Listing specific steps helps remove ambiguity and gets people to using your project as quickly as possible. If it only runs in a specific context like a particular programming language version or operating system or has dependencies that have to be installed manually, also add a Requirements subsection.
|
||||
|
||||
## Usage
|
||||
Use examples liberally, and show the expected output if you can. It's helpful to have inline the smallest example of usage that you can demonstrate, while providing links to more sophisticated examples if they are too long to reasonably include in the README.
|
||||
|
||||
## Support
|
||||
Tell people where they can go to for help. It can be any combination of an issue tracker, a chat room, an email address, etc.
|
||||
|
||||
## Roadmap
|
||||
If you have ideas for releases in the future, it is a good idea to list them in the README.
|
||||
|
||||
## Contributing
|
||||
State if you are open to contributions and what your requirements are for accepting them.
|
||||
|
||||
For people who want to make changes to your project, it's helpful to have some documentation on how to get started. Perhaps there is a script that they should run or some environment variables that they need to set. Make these steps explicit. These instructions could also be useful to your future self.
|
||||
|
||||
You can also document commands to lint the code or run tests. These steps help to ensure high code quality and reduce the likelihood that the changes inadvertently break something. Having instructions for running tests is especially helpful if it requires external setup, such as starting a Selenium server for testing in a browser.
|
||||
|
||||
## Authors and acknowledgment
|
||||
Show your appreciation to those who have contributed to the project.
|
||||
|
||||
## License
|
||||
For open source projects, say how it is licensed.
|
||||
|
||||
## Project status
|
||||
If you have run out of energy or time for your project, put a note at the top of the README saying that development has slowed down or stopped completely. Someone may choose to fork your project or volunteer to step in as a maintainer or owner, allowing your project to keep going. You can also make an explicit request for maintainers.
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,451 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "6a764d92",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Representation Learning\n",
|
||||
"\n",
|
||||
"Often it is very hard either to find similarities within data or even to classify data as belonging to different labels when the data contains so many features. One particular example are images which may contain millions of pixels. Even low-dimensional data may not be easy to visualize in a 2D plane, so bringing the dominant effects in a 2D plane for visualization is in itself a very helpful starting point.\n",
|
||||
"\n",
|
||||
"There are different methods to obtain another view of the data, by performing linear or even non-linear combinations of the data features. The price being paid by such methods is that the new representation of the data may not be that straightforward to digest, loosing therefore some of its scientific interpretation. On the other hand, if one understands the assumptions made in such methods, one can easily imagine the mathematical process required to transform to and from this new view and gain insight from the new view without loosing track of the scientific background.\n",
|
||||
"\n",
|
||||
"We are going to go through a few methods of obtaining an alternative view of the data here and what their assumptions might be."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "fce4d8e8",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We start by loading the necessary Python modules. If you have not yet installed them, run the following cell to install them with pip:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"id": "300cf8d3",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import matplotlib.pyplot as plt\n",
|
||||
"\n",
|
||||
"import pandas as pd\n",
|
||||
"import numpy as np\n",
|
||||
"from sklearn.decomposition import PCA"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "0ecd6a69",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Let's generate the fake data now to have something to cluster."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"id": "4959a292",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"rng = np.random.RandomState(0)\n",
|
||||
"n_samples = 500\n",
|
||||
"cov = [[3, 3], [3, 4]]\n",
|
||||
"data = rng.multivariate_normal(mean=[0, 0], cov=cov, size=n_samples)\n",
|
||||
"data = pd.DataFrame(data, columns=[\"x1\", \"x2\"])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "d8295e8a",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Let's print out the dataset read first."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"id": "024fb65a",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/html": [
|
||||
"<div>\n",
|
||||
"<style scoped>\n",
|
||||
" .dataframe tbody tr th:only-of-type {\n",
|
||||
" vertical-align: middle;\n",
|
||||
" }\n",
|
||||
"\n",
|
||||
" .dataframe tbody tr th {\n",
|
||||
" vertical-align: top;\n",
|
||||
" }\n",
|
||||
"\n",
|
||||
" .dataframe thead th {\n",
|
||||
" text-align: right;\n",
|
||||
" }\n",
|
||||
"</style>\n",
|
||||
"<table border=\"1\" class=\"dataframe\">\n",
|
||||
" <thead>\n",
|
||||
" <tr style=\"text-align: right;\">\n",
|
||||
" <th></th>\n",
|
||||
" <th>x1</th>\n",
|
||||
" <th>x2</th>\n",
|
||||
" </tr>\n",
|
||||
" </thead>\n",
|
||||
" <tbody>\n",
|
||||
" <tr>\n",
|
||||
" <th>0</th>\n",
|
||||
" <td>-3.123062</td>\n",
|
||||
" <td>-3.267402</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>1</th>\n",
|
||||
" <td>-2.775958</td>\n",
|
||||
" <td>-0.929101</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>2</th>\n",
|
||||
" <td>-2.582416</td>\n",
|
||||
" <td>-4.072345</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>3</th>\n",
|
||||
" <td>-1.492453</td>\n",
|
||||
" <td>-1.920361</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>4</th>\n",
|
||||
" <td>-0.041529</td>\n",
|
||||
" <td>0.381166</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>...</th>\n",
|
||||
" <td>...</td>\n",
|
||||
" <td>...</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>495</th>\n",
|
||||
" <td>-0.821492</td>\n",
|
||||
" <td>-0.782416</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>496</th>\n",
|
||||
" <td>1.197165</td>\n",
|
||||
" <td>1.665481</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>497</th>\n",
|
||||
" <td>-0.691309</td>\n",
|
||||
" <td>-0.383494</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>498</th>\n",
|
||||
" <td>0.279317</td>\n",
|
||||
" <td>0.428408</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>499</th>\n",
|
||||
" <td>2.082251</td>\n",
|
||||
" <td>2.082815</td>\n",
|
||||
" </tr>\n",
|
||||
" </tbody>\n",
|
||||
"</table>\n",
|
||||
"<p>500 rows × 2 columns</p>\n",
|
||||
"</div>"
|
||||
],
|
||||
"text/plain": [
|
||||
" x1 x2\n",
|
||||
"0 -3.123062 -3.267402\n",
|
||||
"1 -2.775958 -0.929101\n",
|
||||
"2 -2.582416 -4.072345\n",
|
||||
"3 -1.492453 -1.920361\n",
|
||||
"4 -0.041529 0.381166\n",
|
||||
".. ... ...\n",
|
||||
"495 -0.821492 -0.782416\n",
|
||||
"496 1.197165 1.665481\n",
|
||||
"497 -0.691309 -0.383494\n",
|
||||
"498 0.279317 0.428408\n",
|
||||
"499 2.082251 2.082815\n",
|
||||
"\n",
|
||||
"[500 rows x 2 columns]"
|
||||
]
|
||||
},
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"data"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "1c178424",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We can plot this fairly easily using Matplotlib."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"id": "e63b38c5",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"image/png": "iVBORw0KGgoAAAANSUhEUgAAArIAAAKsCAYAAAAQpV5rAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAACoeElEQVR4nOz9e2xc+X0ffr/PfebMlbchKZESJVG7q9217PVudpNsktqN2zhNkCYPnjR9mhZOUPiPp06b1EXROCiaGg3qFg1+CBAEuQFJgeKXpLfEddMm/WXz+JLY64vW9iqr1UririhRJEWKnOHczpz7ef4YDTUcDud6hjNDvl+AYa3EmfmeM4czn/P5fr6frxAEQQAiIiIiojEjDnsARERERES9YCBLRERERGOJgSwRERERjSUGskREREQ0lhjIEhEREdFYYiBLRERERGOJgSwRERERjSUGskREREQ0lhjIEhEREdFYYiBLRERERGNp7ALZ9fV1/P2///cxNTWFaDSK973vfbh27dqwh0VEREREx0we9gC6kcvl8Oqrr+LDH/4w/uRP/gQzMzO4c+cOJiYmhj00IiIiIjpmQhAEwbAH0amf//mfx5e//GX8xV/8RU+P930fGxsbSCQSEAQh5NERERERUb+CIECxWMSZM2cgiq2LB8YqkH322WfxAz/wA3jw4AG++MUv4uzZs/hH/+gf4eMf/3jTn7csC5Zl7f/3+vo6nn322eMaLhERERH1aG1tDQsLCy1/ZqwC2UgkAgD45Cc/iR//8R/HN77xDfzsz/4sfuM3fgMf+9jHDv38v/7X/xqf/vSnD/392toaksnkwMdLRERERN0pFApYXFzE3t4eUqlUy58dq0BWVVW89NJL+MpXvrL/d//kn/wTfOMb38Drr79+6OcbM7K1E5PP5xnIEhEREY2gQqGAVCrVUbw2Vl0L5ufnD5UGXLlyBffv32/685qmIZlMHvgfEREREZ0MYxXIvvrqq7h169aBv7t9+zbOnz8/pBERERER0bCMVSD7T//pP8VXv/pV/Nt/+2+xsrKC3/u938Nv/dZv4ROf+MSwh0ZEREREx2ysAtnv+I7vwB/90R/h93//9/H888/j3/ybf4Nf+ZVfwU/+5E8Oe2hEREREdMzGarFXv7opHiYiIiKi43diF3sREREREdUwkCUiIiKiscRAloiIiIjGEgNZIiIiIhpLDGSJiIiIaCwxkCUiIiKiscRAloiIiIjGEgNZIiIiIhpLDGSJiIiIaCwxkCUiIiKiscRAloiIiIjGEgNZIiIiIhpLDGSJiIiIaCwxkCUiIiKiscRAloiIiIjGEgNZIiIiIhpL8rAHQERERHSSGbYLw/agqxJ0laFXmHg2iYiIiAbA8Xxcf5DHne0iKraHqCrhciaBqwspKBInxcPAs0hEREQ0ANcf5HFtNQtJEDCbiEASBFxbzeL6g/ywh3ZiMJAlIiIiCplhu7izXcRkTMWErkKVRUzo1T/f2S7CsN1hD/FEYCBLREREFDLD9lCxPcQaamLjmgzT8WDY3pBGdrIwkCUiIiIKma5KiKoSyg2Z15LlIqJI0FVpSCM7WRjIEhEREYVMV2VcziSQLdvIlm3Yro9s2UbOsHE5k2D3gpDwLBIRERENwNWFFADgznYR2yUTEUXCS0uT+39P/WMgS0RERDQAiiTixfMTuDKfYB/ZAeHZJCIiIhogXZUZwA4IzyoRERFRF7hT1+jg2SciIiLqAHfqGj0860REREQd4E5do4eBLBEREVEb3KlrNDGQJSIiImqDO3WNJgayRERERG1wp67RxECWiIiIqA3u1DWaeNaJiIiIOsCdukYPA1kiIiKiDnCnrtHDs09ERETUBe7UNTpYI0tEREREY4mBLBERERGNJQayRERERDSWWOBBREREJ4Zhu1yIdYrwHSYiIqKx53g+rj/I4852ERXbQ1SVcDmTwNWFFBSJE9AnFd9ZIiIiGnvXH+RxbTULSRAwm4hAEgRcW83i+oP8sIdGA8RAloiIiMaaYbu4s13EZEzFhK5ClUVM6NU/39kuwmjYVpZODgayRERENNYM20PF9hBrqImNazJMx4Nhe0MaGQ0aA1kiIiIaa7oqIapKKDdkXkuWi4giQVelIY2MBo2BLBER0ZgxbBc7JYtT5o/pqozLmQSyZRvZsg3b9ZEt28gZNi5nEuxecILxnSUiIhoTXJl/tKsLKQDAne0itksmIoqEl5Ym9/+eTiYGskRERGOitjJ/MqZiNhFB2XZxbTULAHjx/MSQRzdciiTixfMTuDKfYB/ZU4TvMBER0RhoXJkPAKqsIgiqWcgr85xCB6plBjwPp8fpnocgIiIaE1yZT3QYA1kiIqIxwJX5RIcxkCUiIhoDXJlPdBiveiIiojHBlflEBzGQJSIiGhNcmU90EK9+IiKiMcOV+URVrJElIiIiorHEQJaIiIiIxhIDWSIiIiIaSyywISIiIuqSYbtccDcCeOaJiIiIOuR4Pq4/yOPOdhEV20NUlXA5k8DVhRQUiRPdx41nnIiIiE4lw3axU7JgNOyW1sr1B3lcW81CEgTMJiKQBAHXVrO4/iA/wJHSUZiRJSIiohOn1dR/r1lVw3ZxZ7uIyZiKCV0FAKiyiiCoblJxZZ47rB03nm0iIiI6MToJUmtZ1cmYitlEBGXbxbXVLADgxfMTRz63YXuo2B5mE5EDfx/XZGyXzMeB88kLrUa5Hni0RkNERETUh3ZBaj9ZVV2VEFUllG0Xqqzu/33JchFRJOiqNPgDPEbjUA88GqMgIiIi6lNjkKrKIib06p/vbBf3M4sV20OsIViNazJMx4Nhe0c+v67KuJxJIFu2kS3bsF0f2bKNnGHjcubklRWMQz0wA1kiIiI6EToJUuuzqvU6zapeXUjhpaVJ+AiwXTLhI8BLS5O4upAK/XiGqZObglFwsm4diIiI6NTqZOq/llW9tppFEFSD3JLlImfYeGlpsm1WVZFEvHh+AlfmEyNbNxqGcakHZkaWiIiIToROp/7DyKrqqozpuDYSwdwg9Ju5Pi4n8+wTERHRqVQLRu9sF7FdMhFRpENB6mnJqvaj38z1cRmNURARERGFoJsgVVflkQnIRlEnNwXDxnePiIiIThwGqf0bh8z1aI2GiIiIiEbKKN8UcLEXEREREY0lBrJERERENJYYyBIRERHRWGIgS0RERERjiYEsEREREY0lBrJERERENJYYyBIRERHRWBrNpmBERERjwrDdkW0WT3TS8TeOiIioB47n4/qDPO5sF1GxPURVCZczCVxdSEGROOFJdBz4m0ZERNSD6w/yuLaahSQImE1EIAkCrq1mcf1BfthDO1UM28VOyYJhu8MeCg0BM7JERERdMmwXd7aLmIypmNBVAIAqqwgC4M52EVfmEywzGDBmxAlgRpaIiKhrhu2hYnuINQSrcU2G6XgwbG9IIzs9mBEngIEsERFR13RVQlSVUG6Yzi5ZLiKKBF2VhjSy06ExI67KIib06p/vbBdZZnCKMJAlIiLqkq7KuJxJIFu2kS3bsF0f2bKNnGHjcoZlBZ3qtb6VGXGq4W8aERFRD64upABUa2K3SyYiioSXlib3/56O1m99a31GXJXV/b9nRvz0YSBLRETUA0US8eL5CVyZT7CPbJdq9a2TMRWziQjKtotrq1kAwIvnJ9o+vpYRv7aaRRBUM7Ely0XOsPHS0iTfh1NkbEsL/t2/+3cQBAE/93M/N+yhEBHRKaarMqbjGoOnDoVV33p1IYWXlibhI8B2yYSPgBnxU2gsf+u+8Y1v4Dd/8zdx9erVYQ+FiIiIulCrb51NRA78fVyTsV0yH2e324cnzIgTMIYZ2VKphJ/8yZ/Eb//2b2Niov30AxEREY2OsDs+MCN+uo1dIPuJT3wCP/RDP4SPfOQjbX/WsiwUCoUD/yMiIqLhYceH1rhTWXfG6mr5gz/4A3zzm9/EN77xjY5+/jOf+Qw+/elPD3hURERE1A12fDiMO5X1ZmwC2bW1Nfzsz/4s/uzP/gyRSKT9AwB86lOfwic/+cn9/y4UClhcXBzUEImIiKgDrG89rN9ODqeVEARBMOxBdOKzn/0sfuzHfgyS9KR2xvM8CIIAURRhWdaBf2umUCgglUohn88jmUwOeshEREREbRm2i8+9uQFJEDChP+mLmy3b8BHgR95/5lQF+t3Ea2NzVr7/+78ff/VXf3Xg7376p38azzzzDP7Fv/gXbYNYIiIiGh2G7TIb+1hYnRxOo7E5K4lEAs8///yBv4vFYpiamjr090RERDSaWAt6GHcq693pvGKIiIhoKGq1oJIgYDYRgSQIuLaaxfUH+WEPbWjYyaF3Y31mvvCFLwx7CERERNShxl29AECVVQRBtYPBlfnTG7Sxk0NvTufVQkRERMeOtaBHYyeH3vAMERER0bFgLWh7uiozgO0Ca2SJiIjoWLAWlMLGK4aIiIiODWtBKUwMZImIiOjYjEItKHvYnhx894iIiKgn/QSEw6gFZQ/bk4eBLBEREXVlXAPCWg/byZiK2UQEZdvFtdUsAODF8xNDHh31YnSvNiIioi4YtoudkgXDdoc9lBNvHDc1aOxhq8oiJvTqn+9sF3ndjClmZImIaKyNa3ZwVLUrFxjXTQ3Yw/Zk4jtGRERjjdPF4TjqhmA5E4PjBfuB7bgGhOxhezKN3pVGRETUoXHNDo6ixhuCgungs996gHhERiYRORDYjmNAWOthe201iyCoBt4ly0XOsPHS0iSvkzHFORciIhpbtexgrCEIiWsyTMeDYXtDGtl4aVY/umc42C3ZeFSwkI4q+3WwK9vlsd3U4OpCCi8tTcJHgO2SCR8Be9iOudG92oiIiNrgdHE4GssFLMfDZr6CyZgKLwgQAJjQn2S6f/D5OQDjt6nBKPSwpXDx3SMiorHF6eJwNN4QWK4Py/UhCwIUSYQmVydwa3WwjheMXUDYuIht1MdLneG7SEREY41bnvav8YZLine truncated
|
||||
"text/plain": [
|
||||
"<Figure size 800x800 with 1 Axes>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"fig, ax = plt.subplots(figsize=(8, 8))\n",
|
||||
"data.plot.scatter(x=\"x1\", y=\"x2\", alpha=0.3, ax=ax)\n",
|
||||
"ax.set(xlabel=\"$x_1$\", ylabel=r\"$x_2$\", title=\"\")\n",
|
||||
"plt.show()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "bba01cd6",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Clearly there is a strong correlation in this data. One could almost predict $y$ using $x$ alone by fitting a line in the $xy$-plane. Therefore, it may be easier to provide a representation on which only one variable is present but retaining the maximum information possible. This is possible using several methods, which rely on several underlying assumptions. Let us start with Principal Component Analysis.\n",
|
||||
"\n",
|
||||
"In Principal Component Analysis, we set ourselves the objective of maximizing the variance of the data in our new representation. That is, imagine our new representation is given by $p_1$ and $p_2$ and we define them such that $p$ is a linear combination of $x$. The criteria used to find how to make such a linear combination shall be that the variance of $p_1$ is maximal, when we want to find out how to combine $x_1$ and $x_2$ to obtain $p_1$. This can be similarly done for $p_2$ and any other component. Instead of finding $p$ directly for each sample, we will find the direction $u_1$ on which we should project $x$ using $u_1^T x$ to obtain the new space. As we are only interested in the direction of $u_1$, we define it as normalized vector, so $u_1^T u_1 = 1$.\n",
|
||||
"\n",
|
||||
"The mean data value is $\\overline{x} = \\frac{1}{N} \\sum_k x^{(k)}$, where $x^{(k)}$ refers to the vector $(x_1, x_2)$ for the $k$-th sample point. The mean of the projected data in this new dimension $u_1$ is $u_1^T \\overline{x}$. The variance of the projected data in the direction $u_1$ is:\n",
|
||||
"\n",
|
||||
"$\\frac{1}{N}\\sum_k\\left(u_1^T x^{(k)} - u_1^T \\overline{x}\\right)^2 = u_1^T S u_1$,\n",
|
||||
"\n",
|
||||
"where $S = \\frac{1}{N}\\left(x^{(k)} - \\overline{x}\\right)\\left(x^{x(k)} - \\overline{x}\\right)^T$ is the covariance matrix of the data.\n",
|
||||
"\n",
|
||||
"We maximize the variance of the data projected in the direction $u_1$, while imposing a restriction on the maximization procedure, such that $u_1$ is normalized to 1. This can be done by maximizing the following function:\n",
|
||||
"\n",
|
||||
"$u_1^T S u_1 + \\lambda_1 (1 - u_1^T u_1)$,\n",
|
||||
"\n",
|
||||
"where $\\lambda_1$ is a Lagrange multiplier used to enforce the condition that $u_1^T u_1$ is 1.\n",
|
||||
"\n",
|
||||
"Calculating the derivative relative to $u_1$ and setting it to zero to find the maximum we see that:\n",
|
||||
"\n",
|
||||
"$S u_1 = \\lambda_1 u_1$,\n",
|
||||
"\n",
|
||||
"which is an eigenvalue problem! That is, the direction $u_1$, which maximizes the variance of the projected data is the eigenvector of the covariance matrix $S$. Moreover, if we multiply this equation by $u_1^T$ on the left and use $u_1^T u_1 = 1$, we obtain $\\lambda_1 = u_1^T S u_1$. That is, $\\lambda_1$ is the variance of the data projected in the direction $u_1$.\n",
|
||||
"\n",
|
||||
"This gives a simple recipe to find the directions with largest variance: we find the eigenvectors of the covariance matrix $S$ which have highest eigenvalues. By sorting the eigenvalues, we can easily choose which of the directions of the new representations are more important to analyse. We can also discard directions that have low eigenvalues, as they contribute little to the variations observed in the data.\n",
|
||||
"\n",
|
||||
"Naturally, we do not need to write the code to perform all those steps, as scikit-learn implements them for us:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"id": "0837b3ff",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"pca = PCA(n_components=2)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"id": "8798f857",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/html": [
|
||||
"<style>#sk-container-id-1 {color: black;background-color: white;}#sk-container-id-1 pre{padding: 0;}#sk-container-id-1 div.sk-toggleable {background-color: white;}#sk-container-id-1 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-1 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-1 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-1 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-1 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-1 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-1 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-1 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-1 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-1 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-1 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-1 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-1 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-1 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-1 div.sk-item {position: relative;z-index: 1;}#sk-container-id-1 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-1 div.sk-item::before, #sk-container-id-1 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-1 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-1 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-1 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-1 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-1 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-1 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-1 div.sk-label-container {text-align: center;}#sk-container-id-1 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-1 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-1\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>PCA(n_components=2)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-1\" type=\"checkbox\" checked><label for=\"sk-estimator-id-1\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">PCA</label><div class=\"sk-toggleable__content\"><pre>PCA(n_components=2)</pre></div></div></div></div></div>"
|
||||
],
|
||||
"text/plain": [
|
||||
"PCA(n_components=2)"
|
||||
]
|
||||
},
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"pca.fit(data.loc[:, [\"x1\", \"x2\"]])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"id": "cc8fc1f1",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"data_t = pca.transform(data.loc[:, [\"x1\", \"x2\"]])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"id": "88982b21",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"data.loc[:, \"pca1\"] = data_t[:, 0]\n",
|
||||
"data.loc[:, \"pca2\"] = data_t[:, 1]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "3ac27316",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We can start by plotting how the data looks like after this transformation."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 9,
|
||||
"id": "333581b5",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"image/png": "iVBORw0KGgoAAAANSUhEUgAAA1kAAANYCAYAAAA2XSeWAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAEAAElEQVR4nOzdeZxcVZ03/s+tu9Re1dX7mu4kncRACEsiCIwCyiODIIOKOogOrs/4GxhlUGfEmUdFRnHGUfGZRcUNx30DdXB5FNlEMZIEiIEQ0pBO0p3eq7qqa711l98flSq6uqu6q6pr78/79fIlqa7l3Fvd55zvOd9zjmCapgkiIiIiIiIqC0utC0BERERERNRMGGQRERERERGVEYMsIiIiIiKiMmKQRUREREREVEYMsoiIiIiIiMqIQRYREREREVEZMcgiIiIiIiIqIwZZREREREREZcQgi4iIiIiIqIwYZBFRQ/voRz8KQRBqXQwiooaxlnrzrrvugiAIGB0dLW+hFhkdHYUgCLjrrrtKen01yki0GgZZ1BDSFWb6fzabDVu3bsWNN96IqampZc+fmprC+9//frzoRS+Cw+GA0+nErl278M///M+Yn5/P+RnnnnsuBEHA5z//+Qpfzfrz7W9/G3fccUfJr49Go/joRz+KBx98sGxlIiJqNE899RTe/OY3o6+vD1arFb29vbjuuuvw1FNP1bpoTWOt7RVRmmCaplnrQhCt5q677sLb3vY2fOxjH8PGjRsRj8fxyCOP4Bvf+AYGBwdx8OBBOBwOAMBjjz2GV73qVQiHw3jzm9+MXbt2AQD27t2L7373u7jgggvwq1/9Kuv9jxw5gq1bt2JoaAh9fX145JFHqn6NzezKK6/EwYMHSx5VnJ2dRUdHBz7ykY/gox/9aNbPNE2Dpmmw2WxrLygRUZ26++67ce2116K1tRXveMc7sHHjRoyOjuIrX/kK5ubm8N3vfhevec1rCnqvtdSbuq4jmUzCarVWLItgdHQUGzduxNe+9jW89a1vLfr16T7D0aNHMTQ0VNRr19peEaVJtS4AUTEuv/xy7N69GwDwzne+E21tbfjMZz6Dn/zkJ7j22msxPz+P17zmNRBFEY8//jhe9KIXZb3+4x//OL70pS8te99vfvOb6OzsxKc//Wlcc801GB0dLbpiTovH41AUBRZL5SeKDcOAqqrrOsCQJAmSxKqMiJrXc889h7e85S3YtGkTHn74YXR0dGR+9t73vhcvfelL8Za3vAUHDhzApk2b8r5PJBKB0+lcU70piiJEUSzptUTrCdMFqaG9/OUvBwAcPXoUAPDFL34R4+Pj+MxnPrMswAKArq4u/NM//dOyx7/97W/jmmuuwZVXXgmv14tvf/vbBX3+gw8+CEEQ8N3vfhf/9E//hL6+PjgcDoRCIQDAnj178Od//ufwer1wOBy46KKL8Lvf/S7rPdK58c888wze8IY3wOPxoK2tDe9973sRj8eznisIAm688UZ861vfwumnnw6r1Ypf/vKXAIDx8XG8/e1vR1dXF6xWK04//XR89atfXVbmf//3f8fpp58Oh8MBn8+H3bt3L7veQt4rfe3f//738fGPfxz9/f2w2Wx4xStegZGRkczzLr74YvzsZz/DsWPHMume6QBWVVV8+MMfxq5du+D1euF0OvHSl74UDzzwQOb1o6OjmQ7FrbfemnmP9IxWrrUFmqbhtttuw+bNm2G1WjE0NIQPfehDSCQSWc8bGhrClVdeiUceeQTnnnsubDYbNm3ahP/+7/9e/mUTEdXIpz71KUSjUdx5551ZARYAtLe344tf/CIikQj+9V//NfN4um58+umn8aY3vQk+nw9/9md/lvWzxWKxGN7znvegvb0dbrcbV111FcbHx7PqWyD3eqdC61K/34/3v//9OOOMM+ByueDxeHD55ZfjySefLPnePPXUU3j5y18Ou92O/v5+/PM//zMMw1j2vJ/85Ce44oor0NvbC6vVis2bN+O2226DruuZ56y1vSJajMO/1NCee+45AEBbWxsA4Kc//Snsdjuuueaagt9jz549GBkZwde+9jUoioLXvva1+Na3voUPfehDBb/HbbfdBkVR8P73vx+JRAKKouD+++/H5Zdfjl27duEjH/kILBYLvva1r+HlL385fvvb3+Lcc8/Neo83vOENGBoawu23344//OEP+L//9/8iEAgsa6Tuv/9+fP/738eNN96I9vZ2DA0NYWpqCi95yUsyQVhHRwd+8Ytf4B3veAdCoRBuuukmAMCXvvQlvOc978E111yTCeIOHDiAPXv24E1vehMAFPxeaZ/85CdhsVjw/ve/H8FgEP/6r/+K6667Dnv27AEA/OM//iOCwSDGxsbw2c9+FgDgcrkAAKFQCF/+8pdx7bXX4l3vehcWFhbwla98BZdddhn++Mc/4qyzzkJHRwc+//nP4//7//4/vOY1r8FrX/taAMDOnTvzfh/vfOc78fWvfx3XXHMN3ve+92HPnj24/fbbcejQIdxzzz1Zzx0ZGcE111yDd7zjHbj++uvx1a9+FW9961uxa9cunH766QX/DhARVcr//M//YGhoCC996Utz/vxlL3sZhoaG8LOf/WzZz17/+tdjy5Yt+MQnPoGVVoi89a1vxfe//3285S1vwUte8hI89NBDuOKKKwouYyF16fPPP48f//jHeP3rX4+NGzdiamoKX/ziF3HRRRfh6aefRm9vb8GfBwCTk5O45JJLoGkaPvjBD8LpdOLOO++E3W5f9ty77roLLpcLN998M1wuF+6//358+MMfRigUwqc+9SkAa2+viLKYRA3ga1/7mgnAvO+++8yZmRnzxIkT5ne/+12zra3NtNvt5tjYmGmapunz+cwzzzyzqPe+8cYbzYGBAdMwDNM0TfNXv/qVCcB8/PHHV33tAw88YAIwN23aZEaj0czjhmGYW7ZsMS+77LLM+5qmaUajUXPjxo3m//pf/yvz2Ec+8hETgHnVVVdlvfff/M3fmADMJ598MvMYANNisZhPPfVU1nPf8Y53mD09Pebs7GzW43/5l39per3eTNn+4i/+wjz99NNXvKZC3yt97du3bzcTiUTmeZ/73OdMAOaf/vSnzGNXXHGFOTg4uOyzNE3Leq1pmmYgEDC7urrMt7/97ZnHZmZmTADmRz7ykWXvkb5/aU888YQJwHznO9+Z9bz3v//9JgDz/vvvzzw2ODhoAjAffvjhzGPT09Om1Wo13/e+9+W6PUREVTU/P28CMP/iL/5ixeddddVVJgAzFAqZpvlC3Xjttdcue+7SenPfvn0mAPOmm27Ket5b3/rWZXVvuj0+evRo5rFC69J4PG7qup71GUePHjWtVqv5sY99LOsxAObXvva1Fa/5pptuMgGYe/bsyfpcr9e7rIyL2+i0v/7rvzYdDocZj8czj621vSJKY7ogNZRLL70UHR0dGBgYwF/+5V/C5XLhnnvuQV9fH4DUSJPb7S74/TRNw/e+9z288Y1vzKROvPzlL0dnZye+9a1vFfw+119/fdbI2RNPPIEjR47gTW96E+bm5jA7O4vZ2VlEIhG84hWvwMMPP7wsneGGG27I+vff/u3fAgB+/vOfZz1+0UUX4bTTTsv82zRN/OhHP8KrX/1qmKaZ+azZ2VlcdtllCAaD2L9/PwCgpaUFY2NjeOyxx3JeRzHvlfa2t70NiqJk/p0eaX3++edXvW+iKGZeaxgG/H4/NE3D7t27l31OodL36+abb856/H3vex8ALBvpPe2007JGhzs6OrBt27aCyk9EVGkLCwsAsGrblv55Ol097d3vfveqn5FOO/+bv/mbrMfT7VAhCqlLrVZrZr2yruuYm5uDy+XCtm3bSqrzf/7zn+MlL3lJVmZIR0cHrrvuumXPXdxGLywsYHZ2Fi996UsRjUbxzDPPrPpZlWivqLkxXZAayn/+539i69atkCQJXV1d2LZtW9YGEx6PJ9MgFeJXv/oVZmZmcO6552atI7rkkkvwne98B//yL/9S0AYWGzduzPr3kSNHAKSCr3yCwSB8Pl/m31u2bMn6+ebNm2GxWJbtcLT0s2ZmZjA/P48777wTd955Z87Pmp6eBgD8wz/8A+677z6ce+65GB4exitf+Uq86U1vwoUXXlj0e6Vt2LAh69/pawoEAjlfv9TXv/51fPrTn8YzzzyDZDKZ9zoLdezYMVgsFgwPD2c93t3djZaWFhw7dizr8aXlB1LXUGj5iYgqKR08rda25QvGCqlL0/Xm0ucurUdXUkhdahgGPve5z+G//uu/cPTo0az1UOm0/2IcO3YM55133rLHt23btuyxp556Cv/0T/+E+++/f1kgGgwGC/q8crdX1NwYZFFDOffcczO7C+byohe9CE888QRUVc2aXcknPVv1hje8IefPH3roIVxyySWrvs/S/O/0LNWnPvWpvHna6TzvfPJtjZvvs9785jfnDerS65e2b9+Ow4cP495778Uvf/lL/OhHP8J//dd/4cMf/jBuvfXWot4rLd8uU2YBp0N885vfxFvf+lZcffXV+MAHPoDOzk6Ioojbb789s96uVIVuLbyW8hMRVZrX60VPTw8OHDiw4vMOHDiAvr4+eDyerMdzrU+qhELq0k984hP4P//n/+Dtb387brvtNrS2tsJiseCmm27KuVlFuczPz+Oiiy6Cx+PBxz72MWzevBk2mw379+/HP/zDPxT02ZVsr6g5MciipvLqV78ajz76KH70ox/h2muvXfG5kUgEP/nJT/DGN74x50YZ73nPe/Ctb32roCBrqc2bNwNIzaxdeumlBb3myJEjWaNhIyMjMAxj1a3kOzo64Ha7oet6QZ/ldDrxxje+EW984xuhqipe+9rX4uMf/zhuueWWot+rUPkCnh/+8IfYtGkT7r777qznfOQjHyno9bkMDg7CMAwcOXIE27dvzzw+NTWF+fl5DA4OFll6IqLauvLKK/GlL30JjzzySGaHwMVLine truncated
|
||||
"text/plain": [
|
||||
"<Figure size 1000x1000 with 2 Axes>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"fig, ax = plt.subplots(figsize=(10, 10), ncols=2)\n",
|
||||
"data.plot.scatter(x=\"pca1\", y=\"pca2\", alpha=0.3, ax=ax[0])\n",
|
||||
"data.plot.scatter(x=\"x1\", y=\"x2\", alpha=0.3, ax=ax[1])\n",
|
||||
"ax[0].set(xlabel=\"PCA component 1\", ylabel=r\"PCA component 2\", title=\"PCA representation\")\n",
|
||||
"ax[1].set(xlabel=\"$x_1$\", ylabel=r\"$x_2$\", title=\"Original data\")\n",
|
||||
"plt.show()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "54e538c6",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"It is interesting to understand how many PCA components are necessary to explain the variance of the data. This is easily obtainable from the PCA object."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"id": "fa5ebbce",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"image/png": "iVBORw0KGgoAAAANSUhEUgAAA2EAAANXCAYAAAC8L1ePAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA8JUlEQVR4nO3debTVdb3/8dcBZBAZHEBBEREURZzxmjgrieSAdgskBzCzfl41cfhdtZ8EWqFlGVTKzS6BlfwQJzTSyBTRHFPDeSTHRHHiAJKknP37w8X5dS6ocDh9Dh4fj7X2Wp3v97u/+733di17+h12VaVSqQQAAIAimjX2AAAAAJ8lIgwAAKAgEQYAAFCQCAMAAChIhAEAABQkwgAAAAoSYQAAAAWJMAAAgIJEGAAAQEEiDIC1xogRI7LFFlvU67lbbLFFRowY0aDzrKo1mftfpTE/DwA+nggDoI7JkyenqqrqIx/33ntvY48IAJ9qLRp7AADWThdccEF69OixwvJevXo1wjSf7Omnn06zZv7b4nI+D4C1lwgDYKUGDRqUfv36NfYYq6xVq1aNPUKjq1Qqee+999KmTRufB8BazH8iA6BeRo8enWbNmuXWW2+ts/zrX/96WrZsmYcffjhJcvvtt6eqqipXXXVVvvWtb2WTTTZJ27Ztc/jhh+fll1/+xNf54Q9/mP79+2fDDTdMmzZtsuuuu+aaa65ZYbv/eQ3U8tMq77rrrpxxxhnp1KlT2rZtmyOPPDJvvPHGCs+/+eabs/fee6dt27Zp165dDjnkkDz++OMrbDd9+vT07ds3rVu3Tt++fXP99dd/4ntIkkMPPTRbbrnlStftsccedYJ30qRJOeCAA9K5c+e0atUqffr0yYQJE1b6ng899NDMnDkz/fr1S5s2bfLzn/98pZ/H22+/nbPOOivbb7991ltvvbRv3z6DBg2q/Z6WW/59TZs2Ld/73vey2WabpXXr1jnwwAPz3HPPrTDDfffdly984QtZf/3107Zt2+ywww4ZP358nW2eeuqpfOlLX8oGG2yQ1q1bp1+/frnxxhtX6XMDaIocCQNgpaqrq/Pmm2/WWVZVVZUNN9wwSXLeeeflt7/9bU444YQ8+uijadeuXWbOnJlf/OIX+c53vpMdd9yxznO/973vpaqqKmeffXbmz5+fcePGZcCAAZkzZ07atGnzkXOMHz8+hx9+eI4++uj84x//yNSpU/PlL385M2bMyCGHHPKJ7+PUU0/N+uuvn9GjR+eFF17IuHHjcsopp+Sqq66q3ebXv/51hg8fnoEDB+b73/9+lixZkgkTJmSvvfbKX/7yl9qbbvzhD3/Iv//7v6dPnz658MIL89Zbb+X444/PZptt9olzDB06NMcdd1z+/Oc/Z7fddqtd/uKLL+bee+/NxRdfXLtswoQJ2W677XL44YenRYsW+e1vf5v/+I//SE1NTU4++eQ6+3366aczbNiwfOMb38iJJ56Y3r17r/T1//rXv2b69On58pe/nB49euT111/Pz3/+8+y777554okn0rVr1zrbX3TRRWnWrFnOOuusVFdX5wc/+EGOPvro3HfffbXb3HLLLTn00EPTpUuXnHbaadlkk03y5JNPZsaMGTnttNOSJI8//nj23HPPbLrppjnnnHPStm3bTJs2LUcccUSuvfbaHHnkkZ/42QE0ORUA+CeTJk2qJFnpo1WrVnW2ffTRRystW7asfO1rX6u88847lU033bTSr1+/yvvvv1+7zaxZsypJKptuumll4cKFtcunTZtWSVIZP3587bLhw4dXunfvXuc1lixZUufvf/zjH5W+fftWDjjggDrLu3fvXhk+fPgK72PAgAGVmpqa2uWnn356pXnz5pUFCxZUKpVKZdGiRZWOHTtWTjzxxDr7e+211yodOnSos3ynnXaqdOnSpfa5lUql8oc//KGSZIW5/6fq6upKq1atKmeeeWad5T/4wQ8qVVVVlRdffPEj33OlUqkMHDiwsuWWW67wnpNUfv/736+w/f/8PN57773KsmXL6mzz/PPPV1q1alW54IILapct/7623XbbytKlS2uXjx8/vpKk8uijj1YqlUrlgw8+qPTo0aPSvXv3yjvvvFNnv//8eR944IGV7bffvvLee+/VWd+/f//KVltttcLcAJ8FTkcEYKUuvfTS3HLLLXUeN998c51t+vbtm/PPPz///d//nYEDB+bNN9/MFVdckRYtVjzR4rjjjku7du1q//7Sl76ULl265KabbvrYOf75KNk777yT6urq7L333nnooYdW6X18/etfT1VVVe3fe++9d5YtW5YXX3wxyYdHcxYsWJBhw4blzTffrH00b948u+++e2bNmpUkmTdvXubMmZPhw4enQ4cOtfv7/Oc/nz59+nziHMtP/5s2bVoqlUrt8quuuiqf+9znsvnmm6/0PS8/Irnvvvvmr3/9a6qrq+vst0ePHhk4cOAnvn6rVq1qb9SxbNmyvPXWW1lvvfXSu3fvlX6Wxx9/fFq2bFn79957753kwyNqSfKXv/wlzz//fEaOHJmOHTvWee7yz/vtt9/ObbfdliFDhmTRokW1n+1bb72VgQMH5tlnn83f/va3T5wdoKlxOiIAK/Vv//Zvq3Rjjv/9v/93pk6dmvvvvz9jx479yCDZaqut6vxdVVWVXr165YUXXvjY/c+YMSPf/e53M2fOnCxdurTO81fFP8dNkqy//vpJPgy6JHn22WeTJAcccMBKn9++ffskqY22//k+knxkyPxPQ4cOzfTp03PPPfekf//+mTt3bh588MGMGzeuznZ33XVXRo8enXvuuSdLliyps666urpOBK7sDpYrU1NTk/Hjx+eyyy7L888/n2XLltWuW36K6T/7pM9t7ty5ST4M8Y/y3HPPpVKpZNSoURk1atRKt5k/f3423XTTVXoPAE2FCANgjfz1r3+tDZlHH320Qfd955135vDDD88+++yTyy67LF26dMk666yTSZMmZcqUKau0j+bNm690+fKjUTU1NUk+vC5sk002WWG7lR3Vq6/DDjss6667bqZNm5b+/ftn2rRpadasWb785S/XbjN37twceOCB2WabbXLJJZekW7duadmyZW666ab8+Mc/rp13uY+7nu6fjR07NqNGjcpXv/rVfOc738kGG2yQZs2aZeTIkSvsM/nkz21VLN/vWWed9ZFH69bWnzwA+FcSYQDUW01NTUaMGJH27dtn5MiRGTt2bL70pS/li1/84grbLg+15SqVSp577rnssMMOH7n/a6+9Nq1bt87MmTPr3HJ90qRJDfYeevbsmSTp3LlzBgwY8JHbde/ePcmK7yP58OYYq6Jt27Y59NBDc/XVV+eSSy7JVVddlb333rvOTTF++9vfZunSpbnxxhvrHI1aflpkfV1zzTXZf//9M3HixDrLFyxYkI022mi197f8c3vsscc+8nNbfjfIddZZ52M/W4DPGteEAVBvl1xySe6+++5cfvnl+c53vpP+/fvnpJNOWuGuiknyq1/9KosWLar9+5prrsm8efMyaNCgj9x/8+bNU1VVVefUuRdeeCHTp09vsPcwcODAtG/fPmPHjs3777+/wvrlt7Pv0qVLdtppp1xxxRV1rsu65ZZb8sQTT6zy6w0dOjSvvvpq/vu//zsPP/xwhg4dWmf98iNQ/3zEqbq6eo3Ds3nz5iscxbr66qvrfU3WLrvskh49emTcuHFZsGBBnXXLX6dz587Zb7/98vOf/zzz5s1bYR8r+6kAgM8CR8IAWKmbb745Tz311ArL+/fvny233DJPPvlkRo0alREjRuSwww5L8uFvc+200075j//4j0ybNq3O8zbYYIPstddeOf744/P6669n3Lhx6dWrV0488cSPnOGQQw7JJZdckoMPPjhf+cpXMn/+/Fx66aXp1atXHnnkkQZ5n+3bt8+ECRNy7LHHZpdddslRRx2VTp065aWXXsrvfve77LnnnvnZz36WJLnwwgtzyCGHZK+99spXv/rVvP322/npT3+a7bbbLosXL16l1/vCF76Qdu3a5ayzzkrz5s3z7//+73XWH3TQQWnZsmUOO+ywfOMb38jixYvzi1/8Ip07d15pyKyqQw89NBdccEGOP/749O/fP48++miuvPLKj/ztsk/SrFmzTJgwIYcddlh22mmnHH/88enSpUueeuqpPP7445k5c2aSD2/wstdee2X77bfPiSeemC233DKvv/567rnnnrzyyisr/E4ZwGeBCANgpb797W+vdPmkSZPSvXv3DB8+PBtttFGdm0pstdVWufDCC3Paaadl2rRpGTJkSO26b33rW3nkkUdy4YUXZtGiRTnwwANz2WWXZd111/3IGQ444IBMnDgxF110UUaOHJkePXrk+9//fl544YUGi7Ak+cpXvpKuXbvmoosuysUXX5ylS5dm0003zd57753jjz++druDDz44V199dc4777yce+656dmzZyZNmpQbbrght99++yq9VuvWrXP44YfnyiuvzIABA9K5c+c663v37p1rrrkm5513Xs4666xssskmOemkk9KpU6d89atfrfd7/Na3vpV33303U6ZMyVVXXZVddtklv/vd73LOOefUe58DBw7MrFmzcv755+dHP/pRampq0rNnzzph3adPnzzwwAM5//zzM3ny5Lz11lvp3Llzdt5554/8ZwygqauqrM4VtgCwmm6//fbsv//+ufrqq/OlL32psccBgEbnmjAAAICCRBgAAEBBIgwAAKAg14QBAAAU5EgYAABAQSIMAACgIL8TtgZqamry6quvpl27dqmqqmrscQAAgEZSqVSyaNGidO3aNc2affyxLhG2Bl599dV069atsccAAADWEi+//HI222yzj91GhK2Bdu3aJfnwg27fvn0jTwMAADSWhQsXplu3brWN8HFE2BpYfgpi+/btRRgAALBKlym5MQcAAEBBIgwAAKAgEQYAAFCQCAMAAChIhAEAABQLine truncated
|
||||
"text/plain": [
|
||||
"<Figure size 1000x1000 with 1 Axes>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"fig = plt.figure(figsize=(10,10))\n",
|
||||
"plt.barh([\"PCA 1\", \"PCA 2\"], pca.explained_variance_)\n",
|
||||
"plt.gca().set(xlabel=\"Variance\", ylabel=\"Component\", title=\"Explained variance\")\n",
|
||||
"plt.show()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "ff6b13e3",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Clearly the zeroth PCA component holds most of the variance and we could therefore use this component to closely determine where a data sample point should be if we dropped the other component. This is equivalent to fitting a line and using the projection of the point in the line to characterize the data, instead of using the two coordinates.\n",
|
||||
"\n",
|
||||
"While this may seem superfluous in this simple case, if one has hundreds of variables, PCA provides a simple and almost automatic way to reduce the amount of features being examined, by concentrating most of the variance in a few variables. Which variables to choose, can be decided from the `explained_variance_` attribute."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "7bcb1e2c",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Other representation learning methods\n",
|
||||
"\n",
|
||||
"While PCA is a great method for both data visualization and choosing a latent space to condense the information contained in the data, there are many other available, which differ on the assumptions made. Here are some of them and easy-to-read and practical references on how to use them and which assumptions they entail.\n",
|
||||
"\n",
|
||||
" * Kernel PCA: While PCA does a great job at finding the directions where most of the variance is, it focuses on making only *linear* combinations of the features to find the new representation directions. A straightforward method to generalise this idea to non-linear transformations is to apply a non-linear transformation to the data features and after that apply PCA. This is similar to what Kernel PCA does, generalizing the PCA idea. Of course, a non-linearity transformation needs to be chosen for this and this adds extra assumptions in the model. Some more on this can be read here: https://scikit-learn.org/stable/modules/decomposition.html#kernel-pca\n",
|
||||
" \n",
|
||||
" * Independent Component Analysis: PCA looks for the direction containing most of the variance, but this is not always the optimal representation of the data for all goals. In some cases, independence is much more important than decorrelation and this is where the Independent Component Analysis comes in. The ICA also assumes that the new representation can be built from a linear combination of the existing data, but it assumes additionally each observed feature comes from a linear combination of independent latent representations we want to discover. Since independence is a very strong requirement, it is imposed through various different proxy methods. One set of methods focuses on reducing the mutual information (as defined in statistics) between the new features, while an alternative imposes non-Gaussianity requirements on the underlying latent features to be discovered. Since Gaussians do not have statistical moments above the second-order moment (covariance), one can require for example, the fourth order statistical moment (kurtosis) to be maximized in the new representation. More on the scikit-learn implementation can be found here: https://scikit-learn.org/stable/modules/decomposition.html#ica .\n",
|
||||
" \n",
|
||||
" * t-SNE embedding: If the objective is only to visualize the data, there are many alternative solutions which focus on reducing the dimensionality of the data into a 2D representation. The t-SNE method assumes that a Gaussian probability can be used to model similarity between data points in N dimensions and that one should maintain that similarity measure when projecting the data in two dimensions, assuming however that the 2D data points' similarity can be represented with a t-Student distribution. Full details on the method can be seen here: https://jmlr.org/papers/volume9/vandermaaten08a/vandermaaten08a.pdf It can be easily tried in scikit-learn following the procedure here: https://scikit-learn.org/stable/modules/generated/sklearn.manifold.TSNE.html"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "9fbc8126-83f3-4c3c-8e5e-a677461e4c48",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Try them out!\n",
|
||||
"\n",
|
||||
"Try using t-SNE or Kernel PCA for visualizing the data. Try also getting other data sources to explore how they can be visualized. An example of a data source that you can use to play with would be the dataset of hand-written digits (MNIST), often used for tests in ML. You can obtain them as follows:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 12,
|
||||
"id": "81e3af9d-eeb5-4f3a-b3d9-005890e0fa14",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import mnist\n",
|
||||
"\n",
|
||||
"data = mnist.train_images() # the images\n",
|
||||
"labels = mnist.train_labels() # the array with the digit each image corresponds to"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "600d7a0b-7b9e-4b17-a73a-5bc44d3cf682",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.10.6"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 275 KiB |
@@ -0,0 +1,532 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "1bba0128",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Supervised regression\n",
|
||||
"\n",
|
||||
"The goal is to build and optimize neural networks with PyTorch with the objective of fitting a known feature in the training dataset.\n",
|
||||
"\n",
|
||||
"The fit is achieved by minimizing the mean-squared-error between the prediction and the target features. As noticed during the presentation, by minimizing the mean-squared-error, we assume that the underlying error distribution is Gaussian. One could use the mean absolute error instead when assuming the distribution is Laplacian.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"id": "23feddde",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from typing import Tuple\n",
|
||||
"\n",
|
||||
"# import standard PyTorch modules\n",
|
||||
"import torch\n",
|
||||
"import torch.nn as nn\n",
|
||||
"import torch.nn.functional as F\n",
|
||||
"\n",
|
||||
"import pandas as pd\n",
|
||||
"import numpy as np\n",
|
||||
"import matplotlib.pyplot as plt\n",
|
||||
"from mpl_toolkits.mplot3d import axes3d\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "bb1286f0",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We start by generating some fake dataset, which is simple enough that we can visualize the results easily. For this reason, the dataset will contain only one independent variable.\n",
|
||||
"\n",
|
||||
"The simulated example data will be $f(x) = (3 + \\kappa) x^2 + \\epsilon$, where $\\epsilon \\sim \\mathcal{N}(\\mu=0, \\sigma=10)$ and $\\kappa \\sim \\mathcal{N}(\\mu=0, \\sigma=0.03)$.\n",
|
||||
"\n",
|
||||
"In this case we do know the true model, so it is interesting to take some time to pinpoint the role of $\\kappa$ and $\\epsilon$. These variables add fluctuation to the results. $\\epsilon$ adds Gaussian noise in a way that is completely independent from $x$ and cannot be traced down to a particular functional dependence. $\\kappa$ changes a specific parameter of the model, in this case the coefficient 3, by around 1%.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"id": "5d457cd8",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def generate_data(N: int) -> np.ndarray:\n",
|
||||
" x = 2*np.random.randn(N, 1)\n",
|
||||
" epsilon = 10*np.random.randn(N, 1)\n",
|
||||
" kappa = 0.03*np.random.randn(N, 1)\n",
|
||||
" z = (3 + kappa)*x**2 + epsilon\n",
|
||||
" return np.concatenate((x, z), axis=1).astype(np.float32)\n",
|
||||
"\n",
|
||||
"train_data = generate_data(N=1000)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "48433f6f",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"PyTorch allows you to create a class that outputs a single data entry and use that to feed input to your neural network. We will use the following class to feed the data to the neural network. This looks useless if all your data fits in a Numpy array, but notice that if you have a lot of data and cannot load it all in memory, this allows you to read data on demand, as you need it and only the needed samples are stored at a single time."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"id": "30205402",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"class MyDataset(torch.utils.data.Dataset):\n",
|
||||
" def __init__(self, data: np.ndarray):\n",
|
||||
" self.data = data\n",
|
||||
" def __len__(self) -> int:\n",
|
||||
" \"\"\"How many samples do I have?\"\"\"\n",
|
||||
" return len(self.data)\n",
|
||||
" def __getitem__(self, idx):\n",
|
||||
" # give me item with index idx\n",
|
||||
" return {\"data\": self.data[idx, 0:1], \"target\": self.data[idx, 1:]}\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"id": "cc0b0774",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"1000\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"my_dataset = MyDataset(train_data)\n",
|
||||
"print(len(my_dataset))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"id": "6dccfac6",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"{'data': array([-1.5745504], dtype=float32), 'target': array([17.356268], dtype=float32)}\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"print(my_dataset[1])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "527089bd",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Plot some of the data:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"id": "067b8105",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAioAAAGdCAYAAAA8F1jjAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABZ4ElEQVR4nO3dfXwU5bk38N9uSDYhJBuSSDbRAAGxEKPyJhDheASDRCniU7QFsUeRB5SCR8BzRKyIVGuKfYEqHFBUaouA7VFUpKYPb0qh4aWkWEMQAQNSyAYhkIVAXsju80eYZd9mZ2Z3Zmd29/f9fPLR7M7O3llmZ6657+u+bpPL5XKBiIiIyIDMejeAiIiISAwDFSIiIjIsBipERERkWAxUiIiIyLAYqBAREZFhMVAhIiIiw2KgQkRERIbFQIWIiIgMq4PeDQiX0+nEyZMnkZaWBpPJpHdziIiISAaXy4Xz588jLy8PZrN4v0nUByonT55Efn6+3s0gIiKiEBw/fhzXXXed6PNRH6ikpaUBaP9D09PTdW4NERERyeFwOJCfn+++jouJ+kBFGO5JT09noEJERBRlpNI2mExLREREhsVAhYiIiAyLgQoREREZFgMVIiIiMiwGKkRERGRYDFSIiIjIsBioEBERkWExUCEiIiLDivqCb0RERKS+NqcLu2vqcep8E7qkJWNQQSYSzJFfU4+BChEREXkpr6rFgvXVqG1ocj+Wa03G/DGFKC3KjWhbOPRDREREbuVVtZi2qtIrSAEAe0MTpq2qRHlVbUTbw0CFiIiIALQP9yxYXw1XgOeExxasr0abM9AW2mCgQkRERACA3TX1fj0pnlwAahuasLumPmJtYqBCREREAIBT58WDlFC2UwMDFSIiIgIAdElLVnU7NTBQISIiIgDAoIJM5FqTITYJ2YT22T+DCjIj1iYGKkRERAQASDCbMH9MIQD4BSvC7/PHFEa0ngoDFSIiInIrLcrFsof6w2b1Ht6xWZOx7KH+Ea+jwoJvRERE5KW0KBcjC22sTEtERETGlGA2obhnlt7N4NAPERERGRcDFSIiIjIsBipERERkWAxUiIiIyLAYqBAREZFhMVAhIiIiw2KgQkRERIbFQIWIiIgMi4EKERERGRYDFSIiIjIsBipERERkWAxUiIiIyLAYqBAREZFhhRyobNu2DWPGjEFeXh5MJhM+/PBD93Otra2YM2cObrrpJqSmpiIvLw//8R//gZMnT3rto76+HhMnTkR6ejoyMjIwefJkXLhwIeQ/hoiIiGJLyIFKY2MjbrnlFixdutTvuYsXL6KyshLz5s1DZWUlPvjgAxw8eBD33nuv13YTJ07E/v37sXHjRnzyySfYtm0bpk6dGmqTNNXmdKHiyBl8tO8EKo6cQZvTpXeTiIiIYp7J5XKFfcU1mUxYt24d7rvvPtFt9uzZg0GDBuHYsWPo2rUrDhw4gMLCQuzZswcDBw4EAJSXl+Oee+7Bv/71L+Tl5cl6b4fDAavVioaGBqSnp4f7pwRUXlWLBeurUdvQ5H4s15qM+WMKUVqUq8l7EhERxTK51++I5ag0NDTAZDIhIyMDAFBRUYGMjAx3kAIAJSUlMJvN2LVrl+h+mpub4XA4vH60VF5Vi2mrKr2CFACwNzRh2qpKlFfVavr+RERE8SwigUpTUxPmzJmDCRMmuKMmu92OLl26eG3XoUMHZGZmwm63i+6rrKwMVqvV/ZOfn69Zu9ucLixYX41AXU7CYwvWV3MYiIiISCOaByqtra344Q9/CJfLhWXLloW9v7lz56KhocH9c/z4cRVaGdjumnq/nhRPLgC1DU3YXVOvWRuIiIjiWQctdy4EKceOHcOWLVu8xqBsNhtOnTrltf3ly5dRX18Pm80muk+LxQKLxaJZmz2dOi8epISyHRERESmjWY+KEKQcOnQImzZtQlZWltfzxcXFOHfuHPbu3et+bMuWLXA6nRg8eLBWzVKkS1qyqtsRERGRMiH3qFy4cAGHDx92/15TU4N9+/YhMzMTubm5uP/++1FZWYlPPvkEbW1t7ryTzMxMJCUloU+fPigtLcWUKVOwfPlytLa2YsaMGRg/frzsGT9aG1SQiVxrMuwNTQHzVEwAbNZkDCrIjHTTiIiI4kLI05M/++wzDB8+3O/xhx9+GC+88AIKCgoCvm7r1q244447ALQXfJsxYwbWr18Ps9mMcePG4dVXX0WnTp1kt0Pr6cnCrB8AXsGK6cp/lz3Un1OUiYiIFJJ7/ValjoqetAhU2pwu7K6px6nzTeiSloyzjc14ccMB1lEhIiJSidzrt6bJtNFIrLjbvNGF6Jya5A5eBhVkIsFsCrInIiIiChcXJfQQrLjb9NWVaLjUgrF9r0VxzywGKURERBHAQOUKFncjIiIyHgYqV7C4GxERkfEwULmCxd2IiIiMh4HKFSzuRkREZDwMVK4QiruJpcia0D77h8XdiIiIIoeByhUJZhPmjykEAL9gRfh9/phCzvYhIiKKIAYqHkqLcrHsof6wWb2Hd2zWZFagJSIi0gELvvkoLcrFyEKbV2VaFncjIiLSBwOVABLMJhT3zJLekIiIiDTFoR8iIiIyLAYqREREZFgMVIiIiMiwGKgQERGRYTFQISIiIsNioEJERESGxUCFiIiIDIuBChERERkWAxUiIiIyLAYqREREZFgMVIiIiMiwGKgQERGRYTFQISIiIsNioEJERESG1UHvBsSyNqcLu2vqcep8E7qkJWNQQSYSzCa9m0VERBQ1GKhopLyqFgvWV6O2ocn9WK41GfPHFKK0KFfHlhEREUUPDv1ooLyqFtNWVXoFKQBgb2jCtFWVKK+q1allRERE0YWBisranC4sWF8NV4DnhMcWrK9GmzPQFkREROSJgYrKdtfU+/WkeHIBqG1owu6a+sg1ioiIKEqFHKhs27YNY8aMQV5eHkwmEz788EOv510uF55//nnk5uYiJSUFJSUlOHTokNc29fX1mDhxItLT05GRkYHJkyfjwoULoTbJEE6dFw9SQtmOiIgonoUcqDQ2NuKWW27B0qVLAz7/yiuv4NVXX8Xy5cuxa9cupKamYtSoUWhqunqBnjhxIvbv34+NGzfik08+wbZt2zB16tRQm2QIXdKSVd2OiIgonplcLlfYyRImkwnr1q3DfffdB6C9NyUvLw9PPfUU/uu//gsA0NDQgJycHPzud7/D+PHjceDAARQWFmLPnj0YOHAgAKC8vBz33HMP/vWvfyEvL0/WezscDlitVjQ0NCA9PT3cPyVsbU4Xhi3cAntDU8A8FRMAmzUZ2+eM4FRlIiKKW3Kv35rkqNTU1MBut6OkpMT9mNVqxeDBg1FRUQEAqKioQEZGhjtIAYCSkhKYzWbs2rVLdN/Nzc1wOBxeP0aSYDZh/phCAO1BiSfh9/ljChmkEBERyaBJoGK32wEAOTk5Xo/n5OS4n7Pb7ejSpYvX8x06dEBmZqZ7m0DKyspgtVrdP/n5+Sq3PnylRblY9lB/2Kzewzs2azKWPdSfdVSIiIhkirqCb3PnzsXs2bPdvzscDsMGKyMLbaxMS0REFAZNAhWbzQYAqKurQ27u1d6Duro69O3b173NqVOnvF53+fJl1NfXu18fiMVigcViUb/RGkgwm1DcM0vvZhAREUUtTYZ+CgoKYLPZsHnzZvdjDocDu3btQnFxMQCguLgY586dw969e93bbNmyBU6nE4MHD9aiWURERBRlQu5RuXDhAg4fPuz+vaamBvv27UNmZia6du2KmTNn4qWXXkKvXr1QUFCAefPmIS8vzz0zqE+fPigtLcWUKVOwfPlytLa2YsaMGRg/frzsGT9EREQU20IOVP7+979j+PDh7t+FvJGHH34Yv/vd7/D000+jsbERU6dOxblz5zBs2DCUl5cjOflqgum7776LGTNm4M4774TZbMa4cePw6quvhvHnEBERUSxRpY6KnoxWR4WIiIik6VpHhYiIiEgNDFSIiIjIsKKujgoREZGW2pwu1sAyEAYqREREV5RX1WLB+mrUNlxdQDfXmoz5YwpRWpTLIEYHDFSIiIjQHqRMW1Xpt6CsvaEJ01ZVYurtBfj4i1rRIIa0wRwVIiKKe21OFxasrw646r3rys/r22q8ghTgahBTXlUbiWbGJQYqREQU93bX1PsFIXIIgc2C9dVoc0Z1tQ/DYqBCRERx79R55UGKwAWgtqEJu2vq1WsQuTFQISKiuNclLVl6IwnhBDskjoEKERHFvUEFmci1JiOc+TtqBDvkj4EKERHFvQSzCfPHFAKAX7AiFbyY0D77Z1BBphZNi3sMVIiIiACUFuVi2UP9YbN694zYrMl47PYCmCAexMwfU8h6KhphHRWDY3EhIqLIKS3KxchCW8Dzbr+unf2KwdlYR0VzXD3ZwKQqJBIRUWTx5lE9cq/fDFQMSqxCovB1WPZQfwYrREQUteRev5mjYkBSFRIBFhciIjK6NqcLFUfO4KN9J1Bx5AzP2SFijooBSVVI9CwuVNwzK3INIyIiWTh0rx72qBiQ3KJBLC5ERGQ8wtA91wVSBwMVA5JbNIjFhYiIIkfOUA6H7tXHoR8DEiok2huaAh7sJrRPiWNxISKiyJA7lMOhe/WxR8WA5FRIDKW4EBO7iIj8SZ0blQzlcOhefexRMSihQqJaxYWY2EVE5E/q3Cg1lGNC+1DOyEIbEswmDt1rgIGKgQWrkKiEWE0W4W6ANVmIKB7JOTdaU5IUDeVw6F59HPoxuASzCcU9szC277Uo7pkV0nAPE7uIiLzJPTfaHcqGcrQauo9nDFRinJLELiKieCH33Fh5TN650XMoJ9jihuzBVo5DPzGOiV1ERP7knvP+sPNbmE2AWKez2FCOWkP3xEAl5jGxi4jIn5JzXrAgBRAfyhGG7ik8HPqJcUJil1gMb0J7hjsTu4gonkidGwPxjUU4lBMZ7FGJcUJi17RVlTABXoljTOwiongV7NwoxukC5o3ug+wLine truncated
|
||||
"text/plain": [
|
||||
"<Figure size 640x480 with 1 Axes>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"fig = plt.figure()\n",
|
||||
"ax = fig.add_subplot(111)\n",
|
||||
"ax.scatter(train_data[:, 0], train_data[:, 1])\n",
|
||||
"plt.show()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "e517975c",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"And now let us define the neural network. In PyTorch, neural networks always extend `nn.Module`. They define their sub-parts in their constructor, which are convolutional layers and fully connected linear layers in this case, and the method `forward` is expected to receive an input image and output the network target.\n",
|
||||
"\n",
|
||||
"The network parameters are the weights of the `Linear` layers, which are conveniently hidden here, but can be accessed if you try to access their `weights` elements."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"id": "d908ef86",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"class Network(nn.Module):\n",
|
||||
" \"\"\"\n",
|
||||
" This is our parametrized function.\n",
|
||||
" It stores all the parametrized weights theta inside the model object.\n",
|
||||
" For such a simple example data, it was not necessary to have such a complex model:\n",
|
||||
" this was only done here to show the interface provided by PyTorch.\n",
|
||||
" The forward function receives the x values and outputs an estimate of the target.\n",
|
||||
" \n",
|
||||
" The nn.Sequential object allows one to apply each step in the given list of parameter\n",
|
||||
" in a sequential way. An alternative would be to create each of these layers manually\n",
|
||||
" and apply them one after the other in the forward method.\n",
|
||||
" \"\"\"\n",
|
||||
" def __init__(self, input_dimension: int=1, output_dimension: int=1):\n",
|
||||
" \"\"\"\n",
|
||||
" Constructor. Here we initialize the weights.\n",
|
||||
" \"\"\"\n",
|
||||
" super().__init__()\n",
|
||||
"\n",
|
||||
" hidden_layer = 100\n",
|
||||
" self.model = nn.Sequential(\n",
|
||||
" nn.Linear(input_dimension, hidden_layer),\n",
|
||||
" nn.ReLU(),\n",
|
||||
" nn.Linear(hidden_layer, hidden_layer),\n",
|
||||
" nn.ReLU(),\n",
|
||||
" nn.Linear(hidden_layer, output_dimension)\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
" def forward(self, x):\n",
|
||||
" \"\"\"\n",
|
||||
" This function is called when one does my_network(x) and it represents the action\n",
|
||||
" of our parametrized function in the input.\n",
|
||||
" \"\"\"\n",
|
||||
" return self.model(x)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "9c5620dc",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Let us create one instance of this network. We also create an instance of PyTorch's `DataLoader`, which has the task of taking a given number of data elements and outputing it in a single object. This \"mini-batch\" of data is used during training, so that we do not need to load the entire data in memory during the optimization procedure.\n",
|
||||
"\n",
|
||||
"We also create an instance of the Adam optimizer, which is used to tune the parameters of the network."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"id": "988e1979",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"network = Network()\n",
|
||||
"B = 10\n",
|
||||
"loader = torch.utils.data.DataLoader(my_dataset, batch_size=B)\n",
|
||||
"optimizer = torch.optim.Adam(network.parameters(), lr=1e-3)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "3ee54520",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Now we actually repeatedly try to optimize the network parameters. Each time we go through all the data we have, we go through one \"epoch\". For each epoch, we take several \"mini-batches\" of data (given by the `DataLoader` in `loader`) and use it to make one training step."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 9,
|
||||
"id": "d15d655d",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Epoch 0/100: average loss 452.59888\n",
|
||||
"Epoch 1/100: average loss 339.20629\n",
|
||||
"Epoch 2/100: average loss 263.36186\n",
|
||||
"Epoch 3/100: average loss 223.31033\n",
|
||||
"Epoch 4/100: average loss 203.19063\n",
|
||||
"Epoch 5/100: average loss 191.09150\n",
|
||||
"Epoch 6/100: average loss 182.24778\n",
|
||||
"Epoch 7/100: average loss 175.20755\n",
|
||||
"Epoch 8/100: average loss 169.28958\n",
|
||||
"Epoch 9/100: average loss 164.11071\n",
|
||||
"Epoch 10/100: average loss 159.43586\n",
|
||||
"Epoch 11/100: average loss 155.22682\n",
|
||||
"Epoch 12/100: average loss 151.44842\n",
|
||||
"Epoch 13/100: average loss 148.07120\n",
|
||||
"Epoch 14/100: average loss 145.08345\n",
|
||||
"Epoch 15/100: average loss 142.42645\n",
|
||||
"Epoch 16/100: average loss 140.11430\n",
|
||||
"Epoch 17/100: average loss 138.10618\n",
|
||||
"Epoch 18/100: average loss 136.35764\n",
|
||||
"Epoch 19/100: average loss 134.83290\n",
|
||||
"Epoch 20/100: average loss 133.48525\n",
|
||||
"Epoch 21/100: average loss 132.29287\n",
|
||||
"Epoch 22/100: average loss 131.22552\n",
|
||||
"Epoch 23/100: average loss 130.26203\n",
|
||||
"Epoch 24/100: average loss 129.36947\n",
|
||||
"Epoch 25/100: average loss 128.54001\n",
|
||||
"Epoch 26/100: average loss 127.77259\n",
|
||||
"Epoch 27/100: average loss 127.05234\n",
|
||||
"Epoch 28/100: average loss 126.35698\n",
|
||||
"Epoch 29/100: average loss 125.67996\n",
|
||||
"Epoch 30/100: average loss 125.02878\n",
|
||||
"Epoch 31/100: average loss 124.39722\n",
|
||||
"Epoch 32/100: average loss 123.77292\n",
|
||||
"Epoch 33/100: average loss 123.15925\n",
|
||||
"Epoch 34/100: average loss 122.55854\n",
|
||||
"Epoch 35/100: average loss 121.97138\n",
|
||||
"Epoch 36/100: average loss 121.39923\n",
|
||||
"Epoch 37/100: average loss 120.84461\n",
|
||||
"Epoch 38/100: average loss 120.30410\n",
|
||||
"Epoch 39/100: average loss 119.77399\n",
|
||||
"Epoch 40/100: average loss 119.26286\n",
|
||||
"Epoch 41/100: average loss 118.77323\n",
|
||||
"Epoch 42/100: average loss 118.30117\n",
|
||||
"Epoch 43/100: average loss 117.84936\n",
|
||||
"Epoch 44/100: average loss 117.40730\n",
|
||||
"Epoch 45/100: average loss 116.97969\n",
|
||||
"Epoch 46/100: average loss 116.56799\n",
|
||||
"Epoch 47/100: average loss 116.17331\n",
|
||||
"Epoch 48/100: average loss 115.78789\n",
|
||||
"Epoch 49/100: average loss 115.42027\n",
|
||||
"Epoch 50/100: average loss 115.06993\n",
|
||||
"Epoch 51/100: average loss 114.73861\n",
|
||||
"Epoch 52/100: average loss 114.42548\n",
|
||||
"Epoch 53/100: average loss 114.11974\n",
|
||||
"Epoch 54/100: average loss 113.82660\n",
|
||||
"Epoch 55/100: average loss 113.55279\n",
|
||||
"Epoch 56/100: average loss 113.29264\n",
|
||||
"Epoch 57/100: average loss 113.04481\n",
|
||||
"Epoch 58/100: average loss 112.80599\n",
|
||||
"Epoch 59/100: average loss 112.58345\n",
|
||||
"Epoch 60/100: average loss 112.36975\n",
|
||||
"Epoch 61/100: average loss 112.16276\n",
|
||||
"Epoch 62/100: average loss 111.96547\n",
|
||||
"Epoch 63/100: average loss 111.77942\n",
|
||||
"Epoch 64/100: average loss 111.60994\n",
|
||||
"Epoch 65/100: average loss 111.44945\n",
|
||||
"Epoch 66/100: average loss 111.29771\n",
|
||||
"Epoch 67/100: average loss 111.14783\n",
|
||||
"Epoch 68/100: average loss 111.00336\n",
|
||||
"Epoch 69/100: average loss 110.86740\n",
|
||||
"Epoch 70/100: average loss 110.73891\n",
|
||||
"Epoch 71/100: average loss 110.61783\n",
|
||||
"Epoch 72/100: average loss 110.50365\n",
|
||||
"Epoch 73/100: average loss 110.39520\n",
|
||||
"Epoch 74/100: average loss 110.28639\n",
|
||||
"Epoch 75/100: average loss 110.17986\n",
|
||||
"Epoch 76/100: average loss 110.07611\n",
|
||||
"Epoch 77/100: average loss 109.97478\n",
|
||||
"Epoch 78/100: average loss 109.87914\n",
|
||||
"Epoch 79/100: average loss 109.78643\n",
|
||||
"Epoch 80/100: average loss 109.69742\n",
|
||||
"Epoch 81/100: average loss 109.60897\n",
|
||||
"Epoch 82/100: average loss 109.52515\n",
|
||||
"Epoch 83/100: average loss 109.44434\n",
|
||||
"Epoch 84/100: average loss 109.36794\n",
|
||||
"Epoch 85/100: average loss 109.30291\n",
|
||||
"Epoch 86/100: average loss 109.23819\n",
|
||||
"Epoch 87/100: average loss 109.17980\n",
|
||||
"Epoch 88/100: average loss 109.12350\n",
|
||||
"Epoch 89/100: average loss 109.06770\n",
|
||||
"Epoch 90/100: average loss 109.01470\n",
|
||||
"Epoch 91/100: average loss 108.96011\n",
|
||||
"Epoch 92/100: average loss 108.90628\n",
|
||||
"Epoch 93/100: average loss 108.86134\n",
|
||||
"Epoch 94/100: average loss 108.81156\n",
|
||||
"Epoch 95/100: average loss 108.76630\n",
|
||||
"Epoch 96/100: average loss 108.71505\n",
|
||||
"Epoch 97/100: average loss 108.67075\n",
|
||||
"Epoch 98/100: average loss 108.62780\n",
|
||||
"Epoch 99/100: average loss 108.58241\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"epochs = 100\n",
|
||||
"# for each epoch\n",
|
||||
"for epoch in range(epochs):\n",
|
||||
" losses = list()\n",
|
||||
" # for each mini-batch given by the loader:\n",
|
||||
" for batch in loader:\n",
|
||||
" # get the input in the mini-batch\n",
|
||||
" # this has size (B, C)\n",
|
||||
" # where B is the mini-batch size\n",
|
||||
" # C is the number of features (1 in this case)\n",
|
||||
" features = batch[\"data\"]\n",
|
||||
" # get the targets in the mini-batch (there shall be B of them)\n",
|
||||
" target = batch[\"target\"]\n",
|
||||
" # get the output of the neural network:\n",
|
||||
" prediction = network(features)\n",
|
||||
" \n",
|
||||
" # calculate the loss function being minimized\n",
|
||||
" # in this case, it is the mean-squared error between the prediction and the target values\n",
|
||||
" loss = F.mse_loss(prediction, target)\n",
|
||||
" # exactly equivalent to:\n",
|
||||
" #loss = ((prediction - target)**2).mean()\n",
|
||||
"\n",
|
||||
" # clean the optimizer temporary gradient storage\n",
|
||||
" optimizer.zero_grad()\n",
|
||||
" # calculate the gradient of the loss function as a function of the gradients\n",
|
||||
" loss.backward()\n",
|
||||
" # ask the Adam optimizer to change the parameters in the direction of - gradient\n",
|
||||
" # Adam scales the gradient by a constant which is adaptively tuned\n",
|
||||
" # take a look at the Adam paper for more details: https://arxiv.org/abs/1412.6980\n",
|
||||
" optimizer.step()\n",
|
||||
" losses.append(loss.detach().cpu().item())\n",
|
||||
" avg_loss = np.mean(np.array(losses))\n",
|
||||
" print(f\"Epoch {epoch}/{epochs}: average loss {avg_loss:.5f}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "a4980bf4",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Let us check what the network says about some new data it has never seen before."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"id": "09646d29",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"test_data = generate_data(N=1000)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "e315b5dc",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"And now we can plot again the new images, now showing what the network tells us about it."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 11,
|
||||
"id": "7a06a4c0",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"predicted = network(torch.from_numpy(test_data[:,0:1])).detach().numpy()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 12,
|
||||
"id": "bab0ce43",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAkAAAAGwCAYAAABB4NqyAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAB7jklEQVR4nO3dfXyN9f8H8Nd1nZ3d2o252yY3c1PM3N8OFeVmiO5LKEpTQm7qFypJKrpRIiGFSuiWhNbXTZFMRIs1CU3ERoyNse3sXNfvj+McO9u5ua5zf3Zez8djj9o517nOZ8d2zvv6fN6f91uQZVkGERERUQARvT0AIiIiIk9jAEREREQBhwEQERERBRwGQERERBRwGAARERFRwGEARERERAGHARAREREFnCBvD8AXSZKEU6dOITIyEoIgeHs4REREpIAsy7h48SISEhIgirbneBgAWXDq1CnUq1fP28MgIiIiB5w4cQLXXXedzWMYAFkQGRkJwPACRkVFeXk0REREpERhYSHq1atn+hy3hQGQBcZlr6ioKAZAREREfkZJ+gqToImIiCjgMAAiIiKigMMAiIiIiAIOAyAiIiIKOAyAiIiIKOAwACIiIqKAwwCIiIiIAg4DICIiIgo4PhcAbd++HQMHDkRCQgIEQcDatWvN7h8xYgQEQTD7Sk1NNTsmPz8fQ4cORVRUFGJiYjBy5EhcunTJgz8FERER+TKfC4CKiorQunVrLFiwwOoxqampyM3NNX2tWrXK7P6hQ4fijz/+wKZNm7B+/Xps374do0aNcvfQ7dJLMjKOnsM3mSeRcfQc9JLs7SEREREFJJ9rhdGvXz/069fP5jEhISGIi4uzeN/BgweRnp6OPXv2oEOHDgCA+fPno3///njzzTeRkJBQ6TElJSUoKSkxfV9YWOjET2BZelYuZnybjdyCYtNt8dGhmD4wCanJ8S5/PiIiIrLO52aAlPjxxx9Ru3Zt3HDDDRg9ejTOnTtnui8jIwMxMTGm4AcAevXqBVEU8csvv1g836xZsxAdHW36cnUn+PSsXIxesc8s+AGAvIJijF6xD+lZuS59PiIiInc5duwYBEFAZmamt4fiFL8LgFJTU/Hxxx9jy5YteO2117Bt2zb069cPer0eAJCXl4fatWubPSYoKAixsbHIy8uzeM6pU6eioKDA9HXixAmXjVcvyZjxbTYsLXYZb5vxbTaXw4iI/IQn0xkq5rxW/HrxxRedOnfFPFt3GTFiBO644w6PPJdSPrcEZs/gwYNN/9+yZUu0atUKjRs3xo8//ohbb73VoXOGhIQgJCTEVUM0szsnv9LMT3kygNyCYuzOyUdK4xpuGQMREbmGp9MZcnOvrRB89tlneOGFF3Do0CHTbdWqVXP5cwYKv5sBqqhRo0aoWbMmjhw5AgCIi4vDmTNnzI4pKytDfn6+1bwhdzpz0Xrw48hxRETkHd5IZ4iLizN9RUdHQxAEs9tWr16N5s2bIzQ0FM2aNcN7771nemxpaSnGjh2L+Ph4hIaGokGDBpg1axYAoGHDhgCAO++8E4IgmL63ZPfu3Wjbti1CQ0PRoUMH/Pbbb2b36/V6jBw5EomJiQgLC8MNN9yAd955x3T/iy++iI8++gjffPONaebqxx9/BABMnjwZ119/PcLDw9GoUSNMmzYNOp3ONS+eHX43A1TRv//+i3PnziE+3hB5p6Sk4MKFC9i7dy/at28PANi6dSskSULnzp09Pr7akaEuPY6IiDzPXjqDAEM6Q++kOGhEwSNj+vTTT/HCCy/g3XffRdu2bfHbb78hLS0NERERGD58OObNm4d169bh888/R/369XHixAlTiseePXtQu3ZtLFu2DKmpqdBoNBaf49KlS7jtttvQu3dvrFixAjk5ORg/frzZMZIk4brrrsMXX3yBGjVqYOfOnRg1ahTi4+Nx33334emnn8bBgwdRWFiIZcuWAQBiY2MBAJGRkVi+fDkSEhJw4MABpKWlITIyEs8884wbXzkDnwuALl26ZJrNAYCcnBxkZmYiNjYWsbGxmDFjBu6++27ExcXh6NGjeOaZZ9CkSRP07dsXANC8eXOkpqYiLS0NixYtgk6nw9ixYzF48GCLO8DcrVNiLOKjQ5FXUGzxD0cAEBcdik6JsZ4eGhERKeSL6QzTp0/HnDlzcNdddwEAEhMTkZ2djcWLF2P48OE4fvw4mjZtiu7du0MQBDRo0MD02Fq1agEAYmJibK6OrFy5EpIk4cMPP0RoaChatGiBf//9F6NHjzYdo9VqMWPGDNP3iYmJyMjIwOeff4777rsP1apVQ1hYGEpKSio91/PPP2/6/4YNG+Lpp5/G6tWrAzMA+vXXX9GzZ0/T95MmTQIADB8+HAsXLsT+/fvx0Ucf4cKFC0hISECfPn0wc+ZMsxyeTz/9FGPHjsWtt94KURRx9913Y968eR7/WQBAIwqYPjAJo1fsgwCYBUHGa4TpA5M8dsVARETq+Vo6Q1FREY4ePYqRI0ciLS3NdHtZWRmio6MBGBKPe/fujRtuuAGpqam47bbb0KdPH1XPc/DgQbRq1QqhoddWKVJSUiodt2DBAixduhTHjx/HlStXUFpaijZt2tg9/2effYZ58+bh6NGjuHTpEsrKyhAVFaVqjI7yuQCoR48ekGXrGfXff/+93XPExsZi5cqVrhyWU1KT47FwWLtKiXNxrANEROQXfC2dwdjdYMmSJZXSO4zLWe3atUNOTg6+++47bN68Gffddx969eqFL7/80qVjWb16NZ5++mnMmTMHKSkpiIyMxBtvvGG19IxRRkYGhg4dihkzZqBv376Ijo7G6tWrMWfOHJeOzxqfC4CqqtTkePROisPunHycuViM2pGGZS/O/BAR+T5fS2eoU6cOEhIS8Pfff2Po0KFWj4uKisL999+P+++/H/fccw9SU1ORn5+P2NhYaLVaUwkZa5o3b45PPvkExcXFplmgXbt2mR3z888/o2vXrnjiiSdMtx09etTsmODg4ErPtXPnTjRo0ADPPfec6bZ//vnH9g/uQn6/C8yfaEQBKY1r4PY2dZHSuAaDHyIiP2FMZwCupS8YeSudYcaMGZg1axbmzZuHv/76CwcOHMCyZcvw1ltvAQDeeustrFq1Cn/++Sf++usvfPHFF4iLi0NMTAwAQ87Nli1bkJeXh/Pnz1t8jiFDhkAQBKSlpSE7OxsbN27Em2++aXZM06ZN8euvv+L777/HX3/9hWnTpmHPnj1mxzRs2BD79+/HoUOHcPbsWeh0OjRt2hTHjx/H6tWrcfToUcybNw9r1qxx/QtlBQMgIiIiBYzpDHHR5stccdGhWDisncfTGR599FF88MEHWLZsGVq2bImbb74Zy5cvR2JiIgDDDqvXX38dHTp0QMeOHXHs2DFs3LgRomj46J8zZw42bdqEevXqoW3bthafo1q1avj2229x4MABtG3bFs899xxee+01s2Mee+wx3HXXXbj//vvRuXNnnDt3zmw2CADS0tJwww03oEOHDqhVqxZ+/vlnDBo0CBMnTsTYsWPRpk0b7Ny5E9OmTXPDK2WZINtKuAlQhYWFiI6ORkFBgceSsYiIyD/oJZnpDD5Kzec3c4CIiIhUMKYzkH/jEhgREREFHAZAREREFHAYABEREVHAYQBEREREAYcBEBEREQUcBkBEREQUcBgAERERUcBhAEREREQYMWIE7rjjDtP3PXr0wIQJEzw+jh9//BGCIODChQtufR4GQERERD5sxIgREAQBgiAgODgYTZo0wUsvvYSysjK3Pu/XX3+NmTNnKjrWU0GLK7ESNBERkRqSHvhnJ3DpNFCtDtCgKyBq3PqUqampWLZsGUpKSrBx40aMGTMGWq0WU6dONTuutLQUwcHBLnnO2FjPdLb3Fs4AERERKZW9DpibDHx0G/DVSMN/5yYbbnejkJAQxMXFoUGDBhg9ejR69eqFdevWmZatXnnlFSQkJOCGG24AAJw4cQL33XcfYmJiEBsbi9tvvx3Hjh0znU+v12PSpEmIiYlBjRo18Mwzz6Bia9CKS2AlJSWYPHky6tWrh5CQEDRp0gQffvghjh07hp49ewIAqlevDkEQMGLECACAJEmYNWsWEhMTERYWhtatW+PLL780e56NGzfi+uuvR1hYGHr27Gk2TndiAERERKRE9jrg84eAwlPmtxfmGm53cxBUXlhYGEpLSwEAW7ZswaFDh7Bp0yasX78eOp0Offv2RWRkJH766Sf8/PPPqFatGlJTU02PmTNnDpYvX46lS5dix44dyM/Px5o1a2w+50MPPYRVq1Zh3rx5OHjwIBYvXoxq1aqhXr16+OqrrwAAhw4dQm5uLt555x0AwKxZs/Dxxx9j0aJF+OOPPzBx4kQMGzYM27ZtA2AI1O666y4MHDgQmZmZePTRRzFlyhR3vWxmuARGRERkj6QH0icDkC3cKQMQgPQpQLMBbl0Ok2UZW7Zswffff49x48bhv//+Q0REBD744APT0teKFSsgSRI++OADCIKhS/2yZcsQExODH3/8EX369MHcuXMxdepU3HXXXQCARYsW4fvvv7f6vH/99Rc+//xzbNq0Cb169QIANGrUyHS/cbmsdu3aiImJAWCYMXr11VexefNmpKSkmB6zY8cOLF68GDfffDMWLlyIxo0bY86cOQCAG264AQcOHMBrr73mwlfNMgZARERE9vyzs/LMjxkZKDxpOC7xRpc//fr161GtWjXodDpIkoQhQ4bgxRdfxJgxY9CyZUuzvJ/ff/8dR44cQWRkpNk5iouLcfToURQUFCA3NxedO3c23RcUFIQOHTpUWgYzyszMhEajwc0336x4zEeOHMHly5fRu3dvs9tLS0vRtm1bAMDBgwfNxgHAFCy5GwMgIiIiey6ddu1xKvXs2RMLFy5EcHAwEhISEBR07eM7IiLCfAiXLqF9+/b49NNPK52nVq1aDj1/WFiY6sdcunQJALBhwwbUrVvX7L6QkBCLine truncated
|
||||
"text/plain": [
|
||||
"<Figure size 640x480 with 1 Axes>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"fig = plt.figure()\n",
|
||||
"ax = fig.add_subplot(111)\n",
|
||||
"ax.scatter(test_data[:, 0], test_data[:, 1], label=\"Test data\")\n",
|
||||
"ax.scatter(test_data[:, 0], predicted, label=\"Predicted\")\n",
|
||||
"ax.set(xlabel=\"$x$\", ylabel=\"$f(x)$\")\n",
|
||||
"#ax.set_yscale('log')\n",
|
||||
"plt.legend(frameon=False)\n",
|
||||
"plt.show()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "ce33b86f-2de6-481f-a663-7ccb3949946c",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## What is the impact of the loss function?\n",
|
||||
"\n",
|
||||
"What if instead of the mean squared error, you used the mean absolute error (L1 loss)? How would the result change? What does that mean for the assumptions of the model?\n",
|
||||
"\n",
|
||||
"Try replacing `F.mse_loss` above with `F.l1_loss` or `F.huber_loss`. Look for the documentation on those losses to understand what they do."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "98ed597c-ef43-456a-a651-6660d4f7d62b",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "27e5ca02-6df6-4bc9-a72a-6ff38f5d6aeb",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Regularization\n",
|
||||
"\n",
|
||||
"You may try to apply regularization to the fit by setting the `weight_decay` argument of the optimizer. For example:\n",
|
||||
"\n",
|
||||
"```\n",
|
||||
"optimizer = torch.optim.Adam(network.parameters(), lr=1e-3, weight_decay=1e-3)\n",
|
||||
"```\n",
|
||||
"\n",
|
||||
"This adds an extra term in the loss function proportional to the sum of square of weights, similarly to how it was discussed in the lecture.\n",
|
||||
"\n",
|
||||
"If one wants to add other types of regularization terms to the loss function, one may also do that explicitly, by changing the way the loss function is calculated.\n",
|
||||
"For example, to add an L1 regularization (absolute value of the parameters, which corresponds to the Laplace prior of parameters), one may add the following lines after calculating the loss function:\n",
|
||||
"\n",
|
||||
"```\n",
|
||||
"l1_lambda = 0.001\n",
|
||||
"l1_norm = sum(torch.linalg.norm(p, 1) for p in model.parameters())\n",
|
||||
"loss = loss + l1_lambda * l1_norm\n",
|
||||
"```\n",
|
||||
"\n",
|
||||
"Try those out!"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "2d90a6c7-60e1-4622-acee-181d41e46b32",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.10.6"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 77 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 78 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 69 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 69 KiB |
+2536
File diff suppressed because it is too large.
Load diff
Binary file not shown.
|
After Width: | Height: | Size: 16 KiB |
Reference in new issue
Block a user