{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "1bba0128",
   "metadata": {},
   "source": [
    "# Bayesian fits\n",
    "\n",
    "This is a notebook showing a visualization of a Byesian interpretation for the line fit.\n",
    "The idea is that we generate all possible line fits and reject them if they are not compatible with the data.\n",
    "Naturally, this is not how most fits are done, but it illustrates the logic behind the Bayes Theorem when applied for fitting lines.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "23feddde",
   "metadata": {},
   "outputs": [],
   "source": [
    "from typing import Tuple\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb1286f0",
   "metadata": {},
   "source": [
    "We start by generating some fake dataset, which is simple enough that we can visualize the results easily. For this reason, the dataset will contain only two  variables.\n",
    "\n",
    "The simulated example data will be $f(x) = 0.5 x + 0.3 + \\epsilon$, where $\\epsilon \\sim \\mathcal{N}(\\mu=0, \\sigma=0.5)$.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "aef54435-78af-4b13-aee2-2bc576c773cd",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "true_alpha = 0.3\n",
    "true_beta = 0.5\n",
    "true_eps = 0.5\n",
    "def generate_data(N: int) -> np.ndarray:\n",
    "    x = 2*np.random.randn(N, 1)\n",
    "    epsilon = true_eps*np.random.randn(N, 1)\n",
    "    z = true_beta*x + true_alpha + epsilon\n",
    "    return np.concatenate((x, z), axis=1).astype(np.float32)\n",
    "\n",
    "data = generate_data(N=100)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04432613-39cd-4209-8833-7af585e7eed9",
   "metadata": {},
   "source": [
    "Given some parameters $\\alpha$ and $\\beta$, we don't know the true $f(x)$, but we assume it is something like $f(x) = \\beta x + \\alpha$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "a795239a-03eb-483a-8b07-635e3d9dbc55",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def f(x: np.ndarray, alpha: float, beta: float) -> np.ndarray:\n",
    "    return beta*x + alpha"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2fba887-dc1a-4400-ab40-59fec452dc83",
   "metadata": {},
   "source": [
    "Let's plot it for reference:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "fead461d-4d3a-48f2-a525-c518be4f1cf8",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 42328 characters]",
      "text/plain": [
       "<Figure size 800x600 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig = plt.figure(figsize=(8, 6))\n",
    "plt.scatter(data[:,0], data[:,1], label=r\"Data $\\mathcal{D}$\")\n",
    "plt.plot(data[:,0], f(data[:,0], true_alpha, true_beta), c='r', ls='-', alpha=0.5, lw=4, label=r\"True $y=w_1 x + w_0$\")\n",
    "xs = np.linspace(-2, 2, 10)\n",
    "plt.plot(xs, f(xs, true_alpha, true_beta) - 2*true_eps, c='r', ls='--', alpha=0.5, lw=4)\n",
    "plt.plot(xs, f(xs, true_alpha, true_beta) + 2*true_eps, c='r', ls='--', alpha=0.5, lw=4)\n",
    "xa = 2\n",
    "ya = f(xa, true_alpha, true_beta)\n",
    "plt.annotate(xy=(xa, ya + 2*true_eps), xytext=(xa, ya - 2*true_eps), arrowprops=dict(arrowstyle='|-|', lw=3, alpha=0.5, color=\"green\"), text=r\"\")\n",
    "plt.text(xa+0.5, ya, s=r\"$\\epsilon$ noise\", size=16)\n",
    "plt.legend(frameon=False)\n",
    "plt.xlabel(r\"$x$\")\n",
    "plt.ylabel(r\"$y$\")\n",
    "plt.savefig(\"data.png\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "00e098e8-0082-4451-927a-073cf242787e",
   "metadata": {},
   "source": [
    "Let's show several random lines and reject the ones that do not agree with the likelihood.\n",
    "\n",
    "Note that we are producing lines to show in the plots below randomly by taking samples for the posterior and therefore they would differ every time the notebook is executed, but they should still be likely samples that match the data generated above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "08d9daaf-25c5-4d06-b4a6-6cecaaa35651",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import matplotlib.pylab as pylab\n",
    "params = {'legend.fontsize': 'x-large',\n",
    "          'figure.figsize': (15, 5),\n",
    "         'axes.labelsize': 'x-large',\n",
    "         'axes.titlesize':'x-large',\n",
    "         'xtick.labelsize':'x-large',\n",
    "         'ytick.labelsize':'x-large'}\n",
    "pylab.rcParams.update(params)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "4c7002b2-5cb5-4929-8e01-964144ee60b6",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def get_log_prior(A, B):\n",
    "    #return 0.25*A/A/1/5\n",
    "    return -0.5*A**2/1**2 - 0.5*B**2/1**2 #- (2*np.pi*p_width**2)\n",
    "def sample_prior() -> Tuple[float, float]:\n",
    "    #return (2*np.random.rand() - 1)*1, (2*np.random.rand() - 1)*5\n",
    "    return 1*np.random.randn(), 1*np.random.randn()\n",
    "def get_log_likelihood(x, y, A, B, sigma):\n",
    "    error = f(x, A, B) - y\n",
    "    return np.sum(-0.5*error**2/sigma**2, axis=0) # - (2*np.pi*sigma**2)\n",
    "def plot(I: int):\n",
    "    # how many lines to show in last column\n",
    "    K = 20\n",
    "    # assumed noise std. dev.\n",
    "    sigma=0.5\n",
    "    # region of alpha and beta to plot\n",
    "    R = 4\n",
    "    # take a subset of data points\n",
    "    D = data[:I,:]\n",
    "    # make plot\n",
    "    fig = plt.figure(figsize=(4*5, 4))\n",
    "    gs = matplotlib.gridspec.GridSpec(1, 7)\n",
    "\n",
    "    # create alpha and beta grid\n",
    "    Ax = np.linspace(-R, R, 50)\n",
    "    Bx = np.linspace(-R, R, 50)\n",
    "    A, B = np.meshgrid(Ax, Bx)\n",
    "\n",
    "    # calculate the log prior\n",
    "    log_prior = get_log_prior(A, B)\n",
    "    # calculate the log likelihood\n",
    "    log_likelihood = get_log_likelihood(D[:, 0, np.newaxis, np.newaxis],\n",
    "                                        D[:, 1, np.newaxis, np.newaxis],\n",
    "                                        A[np.newaxis, ...],\n",
    "                                        B[np.newaxis, ...],\n",
    "                                        sigma=sigma)\n",
    "    # the log posterior is, except for a constant factor, just the sum of those\n",
    "    log_posterior = log_prior + log_likelihood\n",
    "\n",
    "    gsp = matplotlib.gridspec.GridSpecFromSubplotSpec(2, 2, width_ratios=[3,1], height_ratios=[3,1], subplot_spec=gs[0,0], wspace=0, hspace=0)\n",
    "    ax = plt.subplot(gsp[0,0])\n",
    "    axy = plt.subplot(gsp[0,1], sharey=ax)\n",
    "    axx = plt.subplot(gsp[1,0], sharex=ax)\n",
    "    axy.plot(np.exp(log_prior).sum(axis=1), Ax)\n",
    "    axx.plot(Bx, np.exp(log_prior).sum(axis=0))\n",
    "    axx.set(xlabel=r\"$w_1$\")\n",
    "    axx.get_xaxis().set_major_locator(matplotlib.ticker.MultipleLocator(2))\n",
    "    ax.get_yaxis().set_major_locator(matplotlib.ticker.MultipleLocator(2))\n",
    "    ax.set(title=r\"$p(w_0, w_1)$\", ylabel=r\"$w_0$\")\n",
    "    ax.get_xaxis().set_visible(False)\n",
    "    axy.get_yaxis().set_visible(False)\n",
    "    axy.get_xaxis().set_visible(False)\n",
    "    axx.get_yaxis().set_visible(False)\n",
    "    m=ax.imshow(np.exp(log_prior), extent=(-R, R, -R, R), aspect=\"auto\", origin='lower')\n",
    "    #plt.colorbar(m,ax=ax, orientation=\"horizontal\")\n",
    "\n",
    "    gsp = matplotlib.gridspec.GridSpecFromSubplotSpec(2, 2, width_ratios=[3,1], height_ratios=[3,1], subplot_spec=gs[0,1], wspace=0, hspace=0)\n",
    "    ax = plt.subplot(gsp[0,0])\n",
    "    axy = plt.subplot(gsp[0,1], sharey=ax)\n",
    "    axx = plt.subplot(gsp[1,0], sharex=ax)\n",
    "    axy.plot(np.exp(log_likelihood).sum(axis=1), Ax)\n",
    "    axx.plot(Bx, np.exp(log_likelihood).sum(axis=0))\n",
    "    axx.set(xlabel=r\"$w_1$\")\n",
    "    axx.get_xaxis().set_major_locator(matplotlib.ticker.MultipleLocator(2))\n",
    "    ax.get_yaxis().set_major_locator(matplotlib.ticker.MultipleLocator(2))\n",
    "    ax.set(title=r\"$p(\\mathcal{D}|w_0,w_1)$\", ylabel=r\"$w_0$\")\n",
    "    ax.get_xaxis().set_visible(False)\n",
    "    axy.get_yaxis().set_visible(False)\n",
    "    axy.get_xaxis().set_visible(False)\n",
    "    axx.get_yaxis().set_visible(False)\n",
    "    m=ax.imshow(np.exp(log_likelihood), extent=(-R, R, -R, R), aspect=\"auto\", origin='lower')\n",
    "    #plt.colorbar(m,ax=ax, orientation=\"horizontal\")\n",
    "\n",
    "    gsp = matplotlib.gridspec.GridSpecFromSubplotSpec(2, 2, width_ratios=[3,1], height_ratios=[3,1], subplot_spec=gs[0,2], wspace=0, hspace=0)\n",
    "    ax = plt.subplot(gsp[0,0])\n",
    "    axy = plt.subplot(gsp[0,1], sharey=ax)\n",
    "    axx = plt.subplot(gsp[1,0], sharex=ax)\n",
    "    axy.plot(np.exp(log_posterior).sum(axis=1), Ax)\n",
    "    axx.plot(Bx, np.exp(log_posterior).sum(axis=0))\n",
    "    axx.set(xlabel=r\"$w_1$\")\n",
    "    axx.get_xaxis().set_major_locator(matplotlib.ticker.MultipleLocator(2))\n",
    "    ax.get_yaxis().set_major_locator(matplotlib.ticker.MultipleLocator(2))\n",
    "    ax.set(title=r\"$p(w_1, w_0|\\mathcal{D})$\", ylabel=r\"$w_0$\")\n",
    "    ax.get_xaxis().set_visible(False)\n",
    "    axy.get_yaxis().set_visible(False)\n",
    "    axy.get_xaxis().set_visible(False)\n",
    "    axx.get_yaxis().set_visible(False)\n",
    "    m=ax.imshow(np.exp(log_posterior), extent=(-R, R, -R, R), aspect=\"auto\", origin='lower')\n",
    "    #plt.colorbar(m,ax=ax, orientation=\"horizontal\")\n",
    "\n",
    "    ax = plt.subplot(gs[0,3])\n",
    "\n",
    "    # sample alpha and beta\n",
    "    def get_line():\n",
    "        a, b = sample_prior()\n",
    "        if len(D) == 0:\n",
    "            return a, b\n",
    "        for _ in range(1000):\n",
    "            ap, bp = a + 0.1*np.random.randn(), b + 0.1*np.random.randn()\n",
    "            A = min(1, np.exp(get_log_likelihood(D[:, 0], D[:, 1], ap, bp, sigma=sigma) - get_log_likelihood(D[:, 0], D[:, 1], a, b, sigma=sigma) ))\n",
    "            if np.random.rand() <= A:\n",
    "                a = ap\n",
    "                b = bp\n",
    "        return a, b\n",
    "    for _ in range(K):\n",
    "        a, b = get_line()\n",
    "        ax.axline((0, a), slope=b, lw=2, ls='-', c='r', alpha=0.3)\n",
    "    if I > 0:\n",
    "        ax.scatter(D[:, 0], D[:,1], label=\"Data\", alpha=1.0)\n",
    "    ax.set(xlabel=\"x\", ylabel=\"y\", title=\"Data\", xlim=(-5, 5), ylim=(-5, 5))\n",
    "\n",
    "    #plt.legend(frameon=False)\n",
    "    plt.tight_layout()\n",
    "    plt.savefig(f\"bayes_reg_{I}.png\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "693827d3-c5ac-4b6d-b019-233799cd5d78",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 95144 characters]",
      "text/plain": [
       "<Figure size 2000x400 with 10 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot(I=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "a1fbd00b-e7f2-4eff-90db-e1d44d104fba",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 98708 characters]",
      "text/plain": [
       "<Figure size 2000x400 with 10 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot(I=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "ae81e3fe-0e5d-4d56-86e6-ed5f8e6ec66f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 91476 characters]",
      "text/plain": [
       "<Figure size 2000x400 with 10 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot(I=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "8149698a-81e9-4a28-b6eb-39c5a47a8f13",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 84160 characters]",
      "text/plain": [
       "<Figure size 2000x400 with 10 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot(I=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ea1a699d-8e0c-4652-ac5e-0e68dcc6251a",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "05f0627c-aa65-47e0-8f15-770daa7051e8",
   "metadata": {},
   "source": [
    "## Bayesian fits in practice\n",
    "\n",
    "The plots shown previously look very nice, but in practice this is extremely slow: we are effectively trying all lines that may fit the points and excluding the ones that do not. We can do better than this in the simple case of a line fit.\n",
    "\n",
    "Imagine we are trying to find the weights $w$ for a function $y = f(x) = w_M x_M + \\ldots + w_2 x_2 + w_1 x_1 + w_0$. We assume that any mismatch between data and the prediction is caused by some random noise $\\epsilon$ that is Gaussian and has standard deviation $1/\\sqrt{\\beta}$. In this case, we can assume that the probability that data $\\mathcal{D}$ matches our prediction for a certain $x$ is given by the likelihood $p(\\mathcal{D}|w) = \\mathcal{N}(y=f(x), \\frac{1}{\\beta})$.\n",
    "\n",
    "For a line fit, we can assume that the prior probability of the weights is given by $p(w) = \\mathcal{N}(m_0, S_0)$ (ie: a Gaussian distribution, assuming we expect the weights to be close to $m_0$ and with some covariance $S_0$) and use the data likelihood $p(\\mathcal{D}|w)$ to get the posterior probability $p(w|\\mathcal{D})$.\n",
    "\n",
    "One can use Bayes' theorem to obtain the posterior, which follows exactly a Gaussian distribution for the weights $w$ with mean $m$ and covariance $S$ given by:\n",
    "\n",
    "$m = S \\, (S_0^{-1} m_0 + \\beta \\, \\Phi^T t)$,\n",
    "\n",
    "$S = S_0^{-1} + \\beta \\, \\Phi^T \\Phi$.\n",
    "\n",
    "This has been found by calculating the derivatives of the posterior distribution and setting it to zero to find its maximum.\n",
    "In those equations, $t$ is a vector with the value of $y$ for each point in the dataset, and $\\Phi$ (called the design matrix), includes the values of the vector $x$ for each point in the dataset.\n",
    "\n",
    "$\\Phi = \\begin{pmatrix}\n",
    "x_{M1} & \\ldots & x_{21} & x_{11} \\\\\n",
    "x_{M2} & \\ldots & x_{22} & x_{12} \\\\\n",
    "\\vdots & \\ddots & \\vdots & \\vdots \\\\\n",
    "x_{MN} & \\ldots & x_{2N} & x_{1N}\n",
    "\\end{pmatrix}\n",
    "$\n",
    "\n",
    "For more information, check Chapter 3 of C. Bishop's \"Pattern Recognition and Machine Learning\"."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a52832a1-a1a5-4937-81ea-11ee35041fbf",
   "metadata": {},
   "source": [
    "The issue with this method is that one needs to make a prior assumption about how strongly we believe the weights $w$ to be close to our prior assumption $m_0$. That is: what are $m_0$ and $S_0$? These are very important to regularize the fit, which prevents the fit from trying to fit only outliers.\n",
    "\n",
    "To avoid this, hierarchical Bayesian fits are often used. In this case, we assume that $m_0$ is zero and $S_0$ depends of a result $\\lambda$ of a different probability distribution. That is, if we imagine how data would be produce in a purely theoretical way, we have a cascade of probability distribution: we pick $\\lambda$ from some distribution $p(\\lambda)$, then we pick $w$ from $p(w|\\lambda)$ and that allows us to produce data using $p(\\mathcal{D}|w, \\lambda)$.\n",
    "\n",
    "Several textbooks are dedicated to this method and the choice of such distributions. One common example is the usage of a $\\Gamma$ probability distribution for the variable $\\lambda$. This is done in the `BayesianRidge` method implemented in scikit-learn. You can find more information on this here: \n",
    "\n",
    "https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.BayesianRidge.html#sklearn.linear_model.BayesianRidge"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b95b2b78-60ab-4d7b-91e9-3468c2c3d3d9",
   "metadata": {},
   "source": [
    "Let's try using this method to fit the data in the variable `data`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "46b4cee0-db3c-4352-886f-da9867f7ef53",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-1 {\n",
       "  /* Definition of color scheme common for light and dark mode */\n",
       "  --sklearn-color-text: black;\n",
       "  --sklearn-color-line: gray;\n",
       "  /* Definition of color scheme for unfitted estimators */\n",
       "  --sklearn-color-unfitted-level-0: #fff5e6;\n",
       "  --sklearn-color-unfitted-level-1: #f6e4d2;\n",
       "  --sklearn-color-unfitted-level-2: #ffe0b3;\n",
       "  --sklearn-color-unfitted-level-3: chocolate;\n",
       "  /* Definition of color scheme for fitted estimators */\n",
       "  --sklearn-color-fitted-level-0: #f0f8ff;\n",
       "  --sklearn-color-fitted-level-1: #d4ebff;\n",
       "  --sklearn-color-fitted-level-2: #b3dbfd;\n",
       "  --sklearn-color-fitted-level-3: cornflowerblue;\n",
       "\n",
       "  /* Specific color for light theme */\n",
       "  --sklearn-color-text-on-default-background: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, black)));\n",
       "  --sklearn-color-background: var(--sg-background-color, var(--theme-background, var(--jp-layout-color0, white)));\n",
       "  --sklearn-color-border-box: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, black)));\n",
       "  --sklearn-color-icon: #696969;\n",
       "\n",
       "  @media (prefers-color-scheme: dark) {\n",
       "    /* Redefinition of color scheme for dark theme */\n",
       "    --sklearn-color-text-on-default-background: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, white)));\n",
       "    --sklearn-color-background: var(--sg-background-color, var(--theme-background, var(--jp-layout-color0, #111)));\n",
       "    --sklearn-color-border-box: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, white)));\n",
       "    --sklearn-color-icon: #878787;\n",
       "  }\n",
       "}\n",
       "\n",
       "#sk-container-id-1 {\n",
       "  color: var(--sklearn-color-text);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 pre {\n",
       "  padding: 0;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 input.sk-hidden--visually {\n",
       "  border: 0;\n",
       "  clip: rect(1px 1px 1px 1px);\n",
       "  clip: rect(1px, 1px, 1px, 1px);\n",
       "  height: 1px;\n",
       "  margin: -1px;\n",
       "  overflow: hidden;\n",
       "  padding: 0;\n",
       "  position: absolute;\n",
       "  width: 1px;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-dashed-wrapped {\n",
       "  border: 1px dashed var(--sklearn-color-line);\n",
       "  margin: 0 0.4em 0.5em 0.4em;\n",
       "  box-sizing: border-box;\n",
       "  padding-bottom: 0.4em;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-container {\n",
       "  /* jupyter's `normalize.less` sets `[hidden] { display: none; }`\n",
       "     but bootstrap.min.css set `[hidden] { display: none !important; }`\n",
       "     so we also need the `!important` here to be able to override the\n",
       "     default hidden behavior on the sphinx rendered scikit-learn.org.\n",
       "     See: https://github.com/scikit-learn/scikit-learn/issues/21755 */\n",
       "  display: inline-block !important;\n",
       "  position: relative;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-text-repr-fallback {\n",
       "  display: none;\n",
       "}\n",
       "\n",
       "div.sk-parallel-item,\n",
       "div.sk-serial,\n",
       "div.sk-item {\n",
       "  /* draw centered vertical line to link estimators */\n",
       "  background-image: linear-gradient(var(--sklearn-color-text-on-default-background), var(--sklearn-color-text-on-default-background));\n",
       "  background-size: 2px 100%;\n",
       "  background-repeat: no-repeat;\n",
       "  background-position: center center;\n",
       "}\n",
       "\n",
       "/* Parallel-specific style estimator block */\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item::after {\n",
       "  content: \"\";\n",
       "  width: 100%;\n",
       "  border-bottom: 2px solid var(--sklearn-color-text-on-default-background);\n",
       "  flex-grow: 1;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel {\n",
       "  display: flex;\n",
       "  align-items: stretch;\n",
       "  justify-content: center;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  position: relative;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item {\n",
       "  display: flex;\n",
       "  flex-direction: column;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item:first-child::after {\n",
       "  align-self: flex-end;\n",
       "  width: 50%;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item:last-child::after {\n",
       "  align-self: flex-start;\n",
       "  width: 50%;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item:only-child::after {\n",
       "  width: 0;\n",
       "}\n",
       "\n",
       "/* Serial-specific style estimator block */\n",
       "\n",
       "#sk-container-id-1 div.sk-serial {\n",
       "  display: flex;\n",
       "  flex-direction: column;\n",
       "  align-items: center;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  padding-right: 1em;\n",
       "  padding-left: 1em;\n",
       "}\n",
       "\n",
       "\n",
       "/* Toggleable style: style used for estimator/Pipeline/ColumnTransformer box that is\n",
       "clickable and can be expanded/collapsed.\n",
       "- Pipeline and ColumnTransformer use this feature and define the default style\n",
       "- Estimators will overwrite some part of the style using the `sk-estimator` class\n",
       "*/\n",
       "\n",
       "/* Pipeline and ColumnTransformer style (default) */\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable {\n",
       "  /* Default theme specific background. It is overwritten whether we have a\n",
       "  specific estimator or a Pipeline/ColumnTransformer */\n",
       "  background-color: var(--sklearn-color-background);\n",
       "}\n",
       "\n",
       "/* Toggleable label */\n",
       "#sk-container-id-1 label.sk-toggleable__label {\n",
       "  cursor: pointer;\n",
       "  display: block;\n",
       "  width: 100%;\n",
       "  margin-bottom: 0;\n",
       "  padding: 0.5em;\n",
       "  box-sizing: border-box;\n",
       "  text-align: center;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 label.sk-toggleable__label-arrow:before {\n",
       "  /* Arrow on the left of the label */\n",
       "  content: \"▸\";\n",
       "  float: left;\n",
       "  margin-right: 0.25em;\n",
       "  color: var(--sklearn-color-icon);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 label.sk-toggleable__label-arrow:hover:before {\n",
       "  color: var(--sklearn-color-text);\n",
       "}\n",
       "\n",
       "/* Toggleable content - dropdown */\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable__content {\n",
       "  max-height: 0;\n",
       "  max-width: 0;\n",
       "  overflow: hidden;\n",
       "  text-align: left;\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable__content.fitted {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable__content pre {\n",
       "  margin: 0.2em;\n",
       "  border-radius: 0.25em;\n",
       "  color: var(--sklearn-color-text);\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable__content.fitted pre {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 input.sk-toggleable__control:checked~div.sk-toggleable__content {\n",
       "  /* Expand drop-down */\n",
       "  max-height: 200px;\n",
       "  max-width: 100%;\n",
       "  overflow: auto;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {\n",
       "  content: \"▾\";\n",
       "}\n",
       "\n",
       "/* Pipeline/ColumnTransformer-specific style */\n",
       "\n",
       "#sk-container-id-1 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-label.fitted input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Estimator-specific style */\n",
       "\n",
       "/* Colorize estimator box */\n",
       "#sk-container-id-1 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-estimator.fitted input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-label label.sk-toggleable__label,\n",
       "#sk-container-id-1 div.sk-label label {\n",
       "  /* The background is the default theme color */\n",
       "  color: var(--sklearn-color-text-on-default-background);\n",
       "}\n",
       "\n",
       "/* On hover, darken the color of the background */\n",
       "#sk-container-id-1 div.sk-label:hover label.sk-toggleable__label {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "/* Label box, darken color on hover, fitted */\n",
       "#sk-container-id-1 div.sk-label.fitted:hover label.sk-toggleable__label.fitted {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Estimator label */\n",
       "\n",
       "#sk-container-id-1 div.sk-label label {\n",
       "  font-family: monospace;\n",
       "  font-weight: bold;\n",
       "  display: inline-block;\n",
       "  line-height: 1.2em;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-label-container {\n",
       "  text-align: center;\n",
       "}\n",
       "\n",
       "/* Estimator-specific */\n",
       "#sk-container-id-1 div.sk-estimator {\n",
       "  font-family: monospace;\n",
       "  border: 1px dotted var(--sklearn-color-border-box);\n",
       "  border-radius: 0.25em;\n",
       "  box-sizing: border-box;\n",
       "  margin-bottom: 0.5em;\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-estimator.fitted {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "/* on hover */\n",
       "#sk-container-id-1 div.sk-estimator:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-estimator.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Specification for estimator info (e.g. \"i\" and \"?\") */\n",
       "\n",
       "/* Common style for \"i\" and \"?\" */\n",
       "\n",
       ".sk-estimator-doc-link,\n",
       "a:link.sk-estimator-doc-link,\n",
       "a:visited.sk-estimator-doc-link {\n",
       "  float: right;\n",
       "  font-size: smaller;\n",
       "  line-height: 1em;\n",
       "  font-family: monospace;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  border-radius: 1em;\n",
       "  height: 1em;\n",
       "  width: 1em;\n",
       "  text-decoration: none !important;\n",
       "  margin-left: 1ex;\n",
       "  /* unfitted */\n",
       "  border: var(--sklearn-color-unfitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-unfitted-level-1);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link.fitted,\n",
       "a:link.sk-estimator-doc-link.fitted,\n",
       "a:visited.sk-estimator-doc-link.fitted {\n",
       "  /* fitted */\n",
       "  border: var(--sklearn-color-fitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-fitted-level-1);\n",
       "}\n",
       "\n",
       "/* On hover */\n",
       "div.sk-estimator:hover .sk-estimator-doc-link:hover,\n",
       ".sk-estimator-doc-link:hover,\n",
       "div.sk-label-container:hover .sk-estimator-doc-link:hover,\n",
       ".sk-estimator-doc-link:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "div.sk-estimator.fitted:hover .sk-estimator-doc-link.fitted:hover,\n",
       ".sk-estimator-doc-link.fitted:hover,\n",
       "div.sk-label-container:hover .sk-estimator-doc-link.fitted:hover,\n",
       ".sk-estimator-doc-link.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "/* Span, style for the box shown on hovering the info icon */\n",
       ".sk-estimator-doc-link span {\n",
       "  display: none;\n",
       "  z-index: 9999;\n",
       "  position: relative;\n",
       "  font-weight: normal;\n",
       "  right: .2ex;\n",
       "  padding: .5ex;\n",
       "  margin: .5ex;\n",
       "  width: min-content;\n",
       "  min-width: 20ex;\n",
       "  max-width: 50ex;\n",
       "  color: var(--sklearn-color-text);\n",
       "  box-shadow: 2pt 2pt 4pt #999;\n",
       "  /* unfitted */\n",
       "  background: var(--sklearn-color-unfitted-level-0);\n",
       "  border: .5pt solid var(--sklearn-color-unfitted-level-3);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link.fitted span {\n",
       "  /* fitted */\n",
       "  background: var(--sklearn-color-fitted-level-0);\n",
       "  border: var(--sklearn-color-fitted-level-3);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link:hover span {\n",
       "  display: block;\n",
       "}\n",
       "\n",
       "/* \"?\"-specific style due to the `<a>` HTML tag */\n",
       "\n",
       "#sk-container-id-1 a.estimator_doc_link {\n",
       "  float: right;\n",
       "  font-size: 1rem;\n",
       "  line-height: 1em;\n",
       "  font-family: monospace;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  border-radius: 1rem;\n",
       "  height: 1rem;\n",
       "  width: 1rem;\n",
       "  text-decoration: none;\n",
       "  /* unfitted */\n",
       "  color: var(--sklearn-color-unfitted-level-1);\n",
       "  border: var(--sklearn-color-unfitted-level-1) 1pt solid;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 a.estimator_doc_link.fitted {\n",
       "  /* fitted */\n",
       "  border: var(--sklearn-color-fitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-fitted-level-1);\n",
       "}\n",
       "\n",
       "/* On hover */\n",
       "#sk-container-id-1 a.estimator_doc_link:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 a.estimator_doc_link.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-3);\n",
       "}\n",
       "</style><div id=\"sk-container-id-1\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>BayesianRidge()</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator fitted sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-1\" type=\"checkbox\" checked><label for=\"sk-estimator-id-1\" class=\"sk-toggleable__label fitted sk-toggleable__label-arrow fitted\">&nbsp;&nbsp;BayesianRidge<a class=\"sk-estimator-doc-link fitted\" rel=\"noreferrer\" target=\"_blank\" href=\"https://scikit-learn.org/1.4/modules/generated/sklearn.linear_model.BayesianRidge.html\">?<span>Documentation for BayesianRidge</span></a><span class=\"sk-estimator-doc-link fitted\">i<span>Fitted</span></span></label><div class=\"sk-toggleable__content fitted\"><pre>BayesianRidge()</pre></div> </div></div></div></div>"
      ],
      "text/plain": [
       "BayesianRidge()"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.linear_model import BayesianRidge\n",
    "model = BayesianRidge()\n",
    "model.fit(data[:,[0]], data[:,1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "6b6be7c2-3470-4621-b0dc-1eb90e6ac266",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7fd2d20dae70>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 59808 characters]",
      "text/plain": [
       "<Figure size 1500x500 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# let's create x values to predict y for\n",
    "x_axis = np.linspace(-5, 10, 100)\n",
    "# predict it\n",
    "y_prediction, y_uncertainty = model.predict(x_axis[:, np.newaxis], return_std=True)\n",
    "\n",
    "# now plot it:\n",
    "plt.scatter(data[:,0], data[:,1], c='blue')\n",
    "plt.plot(x_axis, y_prediction, c='red', lw=2, ls='--', label=\"Fit expectation\")\n",
    "plt.fill_between(x_axis, y_prediction + 2*y_uncertainty, y_prediction - 2*y_uncertainty, color=\"red\", alpha=0.2, label=\"95% CL uncertainty\")\n",
    "plt.legend(frameon=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8f30a61e-bcad-425d-ac9b-191c93734c2a",
   "metadata": {},
   "source": [
    "## Outliers"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "29563a43-67d0-4679-a4ea-16e4f883f9de",
   "metadata": {},
   "source": [
    "In the previous notebook, we discussed that outliers are a big problem there, since there is no regularization. Since the method above includes a prior probability on the weights that is Gaussian, this restricts the variation of the weights due to outliers. This is not perfect and the degree of regularization needs to be tuned in case of large outliers, but it is more resilient than the unregularized method.\n",
    "\n",
    "Feel free to try adding outliers to the data and checking what happens to the parameters of the fit."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dc429c63-6255-4c28-ba31-4f2831d039ee",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "b6c0ced7-ce4b-420a-8c50-8b71c6c10e59",
   "metadata": {},
   "source": [
    "## Is this the only way?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13aebeb5-99b4-44e6-9106-2c0ac7c0e874",
   "metadata": {},
   "source": [
    "There are *a lot* of other line fitting methods! They make different assumptions on the priors and likelihoods. We could make the assumption that $\\epsilon$ follows a different distribution, changing the fit entirely: this is helpful to avoid sensitivity to outliers in some cases. Another change is that the priors could also follow a different distribuion. Using the Laplace distribution instead of the Gaussian often makes those parameters much more robust to outliers (this is called the *LASSO* estimator). Other line fitting methods could even assume that the prior probabilities depend on another probability distribution, called the *hyper-prior*, which then would also be included in Bayes' Theorem.\n",
    "\n",
    "One example is called Automatic Relevance Determination, which is a variation of this method, using a slightly different setup for the probability distribution of $\\lambda$. This other method has the advantage of being very resilient to noisy data in multi-dimensional cases and setting several weights $w$ to zero if they are smaller than the uncertainty.\n",
    "\n",
    "More information on an implementation of ARD can be found here: https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.ARDRegression.html#sklearn.linear_model.ARDRegression\n",
    "\n",
    "Why not try it in this simple example?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "acfac850-d650-4f25-ae1e-6108498bb743",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b6c3c7b6-8a2f-4d4e-9ae6-c2ee18232c06",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.3"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {},
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
