{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "1bba0128",
   "metadata": {},
   "source": [
    "# Non-linear fits as extensions of line fits\n",
    "\n",
    "Line fits are really useful and they can be easily modified to perform non-linear fits too.\n",
    "\n",
    "One way of modifying them is to simply \"create\" new variables. For example, if one would like to fit a parabola $y = f(x) = a x^2 + b x + c$, one could define a new variable $u = x^2$ and fit the \"plane\" $y = f(x, u) = a u + b x + c$. This fit may be performed with the methods described previously. There are even tools in standard libraries to aid fitting using such procedures.\n",
    "\n",
    "An alternative is to perform a non-linear transformation on $x$, in such a way that the important non-linear effects are captured.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "896d5976-fe56-4a2b-8093-c01fc53b0068",
   "metadata": {},
   "outputs": [],
   "source": [
    "from typing import Tuple\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "import matplotlib.pylab as pylab\n",
    "params = {'legend.fontsize': 'x-large',\n",
    "          'figure.figsize': (15, 5),\n",
    "         'axes.labelsize': 'x-large',\n",
    "         'axes.titlesize':'x-large',\n",
    "         'xtick.labelsize':'x-large',\n",
    "         'ytick.labelsize':'x-large'}\n",
    "pylab.rcParams.update(params)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "843e1f00-4846-4e15-9861-6e9e866a6d62",
   "metadata": {},
   "source": [
    "## Fitting polynomials\n",
    "\n",
    "scikit-learn offers the possibility of easily creating new variables corresponding to the polynomial expansions of $x$.\n",
    "This can be done using: https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.PolynomialFeatures.html#sklearn.preprocessing.PolynomialFeatures\n",
    "\n",
    "Let's look at an example of how this could be done:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb1286f0",
   "metadata": {},
   "source": [
    "The simulated example data will be $f(x) = 0.01 x^2 + 0.5 x + 0.3 + \\epsilon$, where $\\epsilon \\sim \\mathcal{N}(\\mu=0, \\sigma=0.5)$.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "aef54435-78af-4b13-aee2-2bc576c773cd",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def generate_data(N: int) -> np.ndarray:\n",
    "    x = 2*np.random.randn(N, 1)\n",
    "    epsilon = 0.5*np.random.randn(N, 1)\n",
    "    z = 0.1*x**2 + 0.5*x + 0.3 + epsilon\n",
    "    return np.concatenate((x, z), axis=1).astype(np.float32)\n",
    "\n",
    "data = generate_data(N=100)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "00e098e8-0082-4451-927a-073cf242787e",
   "metadata": {},
   "source": [
    "Let's plot it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "08d9daaf-25c5-4d06-b4a6-6cecaaa35651",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.collections.PathCollection at 0x7f1a8832b4f0>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 24620 characters]",
      "text/plain": [
       "<Figure size 1500x500 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.scatter(data[:,0], data[:,1])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9932b25-ffa6-4690-965e-6822643a8edd",
   "metadata": {},
   "source": [
    "Now, let's fit it with a line:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "50578b88-f210-4a35-bada-71a26e45e689",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-1 {color: black;}#sk-container-id-1 pre{padding: 0;}#sk-container-id-1 div.sk-toggleable {background-color: white;}#sk-container-id-1 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-1 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-1 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-1 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-1 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-1 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-1 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-1 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-1 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-1 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-1 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-1 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-1 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-1 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-1 div.sk-item {position: relative;z-index: 1;}#sk-container-id-1 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-1 div.sk-item::before, #sk-container-id-1 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-1 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-1 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-1 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-1 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-1 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-1 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-1 div.sk-label-container {text-align: center;}#sk-container-id-1 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-1 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-1\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>BayesianRidge()</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-1\" type=\"checkbox\" checked><label for=\"sk-estimator-id-1\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">BayesianRidge</label><div class=\"sk-toggleable__content\"><pre>BayesianRidge()</pre></div></div></div></div></div>"
      ],
      "text/plain": [
       "BayesianRidge()"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.linear_model import BayesianRidge\n",
    "\n",
    "model = BayesianRidge()\n",
    "model.fit(data[:, [0]], data[:, 1])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "9e317638-ed75-418f-af18-da8dd585265a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7f1a7ee36640>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 55972 characters]",
      "text/plain": [
       "<Figure size 1500x500 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "x_axis = np.linspace(-5, 5, 100)\n",
    "y_p, y_u = model.predict(x_axis[:, np.newaxis], return_std=True)\n",
    "\n",
    "plt.scatter(data[:, 0], data[:, 1], c='b', label=\"Data\")\n",
    "plt.plot(x_axis, y_p, c='r', lw=2, ls='--', label=\"Prediction\")\n",
    "plt.fill_between(x_axis, y_p-2*y_u, y_p+2*y_u, color='red', alpha=0.2, label=\"95% CL uncertainty\")\n",
    "plt.legend(frameon=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "446bb34d-7ccc-481a-8514-c1ee20fcc5e0",
   "metadata": {},
   "source": [
    "This has a large uncertainty and it really does not capture the quadratic effect! Let's try to fit it adding one extra variable $u=x^2$. We could just calculate this variable and add it to the dataset, but when working with many dimensions and with higher powers (not only second-order polynomials), it is easier to use the auxiliary object `PolynomialFeatures`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "53a6c63b-0144-424d-b191-3dd8b6a090d4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-3 {color: black;}#sk-container-id-3 pre{padding: 0;}#sk-container-id-3 div.sk-toggleable {background-color: white;}#sk-container-id-3 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-3 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-3 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-3 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-3 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-3 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-3 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-3 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-3 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-3 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-3 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-3 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-3 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-3 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-3 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-3 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-3 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-3 div.sk-item {position: relative;z-index: 1;}#sk-container-id-3 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-3 div.sk-item::before, #sk-container-id-3 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-3 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-3 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-3 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-3 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-3 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-3 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-3 div.sk-label-container {text-align: center;}#sk-container-id-3 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-3 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-3\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>BayesianRidge()</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-3\" type=\"checkbox\" checked><label for=\"sk-estimator-id-3\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">BayesianRidge</label><div class=\"sk-toggleable__content\"><pre>BayesianRidge()</pre></div></div></div></div></div>"
      ],
      "text/plain": [
       "BayesianRidge()"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.linear_model import BayesianRidge\n",
    "from sklearn.preprocessing import PolynomialFeatures\n",
    "\n",
    "feature_extractor = PolynomialFeatures(2)\n",
    "data_preprocessed = feature_extractor.fit_transform(data[:, [0]])\n",
    "\n",
    "model = BayesianRidge()\n",
    "model.fit(data_preprocessed, data[:, 1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "c9fd10d6-1c1b-4707-ad9f-bfed56e3ab9e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7f1a803ee970>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 58068 characters]",
      "text/plain": [
       "<Figure size 1500x500 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "preprocessed_x_axis = feature_extractor.fit_transform(x_axis[:, np.newaxis])\n",
    "y_p, y_u = model.predict(preprocessed_x_axis, return_std=True)\n",
    "\n",
    "plt.scatter(data[:, 0], data[:, 1], c='b', label=\"Data\")\n",
    "plt.plot(x_axis, y_p, c='r', lw=2, ls='--', label=\"Prediction\")\n",
    "plt.fill_between(x_axis, y_p-2*y_u, y_p+2*y_u, color='red', alpha=0.2, label=\"95% CL uncertainty\")\n",
    "plt.legend(frameon=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff9c4999-5ef9-42da-a748-6b1384f5651f",
   "metadata": {},
   "source": [
    "This looks much better! We can also do this in a more streamlined way using the concept of `Pipeline` in scikit-learn, which does this transformation for us in a more transparent way."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "9cc91240-2ad2-4912-b4a3-96e0d12f119a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-4 {color: black;}#sk-container-id-4 pre{padding: 0;}#sk-container-id-4 div.sk-toggleable {background-color: white;}#sk-container-id-4 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-4 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-4 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-4 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-4 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-4 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-4 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-4 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-4 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-4 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-4 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-4 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-4 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-4 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-4 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-4 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-4 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-4 div.sk-item {position: relative;z-index: 1;}#sk-container-id-4 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-4 div.sk-item::before, #sk-container-id-4 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-4 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-4 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-4 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-4 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-4 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-4 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-4 div.sk-label-container {text-align: center;}#sk-container-id-4 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-4 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-4\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>Pipeline(steps=[(&#x27;feature_extractor&#x27;, PolynomialFeatures()),\n",
       "                (&#x27;fit&#x27;, BayesianRidge())])</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-4\" type=\"checkbox\" ><label for=\"sk-estimator-id-4\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">Pipeline</label><div class=\"sk-toggleable__content\"><pre>Pipeline(steps=[(&#x27;feature_extractor&#x27;, PolynomialFeatures()),\n",
       "                (&#x27;fit&#x27;, BayesianRidge())])</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-5\" type=\"checkbox\" ><label for=\"sk-estimator-id-5\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">PolynomialFeatures</label><div class=\"sk-toggleable__content\"><pre>PolynomialFeatures()</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-6\" type=\"checkbox\" ><label for=\"sk-estimator-id-6\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">BayesianRidge</label><div class=\"sk-toggleable__content\"><pre>BayesianRidge()</pre></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "Pipeline(steps=[('feature_extractor', PolynomialFeatures()),\n",
       "                ('fit', BayesianRidge())])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.linear_model import BayesianRidge\n",
    "from sklearn.preprocessing import PolynomialFeatures\n",
    "from sklearn.pipeline import Pipeline\n",
    "\n",
    "model = Pipeline([('feature_extractor', PolynomialFeatures(2)),\n",
    "                  ('fit', BayesianRidge())\n",
    "                 ])\n",
    "model.fit(data[:,[0]], data[:,1])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "a16075c0-d671-4d06-9cfe-ab191cddd24d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7f1a7ee748e0>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 58068 characters]",
      "text/plain": [
       "<Figure size 1500x500 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "y_p, y_u = model.predict(x_axis[:, np.newaxis], return_std=True)\n",
    "\n",
    "plt.scatter(data[:, 0], data[:, 1], c='b', label=\"Data\")\n",
    "plt.plot(x_axis, y_p, c='r', lw=2, ls='--', label=\"Prediction\")\n",
    "plt.fill_between(x_axis, y_p-2*y_u, y_p+2*y_u, color='red', alpha=0.2, label=\"95% CL uncertainty\")\n",
    "plt.legend(frameon=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b0a02e5f-ce76-4149-b51a-e114dccccf3c",
   "metadata": {},
   "source": [
    "## Kernel methods\n",
    "\n",
    "Another simple extension of linear fits is to apply a non-linear transformation on the input $x$ values indirectly. The logic here is the following: when looking at the derivation of the equations for linear fit results, we see that the results don't really depend on the $x$ values of each point directly. They only depend on the dot-product between $x$ values between different points, that is, on $\\vec{x}_1 \\cdot \\vec{x}_2 = \\sum_i x_{1i} x_{2i}$. If we substituted the *definition of dot product* in our space with something else, we are effectively changing how we measure distances in the space of points. This change in the curvature of the space may detect non-linear relationships more easily.\n",
    "\n",
    "When using this method we are effectively performing the fit:\n",
    "\n",
    "$y = f(x) = \\sum_i w_i \\phi_i(x_i) + w_0$\n",
    "\n",
    "The only change here is that the only thing that enters the method are the pre-transformed versions of the input $x_i$ components.\n",
    "We never really have to calculate $\\phi_i$, since the only thing that enters our linear fit equations for $\\phi_i(x_i)$ is actually $\\phi_i(x)_i \\phi_j(x_j)$ for all combinations of $i$ and $j$. We define this quantity kernel $K(x, x^\\prime) = \\phi(x) \\cdot \\phi(x^\\prime)$ and it is conceptualized as a measure of relationship between the different $x$ values."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "396e37cf-5597-468c-9272-30cb80c71e61",
   "metadata": {},
   "source": [
    "Let's look at an example of this using the kernel function $K(x, x^\\prime) = (\\gamma x \\cdot x^\\prime + 1)^3$. This is called the *polynomial* kernel, but you can define your own kernel."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "f294b5e9-cd9a-4c99-b9aa-08c4b0f8bfab",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-5 {color: black;}#sk-container-id-5 pre{padding: 0;}#sk-container-id-5 div.sk-toggleable {background-color: white;}#sk-container-id-5 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-5 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-5 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-5 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-5 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-5 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-5 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-5 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-5 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-5 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-5 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-5 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-5 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-5 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-5 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-5 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-5 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-5 div.sk-item {position: relative;z-index: 1;}#sk-container-id-5 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-5 div.sk-item::before, #sk-container-id-5 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-5 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-5 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-5 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-5 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-5 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-5 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-5 div.sk-label-container {text-align: center;}#sk-container-id-5 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-5 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-5\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>KernelRidge(kernel=&#x27;polynomial&#x27;)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-7\" type=\"checkbox\" checked><label for=\"sk-estimator-id-7\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">KernelRidge</label><div class=\"sk-toggleable__content\"><pre>KernelRidge(kernel=&#x27;polynomial&#x27;)</pre></div></div></div></div></div>"
      ],
      "text/plain": [
       "KernelRidge(kernel='polynomial')"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.kernel_ridge import KernelRidge\n",
    "\n",
    "model = KernelRidge(kernel=\"polynomial\")\n",
    "model.fit(data[:,[0]], data[:,1])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "33a9a495-db13-44d8-95d9-6d0ca1401164",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7f1a7d88e3d0>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 41560 characters]",
      "text/plain": [
       "<Figure size 1500x500 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "y_p = model.predict(x_axis[:, np.newaxis])\n",
    "\n",
    "plt.scatter(data[:, 0], data[:, 1], c='b', label=\"Data\")\n",
    "plt.plot(x_axis, y_p, c='r', lw=2, ls='--', label=\"Prediction\")\n",
    "plt.legend(frameon=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6d78f2e4-c687-4fbc-b5ab-3ad2b1175c03",
   "metadata": {},
   "source": [
    "This particular model implementation does not define uncertainties in the result (more on that later), but notice that the non-linearity is correctly captured by such a method."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc211612-29fc-45f4-bdd4-897778dac430",
   "metadata": {},
   "source": [
    "A different way of looking at this is that the kernel encodes a transformation to a space of functions that could fit this data. That is, there are infinite functions that fit the data, but the choice of the kernel selects some of them.\n",
    "\n",
    "The disadvantage here is that the parabolic nature of the original function is not so obvious, but the entire structure of the non-linearity is shifted to the choice of the kernel function. Choosing a different kernel leads to a very different result.\n",
    "\n",
    "The underlying code used here is actually very similar to the linear fit done previously: the key difference is that the *design* matrix is changed to depend on $K$ instead of the $x$ values directly. What we are doing here is still, behind the scenes, a linear fit!\n",
    "\n",
    "This method is called often the *kernel trick*, since it allows very complex fits with a small change in the fit procedure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9b3ab83e-4be8-41c4-ade8-4ef19c638fbd",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {},
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
