{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "41989e0a-dac9-4b54-9774-0dfbd3ca76c4",
   "metadata": {},
   "source": [
    "# Classification\n",
    "\n",
    "So far we have looking at how to fit functions to data. This is a regression problem. Now we are going to change pace a bit and focus on a classification task.\n",
    "The problem here is not to discover a function that fit the data points. Here we assume that there are two classes of data, labelled as class $1$ or class $-1$. We would like to separate points from those two classes.\n",
    "\n",
    "The idea is to fit a function $f(x)$, which is positive to class 1 and negative for class -1.\n",
    "\n",
    "We are going to take a more heuristic approach here, because the objective is to show one example on which we can apply the kernel trick. We will start first with a function $f(x)$ that is a line (or hyper-plane in $n$-dimensional space). After we establish how that can be done, we will use the kernel trick to obtain a non-linear separation between the two classes.\n"
   ]
  },
  {
   "attachments": {
    "SVM_margin.png": {
     "image/png": "[encoded data, 374868 characters]"
    }
   },
   "cell_type": "markdown",
   "id": "71a8e7d5",
   "metadata": {},
   "source": [
    "# Support Vector Machines\n",
    "\n",
    "Support Vector Machines (SVM) are supervised classification and regression methods which are extremely robust to outliers.\n",
    "\n",
    "Let us start with a linear classification: in this case, we try to choose a line or hyper-plane  that separates the data in two classes. The SVM model chooses this hyperplane based only on the data points close to the decision boundary and any point in time and not based on the entire dataset. Because the method focus only on points close to the decision boundary, any outlier away from the decision boundary does not have a strong influence on the choice of the boundary itself.\n",
    "\n",
    "Let us assume we have some data points $x_1, x_2, x_3, \\ldots, x_N$ and each of them belong to either class $-1$ or class $1$. Their true class is informed by the known target classes $t_1, t_2, t_3, \\ldots, t_N$. We want to choose the parameters $w$ and $b$ of a line (or hyperplane in more than 2 dimensions) such that we can easily estimate the unknown class of a new data point $x^\\prime$. For that, we define the decision boundary as:\n",
    "\n",
    "$y(x) = w^T x + b$,\n",
    "\n",
    "and we say that if $y(x^\\prime) > 0$, the data point belongs to class $1$, otherwise it belongs to class $-1$. The equation of the decision boundary is then $y(x) = 0$. See the figure below (from Wikipedia).\n",
    "\n",
    "![SVM_margin.png](https://raw.githubusercontent.com/European-XFEL/ml-lecture-oct-2023/main/SVM_margin.png)\n"
   ]
  },
  {
   "attachments": {
    "linear_classifier.png": {
     "image/png": "[encoded data, 26996 characters]"
    }
   },
   "cell_type": "markdown",
   "id": "6a764d92",
   "metadata": {},
   "source": [
    "\n",
    "Because of our choice of $t_i \\in \\{+1, -1\\}$, we can also write for all data points for which we know the true class: $t_i y(x_i) > 0$, since $y(x_i) < 0$ if $t_i = -1$ and $y (x_i) > 0$ if $t_i > 0$. The idea in SVMs is choose $w$ and $b$ to maximize the distance between the boundary points and the line. What is this distance between a given point and the decision boundary? This can be seen from simple geometrical calculations in the figure below (from Bishop (2006)). Given a known decision boundary, this distance for a point $x$ shall be $\\frac{|y(x)|}{|w|}$.\n",
    "\n",
    "![linear_classifier.png](https://raw.githubusercontent.com/European-XFEL/ml-lecture-oct-2023/main/linear_classifier.png)\n",
    "\n",
    "As we want that $t_i y(x_i) > 0$ be true for all points close to the boundary, we can use $t_i y(x_i)$ instead of $|y(x_i)|$, using $t_i$ to guarantee that the numerator is always positive, instead of relying on the absolute value. To focus only on the points close to the decision boundary, we try to find $w$ and $b$ which *maximize* the *minimum distance* between the decision boundary and the data points. As the minimum distance is determined only by the points close to the boundary, this makes the method insensitive to what happens away from the decision boundary. Our objective is then to find $w$ and $b$ that satisfy:\n",
    "\n",
    "$\\text{argmax}_{w, b} \\left[\\frac{1}{|w|} \\min_{i} t_i \\left(w^T x + b\\right) \\right]$.\n",
    "\n",
    "This is a very hard problem, but notice that we only care about the distance between the points and the boundary, and if $w$ and $b$ are both multiplied by the same number, this distance does not change. We can therefore choose a normalisation so that $t_i (w^T x_i + b) = 1$ for the point closest to the boundary. This implies that $t_k (w^T x_k + b) \\geq 1$ for all points in the dataset. Since there is always one point closest, the condition is to maximize $\\frac{1}{|w|}$ (that is, the remaining factor) for that point, which is equivalent to minimize $|w|^2$ for the closest point while miximizing the condition above.\n",
    "\n",
    "This can be summarized in simpler form using the Lagrange multiplier formalism, which translates this complex constrained optimization into a simpler unconstrained optimization problem. In that formalism we need to minimize a cost function $\\mathcal{L}$, which introduces the Lagrange multipliers $a_i$:\n",
    "\n",
    "$\\mathcal{L}(w, b, a) = \\frac{1}{2} |w|^2 - \\sum_i a_i \\left[ t_i \\left(w^T x_i + b\\right) - 1\\right]$\n",
    "\n",
    "One can set the derivative of $\\mathcal{L}$ to zero with respect to $w$, $b$ and $a$ to obtain a simpler version of this optimization problem. Further details can be consulted in Bishop (2006) or in the Wikipedia description.\n",
    "\n",
    "It is interesting to note that by using the Kernel trick, one can choose the function $y(x) = w^T \\phi(x) + b$ and with an appropriate choise of $\\phi$, we can apply a linear classification after a non-linear transformation which allows this method to be expanded into non-linear classification. It can be shown following the procedure above that the final algorithm does not depend on $\\phi(x)$, but only on $k(x, x^\\prime) = \\phi(x)\\phi(x^\\prime)$, so one needs only to specify the kernel function $k$.\n",
    "\n",
    "Similar logic can be used for regression, on which only the *worse* prediction of the feature becomes relevant.\n",
    "\n",
    "Fortunately the details of the algorithm are not needed to implement this for classification. In the following example, we show how the algorithm can be easily used from the scikit-learn modules."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fce4d8e8",
   "metadata": {},
   "source": [
    "We start by loading the necessary Python modules. If you have not yet installed them, run the following cell to install them with pip:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "300cf8d3",
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn import svm"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ecd6a69",
   "metadata": {},
   "source": [
    "Let's generate the fake data now to have something to cluster."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "4959a292",
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_data(N: int) ->np.ndarray:\n",
    "    assert N > 1\n",
    "    loc = [np.array([2.0, 2.0]), np.array([-1.0, 1.0])]\n",
    "    scale = [np.array([0.5, 0.5]), np.array([0.5, 0.2])]\n",
    "    data = np.concatenate([np.random.default_rng().laplace(loc=l, scale=s, size=(N, 2))\n",
    "                           for l, s in zip(loc, scale)], axis=0)\n",
    "    source = np.concatenate([k*np.ones([N, 1]) for k in range(len(loc))], axis=0)\n",
    "    return np.concatenate([data, source], axis=1)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "82929490",
   "metadata": {},
   "outputs": [],
   "source": [
    "data = generate_data(N=1000)\n",
    "data = pd.DataFrame(data, columns=[\"x\", \"y\", \"source\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d8295e8a",
   "metadata": {},
   "source": [
    "Let's print out the dataset read first."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "024fb65a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>x</th>\n",
       "      <th>y</th>\n",
       "      <th>source</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.957979</td>\n",
       "      <td>1.963396</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.746422</td>\n",
       "      <td>2.743400</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.853572</td>\n",
       "      <td>1.972891</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2.128815</td>\n",
       "      <td>1.326886</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1.926940</td>\n",
       "      <td>2.328016</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1995</th>\n",
       "      <td>-1.436096</td>\n",
       "      <td>0.892213</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1996</th>\n",
       "      <td>-0.829921</td>\n",
       "      <td>1.677958</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1997</th>\n",
       "      <td>-0.881437</td>\n",
       "      <td>1.086710</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1998</th>\n",
       "      <td>-0.800512</td>\n",
       "      <td>1.293252</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1999</th>\n",
       "      <td>-0.832540</td>\n",
       "      <td>0.900190</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2000 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "             x         y  source\n",
       "0     1.957979  1.963396     0.0\n",
       "1     1.746422  2.743400     0.0\n",
       "2     1.853572  1.972891     0.0\n",
       "3     2.128815  1.326886     0.0\n",
       "4     1.926940  2.328016     0.0\n",
       "...        ...       ...     ...\n",
       "1995 -1.436096  0.892213     1.0\n",
       "1996 -0.829921  1.677958     1.0\n",
       "1997 -0.881437  1.086710     1.0\n",
       "1998 -0.800512  1.293252     1.0\n",
       "1999 -0.832540  0.900190     1.0\n",
       "\n",
       "[2000 rows x 3 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c178424",
   "metadata": {},
   "source": [
    "We can plot this fairly easily using Matplotlib."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "e63b38c5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 106288 characters]",
      "text/plain": [
       "<Figure size 800x800 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots(figsize=(8, 8))\n",
    "data.plot.scatter(x=\"x\", y=\"y\", c=\"source\", colormap='viridis', ax=ax)\n",
    "ax.set(xlabel=\"x\", ylabel=r\"y\", title=\"\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a376636d",
   "metadata": {},
   "source": [
    "The usage of the scikit-learn interface is very standard, and one does not even need to know the details of how the SVM algorithm operates. It is however important to understand the basics, to understand how it operates.\n",
    "\n",
    "The kernel choice is effectively changing the choice for $\\phi(x)$ in the explanation above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "0837b3ff",
   "metadata": {},
   "outputs": [],
   "source": [
    "clf = svm.SVC(kernel=\"linear\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "8798f857",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-1 {\n",
       "  /* Definition of color scheme common for light and dark mode */\n",
       "  --sklearn-color-text: black;\n",
       "  --sklearn-color-line: gray;\n",
       "  /* Definition of color scheme for unfitted estimators */\n",
       "  --sklearn-color-unfitted-level-0: #fff5e6;\n",
       "  --sklearn-color-unfitted-level-1: #f6e4d2;\n",
       "  --sklearn-color-unfitted-level-2: #ffe0b3;\n",
       "  --sklearn-color-unfitted-level-3: chocolate;\n",
       "  /* Definition of color scheme for fitted estimators */\n",
       "  --sklearn-color-fitted-level-0: #f0f8ff;\n",
       "  --sklearn-color-fitted-level-1: #d4ebff;\n",
       "  --sklearn-color-fitted-level-2: #b3dbfd;\n",
       "  --sklearn-color-fitted-level-3: cornflowerblue;\n",
       "\n",
       "  /* Specific color for light theme */\n",
       "  --sklearn-color-text-on-default-background: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, black)));\n",
       "  --sklearn-color-background: var(--sg-background-color, var(--theme-background, var(--jp-layout-color0, white)));\n",
       "  --sklearn-color-border-box: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, black)));\n",
       "  --sklearn-color-icon: #696969;\n",
       "\n",
       "  @media (prefers-color-scheme: dark) {\n",
       "    /* Redefinition of color scheme for dark theme */\n",
       "    --sklearn-color-text-on-default-background: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, white)));\n",
       "    --sklearn-color-background: var(--sg-background-color, var(--theme-background, var(--jp-layout-color0, #111)));\n",
       "    --sklearn-color-border-box: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, white)));\n",
       "    --sklearn-color-icon: #878787;\n",
       "  }\n",
       "}\n",
       "\n",
       "#sk-container-id-1 {\n",
       "  color: var(--sklearn-color-text);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 pre {\n",
       "  padding: 0;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 input.sk-hidden--visually {\n",
       "  border: 0;\n",
       "  clip: rect(1px 1px 1px 1px);\n",
       "  clip: rect(1px, 1px, 1px, 1px);\n",
       "  height: 1px;\n",
       "  margin: -1px;\n",
       "  overflow: hidden;\n",
       "  padding: 0;\n",
       "  position: absolute;\n",
       "  width: 1px;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-dashed-wrapped {\n",
       "  border: 1px dashed var(--sklearn-color-line);\n",
       "  margin: 0 0.4em 0.5em 0.4em;\n",
       "  box-sizing: border-box;\n",
       "  padding-bottom: 0.4em;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-container {\n",
       "  /* jupyter's `normalize.less` sets `[hidden] { display: none; }`\n",
       "     but bootstrap.min.css set `[hidden] { display: none !important; }`\n",
       "     so we also need the `!important` here to be able to override the\n",
       "     default hidden behavior on the sphinx rendered scikit-learn.org.\n",
       "     See: https://github.com/scikit-learn/scikit-learn/issues/21755 */\n",
       "  display: inline-block !important;\n",
       "  position: relative;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-text-repr-fallback {\n",
       "  display: none;\n",
       "}\n",
       "\n",
       "div.sk-parallel-item,\n",
       "div.sk-serial,\n",
       "div.sk-item {\n",
       "  /* draw centered vertical line to link estimators */\n",
       "  background-image: linear-gradient(var(--sklearn-color-text-on-default-background), var(--sklearn-color-text-on-default-background));\n",
       "  background-size: 2px 100%;\n",
       "  background-repeat: no-repeat;\n",
       "  background-position: center center;\n",
       "}\n",
       "\n",
       "/* Parallel-specific style estimator block */\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item::after {\n",
       "  content: \"\";\n",
       "  width: 100%;\n",
       "  border-bottom: 2px solid var(--sklearn-color-text-on-default-background);\n",
       "  flex-grow: 1;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel {\n",
       "  display: flex;\n",
       "  align-items: stretch;\n",
       "  justify-content: center;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  position: relative;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item {\n",
       "  display: flex;\n",
       "  flex-direction: column;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item:first-child::after {\n",
       "  align-self: flex-end;\n",
       "  width: 50%;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item:last-child::after {\n",
       "  align-self: flex-start;\n",
       "  width: 50%;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item:only-child::after {\n",
       "  width: 0;\n",
       "}\n",
       "\n",
       "/* Serial-specific style estimator block */\n",
       "\n",
       "#sk-container-id-1 div.sk-serial {\n",
       "  display: flex;\n",
       "  flex-direction: column;\n",
       "  align-items: center;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  padding-right: 1em;\n",
       "  padding-left: 1em;\n",
       "}\n",
       "\n",
       "\n",
       "/* Toggleable style: style used for estimator/Pipeline/ColumnTransformer box that is\n",
       "clickable and can be expanded/collapsed.\n",
       "- Pipeline and ColumnTransformer use this feature and define the default style\n",
       "- Estimators will overwrite some part of the style using the `sk-estimator` class\n",
       "*/\n",
       "\n",
       "/* Pipeline and ColumnTransformer style (default) */\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable {\n",
       "  /* Default theme specific background. It is overwritten whether we have a\n",
       "  specific estimator or a Pipeline/ColumnTransformer */\n",
       "  background-color: var(--sklearn-color-background);\n",
       "}\n",
       "\n",
       "/* Toggleable label */\n",
       "#sk-container-id-1 label.sk-toggleable__label {\n",
       "  cursor: pointer;\n",
       "  display: block;\n",
       "  width: 100%;\n",
       "  margin-bottom: 0;\n",
       "  padding: 0.5em;\n",
       "  box-sizing: border-box;\n",
       "  text-align: center;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 label.sk-toggleable__label-arrow:before {\n",
       "  /* Arrow on the left of the label */\n",
       "  content: \"▸\";\n",
       "  float: left;\n",
       "  margin-right: 0.25em;\n",
       "  color: var(--sklearn-color-icon);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 label.sk-toggleable__label-arrow:hover:before {\n",
       "  color: var(--sklearn-color-text);\n",
       "}\n",
       "\n",
       "/* Toggleable content - dropdown */\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable__content {\n",
       "  max-height: 0;\n",
       "  max-width: 0;\n",
       "  overflow: hidden;\n",
       "  text-align: left;\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable__content.fitted {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable__content pre {\n",
       "  margin: 0.2em;\n",
       "  border-radius: 0.25em;\n",
       "  color: var(--sklearn-color-text);\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable__content.fitted pre {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 input.sk-toggleable__control:checked~div.sk-toggleable__content {\n",
       "  /* Expand drop-down */\n",
       "  max-height: 200px;\n",
       "  max-width: 100%;\n",
       "  overflow: auto;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {\n",
       "  content: \"▾\";\n",
       "}\n",
       "\n",
       "/* Pipeline/ColumnTransformer-specific style */\n",
       "\n",
       "#sk-container-id-1 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-label.fitted input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Estimator-specific style */\n",
       "\n",
       "/* Colorize estimator box */\n",
       "#sk-container-id-1 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-estimator.fitted input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-label label.sk-toggleable__label,\n",
       "#sk-container-id-1 div.sk-label label {\n",
       "  /* The background is the default theme color */\n",
       "  color: var(--sklearn-color-text-on-default-background);\n",
       "}\n",
       "\n",
       "/* On hover, darken the color of the background */\n",
       "#sk-container-id-1 div.sk-label:hover label.sk-toggleable__label {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "/* Label box, darken color on hover, fitted */\n",
       "#sk-container-id-1 div.sk-label.fitted:hover label.sk-toggleable__label.fitted {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Estimator label */\n",
       "\n",
       "#sk-container-id-1 div.sk-label label {\n",
       "  font-family: monospace;\n",
       "  font-weight: bold;\n",
       "  display: inline-block;\n",
       "  line-height: 1.2em;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-label-container {\n",
       "  text-align: center;\n",
       "}\n",
       "\n",
       "/* Estimator-specific */\n",
       "#sk-container-id-1 div.sk-estimator {\n",
       "  font-family: monospace;\n",
       "  border: 1px dotted var(--sklearn-color-border-box);\n",
       "  border-radius: 0.25em;\n",
       "  box-sizing: border-box;\n",
       "  margin-bottom: 0.5em;\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-estimator.fitted {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "/* on hover */\n",
       "#sk-container-id-1 div.sk-estimator:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-estimator.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Specification for estimator info (e.g. \"i\" and \"?\") */\n",
       "\n",
       "/* Common style for \"i\" and \"?\" */\n",
       "\n",
       ".sk-estimator-doc-link,\n",
       "a:link.sk-estimator-doc-link,\n",
       "a:visited.sk-estimator-doc-link {\n",
       "  float: right;\n",
       "  font-size: smaller;\n",
       "  line-height: 1em;\n",
       "  font-family: monospace;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  border-radius: 1em;\n",
       "  height: 1em;\n",
       "  width: 1em;\n",
       "  text-decoration: none !important;\n",
       "  margin-left: 1ex;\n",
       "  /* unfitted */\n",
       "  border: var(--sklearn-color-unfitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-unfitted-level-1);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link.fitted,\n",
       "a:link.sk-estimator-doc-link.fitted,\n",
       "a:visited.sk-estimator-doc-link.fitted {\n",
       "  /* fitted */\n",
       "  border: var(--sklearn-color-fitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-fitted-level-1);\n",
       "}\n",
       "\n",
       "/* On hover */\n",
       "div.sk-estimator:hover .sk-estimator-doc-link:hover,\n",
       ".sk-estimator-doc-link:hover,\n",
       "div.sk-label-container:hover .sk-estimator-doc-link:hover,\n",
       ".sk-estimator-doc-link:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "div.sk-estimator.fitted:hover .sk-estimator-doc-link.fitted:hover,\n",
       ".sk-estimator-doc-link.fitted:hover,\n",
       "div.sk-label-container:hover .sk-estimator-doc-link.fitted:hover,\n",
       ".sk-estimator-doc-link.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "/* Span, style for the box shown on hovering the info icon */\n",
       ".sk-estimator-doc-link span {\n",
       "  display: none;\n",
       "  z-index: 9999;\n",
       "  position: relative;\n",
       "  font-weight: normal;\n",
       "  right: .2ex;\n",
       "  padding: .5ex;\n",
       "  margin: .5ex;\n",
       "  width: min-content;\n",
       "  min-width: 20ex;\n",
       "  max-width: 50ex;\n",
       "  color: var(--sklearn-color-text);\n",
       "  box-shadow: 2pt 2pt 4pt #999;\n",
       "  /* unfitted */\n",
       "  background: var(--sklearn-color-unfitted-level-0);\n",
       "  border: .5pt solid var(--sklearn-color-unfitted-level-3);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link.fitted span {\n",
       "  /* fitted */\n",
       "  background: var(--sklearn-color-fitted-level-0);\n",
       "  border: var(--sklearn-color-fitted-level-3);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link:hover span {\n",
       "  display: block;\n",
       "}\n",
       "\n",
       "/* \"?\"-specific style due to the `<a>` HTML tag */\n",
       "\n",
       "#sk-container-id-1 a.estimator_doc_link {\n",
       "  float: right;\n",
       "  font-size: 1rem;\n",
       "  line-height: 1em;\n",
       "  font-family: monospace;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  border-radius: 1rem;\n",
       "  height: 1rem;\n",
       "  width: 1rem;\n",
       "  text-decoration: none;\n",
       "  /* unfitted */\n",
       "  color: var(--sklearn-color-unfitted-level-1);\n",
       "  border: var(--sklearn-color-unfitted-level-1) 1pt solid;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 a.estimator_doc_link.fitted {\n",
       "  /* fitted */\n",
       "  border: var(--sklearn-color-fitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-fitted-level-1);\n",
       "}\n",
       "\n",
       "/* On hover */\n",
       "#sk-container-id-1 a.estimator_doc_link:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 a.estimator_doc_link.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-3);\n",
       "}\n",
       "</style><div id=\"sk-container-id-1\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>SVC(kernel=&#x27;linear&#x27;)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator fitted sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-1\" type=\"checkbox\" checked><label for=\"sk-estimator-id-1\" class=\"sk-toggleable__label fitted sk-toggleable__label-arrow fitted\">&nbsp;&nbsp;SVC<a class=\"sk-estimator-doc-link fitted\" rel=\"noreferrer\" target=\"_blank\" href=\"https://scikit-learn.org/1.4/modules/generated/sklearn.svm.SVC.html\">?<span>Documentation for SVC</span></a><span class=\"sk-estimator-doc-link fitted\">i<span>Fitted</span></span></label><div class=\"sk-toggleable__content fitted\"><pre>SVC(kernel=&#x27;linear&#x27;)</pre></div> </div></div></div></div>"
      ],
      "text/plain": [
       "SVC(kernel='linear')"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf.fit(data.loc[:, [\"x\", \"y\"]], data.loc[:, \"source\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e928f498",
   "metadata": {},
   "source": [
    "Only a few vectors are needed to choose the decision boundary, since only they contribute to the minimum distance shown before. Those vectors are called the support vectors and give the name to the method. They can be accessed using the following attribute:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "fb5796e5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.5728445 ,  1.85279962],\n",
       "       [ 0.88952752,  1.29565843],\n",
       "       [-0.29289522,  1.7323941 ],\n",
       "       [ 1.21587184,  0.0298207 ],\n",
       "       [-0.71623808,  1.91730325],\n",
       "       [-1.15606358,  2.29934544],\n",
       "       [ 1.10874972, -0.04472429],\n",
       "       [ 1.86671529,  0.21816809],\n",
       "       [ 0.72221526,  1.3187182 ],\n",
       "       [ 0.43127916,  2.03084387],\n",
       "       [ 0.38754048,  1.14382588],\n",
       "       [ 0.79404296,  0.28318105],\n",
       "       [ 1.77799098, -0.95419067],\n",
       "       [ 1.50726769,  0.01742912],\n",
       "       [ 0.45686066,  1.59590656],\n",
       "       [ 0.22941722,  2.20258659],\n",
       "       [ 1.00158621,  0.70316458],\n",
       "       [-0.14541018,  1.54853915],\n",
       "       [ 0.59450067,  1.29773237],\n",
       "       [-0.55405823,  2.01528005],\n",
       "       [-0.6515804 ,  1.60818611],\n",
       "       [ 1.39258445,  0.74983753],\n",
       "       [ 1.28885338,  0.96748195],\n",
       "       [ 0.64700268,  1.51371681],\n",
       "       [ 0.24078166,  2.254969  ],\n",
       "       [ 1.22797034,  0.95030829],\n",
       "       [ 0.67063657,  1.63610957],\n",
       "       [ 2.86350485, -2.53654902],\n",
       "       [-0.0705547 ,  1.25845596],\n",
       "       [ 2.27703837, -0.5358523 ],\n",
       "       [ 1.65510129,  0.48075801],\n",
       "       [ 0.2393727 ,  1.9192947 ],\n",
       "       [ 0.56761675,  1.90154018],\n",
       "       [ 1.50497946,  0.07693201],\n",
       "       [ 1.23721436,  0.56518652],\n",
       "       [ 0.30829555,  1.84400209],\n",
       "       [-0.30502034,  1.84341234],\n",
       "       [ 1.21258702,  0.42511609],\n",
       "       [ 1.01819711, -0.25035844],\n",
       "       [ 0.8710959 ,  1.42119161],\n",
       "       [ 0.60229855,  1.30261832],\n",
       "       [ 0.15251482,  1.15259001],\n",
       "       [ 1.49920059,  0.49903158],\n",
       "       [ 1.31162278,  0.12175366],\n",
       "       [-0.11236343,  2.00215393],\n",
       "       [ 0.10888562,  1.79202884],\n",
       "       [ 1.82631824,  0.09732397],\n",
       "       [ 0.40265968,  1.73810626],\n",
       "       [ 0.57693124,  0.98148211],\n",
       "       [ 0.02221168,  2.44912934],\n",
       "       [ 0.25020485,  2.23171948],\n",
       "       [ 0.14273132,  2.37234946],\n",
       "       [ 0.59781711,  1.83386377],\n",
       "       [ 0.18067333,  2.07287702],\n",
       "       [ 0.35134927,  0.98834705],\n",
       "       [ 1.62665053,  1.03247395],\n",
       "       [ 0.20444582,  1.19518295],\n",
       "       [ 1.16025063,  1.99899162],\n",
       "       [ 2.06416279,  1.28713614],\n",
       "       [ 0.17740183,  1.09284454],\n",
       "       [ 0.34518529,  0.84383432],\n",
       "       [ 0.02479904,  1.31650692],\n",
       "       [-0.06225827,  1.7871111 ],\n",
       "       [ 1.77185136,  0.9250378 ],\n",
       "       [ 0.73709884,  0.56956188],\n",
       "       [ 0.24821658,  1.21768365],\n",
       "       [ 1.41336706,  1.12781135],\n",
       "       [ 0.51921524,  0.67163378],\n",
       "       [ 0.31757877,  0.99824748],\n",
       "       [ 0.48180318,  1.03682991],\n",
       "       [ 0.74282773,  0.91469184],\n",
       "       [ 0.35690936,  1.48333486],\n",
       "       [ 0.40757143,  1.06863743],\n",
       "       [ 0.7707394 ,  0.90988459],\n",
       "       [ 0.16073531,  1.23179266],\n",
       "       [ 0.50354463,  0.84560193],\n",
       "       [ 0.50289217,  1.00098647],\n",
       "       [ 0.37722618,  0.91720759],\n",
       "       [ 0.39246892,  0.81844045],\n",
       "       [ 0.67345951,  1.04452779],\n",
       "       [ 1.53950282,  1.37052216],\n",
       "       [ 0.91260177,  0.60198605],\n",
       "       [ 0.89150756,  0.66479233],\n",
       "       [ 1.01187864,  0.95802718],\n",
       "       [ 0.87511131,  0.94929959],\n",
       "       [ 1.86423776,  0.79036484],\n",
       "       [-0.18513001,  1.82567638],\n",
       "       [ 1.09088316,  0.70515353],\n",
       "       [-0.32450938,  1.70973872],\n",
       "       [ 0.69654802,  1.08316679],\n",
       "       [ 0.3978825 ,  1.00509957],\n",
       "       [-0.10716825,  1.48886641],\n",
       "       [ 0.25936636,  0.96998539],\n",
       "       [ 1.70496701,  1.03275951],\n",
       "       [ 0.59068219,  0.94649016],\n",
       "       [ 0.1215318 ,  1.57385714],\n",
       "       [ 0.048246  ,  1.24850498],\n",
       "       [ 0.36958743,  1.3984192 ],\n",
       "       [ 0.47891732,  1.30940599],\n",
       "       [ 1.1918804 ,  0.80792963],\n",
       "       [ 0.51690664,  1.01246696],\n",
       "       [ 0.37620589,  1.03860828],\n",
       "       [ 0.52612312,  0.95286716],\n",
       "       [ 0.80854953,  0.93484408],\n",
       "       [-0.22659936,  1.71509577],\n",
       "       [ 0.19339286,  1.45037463],\n",
       "       [ 1.05842415,  1.61146035],\n",
       "       [-0.03830695,  1.4064268 ]])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf.support_vectors_"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b54001fc",
   "metadata": {},
   "source": [
    "We can now predict to which class a new data point belongs to using `clf.predict(new_data_samples)`. It is however interesting to visualize the decision boundary itself.\n",
    "\n",
    "(Taken from https://scikit-learn.org/stable/auto_examples/svm/plot_separating_hyperplane.html#sphx-glr-auto-examples-svm-plot-separating-hyperplane-py -- take a look there for more resources and more examples)\n",
    "\n",
    "The code below also highlights the support vectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "cc8fc1f1",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/daniloefl/miniconda3/envs/ml/lib/python3.12/site-packages/sklearn/base.py:493: UserWarning: X does not have valid feature names, but SVC was fitted with feature names\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 162640 characters]",
      "text/plain": [
       "<Figure size 800x800 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots(figsize=(8, 8))\n",
    "plt.scatter(data.loc[:, \"x\"], data.loc[:, \"y\"], c=data.loc[:, \"source\"], s=30, cmap=plt.cm.Paired)\n",
    "\n",
    "# plot the decision function\n",
    "ax = plt.gca()\n",
    "xlim = ax.get_xlim()\n",
    "ylim = ax.get_ylim()\n",
    "\n",
    "# create grid to evaluate model\n",
    "xx = np.linspace(xlim[0], xlim[1], 30)\n",
    "yy = np.linspace(ylim[0], ylim[1], 30)\n",
    "YY, XX = np.meshgrid(yy, xx)\n",
    "xy = np.vstack([XX.ravel(), YY.ravel()]).T\n",
    "Z = clf.decision_function(xy).reshape(XX.shape)\n",
    "\n",
    "# plot decision boundary and margins\n",
    "ax.contour(\n",
    "    XX, YY, Z, colors=\"k\", levels=[-1, 0, 1], alpha=0.5, linestyles=[\"--\", \"-\", \"--\"]\n",
    ")\n",
    "# plot support vectors\n",
    "ax.scatter(\n",
    "    clf.support_vectors_[:, 0],\n",
    "    clf.support_vectors_[:, 1],\n",
    "    s=100,\n",
    "    linewidth=1,\n",
    "    facecolors=\"none\",\n",
    "    edgecolors=\"k\",\n",
    ")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ee97208",
   "metadata": {},
   "source": [
    "Try the same using another kernel to see how the decision boundary changes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "d98ececa",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/daniloefl/miniconda3/envs/ml/lib/python3.12/site-packages/sklearn/base.py:493: UserWarning: X does not have valid feature names, but SVC was fitted with feature names\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 177576 characters]",
      "text/plain": [
       "<Figure size 800x800 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "clf_kernel = svm.SVC(kernel=\"rbf\")\n",
    "clf_kernel.fit(data.loc[:, [\"x\", \"y\"]], data.loc[:, \"source\"])\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(8, 8))\n",
    "plt.scatter(data.loc[:, \"x\"], data.loc[:, \"y\"], c=data.loc[:, \"source\"], s=30, cmap=plt.cm.Paired)\n",
    "\n",
    "# plot the decision function\n",
    "ax = plt.gca()\n",
    "xlim = ax.get_xlim()\n",
    "ylim = ax.get_ylim()\n",
    "\n",
    "# create grid to evaluate model\n",
    "xx = np.linspace(xlim[0], xlim[1], 30)\n",
    "yy = np.linspace(ylim[0], ylim[1], 30)\n",
    "YY, XX = np.meshgrid(yy, xx)\n",
    "xy = np.vstack([XX.ravel(), YY.ravel()]).T\n",
    "Z = clf_kernel.decision_function(xy).reshape(XX.shape)\n",
    "\n",
    "# plot decision boundary and margins\n",
    "ax.contour(\n",
    "    XX, YY, Z, colors=\"k\", levels=[-1, 0, 1], alpha=0.5, linestyles=[\"--\", \"-\", \"--\"]\n",
    ")\n",
    "# plot support vectors\n",
    "ax.scatter(\n",
    "    clf_kernel.support_vectors_[:, 0],\n",
    "    clf_kernel.support_vectors_[:, 1],\n",
    "    s=100,\n",
    "    linewidth=1,\n",
    "    facecolors=\"none\",\n",
    "    edgecolors=\"k\",\n",
    ")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0099362-0145-44f6-a777-50e09eba20c3",
   "metadata": {},
   "source": [
    "## Other kernels\n",
    "\n",
    "What if you try other kernels? How does this separation curve change?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ce1a3211-4119-42b2-9341-e93e2dbf584b",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.3"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {},
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
