{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "6a764d92",
   "metadata": {},
   "source": [
    "# Mixture Models\n",
    "\n",
    "One common objective is to find similarities in data and cluster it. There are several heuristic methods to cluster them. We are going to focus on a few strongly theoretically motivated methods. Other methods use heuristics to identify similarities in data and are mentioned in the end.\n",
    "\n",
    "For the purposes of this example, we will generate a fake dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fce4d8e8",
   "metadata": {},
   "source": [
    "We start by loading the necessary Python modules."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "300cf8d3",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn.mixture import GaussianMixture, BayesianGaussianMixture\n",
    "from sklearn.cluster import KMeans"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ecd6a69",
   "metadata": {},
   "source": [
    "Let's generate the fake data now to have something to cluster."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "4959a292",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def generate_clusters(mu: np.ndarray, sigma: np.ndarray, N: int) ->np.ndarray:\n",
    "    assert len(mu) == len(sigma)\n",
    "    assert N > 1\n",
    "    D = len(mu[0].shape)\n",
    "    data = np.concatenate([np.random.default_rng().multivariate_normal(mean=mu_k, cov=sigma_k, size=N)\n",
    "                           for mu_k, sigma_k in zip(mu, sigma)], axis=0)\n",
    "    source = np.concatenate([k*np.ones([N, 1]) for k in range(len(mu))], axis=0)\n",
    "    return np.concatenate([data, source], axis=1)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "82929490",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "data = generate_clusters(mu=[np.array([5.0, -2.0]),\n",
    "                             np.array([1.0, 5.0]),\n",
    "                             np.array([-5.0, -1.0])],\n",
    "                         sigma=[np.array([[0.4, 0.2],\n",
    "                                          [0.2, 0.4]]),\n",
    "                                np.array([[1.0, 0.5],\n",
    "                                          [0.5, 1.0]]),\n",
    "                                np.array([[2.0, 0.0],\n",
    "                                          [0.0, 5.0]])],\n",
    "                         N=1000)\n",
    "data = pd.DataFrame(data, columns=[\"x\", \"y\", \"source\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d8295e8a",
   "metadata": {},
   "source": [
    "Let's print out the dataset read first."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "024fb65a",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>x</th>\n",
       "      <th>y</th>\n",
       "      <th>source</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>4.960065</td>\n",
       "      <td>-2.902600</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>6.231718</td>\n",
       "      <td>-1.404399</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3.985313</td>\n",
       "      <td>-2.833486</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5.314968</td>\n",
       "      <td>-1.930364</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5.652647</td>\n",
       "      <td>-1.523516</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2995</th>\n",
       "      <td>-6.289552</td>\n",
       "      <td>-1.923758</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2996</th>\n",
       "      <td>-7.690072</td>\n",
       "      <td>1.032256</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2997</th>\n",
       "      <td>-2.930869</td>\n",
       "      <td>-0.503878</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2998</th>\n",
       "      <td>-3.721900</td>\n",
       "      <td>3.096010</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2999</th>\n",
       "      <td>-5.862448</td>\n",
       "      <td>-0.028824</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3000 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "             x         y  source\n",
       "0     4.960065 -2.902600     0.0\n",
       "1     6.231718 -1.404399     0.0\n",
       "2     3.985313 -2.833486     0.0\n",
       "3     5.314968 -1.930364     0.0\n",
       "4     5.652647 -1.523516     0.0\n",
       "...        ...       ...     ...\n",
       "2995 -6.289552 -1.923758     2.0\n",
       "2996 -7.690072  1.032256     2.0\n",
       "2997 -2.930869 -0.503878     2.0\n",
       "2998 -3.721900  3.096010     2.0\n",
       "2999 -5.862448 -0.028824     2.0\n",
       "\n",
       "[3000 rows x 3 columns]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c178424",
   "metadata": {},
   "source": [
    "We can plot this fairly easily using Matplotlib."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "e63b38c5",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 136544 characters]",
      "text/plain": [
       "<Figure size 800x800 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots(figsize=(8, 8))\n",
    "data.plot.scatter(x=\"x\", y=\"y\", c=\"source\", colormap='viridis', ax=ax)\n",
    "ax.set(xlabel=\"x\", ylabel=r\"y\", title=\"\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a376636d",
   "metadata": {},
   "source": [
    "There are several ways of finding similarities in the data. The Gaussian Mixture Model assumes that the data has been produced exactly as in this example. For each sample, the procedure assumed to be used for the generation of each sample is the following:\n",
    "  * a random integer is chosen according to a discrete probability distribution, which identifies to which cluster the sample belongs to (this is the `source` variable in the generation procedure above);\n",
    "  * that random integer is assumed to be used to choose the mean and covariance matrix for a Gaussian random variable, which is then used to produce the observed sample.\n",
    " \n",
    "Our objective here is then to find out the probability of producing a sample for each cluster (the probabilities with which a sample belongs to one cluster instead of the others), the means and covariance matrices for the model. Using Bayes' theorem, one may write down the posterior probability for the true means, covariances and cluster probabilities ($\\theta$) given the data ($x$), but it is very hard to calculate the parameters from it. Let us call the cluster that a specific sample belongs to, $z$.\n",
    "\n",
    "The true posterior would look like this:\n",
    "\n",
    "$p(\\theta|x) = \\frac{p(x|\\theta) p(\\theta)}{p(x)}$\n",
    "\n",
    "We assume $p(\\theta)$ is constant for all $\\theta$ and we know $p(x)$ is independent of $\\theta$ (because it is just the probability we obtained the existing data). We also know that $p(x|\\theta) = \\sum_z p(x,z|\\theta)$, since summing over all $z$, we obtain a total probability 1 of getting any $z$.\n",
    "\n",
    "$p(\\theta|x) \\propto \\sum_z p(x,z|\\theta) = \\sum_z p(x|\\theta,z) p(z|\\theta)$\n",
    "\n",
    "Where we have expanded it on $z$ using $p(a,b) = p(a|b)p(b)$.\n",
    "\n",
    "Now if we want to find the most probable $\\theta$ from this posterior (the \"maximum a posteriori\", or \"MAP\"), we just need to maximize the right-hand side of the last equation. This is very hard to do, but we can use the Expectation-Maximization (EM) algorithm, on which we iteratively improve our probability estimates. This method works as follows.\n",
    "\n",
    "First we define the log-likelihood $LL(\\theta|z) = \\log L(\\theta|z) = \\log p(x|\\theta,z) = \\sum_{k=\\text{sample}} \\log p(x_k|\\theta,z_k)$. Since we are assuming that the probability for each data sample $x_k$ is a Gaussian distribution, each of the terms in the sum is a Gaussian probability, so if we know $\\theta$ and $z_k$, this full sum can be calculated. The issue is that we do not know $z_k$ (that is the whole point!).\n",
    "\n",
    "We start by assuming $\\theta=\\theta_0$ for some random initial value of the parameters. The EM method iterates on two steps:\n",
    "\n",
    "  1. Calculate the *average* value of $LL(\\theta|z)$ over $z$ assuming the current $\\theta_i$ to calculate $p(z|\\theta)$, that is: $Q(\\theta) = \\sum_z LL(\\theta|z) p(z|\\theta_i)$. This means we calculate the weighted sum of the log-likelihood a point belongs to each Gaussian, weighted by the probability that that Gaussian was the reason that sample was generated. This avoids needing to know the correct $z_k$, since we just sum over all possibilities. The weights of this weighted sum are simply one of the parameters $\\theta$, namely, the cluster probabilities.\n",
    "  \n",
    "  2. Find the $\\theta$ which maximizes $Q(\\theta)$ and use this as the next $\\theta_{i+1}$.\n",
    "\n",
    "These two steps are iterated to improve the $\\theta$ for several iterations. It can be shown that an improvement on $Q$ following this procedure leads to a new $\\theta$ that improves the posterior probability $p(\\theta|x)$ above (see https://en.wikipedia.org/wiki/Expectation%E2%80%93maximization_algorithm#Proof_of_correctness ).\n",
    "\n",
    "We will not write all of this from scratch. Instead, we use the `GaussianMixture` function in `scikit-learn`, which has this ready for us. It is nevertheless important to understand the assumptions made here: the underlying samples are assumed to come from a discrete combination of Gaussians. Other mixture models are possible using other underlying distributions, but note that this method will not work if the samples do not follow this generative pattern."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "0837b3ff",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "gmm = GaussianMixture(n_components=3, covariance_type=\"full\", max_iter=20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "8798f857",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-1 {color: black;background-color: white;}#sk-container-id-1 pre{padding: 0;}#sk-container-id-1 div.sk-toggleable {background-color: white;}#sk-container-id-1 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-1 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-1 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-1 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-1 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-1 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-1 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-1 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-1 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-1 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-1 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-1 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-1 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-1 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-1 div.sk-item {position: relative;z-index: 1;}#sk-container-id-1 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-1 div.sk-item::before, #sk-container-id-1 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-1 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-1 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-1 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-1 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-1 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-1 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-1 div.sk-label-container {text-align: center;}#sk-container-id-1 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-1 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-1\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>GaussianMixture(max_iter=20, n_components=3)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-1\" type=\"checkbox\" checked><label for=\"sk-estimator-id-1\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">GaussianMixture</label><div class=\"sk-toggleable__content\"><pre>GaussianMixture(max_iter=20, n_components=3)</pre></div></div></div></div></div>"
      ],
      "text/plain": [
       "GaussianMixture(max_iter=20, n_components=3)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gmm.fit(data.loc[:, [\"x\", \"y\"]])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e928f498",
   "metadata": {},
   "source": [
    "Now that we have model fit, we can even check the fit parameters $\\theta$, which include the means, the covariance matrices and the probabilities given to each Gaussian in the mixture:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "fb5796e5",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.99087075,  5.04466165],\n",
       "       [-5.06356078, -0.94701357],\n",
       "       [ 4.97867027, -1.98953164]])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gmm.means_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "182904d1",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[[ 9.80327267e-01,  4.34733962e-01],\n",
       "        [ 4.34733962e-01,  9.46532120e-01]],\n",
       "\n",
       "       [[ 1.94905096e+00, -1.15878394e-03],\n",
       "        [-1.15878394e-03,  4.73462596e+00]],\n",
       "\n",
       "       [[ 3.86398255e-01,  1.99692091e-01],\n",
       "        [ 1.99692091e-01,  3.77042067e-01]]])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gmm.covariances_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "2faa1f72",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.33400535, 0.33266132, 0.33333333])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gmm.weights_"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b54001fc",
   "metadata": {},
   "source": [
    "We can predict to which cluster each sample belongs now by selecting the cluster for each sample that maximizes the probability $p(z|\\theta,x)$, now that we know $\\theta$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "cc8fc1f1",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "guess = gmm.predict(data.loc[:, [\"x\", \"y\"]])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "68560fb4",
   "metadata": {},
   "source": [
    "Let's plot it!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "88982b21",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "data.loc[:, \"guess\"] = guess"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "333581b5",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 218728 characters]",
      "text/plain": [
       "<Figure size 1000x1000 with 4 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots(figsize=(10, 10), ncols=2)\n",
    "data.plot.scatter(x=\"x\", y=\"y\", c=\"guess\", colormap='viridis', ax=ax[0])\n",
    "data.plot.scatter(x=\"x\", y=\"y\", c=\"source\", colormap='viridis', ax=ax[1])\n",
    "ax[0].set(xlabel=\"x\", ylabel=r\"y\", title=\"Guessed source\")\n",
    "ax[1].set(xlabel=\"x\", ylabel=r\"y\", title=\"True association\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b1363ec",
   "metadata": {},
   "source": [
    "Note that if the sample clusters were not \"blobs\" of data, but were in concentric circles, the assumption of this method would be false and the method would simply not work well. This is why it is important to understand the underlying assumptions made in the method."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fbee128c",
   "metadata": {},
   "source": [
    "## Determining the number of clusters\n",
    "\n",
    "So far, we have assumed that the number of clusters is known. This is not often the case. One method to determine the number of clusters is to fit the GMM with different number of clusters and after each fit, calculate `gmm.bic()`. This method returns the Bayesian Information Criteria (see also AIC), which should be small when the fit worked well and the number of components is compatible with your sample.\n",
    "\n",
    "See a full example here: https://scikit-learn.org/stable/auto_examples/mixture/plot_gmm_selection.html#sphx-glr-auto-examples-mixture-plot-gmm-selection-py"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7076f779",
   "metadata": {},
   "source": [
    "## K-Means\n",
    "\n",
    "Another common method used for clustering is the K-Means, on which one simply tries to find the cluster centers which minimize in-cluster distances (in an Euclidean sense) while maximizing distances between the centers. It can be shown that this method is a special case of the Gaussian Mixture Model when the covariance matrices are diagonal, which would mean that within each blob, there is no correlation between the variables (see https://en.wikipedia.org/wiki/K-means_clustering#Gaussian_mixture_model and references).\n",
    "\n",
    "While this is an approximation of the GMM model, it is still a very useful approach, since there are faster algorithms to achieve the clustering than for GMMs.\n",
    "\n",
    "The scikit-learn module also provides an easy-to-use implementation of this algorithm:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "2f280e1d",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/daniloefl/workspace/ml-lecture-problems/ml/lib/python3.10/site-packages/sklearn/cluster/_kmeans.py:870: FutureWarning: The default value of `n_init` will change from 10 to 'auto' in 1.4. Set the value of `n_init` explicitly to suppress the warning\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-2 {color: black;background-color: white;}#sk-container-id-2 pre{padding: 0;}#sk-container-id-2 div.sk-toggleable {background-color: white;}#sk-container-id-2 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-2 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-2 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-2 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-2 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-2 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-2 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-2 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-2 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-2 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-2 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-2 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-2 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-2 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-2 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-2 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-2 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-2 div.sk-item {position: relative;z-index: 1;}#sk-container-id-2 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-2 div.sk-item::before, #sk-container-id-2 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-2 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-2 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-2 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-2 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-2 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-2 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-2 div.sk-label-container {text-align: center;}#sk-container-id-2 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-2 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-2\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>KMeans(n_clusters=3)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-2\" type=\"checkbox\" checked><label for=\"sk-estimator-id-2\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">KMeans</label><div class=\"sk-toggleable__content\"><pre>KMeans(n_clusters=3)</pre></div></div></div></div></div>"
      ],
      "text/plain": [
       "KMeans(n_clusters=3)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "kmeans = KMeans(n_clusters=3)\n",
    "kmeans.fit(data.loc[:, [\"x\", \"y\"]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "950a7eec",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "data.loc[:, \"guess_kmeans\"] = kmeans.predict(data.loc[:, [\"x\", \"y\"]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "16a56489",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 223516 characters]",
      "text/plain": [
       "<Figure size 1000x1000 with 4 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots(figsize=(10, 10), ncols=2)\n",
    "data.plot.scatter(x=\"x\", y=\"y\", c=\"guess_kmeans\", colormap='viridis', ax=ax[0])\n",
    "data.plot.scatter(x=\"x\", y=\"y\", c=\"source\", colormap='viridis', ax=ax[1])\n",
    "ax[0].set(xlabel=\"x\", ylabel=r\"y\", title=\"Guessed source using K-Means\")\n",
    "ax[1].set(xlabel=\"x\", ylabel=r\"y\", title=\"True association\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2fe3cad2",
   "metadata": {},
   "source": [
    "## Variational Inference on Gaussian Mixture Models\n",
    "\n",
    "While the Gaussian Mixture Model presented beforehand had a very strong theoretical background, it still assumes that there is a single correct value for the Gaussian parameters. This may not be the case if the Gaussian model is only approximate (as is very often the case!).\n",
    "\n",
    "One improvement to the Gaussian Mixture Models to allow for the usage of prior knowledge (a prior) on the parameters to be discovered. This helps us, for instance, to determine the ideal number of components automatically, since the number of components itself can be fit with it (assuming a prior distribution on them).\n",
    "\n",
    "Optimizing this model becomes even more complicated as the derivation shown before and if we wanted to be fully general, we would need to use very slow algorithms, such as Monte-Carlo sampling to obtain uncertainties with the least amount of extra assumptions. This is often undesirable, since we need fast clustering and often Monte-Carlo sampling is very slow with even more data! An alternative is to assume some underlying prior probability for the means and covariances and find those parameters as well in an optimization algorithm. This is what is done in Variational Inference. Further details can be seen in Bishop (2006), or in a more practical approach, here: https://scikit-learn.org/stable/modules/mixture.html#bgmm\n",
    "\n",
    "We can also easily use this method in our toy data, taking the code from scikit-learn:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "0a8642bd",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "bgmm = BayesianGaussianMixture(n_components=3)\n",
    "bgmm.fit(data.loc[:, [\"x\", \"y\"]])\n",
    "data.loc[:, \"guess_bgmm\"] = bgmm.predict(data.loc[:, [\"x\", \"y\"]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "ccbf4019",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 222980 characters]",
      "text/plain": [
       "<Figure size 1000x1000 with 4 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots(figsize=(10, 10), ncols=2)\n",
    "data.plot.scatter(x=\"x\", y=\"y\", c=\"guess_bgmm\", colormap='viridis', ax=ax[0])\n",
    "data.plot.scatter(x=\"x\", y=\"y\", c=\"source\", colormap='viridis', ax=ax[1])\n",
    "ax[0].set(xlabel=\"x\", ylabel=r\"y\", title=\"Guessed source using Variation Inference GMM\")\n",
    "ax[1].set(xlabel=\"x\", ylabel=r\"y\", title=\"True association\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72508ead",
   "metadata": {},
   "source": [
    "## Other heuristic approaches\n",
    "\n",
    "Many other heuristic approaches for clustering have been developed. Some of them rely on a strong foundation in statistics, others take a more practical approach and make assumptions on how the data samples are expected to be distributed in the $xy$ scatter plot. The following methods deserve a look if you are not familiar with them already:\n",
    "\n",
    "  * DBSCAN (see https://scikit-learn.org/stable/modules/generated/sklearn.cluster.DBSCAN.html ). Uses the euclidean distances between the samples to define if a sample belongs in a cluster or another. The assumption is related to GMMs. While the GMM requires fixing the number of clusters, DBSCAN relies on the maximum distance parameter do define which samples belong to a cluster.\n",
    "  * Agglomerative clustering (see https://scikit-learn.org/stable/modules/generated/sklearn.cluster.AgglomerativeClustering.html#sklearn.cluster.AgglomerativeClustering). It uses the nearest neighbour to each sample to define which samples belong together in the same cluster. Since the cluster association is topological, it does not assume the samples make a \"blob\" of some sort, as in the case of GMM. One could for example have samples close to one another making a circle in the $xy$-scatter plane and those samples would belong to the same cluster (depending on the definition of closeness and other parameters of the algorithm).\n",
    "  * Spectral clustering (see https://scikit-learn.org/stable/modules/clustering.html#spectral-clustering). Uses K-means on another representation of the data. This representation of the data transforms the data points from the $xy$ space shown before into a space where connectedness of nearest neighbours is the criteria for similarity. For the mathematical foundation, this document may be useful: https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.165.9323&rep=rep1&type=pdf\n",
    "  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "941c7cb8",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
