{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "6a764d92",
   "metadata": {},
   "source": [
    "# Representation Learning\n",
    "\n",
    "Often it is very hard either to find similarities within data or even to classify data as belonging to different labels when the data contains so many features. One particular example are images which may contain millions of pixels. Even low-dimensional data may not be easy to visualize in a 2D plane, so bringing the dominant effects in a 2D plane for visualization is in itself a very helpful starting point.\n",
    "\n",
    "There are different methods to obtain another view of the data, by performing linear or even non-linear combinations of the data features. The price being paid by such methods is that the new representation of the data may not be that straightforward to digest, loosing therefore some of its scientific interpretation. On the other hand, if one understands the assumptions made in such methods, one can easily imagine the mathematical process required to transform to and from this new view and gain insight from the new view without loosing track of the scientific background.\n",
    "\n",
    "We are going to go through a few methods of obtaining an alternative view of the data here and what their assumptions might be."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fce4d8e8",
   "metadata": {},
   "source": [
    "We start by loading the necessary Python modules. If you have not yet installed them, run the following cell to install them with pip:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "300cf8d3",
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn.decomposition import PCA"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ecd6a69",
   "metadata": {},
   "source": [
    "Let's generate the fake data now to have something to cluster."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "4959a292",
   "metadata": {},
   "outputs": [],
   "source": [
    "rng = np.random.RandomState(0)\n",
    "n_samples = 500\n",
    "cov = [[3, 3], [3, 4]]\n",
    "data = rng.multivariate_normal(mean=[0, 0], cov=cov, size=n_samples)\n",
    "data = pd.DataFrame(data, columns=[\"x1\", \"x2\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d8295e8a",
   "metadata": {},
   "source": [
    "Let's print out the dataset read first."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "024fb65a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>x1</th>\n",
       "      <th>x2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-3.123062</td>\n",
       "      <td>-3.267402</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-2.775958</td>\n",
       "      <td>-0.929101</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-2.582416</td>\n",
       "      <td>-4.072345</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-1.492453</td>\n",
       "      <td>-1.920361</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-0.041529</td>\n",
       "      <td>0.381166</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>495</th>\n",
       "      <td>-0.821492</td>\n",
       "      <td>-0.782416</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>496</th>\n",
       "      <td>1.197165</td>\n",
       "      <td>1.665481</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>497</th>\n",
       "      <td>-0.691309</td>\n",
       "      <td>-0.383494</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>498</th>\n",
       "      <td>0.279317</td>\n",
       "      <td>0.428408</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>499</th>\n",
       "      <td>2.082251</td>\n",
       "      <td>2.082815</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>500 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "           x1        x2\n",
       "0   -3.123062 -3.267402\n",
       "1   -2.775958 -0.929101\n",
       "2   -2.582416 -4.072345\n",
       "3   -1.492453 -1.920361\n",
       "4   -0.041529  0.381166\n",
       "..        ...       ...\n",
       "495 -0.821492 -0.782416\n",
       "496  1.197165  1.665481\n",
       "497 -0.691309 -0.383494\n",
       "498  0.279317  0.428408\n",
       "499  2.082251  2.082815\n",
       "\n",
       "[500 rows x 2 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c178424",
   "metadata": {},
   "source": [
    "We can plot this fairly easily using Matplotlib."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "e63b38c5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 57700 characters]",
      "text/plain": [
       "<Figure size 800x800 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots(figsize=(8, 8))\n",
    "data.plot.scatter(x=\"x1\", y=\"x2\", alpha=0.3, ax=ax)\n",
    "ax.set(xlabel=\"$x_1$\", ylabel=r\"$x_2$\", title=\"\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bba01cd6",
   "metadata": {},
   "source": [
    "Clearly there is a strong correlation in this data. One could almost predict $y$ using $x$ alone by fitting a line in the $xy$-plane. Therefore, it may be easier to provide a representation on which only one variable is present but retaining the maximum information possible. This is possible using several methods, which rely on several underlying assumptions. Let us start with Principal Component Analysis.\n",
    "\n",
    "In Principal Component Analysis, we set ourselves the objective of maximizing the variance of the data in our new representation. That is, imagine our new representation is given by $p_1$ and $p_2$ and we define them such that $p$ is a linear combination of $x$. The criteria used to find how to make such a linear combination shall be that the variance of $p_1$ is maximal, when we want to find out how to combine $x_1$ and $x_2$ to obtain $p_1$. This can be similarly done for $p_2$ and any other component. Instead of finding $p$ directly for each sample, we will find the direction $u_1$ on which we should project $x$ using $u_1^T x$ to obtain the new space. As we are only interested in the direction of $u_1$, we define it as normalized vector, so $u_1^T u_1 = 1$.\n",
    "\n",
    "The mean data value is $\\overline{x} = \\frac{1}{N} \\sum_k x^{(k)}$, where $x^{(k)}$ refers to the vector $(x_1, x_2)$ for the $k$-th sample point. The mean of the projected data in this new dimension $u_1$ is $u_1^T \\overline{x}$. The variance of the projected data in the direction $u_1$ is:\n",
    "\n",
    "$\\frac{1}{N}\\sum_k\\left(u_1^T x^{(k)} - u_1^T \\overline{x}\\right)^2 = u_1^T S u_1$,\n",
    "\n",
    "where $S = \\frac{1}{N}\\left(x^{(k)} - \\overline{x}\\right)\\left(x^{x(k)} - \\overline{x}\\right)^T$ is the covariance matrix of the data.\n",
    "\n",
    "We maximize the variance of the data projected in the direction $u_1$, while imposing a restriction on the maximization procedure, such that $u_1$ is normalized to 1. This can be done by maximizing the following function:\n",
    "\n",
    "$u_1^T S u_1 + \\lambda_1 (1 - u_1^T u_1)$,\n",
    "\n",
    "where $\\lambda_1$ is a Lagrange multiplier used to enforce the condition that $u_1^T u_1$ is 1.\n",
    "\n",
    "Calculating the derivative relative to $u_1$ and setting it to zero to find the maximum we see that:\n",
    "\n",
    "$S u_1 = \\lambda_1 u_1$,\n",
    "\n",
    "which is an eigenvalue problem! That is, the direction $u_1$, which maximizes the variance of the projected data is the eigenvector of the covariance matrix $S$. Moreover, if we multiply this equation by $u_1^T$ on the left and use $u_1^T u_1 = 1$, we obtain $\\lambda_1 = u_1^T S u_1$. That is, $\\lambda_1$ is the variance of the data projected in the direction $u_1$.\n",
    "\n",
    "This gives a simple recipe to find the directions with largest variance: we find the eigenvectors of the covariance matrix $S$ which have highest eigenvalues. By sorting the eigenvalues, we can easily choose which of the directions of the new representations are more important to analyse. We can also discard directions that have low eigenvalues, as they contribute little to the variations observed in the data.\n",
    "\n",
    "Naturally, we do not need to write the code to perform all those steps, as scikit-learn implements them for us:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "0837b3ff",
   "metadata": {},
   "outputs": [],
   "source": [
    "pca = PCA(n_components=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "8798f857",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-1 {color: black;background-color: white;}#sk-container-id-1 pre{padding: 0;}#sk-container-id-1 div.sk-toggleable {background-color: white;}#sk-container-id-1 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-1 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-1 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-1 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-1 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-1 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-1 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-1 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-1 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-1 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-1 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-1 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-1 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-1 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-1 div.sk-item {position: relative;z-index: 1;}#sk-container-id-1 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-1 div.sk-item::before, #sk-container-id-1 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-1 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-1 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-1 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-1 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-1 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-1 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-1 div.sk-label-container {text-align: center;}#sk-container-id-1 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-1 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-1\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>PCA(n_components=2)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-1\" type=\"checkbox\" checked><label for=\"sk-estimator-id-1\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">PCA</label><div class=\"sk-toggleable__content\"><pre>PCA(n_components=2)</pre></div></div></div></div></div>"
      ],
      "text/plain": [
       "PCA(n_components=2)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pca.fit(data.loc[:, [\"x1\", \"x2\"]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "cc8fc1f1",
   "metadata": {},
   "outputs": [],
   "source": [
    "data_t = pca.transform(data.loc[:, [\"x1\", \"x2\"]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "88982b21",
   "metadata": {},
   "outputs": [],
   "source": [
    "data.loc[:, \"pca1\"] = data_t[:, 0]\n",
    "data.loc[:, \"pca2\"] = data_t[:, 1]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ac27316",
   "metadata": {},
   "source": [
    "We can start by plotting how the data looks like after this transformation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "333581b5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 121200 characters]",
      "text/plain": [
       "<Figure size 1000x1000 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots(figsize=(10, 10), ncols=2)\n",
    "data.plot.scatter(x=\"pca1\", y=\"pca2\", alpha=0.3, ax=ax[0])\n",
    "data.plot.scatter(x=\"x1\", y=\"x2\", alpha=0.3, ax=ax[1])\n",
    "ax[0].set(xlabel=\"PCA component 1\", ylabel=r\"PCA component 2\", title=\"PCA representation\")\n",
    "ax[1].set(xlabel=\"$x_1$\", ylabel=r\"$x_2$\", title=\"Original data\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54e538c6",
   "metadata": {},
   "source": [
    "It is interesting to understand how many PCA components are necessary to explain the variance of the data. This is easily obtainable from the PCA object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "fa5ebbce",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 20728 characters]",
      "text/plain": [
       "<Figure size 1000x1000 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig = plt.figure(figsize=(10,10))\n",
    "plt.barh([\"PCA 1\", \"PCA 2\"], pca.explained_variance_)\n",
    "plt.gca().set(xlabel=\"Variance\", ylabel=\"Component\", title=\"Explained variance\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff6b13e3",
   "metadata": {},
   "source": [
    "Clearly the zeroth PCA component holds most of the variance and we could therefore use this component to closely determine where a data sample point should be if we dropped the other component. This is equivalent to fitting a line and using the projection of the point in the line to characterize the data, instead of using the two coordinates.\n",
    "\n",
    "While this may seem superfluous in this simple case, if one has hundreds of variables, PCA provides a simple and almost automatic way to reduce the amount of features being examined, by concentrating most of the variance in a few variables. Which variables to choose, can be decided from the `explained_variance_` attribute."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7bcb1e2c",
   "metadata": {},
   "source": [
    "## Other representation learning methods\n",
    "\n",
    "While PCA is a great method for both data visualization and choosing a latent space to condense the information contained in the data, there are many other available, which differ on the assumptions made. Here are some of them and easy-to-read and practical references on how to use them and which assumptions they entail.\n",
    "\n",
    "  * Kernel PCA: While PCA does a great job at finding the directions where most of the variance is, it focuses on making only *linear* combinations of the features to find the new representation directions. A straightforward method to generalise this idea to non-linear transformations is to apply a non-linear transformation to the data features and after that apply PCA. This is similar to what Kernel PCA does, generalizing the PCA idea. Of course, a non-linearity transformation needs to be chosen for this and this adds extra assumptions in the model. Some more on this can be read here: https://scikit-learn.org/stable/modules/decomposition.html#kernel-pca\n",
    "  \n",
    "  * Independent Component Analysis: PCA  looks for the direction containing most of the variance, but this is not always the optimal representation of the data for all goals. In some cases, independence is much more important than decorrelation and this is where the Independent Component Analysis comes in. The ICA also assumes that the new representation can be built from a linear combination of the existing data, but it assumes additionally each observed feature comes from a linear combination of independent latent representations we want to discover. Since independence is a very strong requirement, it is imposed through various different proxy methods. One set of methods focuses on reducing the mutual information (as defined in statistics) between the new features, while an alternative imposes non-Gaussianity requirements on the underlying latent features to be discovered. Since Gaussians do not have statistical moments above the second-order moment (covariance), one can require for example, the fourth order statistical moment (kurtosis) to be maximized in the new representation. More on the scikit-learn implementation can be found here: https://scikit-learn.org/stable/modules/decomposition.html#ica .\n",
    "  \n",
    "  * t-SNE embedding: If the objective is only to visualize the data, there are many alternative solutions which focus on reducing the dimensionality of the data into a 2D representation. The t-SNE method assumes that a Gaussian probability can be used to model similarity between data points in N dimensions and that one should maintain that similarity measure when projecting the data in two dimensions, assuming however that the 2D data points' similarity can be represented with a t-Student distribution. Full details on the method can be seen here: https://jmlr.org/papers/volume9/vandermaaten08a/vandermaaten08a.pdf It can be easily tried in scikit-learn following the procedure here: https://scikit-learn.org/stable/modules/generated/sklearn.manifold.TSNE.html"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9fbc8126-83f3-4c3c-8e5e-a677461e4c48",
   "metadata": {},
   "source": [
    "## Try them out!\n",
    "\n",
    "Try using t-SNE or Kernel PCA for visualizing the data. Try also getting other data sources to explore how they can be visualized. An example of a data source that you can use to play with would be the dataset of hand-written digits (MNIST), often used for tests in ML. You can obtain them as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "81e3af9d-eeb5-4f3a-b3d9-005890e0fa14",
   "metadata": {},
   "outputs": [],
   "source": [
    "import mnist\n",
    "\n",
    "data = mnist.train_images() # the images\n",
    "labels = mnist.train_labels() # the array with the digit each image corresponds to"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "600d7a0b-7b9e-4b17-a73a-5bc44d3cf682",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
