{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "1c0d6d14-a97f-41f2-8b3e-673399418107",
   "metadata": {},
   "source": [
    "# Basic statistics\n",
    "\n",
    "Here, the goal is to get used to several concepts in basic statistics needed later on for Machine Learning.\n",
    "We are going to generate a simple dataset ourselves and try to calculate some standard quantities to describe them.\n",
    "\n",
    "First, let's import some standard libraries and generate the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "d5a29abd-6af1-4b18-8bec-8a8402b97dc7",
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "f4bed5ac-0828-4e5f-a166-2e161557b49b",
   "metadata": {},
   "outputs": [],
   "source": [
    "rng = np.random.RandomState(0)\n",
    "n_samples = 10000\n",
    "cov = [[1,       0.7*1*1.3,   0.3*1.3*1],\n",
    "       [0.7*1*1.3,    1.3**2,      0],\n",
    "       [0.3*1.3*1,   0, 1.5**2]]\n",
    "data = rng.multivariate_normal(mean=[1, -1, 0], cov=cov, size=n_samples)\n",
    "data = pd.DataFrame(data, columns=[\"x1\", \"x2\", \"x3\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "87d5cd39-4720-4eba-a268-0c38a561976a",
   "metadata": {},
   "source": [
    "First, make a scatter plot of this dataset using matplotlib to visualise what are the possible values of $x_1$, $x_2$ and $x_3$.\n",
    "One can make a 2D plot of $x_1$ versus $x_2$, $x_2$ versus $x_3$ or $x_1$ versus $x_3$ to understand how they relate to one another.\n",
    "\n",
    "Often this is cumbersome if one has too many variables to plot, but we are going to discuss methods to get around this later in the lecture.\n",
    "\n",
    "One tip: one may use `plt.scatter` for every pair of variables to do this, or use the seaborn module to plot all pairs in one go. Look for documentation on `sns.pairplot`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "7de735cf-3466-41e9-bcd5-fcc756958019",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/daniloefl/miniconda3/envs/ml/lib/python3.12/site-packages/seaborn/_oldcore.py:1119: FutureWarning: use_inf_as_na option is deprecated and will be removed in a future version. Convert inf values to NaN before operating instead.\n",
      "  with pd.option_context('mode.use_inf_as_na', True):\n",
      "/home/daniloefl/miniconda3/envs/ml/lib/python3.12/site-packages/seaborn/_oldcore.py:1119: FutureWarning: use_inf_as_na option is deprecated and will be removed in a future version. Convert inf values to NaN before operating instead.\n",
      "  with pd.option_context('mode.use_inf_as_na', True):\n",
      "/home/daniloefl/miniconda3/envs/ml/lib/python3.12/site-packages/seaborn/_oldcore.py:1119: FutureWarning: use_inf_as_na option is deprecated and will be removed in a future version. Convert inf values to NaN before operating instead.\n",
      "  with pd.option_context('mode.use_inf_as_na', True):\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<seaborn.axisgrid.PairGrid at 0x7f78d9bee210>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 291412 characters]",
      "text/plain": [
       "<Figure size 750x750 with 12 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.pairplot(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "89a1c59f-9ff7-4c16-a85e-c390b9f77b31",
   "metadata": {},
   "source": [
    "What are the means and covariances of each variable?\n",
    "Can you explain how they can be geometrically interpreted from the plots above?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "ee885793-46a8-424c-be66-7b9c642c296a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 57344 characters]",
      "text/plain": [
       "<Figure size 2400x800 with 3 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots(figsize=(24, 8), ncols=3)\n",
    "for i, x in enumerate(data.columns):\n",
    "    mean = data.loc[:, x].mean()\n",
    "    sigma = data.loc[:, x].std()\n",
    "    m_plus = mean + sigma\n",
    "    m_minus = mean - sigma\n",
    "    ax[i].hist(data.loc[:, x], bins=25)\n",
    "    ax[i].axvline(mean, c='r', ls='-', label=\"Mean\")\n",
    "    ax[i].axvline(m_plus, c='g', ls='--', label=r\"Mean + sqrt(variance)\")\n",
    "    ax[i].axvline(m_minus, c='g', ls=':', label=r\"Mean - sqrt(variance)\")\n",
    "    ax[i].legend()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "58a73d63-32a3-469b-bb26-23bb14f49a23",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Correlation[x1, x2] = 0.69 (same as 0.69)\n",
      "Correlation[x1, x3] = 0.27 (same as 0.27)\n",
      "Correlation[x2, x3] = -0.00 (same as -0.00)\n"
     ]
    }
   ],
   "source": [
    "full_cov = data.cov() # covariance matrix\n",
    "full_corr = data.corr() # correlation coefficients\n",
    "for i, x in enumerate(data.columns):\n",
    "    for j, y in enumerate(data.columns):\n",
    "        if j <= i:\n",
    "            continue\n",
    "        # we can recalculate the correlation coefficient from the covariance matrix\n",
    "        rho = full_cov.loc[x, y]/np.sqrt(full_cov.loc[x, x]*full_cov.loc[y, y])\n",
    "        print(f\"Correlation[{x}, {y}] = {rho:.2f} (same as {full_corr.loc[x, y]:.2f})\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "df9f429b-e218-49fd-bc72-fca2e77ba770",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Slope between x1 and x2 = 1.48\n",
      "Slope between x1 and x3 = 3.63\n",
      "Slope between x2 and x3 = -142.59\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 236996 characters]",
      "text/plain": [
       "<Figure size 2400x800 with 3 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots(figsize=(24, 8), ncols=3)\n",
    "count = 0\n",
    "for i, x in enumerate(data.columns):\n",
    "    for j, y in enumerate(data.columns):\n",
    "        if j <= i:\n",
    "            continue\n",
    "        # get the data for the x and y axis\n",
    "        dx = data.loc[:, x]\n",
    "        dy = data.loc[:, y]\n",
    "        \n",
    "        # find their means, so we can draw a line passing through that point\n",
    "        mi = dx.mean()\n",
    "        mj = dy.mean()\n",
    "\n",
    "        # calculate the covariance matrix\n",
    "        cov = np.cov(dx, dy)\n",
    "\n",
    "        # find the eigenvectors of cov:\n",
    "        eigvals, eigvec = np.linalg.eigh(cov)\n",
    "        # find the index of the highest eigenvalue\n",
    "        i_max = np.argmax(np.fabs(eigvals))\n",
    "        # find the slope of the eigenvector corresponding to the highest eigenvalue:\n",
    "        slope = eigvec[1, i_max]/eigvec[0, i_max]\n",
    "        # note: this procedure\n",
    "        # sheds light into an interesting and useful interpretation of the covariance matrix\n",
    "        print(f\"Slope between {x} and {y} = {slope:.2f}\")\n",
    "        # plot the data\n",
    "        ax[count].scatter(dx, dy)\n",
    "        # plot a line with the slope above passing through the mean\n",
    "        ax[count].axline((mi, mj), slope=slope,\n",
    "                         label=f\"Slope of highest eigenvalue eigenvector of cov.\",\n",
    "                         c=\"r\", ls=\"-\", lw=4)\n",
    "        ax[count].set(xlabel=x, ylabel=y)\n",
    "        ax[count].legend(frameon=False)\n",
    "        count += 1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3552a373-ac09-4820-ab33-9c5e0e911dab",
   "metadata": {},
   "source": [
    "Now let's try \"standardizing\" the data and looking at the correlation coefficients:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "54373271-dd1a-4fdd-8f49-b7db7a595695",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 221332 characters]",
      "text/plain": [
       "<Figure size 2400x800 with 3 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots(figsize=(24, 8), ncols=3)\n",
    "count = 0\n",
    "for i, x in enumerate(data.columns):\n",
    "    for j, y in enumerate(data.columns):\n",
    "        if j <= i:\n",
    "            continue\n",
    "        # get the data for the x and y axis\n",
    "        dx = data.loc[:, x]\n",
    "        dy = data.loc[:, y]\n",
    "        # find their means\n",
    "        mi = dx.mean()\n",
    "        mj = dy.mean()\n",
    "        # find the square root of their variances\n",
    "        si = dx.std()\n",
    "        sj = dy.std()\n",
    "        # standardize them:\n",
    "        di = (dx-mi)/si\n",
    "        dj = (dy-mj)/sj\n",
    "        # plot them again\n",
    "        ax[count].scatter(di, dj)\n",
    "        # what is the correlation coefficient?\n",
    "        rho = np.corrcoef(di, dj)[0, 1]\n",
    "        # plot a line with the slope above passing through the mean\n",
    "        ax[count].axline((0, 0), slope=1/rho,\n",
    "                         label=f\"Slope = 1/correlation\",\n",
    "                         c=\"r\", ls=\"-\", lw=4)\n",
    "        ax[count].set(xlabel=x, ylabel=y)\n",
    "        ax[count].legend(frameon=False)\n",
    "        count += 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1e7966d8-8c82-4717-8c8d-49e4a31f338a",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
