{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "1bba0128",
   "metadata": {},
   "source": [
    "# Simple fits\n",
    "\n",
    "Here we are fitting a line from scratch.\n",
    "In the next notebook, we will do fancier fits with neural networks, but let's start with a basic problem and complicate it as we go along.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "23feddde",
   "metadata": {},
   "outputs": [],
   "source": [
    "from typing import Tuple\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb1286f0",
   "metadata": {},
   "source": [
    "We start by generating some fake dataset, which is simple enough that we can visualize the results easily. For this reason, the dataset will contain only two  variables.\n",
    "\n",
    "The simulated example data will be $f(x) = 3 x + \\epsilon$, where $\\epsilon \\sim \\mathcal{N}(\\mu=0, \\sigma=0.5)$.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "5d457cd8",
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_data(N: int) -> np.ndarray:\n",
    "    x = 2*np.random.randn(N, 1)\n",
    "    epsilon = 0.5*np.random.randn(N, 1)\n",
    "    z = 3*x + epsilon\n",
    "    return np.concatenate((x, z), axis=1).astype(np.float32)\n",
    "\n",
    "data = generate_data(N=1000)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48433f6f",
   "metadata": {},
   "source": [
    "We can fit this line from scratch, assuming $y = f(x) = \\beta x + \\alpha + \\epsilon$, where $\\epsilon$ is a zero-mean Gaussian noise."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a170344-34ff-4af6-9a6c-daa44b95408e",
   "metadata": {},
   "source": [
    "We can do this in two ways. In both cases, we start assuming the knowledge of $f(x)$ and use the Bayes Theorem:\n",
    "\n",
    "$ p(\\alpha, \\beta|\\mathcal{D}) \\, \\propto p(\\mathcal{D}|\\alpha, \\beta) \\, p(\\alpha) \\, p(\\beta)$\n",
    "\n",
    "We assume $p(\\alpha)$ and $p(\\beta)$ are constants, so we just need to maximize the likelihood $p(\\mathcal{D}|\\alpha, \\beta)$.\n",
    "That likelihood is a Gaussian distribution, because we assumed $y$ is exactly equals to a line plus some Gaussian noise coming from $\\epsilon$.\n",
    "\n",
    "$p(\\alpha,\\beta|\\mathcal{D}) \\propto \\prod \\mathcal{N}(y_i|\\text{mean}=f(x_i), \\text{std. dev.}=\\sigma_\\epsilon)$\n",
    "\n",
    "This leads to minimizing $L = \\sum_i (y_i - \\beta x_i - \\alpha)^2$, as seen in the lecture (simply take $- \\log p$ and to maximize $p$, one has to minimize $-\\log p$).\n",
    "\n",
    "We can try finding out $\\beta$ and $\\alpha$ by taking the derivative of $L$ and setting it to zero:\n",
    "\n",
    "$\\frac{d L}{d\\beta} = 2 \\sum_i (y_i - \\beta x_i - \\alpha) x_i = 0$\n",
    "\n",
    "$\\frac{d L}{d\\alpha} = -2 \\sum_i (y_i - \\beta x_i - \\alpha) = 0$\n",
    "\n",
    "Re-arranging:\n",
    "\n",
    "$2 \\sum_i (y_i x_i - \\beta x_i^2 - \\alpha x_i) = 0$\n",
    "\n",
    "$2 \\sum_i (y_i - \\beta x_i - \\alpha) = 0$\n",
    "\n",
    "This leads to:\n",
    "\n",
    "$ \\beta \\sum_i x_i^2 = \\sum_i (y_i - \\alpha) x_i$\n",
    "\n",
    "$ \\alpha = 1/N \\sum_i y_i - \\beta x_i = \\mathbb{E}[y - \\beta x]$\n",
    "\n",
    "We can then substitute $\\alpha$ in $\\beta$:\n",
    "\n",
    "$ \\beta \\sum_i x_i^2 = \\sum_i y_i x_i - 1/N \\sum_i \\sum_j (y_j - \\beta x_j) x_i$\n",
    "\n",
    "And expand it:\n",
    "\n",
    "$ \\beta (\\sum_i x_i^2 - 1/N \\sum_i \\sum_j x_i x_j) = \\sum_i y_i x_i - 1/N \\sum_i \\sum_j y_j x_i$\n",
    "\n",
    "Taking $x_j$ from a sum that does not depend on $j$, we see that the second term is just the square of the sum over $x_i$:\n",
    "\n",
    "$ \\beta (\\sum_i x_i^2 - 1/N(\\sum_{i} x_i)^2 ) = \\sum_i y_i x_i - 1/N \\sum_i \\sum_j y_j x_i$\n",
    "\n",
    "Dividing by the number of points $N$, several sums reduce to averages, which make this much easier to interpret if we use some standard identities from statistics.\n",
    "For example $1/N \\sum_i x_i^2 - 1/N^2 (\\sum_i x_i)^2$ is a known expression for the variance of $X$. \n",
    "\n",
    "$ \\beta \\text{var}[X] = \\mathbb{E}[XY] - \\mathbb{E}[X]\\mathbb{E}[Y]$\n",
    "\n",
    "And the right-hand-side is a known expression for the covariance between $X$ and $Y$:\n",
    "\n",
    "$ \\beta \\text{var}[X] = \\text{cov}[X,Y]$\n",
    "\n",
    "This leads to:\n",
    "\n",
    "$\\beta = \\frac{\\text{cov}[X,Y]}{\\text{var}[X]}$\n",
    "\n",
    "This, together with the expression from above for $\\alpha$, allows us to easily fit a line in a few lines of code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "30205402",
   "metadata": {},
   "outputs": [],
   "source": [
    "cov = np.cov(data[:,0], data[:,1])[0,1]\n",
    "varX = np.std(data[:,0])**2\n",
    "beta = cov/varX\n",
    "alpha = np.mean(data[:,1] - beta*data[:,0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "cabdfc12-3033-4a2f-96ed-4ab1ea8312ab",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.lines._AxLine at 0x7f5db8662e30>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 30612 characters]",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.scatter(data[:,0], data[:,1])\n",
    "plt.axline((0, alpha), slope=beta, c='r', lw=2, ls='--', label=\"Line fit\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a1fbd00b-e7f2-4eff-90db-e1d44d104fba",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
